Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bulk Loader: Fix memory usage by JSON parser #3794

Merged
merged 21 commits into from Aug 13, 2019
Merged

Bulk Loader: Fix memory usage by JSON parser #3794

merged 21 commits into from Aug 13, 2019

Commits on Aug 10, 2019

  1. Create a new NQuads struct and move Parse under it.

    @manishrjain
    manishrjain committed Aug 10, 2019
    Configuration menu
    Copy the full SHA
    ded891b View commit details
    Browse the repository at this point in the history

  2. Move all nquads to the new NQuads struct.

    @manishrjain
    manishrjain committed Aug 10, 2019
    Configuration menu
    Copy the full SHA
    9ac7d50 View commit details
    Browse the repository at this point in the history

  3. Make parse tests work

    @manishrjain
    manishrjain committed Aug 10, 2019
    Configuration menu
    Copy the full SHA
    a6047b1 View commit details
    Browse the repository at this point in the history

  4. Code runs and all

    @manishrjain
    manishrjain committed Aug 10, 2019
    Configuration menu
    Copy the full SHA
    6832162 View commit details
    Browse the repository at this point in the history

  5. Run GC every 5s

    @manishrjain
    manishrjain committed Aug 10, 2019
    Configuration menu
    Copy the full SHA
    f4cf4be View commit details
    Browse the repository at this point in the history

  6. GC every 10s

    @manishrjain
    manishrjain committed Aug 10, 2019
    Configuration menu
    Copy the full SHA
    39a00b3 View commit details
    Browse the repository at this point in the history

  7. Set worker goroutines aka mappers to 1/4th of the number of cores.

    @manishrjain
    manishrjain committed Aug 10, 2019
    Configuration menu
    Copy the full SHA
    b81b207 View commit details
    Browse the repository at this point in the history

  8. Only run GC if it hadn't been run. Also mention the RAM usage being p… 

    …roportional to worker threads in shout-case.
    @manishrjain
    manishrjain committed Aug 10, 2019
    Configuration menu
    Copy the full SHA
    fa831be View commit details
    Browse the repository at this point in the history

Commits on Aug 11, 2019

  1. Files moved to one directory to allow easier sharing of the code.

    @manishrjain
    manishrjain committed Aug 11, 2019
    Configuration menu
    Copy the full SHA
    70eb295 View commit details
    Browse the repository at this point in the history

  2. Add batchSize option to Chunker and NQuadBuffer to simplify loader an… 

    …d others.
    @manishrjain
    manishrjain committed Aug 11, 2019
    Configuration menu
    Copy the full SHA
    b7a299b View commit details
    Browse the repository at this point in the history

Commits on Aug 12, 2019

  1. Bring the edgraph/server.go code back. Move multiple RDF parsing to c… 

    …hunker package as well.
    @manishrjain
    manishrjain committed Aug 12, 2019
    Configuration menu
    Copy the full SHA
    17b557b View commit details
    Browse the repository at this point in the history

  2. Hook up NQuadBuffer to RDFChunker.NQuads

    @manishrjain
    manishrjain committed Aug 12, 2019
    Configuration menu
    Copy the full SHA
    3f89c68 View commit details
    Browse the repository at this point in the history

  3. Address golint issues

    @manishrjain
    manishrjain committed Aug 12, 2019
    Configuration menu
    Copy the full SHA
    5e12649 View commit details
    Browse the repository at this point in the history

  4. Fix the live loader test failure.

    @manishrjain
    manishrjain committed Aug 12, 2019
    Configuration menu
    Copy the full SHA
    948bfde View commit details
    Browse the repository at this point in the history

  5. Don't think we need a for loop around chunker.Parse

    @manishrjain
    manishrjain committed Aug 12, 2019
    Configuration menu
    Copy the full SHA
    3e25781 View commit details
    Browse the repository at this point in the history

  6. Move FacetDelimiter to x package to avoid a cyclic import loop.

    @manishrjain
    manishrjain committed Aug 12, 2019
    Configuration menu
    Copy the full SHA
    541c9be View commit details
    Browse the repository at this point in the history

  7. Fix gql package test

    @manishrjain
    manishrjain committed Aug 12, 2019
    Configuration menu
    Copy the full SHA
    297651f View commit details
    Browse the repository at this point in the history

  8. Fix a test failure by handling io.EOF

    @manishrjain
    manishrjain committed Aug 12, 2019
    Configuration menu
    Copy the full SHA
    8f7f8cf View commit details
    Browse the repository at this point in the history

  9. Don't return io.EOF unnecessarily

    @manishrjain
    manishrjain committed Aug 12, 2019
    Configuration menu
    Copy the full SHA
    1d1f529 View commit details
    Browse the repository at this point in the history

  10. Address comments by PR folks. Also, no need to handle io.EOF because … 

    …Parse no longer returns it.
    @manishrjain
    manishrjain committed Aug 12, 2019
    Configuration menu
    Copy the full SHA
    de862d1 View commit details
    Browse the repository at this point in the history

Commits on Aug 13, 2019

  1. Address PR folks comments

    @manishrjain
    manishrjain committed Aug 13, 2019
    Configuration menu
    Copy the full SHA
    5c420fb View commit details
    Browse the repository at this point in the history