Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use duckdb for jsonl conversion to fit into memory #591

Merged
merged 5 commits into from
Jul 1, 2024
Merged

Use duckdb for jsonl conversion to fit into memory #591

merged 5 commits into from
Jul 1, 2024

Commits on Jun 13, 2024

  1. Use duckdb instead of pandas to make the kgx jsonl conversion

    @kevinschaper
    kevinschaper committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    10ec90e View commit details
    Browse the repository at this point in the history

  2. Swap duckdb in for pandas in jsonl writing, wait till the end of the … 

    …pipeline to compress jsonl files into a tar.gz so that it won't have to be unpacked before loading into neo4j.
    @kevinschaper
    kevinschaper committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    40dada6 View commit details
    Browse the repository at this point in the history

  3. merge main

    @kevinschaper
    kevinschaper committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    e1915e8 View commit details
    Browse the repository at this point in the history

  4. Add $ escape that Jenkins is suddenly claiming is a bug

    @kevinschaper
    kevinschaper committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    42f9411 View commit details
    Browse the repository at this point in the history

Commits on Jul 1, 2024

  1. merged main

    @kevinschaper
    kevinschaper committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    8c06542 View commit details
    Browse the repository at this point in the history