Publish the graph as Parquet with a generated dataset card - #50
Merged
Conversation
The graph is published as a tarball of Neo4j import CSVs, which is the right artifact for loading it into Neo4j and the wrong one for anything else. The hub shows it as a download link, the viewer has nothing to render, and nobody can look at the data before committing to half a gigabyte. This adds luatdo export parquet, which converts the CSV export into one Parquet table per file and writes the dataset card next to them. The conversion is driven by the header row rather than by a second projection of the store, for the same reason import.sh is generated rather than listed: a second projection would be a second list of node types and a second place to forget a layer, which has happened three times already. Every neo4j header directive is mapped. The colon names become ordinary ones, so start_id joins to id without anybody having to know how neo4j-admin spells either. Empty fields become null rather than the empty string, arrays become lists, and the multiline provision text stays whole. Large tables shard at a million rows and the shard names carry the total, which is why they are written under a temporary name and renamed at the end. The card is generated because a hand written list of tables goes stale the first time somebody adds a node type, and here it goes worse than silently: a config naming a table with no files stops the whole dataset page from loading rather than degrading. Generating it also settled a question the old card got wrong. It claimed seventeen node labels and the data has thirteen, and it claimed the concept mention layer was populated when that table has no rows at all. The card now counts labels off the rows and says plainly that the concept layer is empty. Verified over the real export: 33 tables, 17,294,357 rows, 623MB of Parquet from 3.4GB of CSV, and the node and relationship totals match the loaded database exactly at 8,175,346 and 9,119,011. Joins across shards in duckdb return the expected counts and the numeric columns come back as BIGINT and DOUBLE rather than text.
9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #49.
The graph has been published on Hugging Face as a tarball of Neo4j import CSVs.
That is the right artifact for loading it into Neo4j and the wrong one for everything else.
The hub renders it as a download link, the viewer has nothing to show, and anybody deciding whether to spend half a gigabyte has to spend it first.
This adds
luatdo export parquet, which converts the CSV export into one Parquet table per file and writes the dataset card next to them.Why convert the CSVs rather than project the store again
A second projection would be a second list of node types, a second set of column names and a second place to forget a layer.
This repository has shipped a graph with a missing layer three times, and each time nothing failed, because nothing asked.
The CSV headers already say what every column is, because
neo4j-adminrequires them to, so a node type added to the export appears in the Parquet without anybody editing anything.Same reasoning as running the export's own
import.shrather than reproducing its file list.What the conversion does
Every header directive is mapped:
:ID,:LABEL,:START_ID,:END_ID,:TYPE,:int,:float,:boolean,:string[].The colon names become ordinary ones, so
start_idjoins toidwithout a reader having to know howneo4j-adminspells either.An unknown directive is refused rather than written as text, because a column of numbers in quotes with nothing to explain it is worse than a failed export.
An empty CSV field becomes null and not the empty string.
A missing deadline and a deadline of nothing are different claims, and every tool that reads Parquet can tell null from
""while none of them can tell it from""in a CSV.Arrays split on the delimiter the export writes, multiline provision text stays whole, and tables shard at a million rows.
The shard names carry the total, which is not known until the last row is read, so files are written under a temporary name and renamed at the end.
A table with no rows is still published.
A layer that came out empty and a layer that was never exported are different facts, and the file list is the only place that difference shows.
Why the card is generated
A hand written list of tables goes stale the first time somebody adds a node type.
Here it goes worse than silently: a config naming a table with no files does not degrade, it stops the whole dataset page from loading.
Generating it also settled two things the previous card had wrong.
It said seventeen node labels and the data carries thirteen, so the card now counts them off the rows.
It said the concept mention layer was populated, and that table has no rows at all, along with concept relations, instance links, concept merges, act participants and conflicts.
The card now says so.
The row counts sit right above the paragraph, so the two had to agree.
Verification
Run over the real export on this machine.
The node and relationship totals are 8,175,346 and 9,119,011, which match the loaded database exactly.
Reading every shard back with pyarrow gives the same 17,294,357 rows.
A duckdb join from
citestodocumentsacross shards returns the expected citation counts, anddeadline_valueandconfidencecome back asBIGINTandDOUBLErather than as text.Tests
At the value level rather than the file level, because every way this can be wrong still produces a valid Parquet file.
A column written under the wrong index is the one that does not:
parquet.Groupis a map andNewSchemaorders fields by name, so a row assembled in header order crashes the process on the first column whose type does not match.That is not a hypothetical, it is what the first two attempts did, and the test that covers it asserts values by column name.
The rest cover null against zero, arrays against pipe separated strings, multiline fields, sharding and the rename that follows it, the empty table, both refusals, and that the card's globs match the file names the shard writer produced.
Not in this change
The tarball stays exactly as it is.
luatdo neo4j installdepends on it and its checksum is pinned, so the two artifacts live side by side and nobody has to download anything again.