Skip to content

fix(sparql-anything): remove stale chunks with five or more digits - #834

Open
ddeboer wants to merge 1 commit into
mainfrom
fix/chunk-cleanup-five-digits
Open

fix(sparql-anything): remove stale chunks with five or more digits#834
ddeboer wants to merge 1 commit into
mainfrom
fix/chunk-cleanup-five-digits

Conversation

@ddeboer

@ddeboer ddeboer commented Sep 8, 2026

Copy link
Copy Markdown
Member

chunk() pads the chunk index to four digits, so the 10,000th chunk onwards gets a five-digit name. removeChunksOf() matched \d{4} exactly and left those behind, so a re-run at a larger chunk size kept the earlier run’s tail in the directory – and in geonames-rdf, in the output.

  • Match four or more digits.
  • Add a test that chunks a 10,001-row input at one row per chunk, re-chunks at a larger size and checks that only the new chunks remain (one character per row keeps it small; it still writes 10,001 files, so it has a 30 s timeout).
  • Docs: take the paths chunk() returns as the list of chunks to convert, rather than listing the directory.

Fix #823

- chunk() pads the index to four digits, so the 10,000th chunk onwards has five;
  removeChunksOf() matched exactly four and left those behind on a re-run
- Cover it with a 10,001-row input chunked at one row and re-chunked at a larger size
- Document that the paths chunk() returns are the list to convert
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

sparql-anything: removeChunksOf() misses chunks past the 10,000th

1 participant