Skip to content

LongDocs lock files v1

Pre-release
Pre-release

Choose a tag to compare

@caciolai caciolai released this 02 Nov 13:33
· 12 commits to main since this release

Per-document sha256 lock files for the LongDocs hub sources (arxiv, doab, gutenberg, hansard, usgpo, wikimedia, swe, code).
They contain no document text.
A lock file lists each document's id, text sha256 and position, so a collection you run yourself can be verified with --verify and a mismatch shows which documents differ.
Total 134,695 documents across the eight sources.
The paper reports 134,740 for native training; the 45 difference has not been investigated.
The code lock reflects a collection of the Stack streamed without a pinned revision, so a later collection may differ.