format: a CSV can be written in the dialect you were handed - #36
Merged
Conversation
Adds delimiter, line_ending and header to the csv format. Separators are named rather than typed - comma, semicolon, tab and pipe - so tab and pipe need no escaping on a command line or in a recipe. The defaults are comma, lf and a header, which is what this tool has always written, so no existing file changes by a byte: checked against a binary built from main, four seeds by three sizes, twelve of twelve identical. These are the three ways a real CSV differs before its contents do. A European spreadsheet exports with semicolons, anything written on Windows ends its rows with CRLF, and a table dumped out of a database has no header. All three are CSV and all three break a reader that assumed the other thing. Two things are not obvious from the list of settings. The filler has to emit the SEPARATOR rather than a comma. The description is a quoted field and it is quoted only because it carries the separator, so a filler that kept dropping commas would leave a semicolon file looking perfect - right size, right separators between the fields, every row the same width - while never exercising the quoted path the setting exists to test. Nothing else here would have seen that, so the guard counts separators in the closing row: six fields need five, and any more are inside the quotes. And the dialect moves the floor. A CRLF row is a byte longer and a header is a whole line, so the four floors are 117, 119, 76 and 77 B rather than one number. The registry announces the floor for the settings left alone and Plan works out the real one, the same arrangement the log format uses. The structural checker is now TOLD the dialect rather than sniffing it, which needed Strict to take optional settings. A checker that guessed the separator would agree with a file that used the wrong one - it would split on whatever it found and report a tidy table either way. Whether the file uses the separator that was ordered is a question for a guard reading the manifest instead, and there is one. That the checker is not a rubber stamp is asked directly: a comma file handed to it as semicolon has to come back refused. Still to come for this format: the quote character and escaping, and the column count. A row count disagrees with the exact size contract for the reason an entry count does in a log, and encoding waits for the piece that does TXT, where it is the headline setting rather than one of several. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
delimiter,line_endingandheaderto thecsvformat.Separators are named rather than typed -
comma,semicolon,tab,pipe- sotabandpipeneed no escaping on a command line or in a recipe.No existing file changes by a byte. The defaults are
comma,lfand a header, which is what this tool has always written. Checked against a binary built frommain: four seeds by three sizes, twelve of twelve identical.These are the three ways a real CSV differs before its contents do. A European spreadsheet exports with semicolons, anything written on Windows ends its rows with CRLF, and a table dumped out of a database has no header. All three are CSV and all three break a reader that assumed the other thing.
Two things that are not obvious from the list of settings
The filler has to emit the separator, not a comma. The description is a quoted field, and it is quoted only because it carries the separator. A filler that kept dropping commas would leave a semicolon file looking perfect - right size, right separators between the fields, every row the same width - while never exercising the quoted path the setting exists to test. Nothing else here would have seen it, so the guard counts separators in the closing row: six fields need five, and any more are inside the quotes.
The dialect moves the floor. A CRLF row is a byte longer and a header is a whole line.
lfcrlfThe separator does not move it, because all four are one byte. The registry announces the floor for the settings left alone and
Planworks out the real one - the same arrangement the log format uses.The checker is told the dialect, not left to guess
oracle.Strictnow takes optional settings. A checker that sniffed the separator would agree with a file that used the wrong one: it would split on whatever it found and report a tidy table either way. Whether the file uses the separator that was ordered is a question for a guard reading the manifest, and there is one.That the checker is not a rubber stamp is asked directly - a comma file handed to it as
semicolonhas to come back refused, and does.What was checked
D11against amainbinary: 12 of 12 byte for byteNew guards:
TestTheCSVDialectIsInTheFileAndMovesTheFloor,TestEveryCSVDialectIsWellFormed,TestTheCSVManifestRecordsTheDialectAsFacts.Still to come for this format
The quote character and escaping, and the column count. A row count disagrees with the exact size contract for the reason an entry count does in a log, and encoding waits for the piece that does TXT, where it is the headline setting rather than one of several.
🤖 Generated with Claude Code