-
Notifications
You must be signed in to change notification settings - Fork 211
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
supporting double quotes #4
Comments
Nice. Thanks for the RFC link. For me in my daily work -- where Miller is pretty stable featurewise -- CSV is a secondary format (hence its not being the default). But I suspected (and it now seems rightly so) that CSV will be the primary for most folks. |
I just found out about miller on the weekend.. Swiss Army knife and looking forward to getting to know it! I would suggest that if supporting double quotes comes at a performance cost, to have a light-weight csv (current operation) option and the full csv support. 99.9% of the time, I don't need double quote support and I'm sure many others are in the same boat. Mostly need it for processing numbers from spreadsheet colleagues and the current operation is just fine as is. Thanks!! |
johnkerl: how easy/difficult would it be to add support for double quotes? |
I came here to request the same, adding support for quoted fields would make this tool awesomely usable for a lot of use cases. |
ftrotter is spot-on -- RFC-compliant CSV is the way to go. Not only on a tool-by-tool basis but because it encourages good data citizenship on the part of producers as well as consumers. ve3ied is (I think) also right -- I suspect a fully compliant CSV parser will run a bit slower than the handles-simple-cases code I have now (which suffices for my personal use cases). I think the right thing to do is make That said, RFC-compliant CSV is going to be a bit of a project, while double quotes per se, as a small step toward that goal, is going to be easy. The RFC states that if double quotes are present, they must wrap the entire field. So it'll be pretty simple to (a) add a state to the state machines in the It's clear I've led a charmed life by having CSV data which is almost entirely double-quote-free. In light of multiple feedbacks, the minimal double-quote-handling feature sketched above is definitely top-of-list. |
I've split out #16 for the more general case; I'll close this when double-quote handling is in place. |
Primary question at this point: how/when to double-quote on output?
I'm OK with both options. Question is if there's any desire for some sort of middle ground, e.g. if someone wants fields which were quoted on input to remain quoted on output. That's a bit more awkward. For transform operations which pass records through, e.g. In short I'm happy to implement output-quote-when-needed and output-quote-always modes; I'm not eager to do output-quote-tracked-from-input-quotes unless it's a really important use-case. |
Perhaps @ftrotter knows of a more proper answer, but I just did a quick survey of some of the more common tools I use and how they handle quoting on output. I hope this adds to the conversation: Pythons CSV ModuleProvides four options, and defaults to QUOTE_MINIMAL:
With the defaults Perls Text::CSVAlso provides four options and the default behaviour is to quote fields if they need to - e.g. they contain the separator:
With the defaults CsvfixProvides two options on quoting, and defaults to quoting all fields:
I did a quick test and the CsvKitProvides four options for quoting, defaults to quoting only when a separator is present.
I did not get time to test the My personal preferences are exactly the two options you laid out (quote minimally and quote everything) and to have Thank you for working on this :) |
Actually @SikhNerd has just pointed out the "real" reason to support double quotes all the time on input, rather than supporting any 'modes'. So many of the standard CSV output systems will default to using no quotes for field exports that contain only numbers, but then automatically include quotes when something "turns alphanumeric" or "turns alphanumeric with commas". For instance a standard input file might go for years with a structure of id,phone_number but then suddenly someone enters a phone number with an extension and then the identical export will get id,phone_number or worse In the same way, you might have an export process that works like this: id,sentence In this case the quotes are only exercised when the comma appears in the string Again thanks for working on such a great tool!! -FT |
@ftrotter thank you! you've got a lot more experience with dirty CSV than I do. :) |
Rethink; I'm duping #16 to this one. No need for a two-task approach. |
Had a nice chunk of time this evening to sketch out the new CSV parser. Proof of concept works. Now just a matter of integrating into the rest of the framework, and plenty of unit-test & regression-test cases, & checking against several found-in-the-wild CSV files. There are several in miller issues, I think enough to work with. |
If you can push up the branch you're working on, I'd be very interested in watching what you're doing, and will try and play around with it to help if I find time :) |
it's all in master -- this evening's pushes -- every bit compiles; there are unit-tests for some stable things; all in backward-compatibility mode so nothing is reachable from the |
Progress point:
( |
At present this CSV reader is about 3.5x slower than the CSV-lite reader. Some performance optimizations can be made after the functionality is delivered but I very much doubt it will ever be as fast as CSV-lite.
|
OK, iterating. If you use
|
Hi @johnkerl, So is this an example of your query working from master compiled mlr?:
or
|
At the moment:
In a day or two once I do some code cleanup & add some test cases:
|
👍 |
Hello. I thank you for this update. I have csv files where the field separator is a space, and even if the --csvex works with "," filed separator, it doesn't seem to work when you want to change the FS. Example : $ mlr --csvex --odkvp cat test This case is ok, but if I change my FS to space 👍 $mlr --csvex -p --odkvp cat test2 How can I use the --csvex with space separator ? Thanks. |
Right. In the next minor, v2.1.0, I'll do a bit more:
|
I'll put the above info in the 2.0.0 release note as well. |
Please see https://github.com/johnkerl/miller/releases/tag/v2.0.0. If you find any issues, please open a new issue as a bugfix request. Thank you thank you thank you for the very informative feedback!!! :) |
Looks good 💯 I haven't been able to break it yet. |
:D |
What would you recommend to do when the input file has a broken line like line 3 below: Is there a way to fix it using miller commands? |
Good question. On @ftrotter's advice I made the CSV parser RFC-compliant. I didn't put any effort into handling non-compliant CSV (although there's a lot of it out there in the world). I don't know how non-compliant your data is beyond the above 3-line example, so I'm unsure how to point you at a general solution, but a few ways to handle the above 3-line example are:
In the latter case I'm suggesting the csvlite parser which isn't RFC-compliant and treats quotes as characters like any others. Also note that for option 3 you'll need head Miller, since I introduced the necessary |
4th option: |
@johnkerl this is awesome 👍
|
:) |
Have not visited this thread in a while and I am thrilled to see the progress. I do think that handling non-standard CSV is a good feature, but I almost think that barfing with an error message is almost a better feature.. that way I can do to CSV producers and say "see... get your shit together"... Would also like to point out that the performance mentioned here is excellent for standards compliant CSV processing, just because it is "worse" does not mean it cannot also be "excellent". It is quite a compliment to have such a complex feature that I suggested closed. I will do my best to promote this to all the right people. Moving to twitter now!! -FT |
I love the idea of Miller. It is clearly a needed tool that is missing from the standard unix toolbox.
However, you really cannot say you have a tool that is designed to support csv, without supporting csv.
CSV is a standard file format, and has an RFC: https://tools.ietf.org/html/rfc4180
Not supporting double quotes is the same thing as saying that you do not support csv, since double quotes are central to the way that the standard handles other characters... comma being just one example. Your tool is young enough that supporting the standard now will make later development much simpler. This will prevent the situation years from now where you have a 'normal mode' and a 'standards mode'. If you make the change now you can just have the one correct mode.
You have an ambitious work-list, but I would suggest taking a pause and thinking about how you will support the RFC version of the file format.
People like me (open data advocates) spend alot of time trying to ensure that organizations that release csv do so under the standard format, rather than releasing unparsable garbage. Having a library like yours that supported the standard too would be a huge boon.. I could say things like:
"See by using the RFC for your data output, all kinds of open tools will work out of the box on your data... like Miller (link)"
Thank you for working on such a clever tool...
Regards,
-FT
The text was updated successfully, but these errors were encountered: