Ignore comments in tokenizer

[hadley]

read.csv() etc support a comment argument to ignore (e.g.) everything after #.

Please 👍 this issue if you'd like this feature.

[lmullen]

👍 I sometimes encounter CSV files with comments. I'm sorry to say I even used to add them myself.

[jimhester]

👎 for me, the only time I have used this feature in read.csv() is when it causes an incorrect parse due to my data having # in it and I have to read the man page to turn it off.

If you do decide to add it please make the default off!

[leondutoit]

👍 default == off, I see people commenting in delimited files all the time

[hadley]

The default would definitely be off - I've also had bad experiences where I had to turn it off

[jennybc]

👍

Some instruments write reasonable delimited files but put metadata in the file itself, not necessarily at the top, so this would be a good complement to skip =. Also more flexible than skip =, which I assume requires a specific number of lines.

[PeteHaitch]

👍 for the reasons @jennybc said.

[davharris]

👍

I was just about to open an issue to suggest this. My current use case is reading in files produced by the Stan package for MCMC (example output; note the comment lines at the beginning, middle, and end of the file).

Currently, I have to make two passes through the file and do some extra fiddling, and it would be nice if all of that could be automated.

If it doesn't add too much complication, it might be nice if the comments (or at least their positions) could be included as an attribute as well, similar to how "problems" are handled.

Thanks for making yet another great package!

[defconst]

👍

Sometimes needed for comments and metadata.

[rpruim]

My bad for just adding an issue to request this -- sorry for the cruft. Seems to me that it does no harm when not used and can be make it or break it when the file you need need to read has comments in it.

Frankly, I'd like to see more people put meta data in commented portions of delimited files because otherwise it tends to go missing.

[sjackman]

👍

[stefano-meschiari]

👍