Support comments for Color scheme

[kecsap]

It would be really nice if the plugin would support the comments in syntax highlighting (Color scheme). Usually, the csv parsers consider hash mark "#" as start of a comment line. I hope it is easy to add.

[SeeSharpSoft]

Thanks for the suggestion.

Comments are not part of RFC 4180, which is the leading standard that this CSV Plugin is based on. IMO the parsing and syntax validation should stay clean and strict, adding comments - even just supporting the highlighting - will lead to a "dirty" CSV format.

Just some examples (consider the following lines part of a CSV file):

# this is not "valid" CSV cause of the "
"# this would be ""valid"" but is probably not wanted"
# this is valid CSV, but the , would still be interpreted as a value separator
# prohibiting double quotes and commas in comments would work - but would it?

This are just some examples when thinking about the text editor and highlighting - the table editor is a whole other topic on top of it. How are comments supposed to be handled there?

[kecsap]

Although it is not in this standard, parsing libraries tend to implement comments for csv reading. E.g. Pandas in python:

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html

I personally don't want the hash mark handling to complicate your parsing. If it is easy to add a simple dumb rule to consider everything to a comment after a hash mark for syntax highlight, it is fine. If it is hard to add you can just close this issue, I understand your arguments.

[SeeSharpSoft]

I will leave it open for now - maybe there are some good ideas how to handle this. As of now, I don't see a proper solution especially for the table editor.

It is not the technical but the semantic aspects that concerns me. Parsing CSV (once) with adjustable settings is a different scope than providing an editor with a defined language, syntax validation, non-text editor etc.

I'd rather promote a clean CSV syntax (e.g. #9, #53 or here) than trying to support all kinds of custom semantics and derivatives. Said this, I am using comments in my CSV files as well but value based, like /* just a comment */... 😀

[Egwor]

It would be very helpful if this plugin did handle the comments!

I think that there are two challenges:

1. Formatting of code. If the auto-formatter ignored commented lines this would be a big win. This would avoid the scenario where the first column is made very wide due to the comment. This is particularly annoying if you use the 'Auto-format' option in commits (because you then need to revert the auto-format). This would be a win in itself.

2. Handling the table editing. I think that there are a few options.

• Ignore the line entirely and the data is lost when saving. Probably not ideal!
• Don't allow editing of CSV files in the table editor if there are comments. Not wonderful, but if the auto-formatting worked then maybe this is ok for a first step.
• Allow existence of the comment line as a row by creating a fake row (grey it out?) with no data in the other columns so that the table editor could work on the rest of the file. If the plugin artificially creates a comment column at the start of the CSV where the comment is displayed, it would allow editing the contents of the comment.
• Link the comment with the line below it (or above it, if the line after is blank?). Ignore this in the table editor and then when save, restore the comment associated with the CSV row. This seems trickier to do to me?

I did start having a look at the code to see if I could add the handling of the comment on a specific branch so that I could get at least 1. but I'm not familiar enough with the IntelliJ lexing. I think I should be able to define the # as a comment. So something very very roughly like this (note: the end of line comment should just be a comment, but that's not a massive concern right now):

(CsvLexer.flex)
END_OF_LINE_COMMENT=("#")[^\r\n]*
...
<YYINITIAL> {END_OF_LINE_COMMENT}                           { yybegin(YYINITIAL); return CsvTypes.COMMENT; }

and then some override in CsvParserDefintion for getCommentTokens()

If you have any pointers on how that could work, I can try having an explore.

[Egwor]

Is there a way to sponsor this change?

[SeeSharpSoft]

As I notice some kind of demand here 😉, I will have a look how to implement it - especially with respect to the Table Editor. Guess for the first iteration, the Table Editor will not be usable if a CSV file contains comments.