Improve usability of CSV format

[dominiklohmann]

Three changes:

• Add a vast import csv --separator=... option that defaults to ,, and tests it for space and tab separators.
• Render enums in their canonical string representation instead of their internal numerical representation.
• Import additional columns in the csv import as strings instead of crashing.

📝 Checklist

• All user-facing changes have changelog entries.
• The changes are reflected on docs.tenzir.com/vast, if necessary.
• The PR description contains instructions for the reviewer, if necessary.

🎯 Review Instructions

@rdettai can you give this a try since you requested these?

For the actual code reviewer, take a look at it commit-by-commit.

[dispanser]

Looks good.

I tested import (tsv, ssv) and export.

Thanks in particular for the extensive integration test coverage!

[rdettai]

I get the following error when I use vast import --type=aws.flowlogs csv --separator=" ":

• vast.format.csv.reader encountered invalid vast.import.csv.separator ''; must be a single character

But the import seems to have worked fine...
Any Idea what might be happening?

[dominiklohmann]

vast import --type=aws.flowlogs csv --separator=" "

That's an issue with how CAF parses the command line. If you use this instead it works:

vast import --type=aws.flowlogs csv '--separator=" "'

[dominiklohmann]

@rdettai and I went over this in a call and discovered another issue with the CSV parser that is independent of the issues solved by this PR. I plan to take a further look tomorrow.

[rdettai]

yes, just to track the full context here: without the quotes arount --separator=" " it actually ignored the option altogether and inserted everything into a single column 😄. More concretely:

• my source file looks like this:

> cat source.csv
version account-id interface-id srcaddr dstaddr srcport dstport protocol packets bytes start end action log-status
2 462855639117 eni-0a0c4004d2115a27f - - - - - - - 1654735756 1654735786 - NODATA

• the schema I have matches these columns
• when I import this as CSV (with "," as separator),
  • the importer sees this file as one column (that is expected)
  • the importer accepts the event in this file and puts its content into the a single column. That shouldn't happen because the name of that mega column (the concatenation version account-id interface-id ... action log-status) does not match any field in the schema

> vast export csv
aws.flowlogs,"2 462855639117 eni-0a0c4004d2115a27f - - - - - - - 1654735756 1654735786 - NODATA"