Enhancements to CSV plugin

[fthevenet]

Hello everyone!

It appears that the CSV plugin bundled in binjr is far more popular than I initially envisaged; as a matter of fact, it was originally built as proof-of-concept for the plugin API and never really refined since then.
It is therefore not a huge surprise than there have been several requests for enhancement on this plugin.

I'm starting this discussion to try and centralize these requests and gather feedback on their resolutions.

Please feel free to chime in.

• support for timestamp in milliseconds (CSVAdapter) #114
  This one is well underway: see 886f41e

• Add support for browsing all csv files in a folder or a zip archive.
  Basically, the same kind of thing that can already be done with log files at the moment

• Use a more robust backend
  Currently, all data from parsed csv files are stored on the heap, in a NavigableMap. While this works fine for small data set, it will become limited very quickly when trying to address larger ones.
  One way to avoir that would be to use a different approach for storing data, off the Java heap and even on disk if need be.
  Possible candidates:

  • RRD4J: Efficient for compact dataset. Not so much for sparse ones. That won't work if the step in the csv timestamp is variable (which is quite likely).
  • Lucene: As used by the log files adapter. Not the most efficient, storage wise, for strict numerical timeseries.
  • Something new? QuestDB looks interesting. No prior implementation, though.

• Add support for timestamps located elsewhere than the first column.

[xrysf03]

The CSV plugin already has quite a number of nice features, such as the user-defineable timestamp format. This rocks! I'd like to stress that CSV is a very convenient and open "data interchange format". You never know, what walks of life your users would come from, and usually as a user you can find a way to convert your data to tame CSV.
What I'm missing sorely, is more tolerance to petty syntactic anomalies in the CSV data - or a more useful error output. Such as, ability to define an ASCII string that's a placeholder for undefined value (and maybe assign a particular constant instead, such as a zero - as a configurable option). The way it is, whenever the CSV loader encounters something marginally outside of correct syntax, it FAILS TO LOAD THE FILE altogether! Produces an error message, that even includes a "stack trace" if memory serves, which however does not tell you the exact location in the CSV file where it broke, and why. So you have to wade through your megabytes of time series data and look for "things that look strange" and infer telepathically, what could possibly make the CSV loader bomb out...

[xrysf03]

Actually, I am now running Preview 3.20.0 on some CSV files that I know contain UNDEF values, and no errors yet loading them. I'll keep trying harder for a while, before I call this glitch gone :-)

[fthevenet]

The current CSV adapter's behavior is (and has been for a long time) to attempt to parse as a double any value in any column other than the one designated as containing the time stamps. If it fails to do that, then it substitutes NaN ("Not A Number") as the content for this cell, and keep on loading the rest of the file.
A NaN value is then interpreted according to the user preference set in "Charts->Treat undefined Y values as 0", i.e. the value is replaced by zero if this is set to true or the sample is ignored if this is set to false (NB: or so it should be... I have since identified a regression that would cause a NaN value to always be replaced by 0. It's fixed in 0896b21)

When it comes to values that are supposed to be parsed as time stamps, however, the behavior has so far been to indeed stop loading the file and display an error message if a single cell could not be parsed. And in such case, the message was indeed not very useful, as it only indicates the column number for the faulty value but not the line number.
I've done two things to try and address that:

1. The message now provides both the column and line numbers as well as the string it attempted (and failed) to parse as a time stamp.
2. I've added an option in the "parsing profile" dialog box where the user can decide to completely ignore any line where it couldn't find a timestamp. Please note that there is a maximum number of error that the parser will tolerate (I've set it to 1000 for now) before it decide to stop loading the whole file, but this is a by-product of the way I implemented it.

There are still other errors that will remain fatal and abort loading the file, such lines not containing value separators, but the error messages are generally useful, from the examples I saw (line number + char position).

All of that is available for testing in the latest preview build.

Cheers!