Timestamp in nmea tag

[joabakk]

This is mostly for logged data. I have some logs of nmea0183 data from kplex with tags for source and epoch time. I would love to get these into influxdb with correct timestamp, and to produce polar diagrams with traceable input.

I'm not sure yet what this would require of changes to implement, but the presence of a tag with timestamp may have to be checked already in lib/index.js and not added in each codec as now.

Currently any sentence with a tag block is ignored, which is perhaps not ideal, since it is valid from v4 of nmea0183. http://www.stripydog.com/kplex/configuration.html#tag

From kplex:
If source identifier and timestamps are both requested for an interface, they are combined into a single TAG block, source identifier first, e.g.: \s:kplex,c:1423133048*5F\

[tkurki]

The ability to handle lines with tag blocks is probably a missing feature and worth implementing on its own right.

Timestamps from tag blocks is another. As there is no time information in the normal NMEA0183 input the parser could just not produce timestamp, as Node Server will fill in missing timestamps for all input. This way the parser would produce timestamps only for data that actually has timestamps.

A third issue would be the ability to play back tag-timestamped data in real time (or speeded up by a factor for example). For N2K data from canboat actisense-serial with timestamps this is implemented in https://github.com/SignalK/signalk-server-node/blob/master/providers/timestamp-throttle.js - you add that to the provider pipeline like in https://github.com/SignalK/signalk-server-node/blob/master/settings/aava-file-settings.json#L27-L32

Node server has optional logging, with the ability to multiplex input from various sources in https://github.com/SignalK/signalk-server-node/blob/master/providers/log.js with the associated provider element that allows playback in real time: https://github.com/SignalK/signalk-server-node/blob/master/providers/multiplexedlog.js#L25-L38. You could take the same approach and handle tag blocks outside the parser: use the timestamp for throttling, hand just the normal NMEA0183 to parser, but you would have to fill in the correct timestamp to the parser output.

For batch oriented import for the use case "import this log file from a day last summer" I am going for still another solution. The system must support backpressure: the pipeline will have no throttling, but the end of the pipe, writing to InfluxDb, will not be able to write as fast as the beginning, reading from a file, will produce data. Node streams, that producer pipeline uses, does that nicely.

The current signalk-to-influxdb plugin does not handle the incoming timestamp and data will be timestamped in InfluxDb per insertion time. Easy to fix, but fyi.

I am not quite sure InfluxDb is the easiest solution to your problem. It will work I guess and signalk-to-influxdb has the code for calculating TWS and TWA. One way to go about this would be to just produce CSV-type files that contain STW, TWS, TWA triplets and feed them to your polar producing code. Then again having the data in a db would be easy for later analysis. A more traditional sql db might be a better fit than a time series database, if you are not using stuff like min/max/average/median heavily.

[joabakk]

Thanks a lot for the pointers. I guess the first would be to strip the line for the tag block and parse the normal nmea as today.

Then I'm thinking of a boolean passed to the parser, for the summer trip import, use tag timestamp, else just ignore the tag timestamp and produce one as is done now.
The parser must of course be able to produce a current timestamp for normal operation, but can this be done globally and not for each sentence type?

Another useful info is the source in the tag. Say you use kplex with different inputs, and pass this to sk. It will be treated as one source in sk, even thought kplex uses several.

I follow you on influx for polars. But the first step is to get the data into influxdb to visualize. We are getting there one step at a time. It might end up with one influx for "log" and one mysql for polar data, dynamically updated.

[joabakk]

Is it even possible to split a line with \ as delimiter in js?
Stackexchange is full of "you don't, you have to escape the backslash in the string".

[timmathews]

Of course you can split a string on \. You just need to escape it in the split function:

someString.split("\\");

[joabakk]

I've been trying for a while now with this sentence: \s:compass,c:1438489697*13\$TIROT,0.1,A*3A

[timmathews]

If you're trying in the browser console or a simple Node script like this:

let x = "\s:compass,c:1438489697*13\$TIROT,0.1,A*3A";
x.split("\\");

It won't work because the \ will get eaten; you need to escape it in string literals.

let x = "\\s:compass,c:1438489697*13\\$TIROT,0.1,A*3A";
x.split("\\");

will do what you expect (well, maybe not exactly what you expect), the resulting array will look like:

[ "", "s:compass,c:1438489697*13", "$TIROT,0.1,A*3A" ]

If you're reading the string from a file or from a network socket, stream, etc, it doesn't need to be escaped. It's no different than reading a newline or any other control character. Must be escaped in string literals, works as expected everywhere else.

[joabakk]

At last an answer I could understand. Thanks @timmathews. Christmas saved!

[fabdrol]

This is implemented in the new version in the development branch. Only downside is that, since NMEA chose to use escape characters (WHY?!?!??!) as the start and end characters of the tag block, it requires these to be escaped before parsing (which is not something that developers of most Providers will think of doing, I suspect).

[joabakk]

That is ok for now. I can do a batch replace in my historic files, once the hooks are in place. Should we close the issue? Doesn't look like we will have a full solution

[fabdrol]

Well, technically they're supported - one could argue that the escaping is not our responsibility since a single \ in a javascript string is essentially not a backslash.

Edit: they're not supported entirely, as I still need to add one thing: verification of the timestamp using the checksum (why does the timestamp have a checksum?)

[fabdrol]

I'll close this for now, and add a new issue for the timestamp checksum.