Open universal json format

[tkurki]

Navgauge uses its own 'universal' format for sending basic navigation data to the browser. 'universal' and 'basic' mean that data from different sources (NMEA 0183, N2K, different sources within a N2K network) are combined on the server and sent in the same format and same fields so that the UI doesn't need to know anything about where the data is coming from. The combining is done in a boat specific configuration file in Navgauge.

This 'universal' way is something I just came up with when making the UI work universally with the data sets from three different boats that I had available. For example units are a mixture of metric & marine unit.

Creating a more universal and standard format would support sharing innovation & functionality between different efforts.

[tkurki]

I am able & willing to change the Navgauge server-to-UI json format, but would really appreciate input & insight from others.

Current format is pretty readable in the boat config files under https://github.com/tkurki/navgauge/tree/master/lib/boats

[keesverruijt]

Hi Tkurki,

A good idea to get a more formal and better JSON format amongst the nautical apps. I can adapt CANboat as needed.

[timmathews]

Hey all,

I too would like to work on formalizing the data format. Have a look at what I'm working on http://pyxis.openseasproject.org and https://github.com/timmathews/pyxis-design-documents. My goal with this is something akin to Maretron's N2KView.

The timing of this is rather fortuitous, as I have been thinking a bit about message specifications this weekend. At a high level, I don't want the client to have to know beforehand how data is formatted or what is available. Instead, the server should provide an API to query for this information. The only thing the client needs to know is how to actually make that query. So once the client makes a request to the server (standard HTTP GET request to a published URL) it will get back a list of all of the individual messages that the server knows how to send. Using this information, our user can build their display pages in the browser and save that configuration to the server. Then when one of those user created pages is displayed, another request is made via a WebSockets connection this time subscribing to the specific data that the client wants (essentially, whatever is needed to drive the gauges, charts, graphs, etc. displayed on the screen). We call this the 'control' channel. The server responds to that request with a URL to a WebSockets connection custom built for that client. The client connects to this new WebSockets URL (the 'data' channel) and begins displaying the data received.

I've pretty much worked all of this out and will be continuing to document it at http://openseasproject.org and on my GitHub page, but what I don't have yet is a good way to format the data going to the client. I know that I want to be able to break down the data packets which come in to the server (n2k, 0183, SeaTalk, whatever) and reconstitute them in a more friendly manner.

Right now, what I'm doing is using my Go port of CANboat to take data from an Actisense NGT-1, pack up the formatted PGNs (using the CANboat format) with MessagePack, send them via ZeroMQ to a very simple web server (again written in Go) which serves the data (this time JSON encoded) via WebSockets to whoever wants to connect. This is all kind of a hack, but it should be a lot better soon.

[rob42]

Yes, Im all for a standard data format, and since the web and HTML5/websockets is my target I'm all for using JSON.

If you look at freeboard (https://github.com/rob42/freeboard-server/blob/master/src/main/java/nz/co/fortytwo/freeboard/server/util/Constants.java) you will see Ive used a 3-char key and associated value. I just create an array of them and convert it to JSON.

Its a basic format, and could be improved by sub-grouping by use (eg position, engine, etc) to more efficiently handle long arrays.

The protocol is also easily used by the arduino micro-controller as a non JSON serial stream. I just chop it [0-2][3-EOL] and I have key/value for minimal CPU. The consistent 3-char key and CR between each make it easy to send or process at the arduino end. Even on the RPi or BBB there is a minimum of spare CPU to parse data, so it needs to be easy to handle.

Also in my protocol if the label starts with # its a command, which I also required. Plus its still reasonably readable for dev and debug. Probably I should add some sort of message checksum too.

So I think the protocol should have:

• JSON format
• simple key=value structure, rather than NMEA0183 style complex combinations of values.
• Human readable
• Checksum field
• fixed length short field labels to aid efficient handing on microprocessors.
• allow command key variant
• allow query key variant
• have a global repository for keys, so we dont collide when we add them.

At the client end I just pass the keys past the various gui objects and they use them if they are interested - that would work nicely for assembling custom displays. Maybe the display should also register to receive interesting keys?

Same with sending keys, a given device or gui component just issues a key=value whenever necessary, its passed to the webserver (via websockets) and routed accordingly to others.

Keep in mind there may be more than one 'server', we should think peer-2-peer.

I dont think there is much need to create a complex formal spec at this stage, just that we all work on a similar basis and co-ordinate our keys. I expect we will find new needs and opportunities, so things will change. And I like the idea (TimMatthews) that the device can be queried as to its capabilities - need a key for that.

Robert (freeboard)

[rob42]

The key is inter-operability. While we remain unique and isolated our projects will never get critical mass ...we need to get OpenCPN into this too.
Rob

[timmathews]

Totally agree we need to get OpenCPN onboard and also OpenSkipper if they're still actively developing (it's been a year since their last update).

There is another project that I've been watching at http://sailboatinstruments.blogspot.com which is pretty interesting. I'll reach out to him and see if he wants to be involved.

[tkurki]

Whoa, seems like there is some demand for this sort of thing. Making things interoperable is in my interest as well.

Some thoughts on the discussion so far:

What are we actually talking about when we want common json? CANboat and gpsd for example already produce json. In fact @canboat's comment about changing the json format sounded to me more like the tail wagging the dog - why not pick the N2K json format for everybody? Navgauge is not using it for the fact that half of my data is NMEA 0183. Furthermore when I got hold of datasets for more than one boat I realised that my first interest, a SailSteer clone gauge, needs some basic data and where they are coming from is different on every boat: Freya has several N2K sources for some of the data, Cassiopeia has both NMEA and N2K and Plaka just NMEA. All this lead me to create a basic stream of data that has just the needed fields and nothing extra.

Then again Navgauge server can serve also the raw data in json format - see http://navgauge.herokuapp.com/swipe/bus.

For N2K this is just CANboat format. For NMEA it is the dead simple format in node-nmea. Node-nmea suited my purposes most easily, so I just went with that. I need to add support for some more sentences to get my N2K tridata available as NMEA for iNavX and iRegatta.

For me the use case is really that if somebody comes up with a great display/gauge/overview I would like to use that on my boat with the minimum of hassle. Also the reverse is true: @rob42 said kind words about my multiuse gauge - if Freeboard would provide the same basic data in the same format over websocket he could just copypaste some Javascript & css and use the gauge on his system. Client side browser component infrastructure is developing quickly right now, which means that there is no one way to package a component, but I am confident something will emerge. With a little bit of extra work we could extract the gauge component and use it as part of Freeboard and Navgauge. Maybe somebody with more skills for graphics could make it prettier, now it's more functional the esthetic.

For this we need standard json format for the basic nav data or the component needs configuration options or intelligent fallback with defined precedence for the different formats.

I've also thought about providing a "list capabilities" type of call. BTW the data set is not static - if you had yesterday a certain PGN available it may not be there today. Or querying for capabilities right now doesn't give you the PGNs from the equipment that is not on.

As for the configurability evident in Pyxis plans I've decided not to go that way at all: so far Javascript/HTML/CSS is the way I configure my display. No doubt this will change, but I assume by just using javascript to instantiate more high level widgets and connect them to certain selected data from the server. Combining data items is imho pretty straightforward as well.

I am not sure about the "simple key=value structure, rather than NMEA0183 style complex combinations of values" - some items have structure and losing that structure might not be a good idea, for example yaw-pitch-roll and rates of turn for example. Also handling structure in json is dead simple.

"Human readable, Checksum field, fixed length short field labels" sound conflicting - the greatest joy of working with json as opposed to xml or some other format is that it really is human readable. I am definitely not fond of three letter abbreviations and it sounds like NMEA all over again. I plan to rely on tcp/ip instead of checking checksums and Mr Moore will take of the extra effort in parsing.

Keys on the level of "these data items I'm interested in" make sense, as in a subscription request. Something that I've run against is that the UI might be interested in a time series of some data - say STW and SOG (see me doing the three-letter-acronym thing ;-) or a single item from two different sensor - to create a plot. So having access to history data is something I'll be looking at in the client 2 server communications. I suppose my case is a bit different, since the UI code runs on the browser and the tablet or whatever might be switched off for periods and I would still like to get history graphs.

I have been thinking about in terms of client and server, not peer to peer, but can imagine scenarios for that. I suppose the json could be the same, but something other than websockets for reliable communications with queuing & retry.

[tkurki]

@timmathews I'm curious:

• Why such a long chain: binary (N2K) to parsed (analyzer json output I assume) to binary (MessagePack) to broker to webserver to json via websockets?
• Why port CANboat and not use the original?

[tkurki]

What would be the use case for this for OpenCPN? As it is a single application with direct access to the different data streams I don't see the need, but is there something I'm missing?

[tkurki]

@timmathews Navgauge server has the possibility to subscribe to

• composite feed of basic nav data ('gaugedata')
• messages by pgn (haven't used this for anything, but seemed like a good idea at the time)
• the raw data

[tkurki]

Does anybody have access to GoFree data in json format? I think we should consider being compatible with commercial stuff as well.

And btw I would be really interested in looking at other data captures. At least for me adding configurations for different boats changed the software quite a bit.

[keesverruijt]

Yes. Good questions :-)

On 17 Mar 2014, at 18:13, tkurki notifications@github.com wrote:

@timmathews I'm curious:

Why such a long chain: binary (N2K) to parsed (analyzer json output I assume) to binary (MessagePack) to broker to webserver to json via websockets?
Why port CANboat and not use the original?

My own solution uses (as you guys probably know) uses actisense-serial | analyser -json | n2kd

which is usable directly by a HTML web client. My code predates web sockets. Web sockets is pretty cool tech, but I am glad I didn’t use it at the time as the spec kept changing.

I am still happy with json as a format, and I am also happy with choosing C for the analysis — it is not a trivial bit of code and the amount of processing power on my server is limited.

One other thing that we may want to discuss is whether we can’t come up with a common server side infrastructure. Even with this small group we already have Go, PHP, node.js and Java if my info is correct. Boy. Sure sounds like a lot of wasted effort. I am happy to move from PHP to something else as long as it is reasonably light weight and will run in a few megabytes.

One thing that kept me from publishing my PHP code is that (a) there is a lot in there that doesn’t work with other vessels, unless you want to invest in a Wago PLC running Linux. (b) You need to configure a lot in json config files, and I didn’t want to field so many questions.

What I would like is to reuse those beautiful gauges and SailSteer pages that you guys have been developing.

Kees

[tkurki]

Another idea might be to make a minimal generic translator from NMEA 0183 on a wifi to an open json format.

Use case is that some existing solution puts out NMEA over wifi and one would like to have a browser app connect to it. An easy to install translator that connects to the NMEA 0183 feed and provides json over http and/or websocket would make the json format more approachable.

[tkurki]

I went for Node because I wanted to know what the fuss was about. It turned out to be a really nice experience, @canboat I urge you to try it. And like it says on the Github page, it is CANboat and gpsd/gpspipe underneath on the data acquisition side.

I'm still not sure about parsing NMEA on the server. Should probably drop it in favor of the gpsd json format. Need NMEA encoding though for N2K to NMEA conversion.