Open universal json format #4

Open
tkurki opened this Issue Mar 16, 2014 · 165 comments

Comments

Projects
None yet
@tkurki
Owner

tkurki commented Mar 16, 2014

Navgauge uses its own 'universal' format for sending basic navigation data to the browser. 'universal' and 'basic' mean that data from different sources (NMEA 0183, N2K, different sources within a N2K network) are combined on the server and sent in the same format and same fields so that the UI doesn't need to know anything about where the data is coming from. The combining is done in a boat specific configuration file in Navgauge.

This 'universal' way is something I just came up with when making the UI work universally with the data sets from three different boats that I had available. For example units are a mixture of metric & marine unit.

Creating a more universal and standard format would support sharing innovation & functionality between different efforts.

@tkurki tkurki added the enhancement label Mar 16, 2014

@tkurki

This comment has been minimized.

Show comment
Hide comment
@tkurki

tkurki Mar 16, 2014

Owner

I am able & willing to change the Navgauge server-to-UI json format, but would really appreciate input & insight from others.

Current format is pretty readable in the boat config files under https://github.com/tkurki/navgauge/tree/master/lib/boats

Owner

tkurki commented Mar 16, 2014

I am able & willing to change the Navgauge server-to-UI json format, but would really appreciate input & insight from others.

Current format is pretty readable in the boat config files under https://github.com/tkurki/navgauge/tree/master/lib/boats

@keesverruijt

This comment has been minimized.

Show comment
Hide comment
@keesverruijt

keesverruijt Mar 16, 2014

Hi Tkurki,

A good idea to get a more formal and better JSON format amongst the nautical apps. I can adapt CANboat as needed.

Hi Tkurki,

A good idea to get a more formal and better JSON format amongst the nautical apps. I can adapt CANboat as needed.

@timmathews

This comment has been minimized.

Show comment
Hide comment
@timmathews

timmathews Mar 17, 2014

Hey all,

I too would like to work on formalizing the data format. Have a look at what I'm working on http://pyxis.openseasproject.org and https://github.com/timmathews/pyxis-design-documents. My goal with this is something akin to Maretron's N2KView.

The timing of this is rather fortuitous, as I have been thinking a bit about message specifications this weekend. At a high level, I don't want the client to have to know beforehand how data is formatted or what is available. Instead, the server should provide an API to query for this information. The only thing the client needs to know is how to actually make that query. So once the client makes a request to the server (standard HTTP GET request to a published URL) it will get back a list of all of the individual messages that the server knows how to send. Using this information, our user can build their display pages in the browser and save that configuration to the server. Then when one of those user created pages is displayed, another request is made via a WebSockets connection this time subscribing to the specific data that the client wants (essentially, whatever is needed to drive the gauges, charts, graphs, etc. displayed on the screen). We call this the 'control' channel. The server responds to that request with a URL to a WebSockets connection custom built for that client. The client connects to this new WebSockets URL (the 'data' channel) and begins displaying the data received.

I've pretty much worked all of this out and will be continuing to document it at http://openseasproject.org and on my GitHub page, but what I don't have yet is a good way to format the data going to the client. I know that I want to be able to break down the data packets which come in to the server (n2k, 0183, SeaTalk, whatever) and reconstitute them in a more friendly manner.

Right now, what I'm doing is using my Go port of CANboat to take data from an Actisense NGT-1, pack up the formatted PGNs (using the CANboat format) with MessagePack, send them via ZeroMQ to a very simple web server (again written in Go) which serves the data (this time JSON encoded) via WebSockets to whoever wants to connect. This is all kind of a hack, but it should be a lot better soon.

Hey all,

I too would like to work on formalizing the data format. Have a look at what I'm working on http://pyxis.openseasproject.org and https://github.com/timmathews/pyxis-design-documents. My goal with this is something akin to Maretron's N2KView.

The timing of this is rather fortuitous, as I have been thinking a bit about message specifications this weekend. At a high level, I don't want the client to have to know beforehand how data is formatted or what is available. Instead, the server should provide an API to query for this information. The only thing the client needs to know is how to actually make that query. So once the client makes a request to the server (standard HTTP GET request to a published URL) it will get back a list of all of the individual messages that the server knows how to send. Using this information, our user can build their display pages in the browser and save that configuration to the server. Then when one of those user created pages is displayed, another request is made via a WebSockets connection this time subscribing to the specific data that the client wants (essentially, whatever is needed to drive the gauges, charts, graphs, etc. displayed on the screen). We call this the 'control' channel. The server responds to that request with a URL to a WebSockets connection custom built for that client. The client connects to this new WebSockets URL (the 'data' channel) and begins displaying the data received.

I've pretty much worked all of this out and will be continuing to document it at http://openseasproject.org and on my GitHub page, but what I don't have yet is a good way to format the data going to the client. I know that I want to be able to break down the data packets which come in to the server (n2k, 0183, SeaTalk, whatever) and reconstitute them in a more friendly manner.

Right now, what I'm doing is using my Go port of CANboat to take data from an Actisense NGT-1, pack up the formatted PGNs (using the CANboat format) with MessagePack, send them via ZeroMQ to a very simple web server (again written in Go) which serves the data (this time JSON encoded) via WebSockets to whoever wants to connect. This is all kind of a hack, but it should be a lot better soon.

@rob42

This comment has been minimized.

Show comment
Hide comment
@rob42

rob42 Mar 17, 2014

Yes, Im all for a standard data format, and since the web and HTML5/websockets is my target I'm all for using JSON.

If you look at freeboard (https://github.com/rob42/freeboard-server/blob/master/src/main/java/nz/co/fortytwo/freeboard/server/util/Constants.java) you will see Ive used a 3-char key and associated value. I just create an array of them and convert it to JSON.

Its a basic format, and could be improved by sub-grouping by use (eg position, engine, etc) to more efficiently handle long arrays.

The protocol is also easily used by the arduino micro-controller as a non JSON serial stream. I just chop it [0-2][3-EOL] and I have key/value for minimal CPU. The consistent 3-char key and CR between each make it easy to send or process at the arduino end. Even on the RPi or BBB there is a minimum of spare CPU to parse data, so it needs to be easy to handle.

Also in my protocol if the label starts with # its a command, which I also required. Plus its still reasonably readable for dev and debug. Probably I should add some sort of message checksum too.

So I think the protocol should have:

  • JSON format
  • simple key=value structure, rather than NMEA0183 style complex combinations of values.
  • Human readable
  • Checksum field
  • fixed length short field labels to aid efficient handing on microprocessors.
  • allow command key variant
  • allow query key variant
  • have a global repository for keys, so we dont collide when we add them.

At the client end I just pass the keys past the various gui objects and they use them if they are interested - that would work nicely for assembling custom displays. Maybe the display should also register to receive interesting keys?

Same with sending keys, a given device or gui component just issues a key=value whenever necessary, its passed to the webserver (via websockets) and routed accordingly to others.

Keep in mind there may be more than one 'server', we should think peer-2-peer.

I dont think there is much need to create a complex formal spec at this stage, just that we all work on a similar basis and co-ordinate our keys. I expect we will find new needs and opportunities, so things will change. And I like the idea (TimMatthews) that the device can be queried as to its capabilities - need a key for that.

Robert (freeboard)

rob42 commented Mar 17, 2014

Yes, Im all for a standard data format, and since the web and HTML5/websockets is my target I'm all for using JSON.

If you look at freeboard (https://github.com/rob42/freeboard-server/blob/master/src/main/java/nz/co/fortytwo/freeboard/server/util/Constants.java) you will see Ive used a 3-char key and associated value. I just create an array of them and convert it to JSON.

Its a basic format, and could be improved by sub-grouping by use (eg position, engine, etc) to more efficiently handle long arrays.

The protocol is also easily used by the arduino micro-controller as a non JSON serial stream. I just chop it [0-2][3-EOL] and I have key/value for minimal CPU. The consistent 3-char key and CR between each make it easy to send or process at the arduino end. Even on the RPi or BBB there is a minimum of spare CPU to parse data, so it needs to be easy to handle.

Also in my protocol if the label starts with # its a command, which I also required. Plus its still reasonably readable for dev and debug. Probably I should add some sort of message checksum too.

So I think the protocol should have:

  • JSON format
  • simple key=value structure, rather than NMEA0183 style complex combinations of values.
  • Human readable
  • Checksum field
  • fixed length short field labels to aid efficient handing on microprocessors.
  • allow command key variant
  • allow query key variant
  • have a global repository for keys, so we dont collide when we add them.

At the client end I just pass the keys past the various gui objects and they use them if they are interested - that would work nicely for assembling custom displays. Maybe the display should also register to receive interesting keys?

Same with sending keys, a given device or gui component just issues a key=value whenever necessary, its passed to the webserver (via websockets) and routed accordingly to others.

Keep in mind there may be more than one 'server', we should think peer-2-peer.

I dont think there is much need to create a complex formal spec at this stage, just that we all work on a similar basis and co-ordinate our keys. I expect we will find new needs and opportunities, so things will change. And I like the idea (TimMatthews) that the device can be queried as to its capabilities - need a key for that.

Robert (freeboard)

@rob42

This comment has been minimized.

Show comment
Hide comment
@rob42

rob42 Mar 17, 2014

The key is inter-operability. While we remain unique and isolated our projects will never get critical mass ...we need to get OpenCPN into this too.
Rob

rob42 commented Mar 17, 2014

The key is inter-operability. While we remain unique and isolated our projects will never get critical mass ...we need to get OpenCPN into this too.
Rob

@timmathews

This comment has been minimized.

Show comment
Hide comment
@timmathews

timmathews Mar 17, 2014

Totally agree we need to get OpenCPN onboard and also OpenSkipper if they're still actively developing (it's been a year since their last update).

There is another project that I've been watching at http://sailboatinstruments.blogspot.com which is pretty interesting. I'll reach out to him and see if he wants to be involved.

Totally agree we need to get OpenCPN onboard and also OpenSkipper if they're still actively developing (it's been a year since their last update).

There is another project that I've been watching at http://sailboatinstruments.blogspot.com which is pretty interesting. I'll reach out to him and see if he wants to be involved.

@tkurki

This comment has been minimized.

Show comment
Hide comment
@tkurki

tkurki Mar 17, 2014

Owner

Whoa, seems like there is some demand for this sort of thing. Making things interoperable is in my interest as well.

Some thoughts on the discussion so far:

What are we actually talking about when we want common json? CANboat and gpsd for example already produce json. In fact @canboat's comment about changing the json format sounded to me more like the tail wagging the dog - why not pick the N2K json format for everybody? Navgauge is not using it for the fact that half of my data is NMEA 0183. Furthermore when I got hold of datasets for more than one boat I realised that my first interest, a SailSteer clone gauge, needs some basic data and where they are coming from is different on every boat: Freya has several N2K sources for some of the data, Cassiopeia has both NMEA and N2K and Plaka just NMEA. All this lead me to create a basic stream of data that has just the needed fields and nothing extra.

Then again Navgauge server can serve also the raw data in json format - see http://navgauge.herokuapp.com/swipe/bus.

For N2K this is just CANboat format. For NMEA it is the dead simple format in node-nmea. Node-nmea suited my purposes most easily, so I just went with that. I need to add support for some more sentences to get my N2K tridata available as NMEA for iNavX and iRegatta.

For me the use case is really that if somebody comes up with a great display/gauge/overview I would like to use that on my boat with the minimum of hassle. Also the reverse is true: @rob42 said kind words about my multiuse gauge - if Freeboard would provide the same basic data in the same format over websocket he could just copypaste some Javascript & css and use the gauge on his system. Client side browser component infrastructure is developing quickly right now, which means that there is no one way to package a component, but I am confident something will emerge. With a little bit of extra work we could extract the gauge component and use it as part of Freeboard and Navgauge. Maybe somebody with more skills for graphics could make it prettier, now it's more functional the esthetic.

For this we need standard json format for the basic nav data or the component needs configuration options or intelligent fallback with defined precedence for the different formats.

I've also thought about providing a "list capabilities" type of call. BTW the data set is not static - if you had yesterday a certain PGN available it may not be there today. Or querying for capabilities right now doesn't give you the PGNs from the equipment that is not on.

As for the configurability evident in Pyxis plans I've decided not to go that way at all: so far Javascript/HTML/CSS is the way I configure my display. No doubt this will change, but I assume by just using javascript to instantiate more high level widgets and connect them to certain selected data from the server. Combining data items is imho pretty straightforward as well.

I am not sure about the "simple key=value structure, rather than NMEA0183 style complex combinations of values" - some items have structure and losing that structure might not be a good idea, for example yaw-pitch-roll and rates of turn for example. Also handling structure in json is dead simple.

"Human readable, Checksum field, fixed length short field labels" sound conflicting - the greatest joy of working with json as opposed to xml or some other format is that it really is human readable. I am definitely not fond of three letter abbreviations and it sounds like NMEA all over again. I plan to rely on tcp/ip instead of checking checksums and Mr Moore will take of the extra effort in parsing.

Keys on the level of "these data items I'm interested in" make sense, as in a subscription request. Something that I've run against is that the UI might be interested in a time series of some data - say STW and SOG (see me doing the three-letter-acronym thing ;-) or a single item from two different sensor - to create a plot. So having access to history data is something I'll be looking at in the client 2 server communications. I suppose my case is a bit different, since the UI code runs on the browser and the tablet or whatever might be switched off for periods and I would still like to get history graphs.

I have been thinking about in terms of client and server, not peer to peer, but can imagine scenarios for that. I suppose the json could be the same, but something other than websockets for reliable communications with queuing & retry.

Owner

tkurki commented Mar 17, 2014

Whoa, seems like there is some demand for this sort of thing. Making things interoperable is in my interest as well.

Some thoughts on the discussion so far:

What are we actually talking about when we want common json? CANboat and gpsd for example already produce json. In fact @canboat's comment about changing the json format sounded to me more like the tail wagging the dog - why not pick the N2K json format for everybody? Navgauge is not using it for the fact that half of my data is NMEA 0183. Furthermore when I got hold of datasets for more than one boat I realised that my first interest, a SailSteer clone gauge, needs some basic data and where they are coming from is different on every boat: Freya has several N2K sources for some of the data, Cassiopeia has both NMEA and N2K and Plaka just NMEA. All this lead me to create a basic stream of data that has just the needed fields and nothing extra.

Then again Navgauge server can serve also the raw data in json format - see http://navgauge.herokuapp.com/swipe/bus.

For N2K this is just CANboat format. For NMEA it is the dead simple format in node-nmea. Node-nmea suited my purposes most easily, so I just went with that. I need to add support for some more sentences to get my N2K tridata available as NMEA for iNavX and iRegatta.

For me the use case is really that if somebody comes up with a great display/gauge/overview I would like to use that on my boat with the minimum of hassle. Also the reverse is true: @rob42 said kind words about my multiuse gauge - if Freeboard would provide the same basic data in the same format over websocket he could just copypaste some Javascript & css and use the gauge on his system. Client side browser component infrastructure is developing quickly right now, which means that there is no one way to package a component, but I am confident something will emerge. With a little bit of extra work we could extract the gauge component and use it as part of Freeboard and Navgauge. Maybe somebody with more skills for graphics could make it prettier, now it's more functional the esthetic.

For this we need standard json format for the basic nav data or the component needs configuration options or intelligent fallback with defined precedence for the different formats.

I've also thought about providing a "list capabilities" type of call. BTW the data set is not static - if you had yesterday a certain PGN available it may not be there today. Or querying for capabilities right now doesn't give you the PGNs from the equipment that is not on.

As for the configurability evident in Pyxis plans I've decided not to go that way at all: so far Javascript/HTML/CSS is the way I configure my display. No doubt this will change, but I assume by just using javascript to instantiate more high level widgets and connect them to certain selected data from the server. Combining data items is imho pretty straightforward as well.

I am not sure about the "simple key=value structure, rather than NMEA0183 style complex combinations of values" - some items have structure and losing that structure might not be a good idea, for example yaw-pitch-roll and rates of turn for example. Also handling structure in json is dead simple.

"Human readable, Checksum field, fixed length short field labels" sound conflicting - the greatest joy of working with json as opposed to xml or some other format is that it really is human readable. I am definitely not fond of three letter abbreviations and it sounds like NMEA all over again. I plan to rely on tcp/ip instead of checking checksums and Mr Moore will take of the extra effort in parsing.

Keys on the level of "these data items I'm interested in" make sense, as in a subscription request. Something that I've run against is that the UI might be interested in a time series of some data - say STW and SOG (see me doing the three-letter-acronym thing ;-) or a single item from two different sensor - to create a plot. So having access to history data is something I'll be looking at in the client 2 server communications. I suppose my case is a bit different, since the UI code runs on the browser and the tablet or whatever might be switched off for periods and I would still like to get history graphs.

I have been thinking about in terms of client and server, not peer to peer, but can imagine scenarios for that. I suppose the json could be the same, but something other than websockets for reliable communications with queuing & retry.

@tkurki

This comment has been minimized.

Show comment
Hide comment
@tkurki

tkurki Mar 17, 2014

Owner

@timmathews I'm curious:

  • Why such a long chain: binary (N2K) to parsed (analyzer json output I assume) to binary (MessagePack) to broker to webserver to json via websockets?
  • Why port CANboat and not use the original?
Owner

tkurki commented Mar 17, 2014

@timmathews I'm curious:

  • Why such a long chain: binary (N2K) to parsed (analyzer json output I assume) to binary (MessagePack) to broker to webserver to json via websockets?
  • Why port CANboat and not use the original?
@tkurki

This comment has been minimized.

Show comment
Hide comment
@tkurki

tkurki Mar 17, 2014

Owner

What would be the use case for this for OpenCPN? As it is a single application with direct access to the different data streams I don't see the need, but is there something I'm missing?

Owner

tkurki commented Mar 17, 2014

What would be the use case for this for OpenCPN? As it is a single application with direct access to the different data streams I don't see the need, but is there something I'm missing?

@tkurki

This comment has been minimized.

Show comment
Hide comment
@tkurki

tkurki Mar 17, 2014

Owner

@timmathews Navgauge server has the possibility to subscribe to

  • composite feed of basic nav data ('gaugedata')
  • messages by pgn (haven't used this for anything, but seemed like a good idea at the time)
  • the raw data
Owner

tkurki commented Mar 17, 2014

@timmathews Navgauge server has the possibility to subscribe to

  • composite feed of basic nav data ('gaugedata')
  • messages by pgn (haven't used this for anything, but seemed like a good idea at the time)
  • the raw data
@tkurki

This comment has been minimized.

Show comment
Hide comment
@tkurki

tkurki Mar 17, 2014

Owner

Does anybody have access to GoFree data in json format? I think we should consider being compatible with commercial stuff as well.

And btw I would be really interested in looking at other data captures. At least for me adding configurations for different boats changed the software quite a bit.

Owner

tkurki commented Mar 17, 2014

Does anybody have access to GoFree data in json format? I think we should consider being compatible with commercial stuff as well.

And btw I would be really interested in looking at other data captures. At least for me adding configurations for different boats changed the software quite a bit.

@keesverruijt

This comment has been minimized.

Show comment
Hide comment
@keesverruijt

keesverruijt Mar 17, 2014

Yes. Good questions :-)

On 17 Mar 2014, at 18:13, tkurki notifications@github.com wrote:

@timmathews I'm curious:

Why such a long chain: binary (N2K) to parsed (analyzer json output I assume) to binary (MessagePack) to broker to webserver to json via websockets?
Why port CANboat and not use the original?

My own solution uses (as you guys probably know) uses actisense-serial | analyser -json | n2kd

which is usable directly by a HTML web client. My code predates web sockets. Web sockets is pretty cool tech, but I am glad I didn’t use it at the time as the spec kept changing.

I am still happy with json as a format, and I am also happy with choosing C for the analysis — it is not a trivial bit of code and the amount of processing power on my server is limited.

One other thing that we may want to discuss is whether we can’t come up with a common server side infrastructure. Even with this small group we already have Go, PHP, node.js and Java if my info is correct. Boy. Sure sounds like a lot of wasted effort. I am happy to move from PHP to something else as long as it is reasonably light weight and will run in a few megabytes.

One thing that kept me from publishing my PHP code is that (a) there is a lot in there that doesn’t work with other vessels, unless you want to invest in a Wago PLC running Linux. (b) You need to configure a lot in json config files, and I didn’t want to field so many questions.

What I would like is to reuse those beautiful gauges and SailSteer pages that you guys have been developing.

Kees

Yes. Good questions :-)

On 17 Mar 2014, at 18:13, tkurki notifications@github.com wrote:

@timmathews I'm curious:

Why such a long chain: binary (N2K) to parsed (analyzer json output I assume) to binary (MessagePack) to broker to webserver to json via websockets?
Why port CANboat and not use the original?

My own solution uses (as you guys probably know) uses actisense-serial | analyser -json | n2kd

which is usable directly by a HTML web client. My code predates web sockets. Web sockets is pretty cool tech, but I am glad I didn’t use it at the time as the spec kept changing.

I am still happy with json as a format, and I am also happy with choosing C for the analysis — it is not a trivial bit of code and the amount of processing power on my server is limited.

One other thing that we may want to discuss is whether we can’t come up with a common server side infrastructure. Even with this small group we already have Go, PHP, node.js and Java if my info is correct. Boy. Sure sounds like a lot of wasted effort. I am happy to move from PHP to something else as long as it is reasonably light weight and will run in a few megabytes.

One thing that kept me from publishing my PHP code is that (a) there is a lot in there that doesn’t work with other vessels, unless you want to invest in a Wago PLC running Linux. (b) You need to configure a lot in json config files, and I didn’t want to field so many questions.

What I would like is to reuse those beautiful gauges and SailSteer pages that you guys have been developing.

Kees

@tkurki

This comment has been minimized.

Show comment
Hide comment
@tkurki

tkurki Mar 17, 2014

Owner

Another idea might be to make a minimal generic translator from NMEA 0183 on a wifi to an open json format.

Use case is that some existing solution puts out NMEA over wifi and one would like to have a browser app connect to it. An easy to install translator that connects to the NMEA 0183 feed and provides json over http and/or websocket would make the json format more approachable.

Owner

tkurki commented Mar 17, 2014

Another idea might be to make a minimal generic translator from NMEA 0183 on a wifi to an open json format.

Use case is that some existing solution puts out NMEA over wifi and one would like to have a browser app connect to it. An easy to install translator that connects to the NMEA 0183 feed and provides json over http and/or websocket would make the json format more approachable.

@tkurki

This comment has been minimized.

Show comment
Hide comment
@tkurki

tkurki Mar 17, 2014

Owner

I went for Node because I wanted to know what the fuss was about. It turned out to be a really nice experience, @canboat I urge you to try it. And like it says on the Github page, it is CANboat and gpsd/gpspipe underneath on the data acquisition side.

I'm still not sure about parsing NMEA on the server. Should probably drop it in favor of the gpsd json format. Need NMEA encoding though for N2K to NMEA conversion.

Owner

tkurki commented Mar 17, 2014

I went for Node because I wanted to know what the fuss was about. It turned out to be a really nice experience, @canboat I urge you to try it. And like it says on the Github page, it is CANboat and gpsd/gpspipe underneath on the data acquisition side.

I'm still not sure about parsing NMEA on the server. Should probably drop it in favor of the gpsd json format. Need NMEA encoding though for N2K to NMEA conversion.

@nohal

This comment has been minimized.

Show comment
Hide comment
@nohal

nohal Mar 17, 2014

For OpenCPN the benefit of this is not to have to reinvent the wheel - we must implement N2K (and SeaTalk) in some way. And even though I was looking at extending @canboat conversion to NMEA-0183 for now, implementing a format adopted by all of the other projects seems to be a better idea long term. But of course, we will stay away from WebSockets and alike as they bring nothing in our case...

nohal commented Mar 17, 2014

For OpenCPN the benefit of this is not to have to reinvent the wheel - we must implement N2K (and SeaTalk) in some way. And even though I was looking at extending @canboat conversion to NMEA-0183 for now, implementing a format adopted by all of the other projects seems to be a better idea long term. But of course, we will stay away from WebSockets and alike as they bring nothing in our case...

@tkurki

This comment has been minimized.

Show comment
Hide comment
@tkurki

tkurki Mar 17, 2014

Owner

@canboat: speaking of peer to peer: If you replace the line that forks the actisense-serial | analyzer -json process with something that reads from your n2kd you should be able to use Navgauge with very little effort. See freya.js for an example on routing data from PGNs to the 'normalized' format that the UI expects.

Which all makes me think of running CANboat like you do and just have Navgauge server connect to it. Do you happen to have an init script for CANboat?

I think this reflects the main idea of this thread: there is already a de facto open source standard for N2K data and several for NMEA in json format. Where should translation from these formats to the representation usable by a UI component should take place and what format should communication between these components use?

Owner

tkurki commented Mar 17, 2014

@canboat: speaking of peer to peer: If you replace the line that forks the actisense-serial | analyzer -json process with something that reads from your n2kd you should be able to use Navgauge with very little effort. See freya.js for an example on routing data from PGNs to the 'normalized' format that the UI expects.

Which all makes me think of running CANboat like you do and just have Navgauge server connect to it. Do you happen to have an init script for CANboat?

I think this reflects the main idea of this thread: there is already a de facto open source standard for N2K data and several for NMEA in json format. Where should translation from these formats to the representation usable by a UI component should take place and what format should communication between these components use?

@keesverruijt

This comment has been minimized.

Show comment
Hide comment
@keesverruijt

keesverruijt Mar 17, 2014

Some more comments (random ramblings):

  • When I said I was willing to change the output format of analyser, I meant that I would be willing to change the json structure in terms of fields etc. -- not change away from JSON completely. The current format is a minimal conversion from the PGN format that I happened to start out with.
  • In general the clients have more CPU power than the servers, at least for my clients and my servers.
  • I experimented with filtering and queries, but in the end it was simpler to have the server just send all data that it has, and the client doing the filtering. JSON compresses extremely well, and if you have a generic data handler on the client device that distributes the data to the various consumers (gauges) it is a very efficient solution.
  • My current solution for history is to have a separate set of code that runs the same HTTP requests and then stores the data in RRD format and then generates RRDtool images. This is an "old fashioned" solution, and not very pretty. I'd love something better.
  • node.js does sound cool.

I will try to run navgauge soon. (So many things to do...)

Some more comments (random ramblings):

  • When I said I was willing to change the output format of analyser, I meant that I would be willing to change the json structure in terms of fields etc. -- not change away from JSON completely. The current format is a minimal conversion from the PGN format that I happened to start out with.
  • In general the clients have more CPU power than the servers, at least for my clients and my servers.
  • I experimented with filtering and queries, but in the end it was simpler to have the server just send all data that it has, and the client doing the filtering. JSON compresses extremely well, and if you have a generic data handler on the client device that distributes the data to the various consumers (gauges) it is a very efficient solution.
  • My current solution for history is to have a separate set of code that runs the same HTTP requests and then stores the data in RRD format and then generates RRDtool images. This is an "old fashioned" solution, and not very pretty. I'd love something better.
  • node.js does sound cool.

I will try to run navgauge soon. (So many things to do...)

@rob42

This comment has been minimized.

Show comment
Hide comment
@rob42

rob42 Mar 17, 2014

The use case for peer2peer is multiple producers, multiple consumers.

  • In freeboard I have an anemometer that reads wind data from the masthead unit. Via arduino that gives apparent wind. If it knew the boat speed it could calculate true wind. So it listens for SOG values, and when it has one it puts out true wind too. Remember this protocol will potentially be used by the sensors too (log, wind, depth, etc) plus it needs to control autopilots etc.
  • I have multiple clients on (potentially) multiple servers. They need to update each other. So I send the event stream (autopilot on/off etc) via JSON, or simple serial to the arduinos.

This is a much better model than a single central server that does all. It means that actions on any one client or device is propagated across all the devices, and things still work when others break.

key=value ... yes we should use complex objects in the JSON. (I already use this for AIS data). By key/value pairs I meant that each value should be a separate key, not comma separated bundles like NMEA0183.

short keys...as a unix geek 3 letters is enough:-) but I guess that we can have normal names for keys, it does make it easy to read. Maybe we could have a 3 letter abbrev for a key for special cases?

Rob

rob42 commented Mar 17, 2014

The use case for peer2peer is multiple producers, multiple consumers.

  • In freeboard I have an anemometer that reads wind data from the masthead unit. Via arduino that gives apparent wind. If it knew the boat speed it could calculate true wind. So it listens for SOG values, and when it has one it puts out true wind too. Remember this protocol will potentially be used by the sensors too (log, wind, depth, etc) plus it needs to control autopilots etc.
  • I have multiple clients on (potentially) multiple servers. They need to update each other. So I send the event stream (autopilot on/off etc) via JSON, or simple serial to the arduinos.

This is a much better model than a single central server that does all. It means that actions on any one client or device is propagated across all the devices, and things still work when others break.

key=value ... yes we should use complex objects in the JSON. (I already use this for AIS data). By key/value pairs I meant that each value should be a separate key, not comma separated bundles like NMEA0183.

short keys...as a unix geek 3 letters is enough:-) but I guess that we can have normal names for keys, it does make it easy to read. Maybe we could have a 3 letter abbrev for a key for special cases?

Rob

@keesverruijt

This comment has been minimized.

Show comment
Hide comment
@keesverruijt

keesverruijt Mar 17, 2014

If you are going to do that you will probably want a sender (origin) in every message, and timestamps to check that the data is current, as well as something to have multiple instances of the same data type. Hey, we’re re-inventing N2K!

Maybe the 0183 JSON format should be more like a N2K message: - have an origin (= N2K src) and timestamp added by the device that translates it to JSON.

One solution is that we develop a 0183 -> N2K converter and do away with all 0183, and/or the other way around as well. Note that n2kd has the beginning of some N2K -> NMEA0183 conversion. It only needs more work :-)

Rob,

For the peer-to-peer part you may want to look into multicast. It dramatically simplifies network configuration (aka there isn’t any.) Unfortunately our browsers won’t be able to use multicast.

Kees

On 17 Mar 2014, at 20:33, rob42 notifications@github.com wrote

The use case for peer2peer is multiple producers, multiple consumers.

In freeboard I have an anemometer that reads wind data from the masthead unit. Via arduino that gives apparent wind. If it knew the boat speed it could calculate true wind. So it listens for SOG values, and when it has one it puts out true wind too. Remember this protocol will potentially be used by the sensors too (log, wind, depth, etc) plus it needs to control autopilots etc.

I have multiple clients on (potentially) multiple servers. They need to update each other. So I send the event stream (autopilot on/off etc) via JSON, or simple serial to the arduinos.

This is a much better model than a single central server that does all. It means that actions on any one client or device is propagated across all the devices, and things still work when others break.

key=value ... yes we should use complex objects in the JSON. (I already use this for AIS data). By key/value pairs I meant that each value should be a separate key, not comma separated bundles like NMEA0183.

short keys...as a unix geek 3 letters is enough:-) but I guess that we can have normal names for keys, it does make it easy to read. Maybe we could have a 3 letter abbrev for a key for special cases?

Nah, assembly is 3 letters. UNIX does it with two: cc, as, ld, nm, od, tr, cd, cp, rm, ln, ls, ...

If you are going to do that you will probably want a sender (origin) in every message, and timestamps to check that the data is current, as well as something to have multiple instances of the same data type. Hey, we’re re-inventing N2K!

Maybe the 0183 JSON format should be more like a N2K message: - have an origin (= N2K src) and timestamp added by the device that translates it to JSON.

One solution is that we develop a 0183 -> N2K converter and do away with all 0183, and/or the other way around as well. Note that n2kd has the beginning of some N2K -> NMEA0183 conversion. It only needs more work :-)

Rob,

For the peer-to-peer part you may want to look into multicast. It dramatically simplifies network configuration (aka there isn’t any.) Unfortunately our browsers won’t be able to use multicast.

Kees

On 17 Mar 2014, at 20:33, rob42 notifications@github.com wrote

The use case for peer2peer is multiple producers, multiple consumers.

In freeboard I have an anemometer that reads wind data from the masthead unit. Via arduino that gives apparent wind. If it knew the boat speed it could calculate true wind. So it listens for SOG values, and when it has one it puts out true wind too. Remember this protocol will potentially be used by the sensors too (log, wind, depth, etc) plus it needs to control autopilots etc.

I have multiple clients on (potentially) multiple servers. They need to update each other. So I send the event stream (autopilot on/off etc) via JSON, or simple serial to the arduinos.

This is a much better model than a single central server that does all. It means that actions on any one client or device is propagated across all the devices, and things still work when others break.

key=value ... yes we should use complex objects in the JSON. (I already use this for AIS data). By key/value pairs I meant that each value should be a separate key, not comma separated bundles like NMEA0183.

short keys...as a unix geek 3 letters is enough:-) but I guess that we can have normal names for keys, it does make it easy to read. Maybe we could have a 3 letter abbrev for a key for special cases?

Nah, assembly is 3 letters. UNIX does it with two: cc, as, ld, nm, od, tr, cd, cp, rm, ln, ls, ...

@timmathews

This comment has been minimized.

Show comment
Hide comment
@timmathews

timmathews Mar 17, 2014

@tkurki So basically I reimplemented CANboat in Go (CANboatGo anyone?) in order to learn Go. I like the language and the concurrency patterns baked in make it a good fit for this sort of application. My convoluted data path from device to browser is a result of having a working application which could read N2k and spit out MsgPack and having a simple fake data generator that could speak JSON over WebSockets. It took about 10 lines of code to convert the data generator into a MsgPack/ZeroMQ to JSON/Websockets bridge, so that's what I did.

My two-cents on peer-to-peer: there's nothing in current browsers or in any W3C draft that I'm aware of that allows a browser to make any p2p connections, they're still strictly client-server. P2P between servers or devices which collect and propagate data is a good thing IMO though and should be investigated further. My recommendation is to use a message queuing system there (either brokered like RabbitMQ or brokerless like ZeroMQ) instead of multicast/anycast.

@tkurki So basically I reimplemented CANboat in Go (CANboatGo anyone?) in order to learn Go. I like the language and the concurrency patterns baked in make it a good fit for this sort of application. My convoluted data path from device to browser is a result of having a working application which could read N2k and spit out MsgPack and having a simple fake data generator that could speak JSON over WebSockets. It took about 10 lines of code to convert the data generator into a MsgPack/ZeroMQ to JSON/Websockets bridge, so that's what I did.

My two-cents on peer-to-peer: there's nothing in current browsers or in any W3C draft that I'm aware of that allows a browser to make any p2p connections, they're still strictly client-server. P2P between servers or devices which collect and propagate data is a good thing IMO though and should be investigated further. My recommendation is to use a message queuing system there (either brokered like RabbitMQ or brokerless like ZeroMQ) instead of multicast/anycast.

@rob42

This comment has been minimized.

Show comment
Hide comment
@rob42

rob42 Mar 17, 2014

Yeah re-inventing NMEA200 would be great - its problem is its closed proprietary nature. If we implemented it we would be forever caught up with conflicting PGM's (theirs/ours) and susceptible to them changing spec on us. So we should 'embrace and extend' it :-)
A mapping of PGMs to our keys, and learning from their implementation is a good thing. Building bi-directional message converters (NMEA0183, NMEA2000, etc) to our JSON is the way to go.

Multi-cast isnt really necessary IMO although its quite interesting, I use Apache Camel to do smart processing/routing/conversion which is pretty easy and powerful. The browser will never be a full p2p client - its always going to have to be pushed by the webserver although websockets makes that pretty easy.

But the server implementation should be agnostic - we all have our preferences there. Also the transport, be it wifi, wired network, CANbus, or serial should not matter if we send the same JSON string.

Rob

rob42 commented Mar 17, 2014

Yeah re-inventing NMEA200 would be great - its problem is its closed proprietary nature. If we implemented it we would be forever caught up with conflicting PGM's (theirs/ours) and susceptible to them changing spec on us. So we should 'embrace and extend' it :-)
A mapping of PGMs to our keys, and learning from their implementation is a good thing. Building bi-directional message converters (NMEA0183, NMEA2000, etc) to our JSON is the way to go.

Multi-cast isnt really necessary IMO although its quite interesting, I use Apache Camel to do smart processing/routing/conversion which is pretty easy and powerful. The browser will never be a full p2p client - its always going to have to be pushed by the webserver although websockets makes that pretty easy.

But the server implementation should be agnostic - we all have our preferences there. Also the transport, be it wifi, wired network, CANbus, or serial should not matter if we send the same JSON string.

Rob

@timmathews

This comment has been minimized.

Show comment
Hide comment
@timmathews

timmathews Mar 17, 2014

So peer pressure has made me clean up my server a bit and push it to GitHub. I called it argo. It's really pretty simple stuff, running it on a machine which has an NGT-1 connected, it will take n2k data from that and publish it to a WebSockets stream at /ws/v1/data. You can check it out now at http://pyxis.openseasproject.org/n2k.html

That's live data from a Maretron WSO100 in my basement with a fan blowing on it.

It needs to be cleaned up a bit and I need to do some housekeeping like adding a README and licensing and a nice thank you letter to Kees for doing all the hard work of figuring out just what the heck the NGT-1 was saying.

So peer pressure has made me clean up my server a bit and push it to GitHub. I called it argo. It's really pretty simple stuff, running it on a machine which has an NGT-1 connected, it will take n2k data from that and publish it to a WebSockets stream at /ws/v1/data. You can check it out now at http://pyxis.openseasproject.org/n2k.html

That's live data from a Maretron WSO100 in my basement with a fan blowing on it.

It needs to be cleaned up a bit and I need to do some housekeeping like adding a README and licensing and a nice thank you letter to Kees for doing all the hard work of figuring out just what the heck the NGT-1 was saying.

@rob42

This comment has been minimized.

Show comment
Hide comment
@rob42

rob42 Mar 17, 2014

@timmathews - nice - I can kind of see what its saying, but there is a lot of misc data in there. So in freeboard I would send something like 'WSA=20[CR]WDA=295[CR]' in the simple serial version, and in JSON it would ideally become something like:
{"wind":{"apparentSpeed":20,"apparentDirection":295}}

Then any interested listener can look at a JSON message and if it has a 'wind' object, get the 'wind.apparentSpeed'

I do this for AIS data, using an array of aisinfo objects in an ais header object. each aisinfo is one vessel, and holds speed dir, lon. lat etc.

Rob

rob42 commented Mar 17, 2014

@timmathews - nice - I can kind of see what its saying, but there is a lot of misc data in there. So in freeboard I would send something like 'WSA=20[CR]WDA=295[CR]' in the simple serial version, and in JSON it would ideally become something like:
{"wind":{"apparentSpeed":20,"apparentDirection":295}}

Then any interested listener can look at a JSON message and if it has a 'wind' object, get the 'wind.apparentSpeed'

I do this for AIS data, using an array of aisinfo objects in an ais header object. each aisinfo is one vessel, and holds speed dir, lon. lat etc.

Rob

@timmathews

This comment has been minimized.

Show comment
Hide comment
@timmathews

timmathews Mar 17, 2014

@rob42 So yeah, there is a lot of info in each packet. The structure is something like this for wind (130306):

{"Header": {
  "Timestamp": "2014-03-17T17:34:16.978571418-04:00", // ISO-8601 Format timestamp
  "Priority": 2, // Message priority on the wire as defined by NMEA
  "Source": 128, // Device ID claimed by the WSO100 at bootup
  "Destination": 255, // Every listener on the bus
  "Pgn": 130306,  // PGN ID
  "Length": 8, // Number of bytes on the wire
  "Data":"7LcAzUz6//8=" // Raw data un-decoded
  },
  "Index": 159, // The index of the definition of the PGN in my map of PGN definitions
  "Data": {
    "0":236, // Field 0, in this case the SID
    "1":1.83, // Field 1, wind speed
    "2":112.64932725148729, // Field 2, wind direction
    "3":"Apparent" // Field 3, measurement type
  }
}

One of the features that I intended to implement originally was a message definition query API, so if you saw this data come in, you would make an HTTP GET request to /api/v1/pgn/:index: where :index: is the number from the packet above and you would get JSON back describing the data: field names, precision, range, units, etc. So that's why Index is there. As for the rest, well I already had that data from the n2k bus, so until I know what I need and what I don't, lets send it along.

@rob42 So yeah, there is a lot of info in each packet. The structure is something like this for wind (130306):

{"Header": {
  "Timestamp": "2014-03-17T17:34:16.978571418-04:00", // ISO-8601 Format timestamp
  "Priority": 2, // Message priority on the wire as defined by NMEA
  "Source": 128, // Device ID claimed by the WSO100 at bootup
  "Destination": 255, // Every listener on the bus
  "Pgn": 130306,  // PGN ID
  "Length": 8, // Number of bytes on the wire
  "Data":"7LcAzUz6//8=" // Raw data un-decoded
  },
  "Index": 159, // The index of the definition of the PGN in my map of PGN definitions
  "Data": {
    "0":236, // Field 0, in this case the SID
    "1":1.83, // Field 1, wind speed
    "2":112.64932725148729, // Field 2, wind direction
    "3":"Apparent" // Field 3, measurement type
  }
}

One of the features that I intended to implement originally was a message definition query API, so if you saw this data come in, you would make an HTTP GET request to /api/v1/pgn/:index: where :index: is the number from the packet above and you would get JSON back describing the data: field names, precision, range, units, etc. So that's why Index is there. As for the rest, well I already had that data from the n2k bus, so until I know what I need and what I don't, lets send it along.

@tkurki

This comment has been minimized.

Show comment
Hide comment
@tkurki

tkurki Mar 18, 2014

Owner

Your example shows what @canboat mentioned: it is often easier to send everything. Bandwidth on a local wifi hotspot is a nonissue and browsers running on tablets and phones have ample cpu for handling the data.

In the interest of a common json format this doesn't sound too promising though: you've come up with an alternate json structure to N2K data that is different from the one CANboat analyzer uses.

Getting data flowing and visible on the screen gives you the moment of satisfaction (at least it did for me!). When you start dealing with multiple sources and multiple formats it makes you start thinking about interoperability.

A concrete example would be that if I wanted to put your data as a sample data set for Navgauge I would have to deal with one more format. If you were using N2K json format I would just try freya.js and it would probably work with a parameter change or two. After that I would be a bit wiser, extract the common logic for pure N2K type boats and make it easier to adapt for such boats. And your data would be available to the wind history plot in Navgauge should you choose to try it out.

I don't see the need for separate metadata api, as I don't see why the format needs to be so verbose in other aspects but the data fields are just "0", "1" instead of properly named fields like they are in the CANboat format. Btw an array would more idiomatic for a list of fields.

As for the 'simple' json Rob mentioned: this is what Navgauge has for 'gaugeData' (as opposed to raw data):

{type:wind,reference:apparent,angle:54,speed:6.32}
{type:wind,reference:true boat,speed:5.172514148866067,angle:81.29917903663596} 
{type:depth,depth:13.38} 
{type:course,heading:167.6}
{type:speed,knots:5.7} 

A separate 'type' field like this has works well for switch type handling logic, but Rob's proposed typed subelement would allow one top level message to contain several different types. For now I don't need timestamp - whatever is received is displayed immediately. Once I delve into fetching history from the server this will change.

Several inconsistencies that need to be fixed: boat speed item named 'knots', but wind speed is just speed without unit, depth depth..(sheesh who did this stuff..).

Owner

tkurki commented Mar 18, 2014

Your example shows what @canboat mentioned: it is often easier to send everything. Bandwidth on a local wifi hotspot is a nonissue and browsers running on tablets and phones have ample cpu for handling the data.

In the interest of a common json format this doesn't sound too promising though: you've come up with an alternate json structure to N2K data that is different from the one CANboat analyzer uses.

Getting data flowing and visible on the screen gives you the moment of satisfaction (at least it did for me!). When you start dealing with multiple sources and multiple formats it makes you start thinking about interoperability.

A concrete example would be that if I wanted to put your data as a sample data set for Navgauge I would have to deal with one more format. If you were using N2K json format I would just try freya.js and it would probably work with a parameter change or two. After that I would be a bit wiser, extract the common logic for pure N2K type boats and make it easier to adapt for such boats. And your data would be available to the wind history plot in Navgauge should you choose to try it out.

I don't see the need for separate metadata api, as I don't see why the format needs to be so verbose in other aspects but the data fields are just "0", "1" instead of properly named fields like they are in the CANboat format. Btw an array would more idiomatic for a list of fields.

As for the 'simple' json Rob mentioned: this is what Navgauge has for 'gaugeData' (as opposed to raw data):

{type:wind,reference:apparent,angle:54,speed:6.32}
{type:wind,reference:true boat,speed:5.172514148866067,angle:81.29917903663596} 
{type:depth,depth:13.38} 
{type:course,heading:167.6}
{type:speed,knots:5.7} 

A separate 'type' field like this has works well for switch type handling logic, but Rob's proposed typed subelement would allow one top level message to contain several different types. For now I don't need timestamp - whatever is received is displayed immediately. Once I delve into fetching history from the server this will change.

Several inconsistencies that need to be fixed: boat speed item named 'knots', but wind speed is just speed without unit, depth depth..(sheesh who did this stuff..).

@rob42

This comment has been minimized.

Show comment
Hide comment
@rob42

rob42 Mar 18, 2014

Ah yes units of measure...lets just chose one consistent set for marine and convert when needed, then we dont have to mess with 'units' tags and extra code to deal with it.

So we could use knots for SOG and wind speeds, decimal degrees for angles, meters for distance and depth, UTC for time. Assume numbers may have decimals, eg floats

{
  timestamp: ,//possibly use this on any level
  source: ,//possibly use this on any level
  checksum:,
  wind:{ 
    apparent:{
      angle:54,
      speed:6.32
    },
    true: { //aackk - reserved word?
      speed:5.172514148866067,
      angle:81.29917903663596
    } 
    },
  depth:13.38,
  course:{
    heading:167.6,
    speed:5.7
    },
  position:{
    latitude:-41.5678,
    longitude: 173.23456
    },
  ais:{
    aisinfo:{
      name: ,
      latitude:,
      longitude:,
      heading:,
      speed:
      },
    aisinfo:{
      name: ,
      latitude:,
      longitude:,
      heading:,
      speed:
      },
    aisinfo:{
      name: ,
      latitude:,
      longitude:,
      heading:,
      speed:
    }
  }
}

Maybe even do meters/sec for speeds - would that be an advantage for complex analysis/comparison later?

Rob

rob42 commented Mar 18, 2014

Ah yes units of measure...lets just chose one consistent set for marine and convert when needed, then we dont have to mess with 'units' tags and extra code to deal with it.

So we could use knots for SOG and wind speeds, decimal degrees for angles, meters for distance and depth, UTC for time. Assume numbers may have decimals, eg floats

{
  timestamp: ,//possibly use this on any level
  source: ,//possibly use this on any level
  checksum:,
  wind:{ 
    apparent:{
      angle:54,
      speed:6.32
    },
    true: { //aackk - reserved word?
      speed:5.172514148866067,
      angle:81.29917903663596
    } 
    },
  depth:13.38,
  course:{
    heading:167.6,
    speed:5.7
    },
  position:{
    latitude:-41.5678,
    longitude: 173.23456
    },
  ais:{
    aisinfo:{
      name: ,
      latitude:,
      longitude:,
      heading:,
      speed:
      },
    aisinfo:{
      name: ,
      latitude:,
      longitude:,
      heading:,
      speed:
      },
    aisinfo:{
      name: ,
      latitude:,
      longitude:,
      heading:,
      speed:
    }
  }
}

Maybe even do meters/sec for speeds - would that be an advantage for complex analysis/comparison later?

Rob

@keesverruijt

This comment has been minimized.

Show comment
Hide comment
@keesverruijt

keesverruijt Mar 18, 2014

Hmm. “most convenient” for one is “hate” for others. I happen to agree with your “most convenient” units, but our set is just one of many possibilities.

There is one way out of that conundrum: use scientific units, e.g. SI.

The seven basic units in SI are: metre, kilogram, second, ampere, kelvin, mole, candela.
Derived units are (limiting myself to what I think we’ll need here): radian, hertz, newton, pascal, joule, watt, coulomb, volt, farad, ohm.

If the format does not use units then the implied units should always be an SI unit. In other words: m/s, m, liters, degrees K, radians, radians/s. Floats is a good idea, it gets us out of the silly units that N2K uses for some datagram components. Time always expressed as UTC, and maybe always as seconds since 1970 (UNIX epoch) with an optional fractional part?

Oh, and we define that the locale is “POSIX", e.g. numbers are expressed as nnn.ddd, e.g. with a decimal point and not any other character, like my native comma (in Dutch we swap . and , in numbers compared to English.) And we define that the character set is always UTF-8.

@canboat

On 18 Mar 2014, at 08:53, rob42 notifications@github.com wrote:

Ah yes units of measure...lets just chose one consistent set for marine and convert when needed, then we dont have to mess with 'units' tags and extra code to deal with it.

So we could use knots for SOG and wind speeds, decimal degrees for angles, meters for distance and depth, UTC for time. Assume numbers may have decimals, eg floats

Hmm. “most convenient” for one is “hate” for others. I happen to agree with your “most convenient” units, but our set is just one of many possibilities.

There is one way out of that conundrum: use scientific units, e.g. SI.

The seven basic units in SI are: metre, kilogram, second, ampere, kelvin, mole, candela.
Derived units are (limiting myself to what I think we’ll need here): radian, hertz, newton, pascal, joule, watt, coulomb, volt, farad, ohm.

If the format does not use units then the implied units should always be an SI unit. In other words: m/s, m, liters, degrees K, radians, radians/s. Floats is a good idea, it gets us out of the silly units that N2K uses for some datagram components. Time always expressed as UTC, and maybe always as seconds since 1970 (UNIX epoch) with an optional fractional part?

Oh, and we define that the locale is “POSIX", e.g. numbers are expressed as nnn.ddd, e.g. with a decimal point and not any other character, like my native comma (in Dutch we swap . and , in numbers compared to English.) And we define that the character set is always UTF-8.

@canboat

On 18 Mar 2014, at 08:53, rob42 notifications@github.com wrote:

Ah yes units of measure...lets just chose one consistent set for marine and convert when needed, then we dont have to mess with 'units' tags and extra code to deal with it.

So we could use knots for SOG and wind speeds, decimal degrees for angles, meters for distance and depth, UTC for time. Assume numbers may have decimals, eg floats

@timmathews

This comment has been minimized.

Show comment
Hide comment
@timmathews

timmathews Mar 18, 2014

In the interest of a common json format this doesn't sound too promising though: you've come up with an alternate json structure to N2K data that is different from the one CANboat analyzer uses.

I'm not set on this format, in fact I pretty much like @canboat's format better. On the other hand, I think we should think long and hard before fixing our JSON representation to n2k PGN definitions. NMEA2000 is essentially an extension of SAE J1939 and the engineers who designed those standards had to make certain decisions informed by their transmission medium and available bandwidth. Our answers to those same questions will be different because we're working in a different medium with different rules.

I don't see the need for separate metadata api, as I don't see why the format needs to be so verbose in other aspects but the data fields are just "0", "1" instead of properly named fields like they are in the CANboat format.

Having a separate metadata API allows me to communicate a lot of static info to the client one time. Things like a nice display name for each field, an abbreviated version of the display name, the units that the field reports (or in the case of what lookup fields (enums) the possible values for the field), min and max values and what values are reported for errors. The field IDs in my current code are just numbers because I was being lazy at the time.

Hmm. “most convenient” for one is “hate” for others. I happen to agree with your “most convenient” units, but our set is just one of many possibilities.

As far as units go, consider this my vote for SI units, ISO-8601 timestamps, POSIX locale, and UTF-8.

In the interest of a common json format this doesn't sound too promising though: you've come up with an alternate json structure to N2K data that is different from the one CANboat analyzer uses.

I'm not set on this format, in fact I pretty much like @canboat's format better. On the other hand, I think we should think long and hard before fixing our JSON representation to n2k PGN definitions. NMEA2000 is essentially an extension of SAE J1939 and the engineers who designed those standards had to make certain decisions informed by their transmission medium and available bandwidth. Our answers to those same questions will be different because we're working in a different medium with different rules.

I don't see the need for separate metadata api, as I don't see why the format needs to be so verbose in other aspects but the data fields are just "0", "1" instead of properly named fields like they are in the CANboat format.

Having a separate metadata API allows me to communicate a lot of static info to the client one time. Things like a nice display name for each field, an abbreviated version of the display name, the units that the field reports (or in the case of what lookup fields (enums) the possible values for the field), min and max values and what values are reported for errors. The field IDs in my current code are just numbers because I was being lazy at the time.

Hmm. “most convenient” for one is “hate” for others. I happen to agree with your “most convenient” units, but our set is just one of many possibilities.

As far as units go, consider this my vote for SI units, ISO-8601 timestamps, POSIX locale, and UTF-8.

@timmathews

This comment has been minimized.

Show comment
Hide comment
@timmathews

timmathews Mar 18, 2014

Here's the metadata API in action: http://pyxis.openseasproject.org/api/v1/messages for the whole list and http://pyxis.openseasproject.org/api/v1/messages/130310 for a specific PGN.

It doesn't work for all PGNs currently because of a limitation with Go's JSON encoder. Hopefully I'll have a chance to work that out tonight. It also only has the data that @canboat included in the original array of PGNs. I haven't added all of the other data that I'd like to see (min, max, and error values for example) mostly because I don't know what they are.

Here's the metadata API in action: http://pyxis.openseasproject.org/api/v1/messages for the whole list and http://pyxis.openseasproject.org/api/v1/messages/130310 for a specific PGN.

It doesn't work for all PGNs currently because of a limitation with Go's JSON encoder. Hopefully I'll have a chance to work that out tonight. It also only has the data that @canboat included in the original array of PGNs. I haven't added all of the other data that I'd like to see (min, max, and error values for example) mostly because I don't know what they are.

@rob42

This comment has been minimized.

Show comment
Hide comment
@rob42

rob42 Mar 18, 2014

I'm ok with SI units but we should use degrees rather than radians, and knots ( Wikipedia - The knot is a non-SI unit that is "accepted for use with the SI".).

This is mainly because of the relationship between navigational maths ( 1 knot=1degree, etc), boaties general knowledge (for support), and to save constant conversion to and from existing defacto nautical units.

Also I looked into JSON on the arduino - as I expected it uses a lot of scarce RAM, and a good slice of precious CPU. In general micro-controllers wont be able to deal with it easily. However the alternative key/value I currently use extends nicely to POS.LAT=xxxx.nn[CR] etc. And that can be easily parsed by a finite state machine concept, so its possibly more efficient than my current version.

I know just the person to build NMEA0183<>JSON conversion, I'll invite him in.

rob42 commented Mar 18, 2014

I'm ok with SI units but we should use degrees rather than radians, and knots ( Wikipedia - The knot is a non-SI unit that is "accepted for use with the SI".).

This is mainly because of the relationship between navigational maths ( 1 knot=1degree, etc), boaties general knowledge (for support), and to save constant conversion to and from existing defacto nautical units.

Also I looked into JSON on the arduino - as I expected it uses a lot of scarce RAM, and a good slice of precious CPU. In general micro-controllers wont be able to deal with it easily. However the alternative key/value I currently use extends nicely to POS.LAT=xxxx.nn[CR] etc. And that can be easily parsed by a finite state machine concept, so its possibly more efficient than my current version.

I know just the person to build NMEA0183<>JSON conversion, I'll invite him in.

@timmathews

This comment has been minimized.

Show comment
Hide comment
@timmathews

timmathews Mar 18, 2014

@rob42 In my client registration document I describe how the client can request from the server which units it would like to see it's data in. This way we can let the server do all of the heavy lifting of unit conversion and the client can just display what it gets. One of the other ideas which I need to explain better is that instead of every client getting the full firehose of every message available all the time, every client would get it's own unique stream of data from the server which would only contain the data that it specifically asked for.

So say I have an iPad app which has several different pages of data displayed (perhaps an engine page, a wind/boat speed page, an environmental page, and a routing page) and each page has eight or ten data points that it displays. In my contrived example here, that's about forty different data points that the client needs to handle. In reality it may be hundreds (just look at the example pages put together by Maretron for N2KView), but at any rate the client is only displaying a small number at any one time. Therefore, we only need to listen to that small number.

Upon loading a display page the client tells the server which data it's currently displaying and the server builds a custom data stream just for it. In that stream, data is formatted exactly how the client requested it and (optionally) it arrives at a rate requested by the client.

This also helps little RAM-starved micro-controller clients, since they only have to deal with a small subset of the data available. Of course the flip side is that it pretty much makes it impossible for one to be the central server in a system like this, but I was targeting ARM-based platforms when I thought all this up.

@rob42 In my client registration document I describe how the client can request from the server which units it would like to see it's data in. This way we can let the server do all of the heavy lifting of unit conversion and the client can just display what it gets. One of the other ideas which I need to explain better is that instead of every client getting the full firehose of every message available all the time, every client would get it's own unique stream of data from the server which would only contain the data that it specifically asked for.

So say I have an iPad app which has several different pages of data displayed (perhaps an engine page, a wind/boat speed page, an environmental page, and a routing page) and each page has eight or ten data points that it displays. In my contrived example here, that's about forty different data points that the client needs to handle. In reality it may be hundreds (just look at the example pages put together by Maretron for N2KView), but at any rate the client is only displaying a small number at any one time. Therefore, we only need to listen to that small number.

Upon loading a display page the client tells the server which data it's currently displaying and the server builds a custom data stream just for it. In that stream, data is formatted exactly how the client requested it and (optionally) it arrives at a rate requested by the client.

This also helps little RAM-starved micro-controller clients, since they only have to deal with a small subset of the data available. Of course the flip side is that it pretty much makes it impossible for one to be the central server in a system like this, but I was targeting ARM-based platforms when I thought all this up.

@rob42

This comment has been minimized.

Show comment
Hide comment
@rob42

rob42 Mar 19, 2014

@timmatthews I understand the concept, its a listener pattern similar to a gui, where a component adds itself as a listener for certain types of events. Its a good pattern, but I think we should be careful to assume we have power to burn at the server.

Freeboard runs on a Raspberry Pi, and is quite capable of running a steady 50% CPU when IMU, GPS, Wind, log, et al are pushing data, and its converting/routing to a browser on websockets, and TCP port for OpenCpn. Its also running the wifi hostap, webserver for browser, and misc tasks. Its not due to slow code either, (jdk8 hard float on ARM, java is actually only about 8% cpu), a lot of the CPU is USB IO stuff.

The point is that to get low power consumption you need efficient systems on small CPUs, hence we cant assume a laptop or PC etc. In that vein I agree with tkurki that the clients have more CPU than the server, so pushing out tasks to them is a good move.

The only filtering I would use would be at the 'wind.*' level, especially for arduinos over USB serial.

Rob

rob42 commented Mar 19, 2014

@timmatthews I understand the concept, its a listener pattern similar to a gui, where a component adds itself as a listener for certain types of events. Its a good pattern, but I think we should be careful to assume we have power to burn at the server.

Freeboard runs on a Raspberry Pi, and is quite capable of running a steady 50% CPU when IMU, GPS, Wind, log, et al are pushing data, and its converting/routing to a browser on websockets, and TCP port for OpenCpn. Its also running the wifi hostap, webserver for browser, and misc tasks. Its not due to slow code either, (jdk8 hard float on ARM, java is actually only about 8% cpu), a lot of the CPU is USB IO stuff.

The point is that to get low power consumption you need efficient systems on small CPUs, hence we cant assume a laptop or PC etc. In that vein I agree with tkurki that the clients have more CPU than the server, so pushing out tasks to them is a good move.

The only filtering I would use would be at the 'wind.*' level, especially for arduinos over USB serial.

Rob

@rob42

This comment has been minimized.

Show comment
Hide comment
@rob42

rob42 Mar 19, 2014

@timmatthews ...request from the server which units...
Sorry I missed addressing this...the problem is tracking who wants what in which units, and routing/converting it an efficient way. The complexity it adds to implementations and the protocol is substantial.
In terms of interoperability with other projects a simple protocol is a big advantage.
Rob

rob42 commented Mar 19, 2014

@timmatthews ...request from the server which units...
Sorry I missed addressing this...the problem is tracking who wants what in which units, and routing/converting it an efficient way. The complexity it adds to implementations and the protocol is substantial.
In terms of interoperability with other projects a simple protocol is a big advantage.
Rob

@timmathews

This comment has been minimized.

Show comment
Hide comment
@timmathews

timmathews Mar 19, 2014

Let me apologize in advance for a longish post (with a bit of a rant towards the end).

Freeboard runs on a Raspberry Pi, and is quite capable of running a steady 50% CPU when IMU, GPS, Wind, log, et al are pushing data, and its converting/routing to a browser on websockets, and TCP port for OpenCpn. Its also running the wifi hostap, webserver for browser, and misc tasks. Its not due to slow code either, (jdk8 hard float on ARM, java is actually only about 8% cpu), a lot of the CPU is USB IO stuff.

If I had to hazard a guess here, the root cause of that level of CPU consumption is that on the rPi the Ethernet interface is a USB device connected to the onboard USB hub. You'd probably see significantly reduced CPU consumption on something like a Beaglebone Black which has a dedicated Ethernet interface. Just speculation as I don't have either device available to test on.

The point is that to get low power consumption you need efficient systems on small CPUs, hence we cant assume a laptop or PC etc. In that vein I agree with tkurki that the clients have more CPU than the server, so pushing out tasks to them is a good move.

Assuming we're powering and charging the clients on the boat, the net power consumption is probably about the same regardless of where the conversion is happening. Whether device A or device B is burning CPU cycles to do unit conversion doesn't really matter if they're both using a common power source of the boat's house batteries.

[T]he problem is tracking who wants what in which units, and routing/converting it an efficient way. The complexity it adds to implementations and the protocol is substantial.

You're right, it is a complex problem. A better approach to the units question may be to let the user set their preferences globally. After all, we're probably going to be doing some conversion because the n2k data (while pretty good) isn't always consistent or in a format that most people understand (radians for instance) and 0183 is all over the map. Internally, we need to pick a format and stick with it for calculations and storage, but as a last step before broadcasting, we could perform the conversions that the user wanted. It's probably better than doing a different conversion for each client and from an end user perspective, it's better than having to configure my unit preferences on every device.

Clients could obviously still do whatever conversions they want (which would be useful if you have a guest who prefers fathoms over meters or wants to know your fuel consumption in rods/hogshead).

As far as routing goes, topic-based subscriptions are also an option. See the Topics Tutorial from RabbitMQ for a primer (unless enterprise messaging is your métier too, in which case we may have more in common than open source boat bits). Depending on the implementation, this can be done client side or server side and it's usually up to the messaging stack how that's handled, ZeroMQ does it at the Publisher and RabbitMQ (and probably most other brokers) handle it at the broker.

In terms of interoperability with other projects a simple protocol is a big advantage.

I agree completely. However, one of the things that I don't like about n2k and the upcoming OneNet from NMEA is that every consumer of the data available has to already know how that data is formatted, what all the lookup fields mean, what units are being used, etc. A very real consequence of this is that in order for my Garmin chart plotter to be able to display audio track information from my Fusion stereo, Garmin has to spend engineering hours getting a spec from Fusion, implementing and testing said spec and then they have to release a firmware update which I have to download at home, put on an SD card, drive to the boat, stick in my chart plotter and then wait 30 minutes or so while it updates. This is dumb (plus Garmin has chosen not to do this for my particular model, so I'm screwed). The plotter already knows how to display audio track information on the screen in a useful manner because it can get that from a Garmin XM receiver. It should be able to query my stereo and find out that a) it provides audio track information and b) how that information is formatted. Now, I as the end user might be required to provide some help viz. mapping the specific Fusion-fields to their respective Garmin-fields, but provided each field has a human-parseable description, this is easy (and would only need to be done once).

And that is why I am proposing serving up metadata. Because it means that new players can produce messages which everyone can make use of without them having to go through a review process and become a "standard" message or be relegated to "manufacturer specific" messages that only that manufacturer and their friends can read. And most importantly, it get's closer to a real plug'n'play environment.

We focus on the how of the formatting, but not the what. We don't need canonical message IDs (PGNs), and if IDs exist at all, they can be defined by the server(s) on the network. We tell the client up-front what units we're using and then never transmit that information again unless they ask for it.

Another benefit of metadata is the ability for the end-user to globally configure boat-specific things. For instance, let's say I have an engine on my boat that tops out at 4000 RPM, and I know my fuel consumption goes all to hell above 3000 RPM, idle is 500 RPM and that the sweet-spot performance-wise is about 2200 RPM. I might want to configure my server with that information so that when a client requests RPM info, it knows automatically that it should display a gauge from 0 - 4000 RPM, that it should color 0-500 red, 2000-2400 green, and 3000-4000 red.

Let me apologize in advance for a longish post (with a bit of a rant towards the end).

Freeboard runs on a Raspberry Pi, and is quite capable of running a steady 50% CPU when IMU, GPS, Wind, log, et al are pushing data, and its converting/routing to a browser on websockets, and TCP port for OpenCpn. Its also running the wifi hostap, webserver for browser, and misc tasks. Its not due to slow code either, (jdk8 hard float on ARM, java is actually only about 8% cpu), a lot of the CPU is USB IO stuff.

If I had to hazard a guess here, the root cause of that level of CPU consumption is that on the rPi the Ethernet interface is a USB device connected to the onboard USB hub. You'd probably see significantly reduced CPU consumption on something like a Beaglebone Black which has a dedicated Ethernet interface. Just speculation as I don't have either device available to test on.

The point is that to get low power consumption you need efficient systems on small CPUs, hence we cant assume a laptop or PC etc. In that vein I agree with tkurki that the clients have more CPU than the server, so pushing out tasks to them is a good move.

Assuming we're powering and charging the clients on the boat, the net power consumption is probably about the same regardless of where the conversion is happening. Whether device A or device B is burning CPU cycles to do unit conversion doesn't really matter if they're both using a common power source of the boat's house batteries.

[T]he problem is tracking who wants what in which units, and routing/converting it an efficient way. The complexity it adds to implementations and the protocol is substantial.

You're right, it is a complex problem. A better approach to the units question may be to let the user set their preferences globally. After all, we're probably going to be doing some conversion because the n2k data (while pretty good) isn't always consistent or in a format that most people understand (radians for instance) and 0183 is all over the map. Internally, we need to pick a format and stick with it for calculations and storage, but as a last step before broadcasting, we could perform the conversions that the user wanted. It's probably better than doing a different conversion for each client and from an end user perspective, it's better than having to configure my unit preferences on every device.

Clients could obviously still do whatever conversions they want (which would be useful if you have a guest who prefers fathoms over meters or wants to know your fuel consumption in rods/hogshead).

As far as routing goes, topic-based subscriptions are also an option. See the Topics Tutorial from RabbitMQ for a primer (unless enterprise messaging is your métier too, in which case we may have more in common than open source boat bits). Depending on the implementation, this can be done client side or server side and it's usually up to the messaging stack how that's handled, ZeroMQ does it at the Publisher and RabbitMQ (and probably most other brokers) handle it at the broker.

In terms of interoperability with other projects a simple protocol is a big advantage.

I agree completely. However, one of the things that I don't like about n2k and the upcoming OneNet from NMEA is that every consumer of the data available has to already know how that data is formatted, what all the lookup fields mean, what units are being used, etc. A very real consequence of this is that in order for my Garmin chart plotter to be able to display audio track information from my Fusion stereo, Garmin has to spend engineering hours getting a spec from Fusion, implementing and testing said spec and then they have to release a firmware update which I have to download at home, put on an SD card, drive to the boat, stick in my chart plotter and then wait 30 minutes or so while it updates. This is dumb (plus Garmin has chosen not to do this for my particular model, so I'm screwed). The plotter already knows how to display audio track information on the screen in a useful manner because it can get that from a Garmin XM receiver. It should be able to query my stereo and find out that a) it provides audio track information and b) how that information is formatted. Now, I as the end user might be required to provide some help viz. mapping the specific Fusion-fields to their respective Garmin-fields, but provided each field has a human-parseable description, this is easy (and would only need to be done once).

And that is why I am proposing serving up metadata. Because it means that new players can produce messages which everyone can make use of without them having to go through a review process and become a "standard" message or be relegated to "manufacturer specific" messages that only that manufacturer and their friends can read. And most importantly, it get's closer to a real plug'n'play environment.

We focus on the how of the formatting, but not the what. We don't need canonical message IDs (PGNs), and if IDs exist at all, they can be defined by the server(s) on the network. We tell the client up-front what units we're using and then never transmit that information again unless they ask for it.

Another benefit of metadata is the ability for the end-user to globally configure boat-specific things. For instance, let's say I have an engine on my boat that tops out at 4000 RPM, and I know my fuel consumption goes all to hell above 3000 RPM, idle is 500 RPM and that the sweet-spot performance-wise is about 2200 RPM. I might want to configure my server with that information so that when a client requests RPM info, it knows automatically that it should display a gauge from 0 - 4000 RPM, that it should color 0-500 red, 2000-2400 green, and 3000-4000 red.

@rob42

This comment has been minimized.

Show comment
Hide comment
@rob42

rob42 Mar 19, 2014

@timmatthews ...unless enterprise messaging is your métier too, in which case we may have more in common...
I am sitting at a desk right now building soa composites, messing with endless xml transforms, reading convoluted corporate specs, and wishing for something simple...
:-(
Rob

rob42 commented Mar 19, 2014

@timmatthews ...unless enterprise messaging is your métier too, in which case we may have more in common...
I am sitting at a desk right now building soa composites, messing with endless xml transforms, reading convoluted corporate specs, and wishing for something simple...
:-(
Rob

@timmathews

This comment has been minimized.

Show comment
Hide comment
@timmathews

timmathews Mar 19, 2014

@rob42 I wish I were so lucky. These days I'm working in healthcare, which means everything is EDI, so I get to work with fun things like EDI 834 and EDI 999. At least XML has structure, EDI has asterisks.

Anyway, my metadata API works for all PGNs now: http://pyxis.openseasproject.org/api/v1/messages, so all of those "@details" links work.

@rob42 I wish I were so lucky. These days I'm working in healthcare, which means everything is EDI, so I get to work with fun things like EDI 834 and EDI 999. At least XML has structure, EDI has asterisks.

Anyway, my metadata API works for all PGNs now: http://pyxis.openseasproject.org/api/v1/messages, so all of those "@details" links work.

@tkurki

This comment has been minimized.

Show comment
Hide comment
@tkurki

tkurki Mar 19, 2014

Owner

At this point I'd like to thank everybody for great discussion! Lots of food for though and this has broadened my horizons a lot. I am thinking of putting up a wiki page for the accumulating use cases. For example the RPM gauge colored sectors example is a clear use case for shared global configuration type metadata.

Also the mental model of a separate N2KView-like application bootstrapping itself is pretty good. It also underlines the current problem with NMEA vs N2K over wifi: all the decent navigation applications support NMEA over tcp and udp, but I am not aware of anything non proprietary supporting N2K. BTW there are many small nav apps in Google Play store, would be really good to have some app authors involved here.

As for using PGN's and N2K spec I fail to see a huge problem. The Fusion/Garmin use case underlines N2K's standardisation motivation, as if the PGN for this had been established both vendors could have used the same stuff instead of going for vendor-specific extensions. Your point about never getting the firmware upgrade is one motivational factor for me: with closed source there is no way to fix these things.

My main point here is that CANboat already provides a de facto standard with open source implementation. N2kd also provides a json streaming server - is there something more needed? Should I just bite the bullet and translate all my NMEA data to N2K-like json and modify my UI code to use that? One benefit of Node stack is that all code can be used on server and/or client at will.

Re:units - knots isn't universal, I for one like to see wind speed in m/s, but I understand I am in minority globally. The good thing about json is that there is no formal schema: if knots exist then use them, otherwise convert from m/s, as long as it is always clear what the units are so we don't miss our destination, be it a planet or just a buoy.

Also thumbs up for @timmathews for putting up the demo - working software is worth more than specs!

Owner

tkurki commented Mar 19, 2014

At this point I'd like to thank everybody for great discussion! Lots of food for though and this has broadened my horizons a lot. I am thinking of putting up a wiki page for the accumulating use cases. For example the RPM gauge colored sectors example is a clear use case for shared global configuration type metadata.

Also the mental model of a separate N2KView-like application bootstrapping itself is pretty good. It also underlines the current problem with NMEA vs N2K over wifi: all the decent navigation applications support NMEA over tcp and udp, but I am not aware of anything non proprietary supporting N2K. BTW there are many small nav apps in Google Play store, would be really good to have some app authors involved here.

As for using PGN's and N2K spec I fail to see a huge problem. The Fusion/Garmin use case underlines N2K's standardisation motivation, as if the PGN for this had been established both vendors could have used the same stuff instead of going for vendor-specific extensions. Your point about never getting the firmware upgrade is one motivational factor for me: with closed source there is no way to fix these things.

My main point here is that CANboat already provides a de facto standard with open source implementation. N2kd also provides a json streaming server - is there something more needed? Should I just bite the bullet and translate all my NMEA data to N2K-like json and modify my UI code to use that? One benefit of Node stack is that all code can be used on server and/or client at will.

Re:units - knots isn't universal, I for one like to see wind speed in m/s, but I understand I am in minority globally. The good thing about json is that there is no formal schema: if knots exist then use them, otherwise convert from m/s, as long as it is always clear what the units are so we don't miss our destination, be it a planet or just a buoy.

Also thumbs up for @timmathews for putting up the demo - working software is worth more than specs!

@timmathews

This comment has been minimized.

Show comment
Hide comment
@timmathews

timmathews Mar 19, 2014

I am thinking of putting up a wiki page for the accumulating use cases

Please do. Or if you're not opposed to the OpenSeas Project name: http://wiki.openseasproject.org

Should I just bite the bullet and translate all my NMEA data to N2K-like json and modify my UI code to use that?

Yes, probably. Everything that I am proposing can be built on top of what CANboat provides, so building clients for it will not be wasted effort. Since @canboat has mentioned that he would be willing to make some changes to the format, I believe that all of the field names should be converted to camelCase or snake_case instead of proper names. That would let JavaScript consumers do something like this:

var conn = new WebSocket("ws://pyxis.openseasproject.org/ws/v1/data")
conn.onmessage = function(evt) {
  var data = JSON.parse(evt.data);
  var ref = data.fields.reference;
  var windSpeed = data.fields.windSpeed;

  if(ref === 'apparent') {
    aws_kts = 1.9439 * windSpeed;
  } else if(ref === 'trueBoatReferenced') {
    tws_kts = 1.9439 * windSpeed;
  }
}

otherwise the same code looks like:

var conn = new WebSocket("ws://pyxis.openseasproject.org/ws/v1/data")
conn.onmessage = function(evt) {
  var data = JSON.parse(evt.data);
  var ref = data.['fields'].['Reference'];
  var windSpeed = data.['fields'].['Wind Speed'];

  if(ref === 'Apparent') {
    aws_kts = 1.9439 * windSpeed;
  } else if(ref === 'True (boat referenced)') {
    tws_kts = 1.9439 * windSpeed;
  }
}

I find the first version easier to read, but that's just personal preference.

[I]f knots exist then use them, otherwise convert from m/s, as long as it is always clear what the units are so we don't miss our destination, be it a planet or just a buoy.

FWIW, most (all?) n2k PGNs which deal with speed values report in m/s already.

I am thinking of putting up a wiki page for the accumulating use cases

Please do. Or if you're not opposed to the OpenSeas Project name: http://wiki.openseasproject.org

Should I just bite the bullet and translate all my NMEA data to N2K-like json and modify my UI code to use that?

Yes, probably. Everything that I am proposing can be built on top of what CANboat provides, so building clients for it will not be wasted effort. Since @canboat has mentioned that he would be willing to make some changes to the format, I believe that all of the field names should be converted to camelCase or snake_case instead of proper names. That would let JavaScript consumers do something like this:

var conn = new WebSocket("ws://pyxis.openseasproject.org/ws/v1/data")
conn.onmessage = function(evt) {
  var data = JSON.parse(evt.data);
  var ref = data.fields.reference;
  var windSpeed = data.fields.windSpeed;

  if(ref === 'apparent') {
    aws_kts = 1.9439 * windSpeed;
  } else if(ref === 'trueBoatReferenced') {
    tws_kts = 1.9439 * windSpeed;
  }
}

otherwise the same code looks like:

var conn = new WebSocket("ws://pyxis.openseasproject.org/ws/v1/data")
conn.onmessage = function(evt) {
  var data = JSON.parse(evt.data);
  var ref = data.['fields'].['Reference'];
  var windSpeed = data.['fields'].['Wind Speed'];

  if(ref === 'Apparent') {
    aws_kts = 1.9439 * windSpeed;
  } else if(ref === 'True (boat referenced)') {
    tws_kts = 1.9439 * windSpeed;
  }
}

I find the first version easier to read, but that's just personal preference.

[I]f knots exist then use them, otherwise convert from m/s, as long as it is always clear what the units are so we don't miss our destination, be it a planet or just a buoy.

FWIW, most (all?) n2k PGNs which deal with speed values report in m/s already.

@tkurki

This comment has been minimized.

Show comment
Hide comment
@tkurki

tkurki Mar 19, 2014

Owner

+1 for CamelCase. Should probably start with CANboat, so that I would have a reference. Speaking of which, see also canboat/canboat#1

Owner

tkurki commented Mar 19, 2014

+1 for CamelCase. Should probably start with CANboat, so that I would have a reference. Speaking of which, see also canboat/canboat#1

@keesverruijt

This comment has been minimized.

Show comment
Hide comment
@keesverruijt

keesverruijt Apr 1, 2014

@rob42 My favorite signal flag is J-Juliet: I am on fire and have dangerous cargo on board, but K-Kilo is probably a better logo.

One could also argue that the description of X is appropriate: Stop carrying out your intentions and watch for my signals, but “Xray” doesn’t really do it as a name. So I agree with Kilo as well. Short, to the point, and not used for anything else in IT that I know of. Apparently the prefix is so dominant that nobody else has dared name a product or idea “Kilo”.

There is one issue with it, there is already a github.com/kilo user, so we’ll have to settle for /kilo.

Shall we use github.com/canboat/kilo as the external go-to point?

@fabdrol Welcome! I don't believe that any toes have been stepped on. You've pretty much gotten to parity with what we have so far just by thinking, so I think that's good.

Welcome, and your physical location is fortuitous as well as you’re only one hour away from me (Harlingen.)
As far as controlling things, there's no reason that your Consumers can't also be Producers (essentially RPC over WebSockets). I hadn't put a lot of thought into controlling things yet, but it is certainly a requirement.

Indeed. The parts of CANboat that aren’t public yet concern two other “sources”. I use a Maxim 1 Wire network to measure temperature, which is read only. I use a WAGO Linux PLC running my own program that communicates with HTTP (sort of) and REST and JSON, which is very much two way as it is able to monitor and control all onboard circuits. In fact this is the reason that I started the entire project, not the N2K side of it.

But there is so much more that we can share. Last night I thought it would also be nice if we would be able to come up with a javascript interface that encapsulates a gauge (browser side). Then we can share our gauges, or collaborate on building the absolute best-ever multi faceted super gauge.

We are pretty strung out on the server side in technologies, and haven’t yet discussed much whether and if so which JS libraries you use. As CANboat is the oldest, it uses JQuery as that was the best player in field at the time (2009).


Kees

@rob42 My favorite signal flag is J-Juliet: I am on fire and have dangerous cargo on board, but K-Kilo is probably a better logo.

One could also argue that the description of X is appropriate: Stop carrying out your intentions and watch for my signals, but “Xray” doesn’t really do it as a name. So I agree with Kilo as well. Short, to the point, and not used for anything else in IT that I know of. Apparently the prefix is so dominant that nobody else has dared name a product or idea “Kilo”.

There is one issue with it, there is already a github.com/kilo user, so we’ll have to settle for /kilo.

Shall we use github.com/canboat/kilo as the external go-to point?

@fabdrol Welcome! I don't believe that any toes have been stepped on. You've pretty much gotten to parity with what we have so far just by thinking, so I think that's good.

Welcome, and your physical location is fortuitous as well as you’re only one hour away from me (Harlingen.)
As far as controlling things, there's no reason that your Consumers can't also be Producers (essentially RPC over WebSockets). I hadn't put a lot of thought into controlling things yet, but it is certainly a requirement.

Indeed. The parts of CANboat that aren’t public yet concern two other “sources”. I use a Maxim 1 Wire network to measure temperature, which is read only. I use a WAGO Linux PLC running my own program that communicates with HTTP (sort of) and REST and JSON, which is very much two way as it is able to monitor and control all onboard circuits. In fact this is the reason that I started the entire project, not the N2K side of it.

But there is so much more that we can share. Last night I thought it would also be nice if we would be able to come up with a javascript interface that encapsulates a gauge (browser side). Then we can share our gauges, or collaborate on building the absolute best-ever multi faceted super gauge.

We are pretty strung out on the server side in technologies, and haven’t yet discussed much whether and if so which JS libraries you use. As CANboat is the oldest, it uses JQuery as that was the best player in field at the time (2009).


Kees

@tkurki

This comment has been minimized.

Show comment
Hide comment
@tkurki

tkurki Apr 1, 2014

Owner

+1 for Kilo, but I think it needs something else to be googlable. I'm terrible with names, no good suggestions here. KiloSpec is name of a company, KiloJSON (as in GeoJSON) brings up only two hits?

+1 for the separate org on Github. Kilo is also already taken on Github.

Owner

tkurki commented Apr 1, 2014

+1 for Kilo, but I think it needs something else to be googlable. I'm terrible with names, no good suggestions here. KiloSpec is name of a company, KiloJSON (as in GeoJSON) brings up only two hits?

+1 for the separate org on Github. Kilo is also already taken on Github.

@tkurki

This comment has been minimized.

Show comment
Hide comment
@tkurki

tkurki Apr 1, 2014

Owner

@fabdrol IMHO your thoughts are exactly in line with what has been discussed. Also a bit of diversity is a good thing. If I knew all I'd just stick to my own thoughts.

Speaking of which: I changed apparent|trueWind to windApparent|True and speedOverWater to speedThroughWater (STW).

Owner

tkurki commented Apr 1, 2014

@fabdrol IMHO your thoughts are exactly in line with what has been discussed. Also a bit of diversity is a good thing. If I knew all I'd just stick to my own thoughts.

Speaking of which: I changed apparent|trueWind to windApparent|True and speedOverWater to speedThroughWater (STW).

@tkurki

This comment has been minimized.

Show comment
Hide comment
@tkurki

tkurki Apr 1, 2014

Owner

@canboat: I've tried to move towards widgets-as-components direction. Take a look at for example windgraph.js: it assumes that d3.js and bacon.js are available, takes in a selector as init argument (specifies where in the DOM tree to put the graph) and expects data to come in as onData(msg). Apart from that it is self-contained, client side javascript. This is exactly where the KiloJSON comes into play: for it to be pluggable incoming data should be in a common format that the gauge can understand.

And I have a 10 DOF IMU on my desk connected to my Raspberry, so multiple sources and sensors is a definite reality.

I would love to have a proof of concept running on the net - for this to be viable we need at least 2 reference implementations that actually interoperate...anybody pair with me to set this up?

Isn't the autopilot control channel best served with a REST api: websockets for streaming updates, rest for one shot control or query calls.

Owner

tkurki commented Apr 1, 2014

@canboat: I've tried to move towards widgets-as-components direction. Take a look at for example windgraph.js: it assumes that d3.js and bacon.js are available, takes in a selector as init argument (specifies where in the DOM tree to put the graph) and expects data to come in as onData(msg). Apart from that it is self-contained, client side javascript. This is exactly where the KiloJSON comes into play: for it to be pluggable incoming data should be in a common format that the gauge can understand.

And I have a 10 DOF IMU on my desk connected to my Raspberry, so multiple sources and sensors is a definite reality.

I would love to have a proof of concept running on the net - for this to be viable we need at least 2 reference implementations that actually interoperate...anybody pair with me to set this up?

Isn't the autopilot control channel best served with a REST api: websockets for streaming updates, rest for one shot control or query calls.

@rob42

This comment has been minimized.

Show comment
Hide comment
@rob42

rob42 Apr 1, 2014

The name - What about just 'K', or 'Signal-K', 'Signal K', 'SignalK'- that has no hits on google.
The meaning is just so well aligned with our direction, makes a great slogan: "I wish to communicate with you."

In terms of the convergence of our

rob42 commented Apr 1, 2014

The name - What about just 'K', or 'Signal-K', 'Signal K', 'SignalK'- that has no hits on google.
The meaning is just so well aligned with our direction, makes a great slogan: "I wish to communicate with you."

In terms of the convergence of our

@fabdrol

This comment has been minimized.

Show comment
Hide comment
@fabdrol

fabdrol Apr 1, 2014

@timmathews Thanks! I agree, that would be the best solution. As long as the communication back (Consumer -> Producer) doesn't go through the central web server (that way, the goal decoupling is still archieved without and the Consumer doesn't really need to know about a boat's capabilities. They just do their thing with the data they receive).

Last night I wrote that Providers could be registered using a list in a configuration file. Come to think about it, and that is maybe not the best solution. We'd end up with a lot of configuration per boat, which would have to be altered whenever a Producer is added. Instead a standard way of registering would prob be better, but then how'd we handle process management.. Also, if it runs on the server a user would have to go into that anyway. Final thought on registering: something like NPM specifically for Kilo Producers and Kilo Consumers would be cool (and could be handled by an "install" page in settings!).. Not important for now, but would be a cool addition in the future.

@canboat How about org name /kilo-project/? Regarding an encapsulated way of handling gauges (or Consumers in the browser in general), Web Components/Polymer comes to mind. Not fully standardised yet, bit very promising and it provides the necessary encapsulation and more.
That settles it, the first meet-up will be here! I'm sure Tim doesn't mind sailing to the Netherlands! I'm originally from Friesland too, btw.

I'll set up a project implementing what I talked about. Goal mostly is standardising the pattern and the interface (but also a stable implementation). If the pattern/interface is standard and agreed upon by most, the language of the implementation won't matter anymore. Once I have a simple server running, I'll try to get some sample Providers up and some basic Consumers.
One issue for me is hardware. On board I only have a tiller pilot accepting SeaTalk, a GPS emitting NMEA 0183, and some instruments emitting NMEA 0183. Furthermore, this would move way too slow if I had to come to the boat in order to work on it. How did you guys solve this/what recommendations can you give me?

fabdrol commented Apr 1, 2014

@timmathews Thanks! I agree, that would be the best solution. As long as the communication back (Consumer -> Producer) doesn't go through the central web server (that way, the goal decoupling is still archieved without and the Consumer doesn't really need to know about a boat's capabilities. They just do their thing with the data they receive).

Last night I wrote that Providers could be registered using a list in a configuration file. Come to think about it, and that is maybe not the best solution. We'd end up with a lot of configuration per boat, which would have to be altered whenever a Producer is added. Instead a standard way of registering would prob be better, but then how'd we handle process management.. Also, if it runs on the server a user would have to go into that anyway. Final thought on registering: something like NPM specifically for Kilo Producers and Kilo Consumers would be cool (and could be handled by an "install" page in settings!).. Not important for now, but would be a cool addition in the future.

@canboat How about org name /kilo-project/? Regarding an encapsulated way of handling gauges (or Consumers in the browser in general), Web Components/Polymer comes to mind. Not fully standardised yet, bit very promising and it provides the necessary encapsulation and more.
That settles it, the first meet-up will be here! I'm sure Tim doesn't mind sailing to the Netherlands! I'm originally from Friesland too, btw.

I'll set up a project implementing what I talked about. Goal mostly is standardising the pattern and the interface (but also a stable implementation). If the pattern/interface is standard and agreed upon by most, the language of the implementation won't matter anymore. Once I have a simple server running, I'll try to get some sample Providers up and some basic Consumers.
One issue for me is hardware. On board I only have a tiller pilot accepting SeaTalk, a GPS emitting NMEA 0183, and some instruments emitting NMEA 0183. Furthermore, this would move way too slow if I had to come to the boat in order to work on it. How did you guys solve this/what recommendations can you give me?

@fabdrol

This comment has been minimized.

Show comment
Hide comment
@fabdrol

fabdrol Apr 1, 2014

@tkurki As I'm starting from scratch, I think I can easily pair with you on this. How do you want to proceed?

As for your example, you assume that d3.js is there.. I'm the world of widgets that can be dangerous - especially if you're not the only one writing widgets (e.g multiple versions of jquery conflicting). Another solution to the one I mentioned (web components) would be to use something like Browserify or AMD to define dependencies including version. This doesn't take care of encapsulation, that would be the responsibility of the widget author (e.g calling $.noConflict within the widget's own encapsulated namespace).

fabdrol commented Apr 1, 2014

@tkurki As I'm starting from scratch, I think I can easily pair with you on this. How do you want to proceed?

As for your example, you assume that d3.js is there.. I'm the world of widgets that can be dangerous - especially if you're not the only one writing widgets (e.g multiple versions of jquery conflicting). Another solution to the one I mentioned (web components) would be to use something like Browserify or AMD to define dependencies including version. This doesn't take care of encapsulation, that would be the responsibility of the widget author (e.g calling $.noConflict within the widget's own encapsulated namespace).

@ktuukkan

This comment has been minimized.

Show comment
Hide comment
@ktuukkan

ktuukkan Apr 1, 2014

Been a bit absent and haven't read all messages with deep thought, but I like "Kilo" as well, very nice. Also very well put @fabdrol, I agree with almost everything. I thinki we should concentrate less on the consumer side because the common format is our beef. Clients can use the data as they wish, and of course based on your experience we can shape the model better support different kind of consumers, but I think we shouldn't design for any particular use case except "sharing what the vessel knows". My project is mainly a data producer, so I have quite little to comment on consumer side (i.e. not not much personal experience) and it already seems to be forming very well.

Anyways, here's some more general guidelines I've been thinking:

  • "forget" conventions and practices of NMEA et.al. as we have totally clean table here
  • field names with camelCase
  • fixed values/enums with CamelCase
  • be consistent with naming, units, symbols and everything. For every occurence of "x", "x" means "x".
  • sticking with chosen units avoids confusion on client side (e.g. depth always in meters)
  • no irrelevant symbols. For example, NMEA 0183 transmits a lot unnecessary characters for units and they are useless if the spec defines which units are used in each field.
  • avoid using abbreviations unless it's absolutely clear by common knowledge
  • avoid arrays with plain indexing, e.g. vessels[0], if it requires consumer to "know" position of each block in beforehand

What do you think? Should we document this kind of things as well?

This thread is getting pretty long.. IRC, Skype anyone? :)

ktuukkan commented Apr 1, 2014

Been a bit absent and haven't read all messages with deep thought, but I like "Kilo" as well, very nice. Also very well put @fabdrol, I agree with almost everything. I thinki we should concentrate less on the consumer side because the common format is our beef. Clients can use the data as they wish, and of course based on your experience we can shape the model better support different kind of consumers, but I think we shouldn't design for any particular use case except "sharing what the vessel knows". My project is mainly a data producer, so I have quite little to comment on consumer side (i.e. not not much personal experience) and it already seems to be forming very well.

Anyways, here's some more general guidelines I've been thinking:

  • "forget" conventions and practices of NMEA et.al. as we have totally clean table here
  • field names with camelCase
  • fixed values/enums with CamelCase
  • be consistent with naming, units, symbols and everything. For every occurence of "x", "x" means "x".
  • sticking with chosen units avoids confusion on client side (e.g. depth always in meters)
  • no irrelevant symbols. For example, NMEA 0183 transmits a lot unnecessary characters for units and they are useless if the spec defines which units are used in each field.
  • avoid using abbreviations unless it's absolutely clear by common knowledge
  • avoid arrays with plain indexing, e.g. vessels[0], if it requires consumer to "know" position of each block in beforehand

What do you think? Should we document this kind of things as well?

This thread is getting pretty long.. IRC, Skype anyone? :)

@fabdrol

This comment has been minimized.

Show comment
Hide comment
@fabdrol

fabdrol Apr 1, 2014

@ktuukkan I agree, a Consumer could be whatever.. After all, there isn't a lot more they can do but to simply read the WS feed and do there thing.

fabdrol commented Apr 1, 2014

@ktuukkan I agree, a Consumer could be whatever.. After all, there isn't a lot more they can do but to simply read the WS feed and do there thing.

@rob42

This comment has been minimized.

Show comment
Hide comment
@rob42

rob42 Apr 1, 2014

@fabdrol - hmm - REST interface for replies from the browser could work...

Some background here - I originally started freeboard because there was no way to provide instruments without buying into the big names. Some open source small projects - but all alone - no common structure. Only OpenCPN had escape velocity - but no suitable sensors/integration/hardware/etc. So I started freeboard partially to create a common platform.

But I have to code front-end (javascript), backend(java), microcontrollers (c, arduino), setup Rpi's, and Ive designed boards to make the physical connections from boat stuff to arduinos - too much!

But I think we are there (common platform) now - we all have our interests and strengths, and a common platform to work too - so if you create a cool client, there is data to show - and if you decode locked formats - there is a client to benefit. Plus new-comers with limited interests or skillsets can find a place to fit in.

So I think the next step is to start a new shared project (Signal K?), and link our projects into it. Then I think we pick the areas we are keen on, and work towards, integrating all our stuff together. Meanwhile we spread the word about - and see if we can reach that elusive escape velocity of the big projects!

If you agree we need to vote on the name - its going to be important - has to be memorable, easy to say, spell, google friendly, code friendly, what else?

Kilo (taken on Github already, flag and all)

  1. K
  2. Signal K - gets my vote
  3. k-project
  4. ?

Rob

rob42 commented Apr 1, 2014

@fabdrol - hmm - REST interface for replies from the browser could work...

Some background here - I originally started freeboard because there was no way to provide instruments without buying into the big names. Some open source small projects - but all alone - no common structure. Only OpenCPN had escape velocity - but no suitable sensors/integration/hardware/etc. So I started freeboard partially to create a common platform.

But I have to code front-end (javascript), backend(java), microcontrollers (c, arduino), setup Rpi's, and Ive designed boards to make the physical connections from boat stuff to arduinos - too much!

But I think we are there (common platform) now - we all have our interests and strengths, and a common platform to work too - so if you create a cool client, there is data to show - and if you decode locked formats - there is a client to benefit. Plus new-comers with limited interests or skillsets can find a place to fit in.

So I think the next step is to start a new shared project (Signal K?), and link our projects into it. Then I think we pick the areas we are keen on, and work towards, integrating all our stuff together. Meanwhile we spread the word about - and see if we can reach that elusive escape velocity of the big projects!

If you agree we need to vote on the name - its going to be important - has to be memorable, easy to say, spell, google friendly, code friendly, what else?

Kilo (taken on Github already, flag and all)

  1. K
  2. Signal K - gets my vote
  3. k-project
  4. ?

Rob

@tkurki

This comment has been minimized.

Show comment
Hide comment
@tkurki

tkurki Apr 1, 2014

Owner

@fabdrol Take a look at http://navgauge.herokuapp.com/gauges.html . Ideally you should be able to enter your own server's url in the box, press connect and the gauges would start showing data from your server. (the box is now a placeholder, no functionality)

Navgauge has the ability to play back raw NMEA captures, Canboat analyzer output and Navgauge's own json capture. Handling raw Canboat actisense output and adding support for several NMEA sources is pretty straightforward. Whatever you do I suggest getting some data captures from real sailing, captures from a boat sitting still get boring pretty quick ;-). If you don't mind Node you can just start with cloning Navgauge and playing around with it.

Browserify, AMD, Bower etc +1, haven't done it because it's been just me as the user and not sure what to pick. Web Components seems to be still in the future, but maybe you have better info on that.

Owner

tkurki commented Apr 1, 2014

@fabdrol Take a look at http://navgauge.herokuapp.com/gauges.html . Ideally you should be able to enter your own server's url in the box, press connect and the gauges would start showing data from your server. (the box is now a placeholder, no functionality)

Navgauge has the ability to play back raw NMEA captures, Canboat analyzer output and Navgauge's own json capture. Handling raw Canboat actisense output and adding support for several NMEA sources is pretty straightforward. Whatever you do I suggest getting some data captures from real sailing, captures from a boat sitting still get boring pretty quick ;-). If you don't mind Node you can just start with cloning Navgauge and playing around with it.

Browserify, AMD, Bower etc +1, haven't done it because it's been just me as the user and not sure what to pick. Web Components seems to be still in the future, but maybe you have better info on that.

@fabdrol

This comment has been minimized.

Show comment
Hide comment
@fabdrol

fabdrol Apr 1, 2014

@tkurki Node is my language of choice as well, so that is a good start. Here's what I'm thinking:
I'll set up a first implementation of the decoupled web server in my idea. It'll take a list of Providers as it's input, multiplex this data and emit it on WS. I'll start writing a simple Provider that takes GPS input (easy starting point, as there are already projects for that written in node). I'll make the WS stream available cross-domain, so you can start playing with it in Navgauge.
That way we'll actually have two projects that are different in implementation, but communicate over WS in the same, shared protocol - which should be good for testing.

More practical: I think we need to fork a good (NMEA) parser and start work on the implementation of our new protocol - evolving the parsers as the protocol evolves. My thinking is, if we have a few basic parsers available (SeaTalk, NMEA 0183 & 2000 to start with?) of which we make sure they follow the specification of the protocol with each new version, new contributors have a good place to get started - either as contributor to the parser or building something on top of a parser.

That said; maybe we should vote for a name quickly, set up the new organisation on github and set up a few basic things for collaboration (e.g. an IRC channel for communicating, a Trello board to manage what needs to be done etc). Then we can decide on what to do next and start making versions of the parsers and the specification, testing the spec with the parsers in our individual projects as we go along.

@rob42 my vote's for Signal K for the project/organisation and "KJSON" as the name for the format (mainly to distinguish between JSON and KJSON when we're talking about stuff).

fabdrol commented Apr 1, 2014

@tkurki Node is my language of choice as well, so that is a good start. Here's what I'm thinking:
I'll set up a first implementation of the decoupled web server in my idea. It'll take a list of Providers as it's input, multiplex this data and emit it on WS. I'll start writing a simple Provider that takes GPS input (easy starting point, as there are already projects for that written in node). I'll make the WS stream available cross-domain, so you can start playing with it in Navgauge.
That way we'll actually have two projects that are different in implementation, but communicate over WS in the same, shared protocol - which should be good for testing.

More practical: I think we need to fork a good (NMEA) parser and start work on the implementation of our new protocol - evolving the parsers as the protocol evolves. My thinking is, if we have a few basic parsers available (SeaTalk, NMEA 0183 & 2000 to start with?) of which we make sure they follow the specification of the protocol with each new version, new contributors have a good place to get started - either as contributor to the parser or building something on top of a parser.

That said; maybe we should vote for a name quickly, set up the new organisation on github and set up a few basic things for collaboration (e.g. an IRC channel for communicating, a Trello board to manage what needs to be done etc). Then we can decide on what to do next and start making versions of the parsers and the specification, testing the spec with the parsers in our individual projects as we go along.

@rob42 my vote's for Signal K for the project/organisation and "KJSON" as the name for the format (mainly to distinguish between JSON and KJSON when we're talking about stuff).

@fabdrol

This comment has been minimized.

Show comment
Hide comment
@fabdrol

fabdrol Apr 1, 2014

@tkurki btw, a few notes on client side:

  • web components are indeed still in a very early stage of implementation and mainly work in newer browsers (see http://www.polymer-project.org/resources/compatibility.html). However, since most Consumer/client implementations would be for a computer, tablet etc on a boat (and not the general public on the internet), one could be quite picky about which browsers they support (and besides, most tablets and phones are webkit-based anyway so that's not really an issue).
  • AMD: I hate it's syntax and I personally would never use it. But, that's a choice, of course.
  • Browserify: doesn't do on-demand loading of scripts but minifies them all into a big bundle file. Not a problem in our case as bandwidth is plenty. Main upside: very nice syntax (just do require('module-name') and browserify does the rest) and the ability to use Node.js modules on the browser. Actually, most modules on NPM "Just Work" in the browser!

But, this is all a bit off-topic - as @ktuukkan pointed out. The Consumer implementation is entirely up to the developer's preferences, I suppose.

fabdrol commented Apr 1, 2014

@tkurki btw, a few notes on client side:

  • web components are indeed still in a very early stage of implementation and mainly work in newer browsers (see http://www.polymer-project.org/resources/compatibility.html). However, since most Consumer/client implementations would be for a computer, tablet etc on a boat (and not the general public on the internet), one could be quite picky about which browsers they support (and besides, most tablets and phones are webkit-based anyway so that's not really an issue).
  • AMD: I hate it's syntax and I personally would never use it. But, that's a choice, of course.
  • Browserify: doesn't do on-demand loading of scripts but minifies them all into a big bundle file. Not a problem in our case as bandwidth is plenty. Main upside: very nice syntax (just do require('module-name') and browserify does the rest) and the ability to use Node.js modules on the browser. Actually, most modules on NPM "Just Work" in the browser!

But, this is all a bit off-topic - as @ktuukkan pointed out. The Consumer implementation is entirely up to the developer's preferences, I suppose.

@timmathews

This comment has been minimized.

Show comment
Hide comment
@timmathews

timmathews Apr 1, 2014

Consider this my vote for SignalK as well!

Consider this my vote for SignalK as well!

@keesverruijt

This comment has been minimized.

Show comment
Hide comment
@keesverruijt

keesverruijt Apr 1, 2014

+1 for SignalK.

2014-04-01 14:06 GMT+02:00 Tim Mathews notifications@github.com:

Consider this my vote for SignalK as well!


Reply to this email directly or view it on GitHubhttps://github.com/tkurki/navgauge/issues/4#issuecomment-39197599
.

+1 for SignalK.

2014-04-01 14:06 GMT+02:00 Tim Mathews notifications@github.com:

Consider this my vote for SignalK as well!


Reply to this email directly or view it on GitHubhttps://github.com/tkurki/navgauge/issues/4#issuecomment-39197599
.

@timmathews

This comment has been minimized.

Show comment
Hide comment
@timmathews

timmathews Apr 1, 2014

In the interest of getting sh*t done, github.com/SignalK is now a project. You should have all gotten notification in one form or another.

EDIT: I meant organization of course

Now, @rob42 since you came up with this name and mentioned a logo, we need a logo. 😄

In the interest of getting sh*t done, github.com/SignalK is now a project. You should have all gotten notification in one form or another.

EDIT: I meant organization of course

Now, @rob42 since you came up with this name and mentioned a logo, we need a logo. 😄

@timmathews

This comment has been minimized.

Show comment
Hide comment
@timmathews

timmathews Apr 1, 2014

One more thing before I'm off to work. When do we ask Ben at Panbo to blog about us? The resultant conversation should be, erm, interesting.

One more thing before I'm off to work. When do we ask Ben at Panbo to blog about us? The resultant conversation should be, erm, interesting.

@tkurki

This comment has been minimized.

Show comment
Hide comment
@tkurki

tkurki Apr 1, 2014

Owner

Get something that actually interoperates done first? Or something that has
been impossible/hard/expensive without SignalK.

Speaking of communication: this issue is getting a bit longwinded. Ideas on
how to move forward?

tiistai 1. huhtikuuta 2014 Tim Mathews notifications@github.com kirjoitti:

One more thing before I'm off to work. When do we ask Ben at Panbo to blog
about us? The resultant conversation should be, erm, interesting.

Reply to this email directly or view it on GitHubhttps://github.com/tkurki/navgauge/issues/4#issuecomment-39200557
.

Owner

tkurki commented Apr 1, 2014

Get something that actually interoperates done first? Or something that has
been impossible/hard/expensive without SignalK.

Speaking of communication: this issue is getting a bit longwinded. Ideas on
how to move forward?

tiistai 1. huhtikuuta 2014 Tim Mathews notifications@github.com kirjoitti:

One more thing before I'm off to work. When do we ask Ben at Panbo to blog
about us? The resultant conversation should be, erm, interesting.

Reply to this email directly or view it on GitHubhttps://github.com/tkurki/navgauge/issues/4#issuecomment-39200557
.

@fabdrol

This comment has been minimized.

Show comment
Hide comment
@fabdrol

fabdrol Apr 1, 2014

How about IRC + Trello for managing tasks/features/whatever?

fabdrol commented Apr 1, 2014

How about IRC + Trello for managing tasks/features/whatever?

@timmathews

This comment has been minimized.

Show comment
Hide comment
@timmathews

timmathews Apr 1, 2014

Perhaps a google group as well?

Perhaps a google group as well?

@keesverruijt

This comment has been minimized.

Show comment
Hide comment
@keesverruijt

keesverruijt Apr 1, 2014

I don't mind IRC but I don't have a reader open for that normally, so I'd
prefer a service that allows reply-via-email or supports XMPP.

Do we want privacy?

2014-04-01 16:11 GMT+02:00 Tim Mathews notifications@github.com:

Perhaps a google group as well?


Reply to this email directly or view it on GitHubhttps://github.com/tkurki/navgauge/issues/4#issuecomment-39209514
.

I don't mind IRC but I don't have a reader open for that normally, so I'd
prefer a service that allows reply-via-email or supports XMPP.

Do we want privacy?

2014-04-01 16:11 GMT+02:00 Tim Mathews notifications@github.com:

Perhaps a google group as well?


Reply to this email directly or view it on GitHubhttps://github.com/tkurki/navgauge/issues/4#issuecomment-39209514
.

@keesverruijt

This comment has been minimized.

Show comment
Hide comment
@keesverruijt

keesverruijt Apr 1, 2014

BTW,

+1 for Google Group
+1 for Trello, it uses Kanban which I am a great fan of.

BTW,

+1 for Google Group
+1 for Trello, it uses Kanban which I am a great fan of.

@fabdrol

This comment has been minimized.

Show comment
Hide comment
@fabdrol

fabdrol Apr 1, 2014

Okay, let's turn this around. Which services are you guys on already - apart from Github and e-mail.
I'm on:

  • Jabber (google talk), but poor support for groups
  • e-mail (but that get's cluttered to quickly)
  • IRC
  • Skype

fabdrol commented Apr 1, 2014

Okay, let's turn this around. Which services are you guys on already - apart from Github and e-mail.
I'm on:

  • Jabber (google talk), but poor support for groups
  • e-mail (but that get's cluttered to quickly)
  • IRC
  • Skype
@keesverruijt

This comment has been minimized.

Show comment
Hide comment
@keesverruijt

keesverruijt Apr 1, 2014

Same here except for IRC.

BTW I am looking for a new cross service chat client anyway, preferably with iOS and Android support.

On 01 Apr 2014, at 16:39, Fabian Tollenaar notifications@github.com wrote:

Okay, let's turn this around. Which services are you guys on already - apart from Github and e-mail.
I'm on:

Jabber (google talk), but poor support for groups
e-mail (but that get's cluttered to quickly)
IRC
Skype

Reply to this email directly or view it on GitHub.

Same here except for IRC.

BTW I am looking for a new cross service chat client anyway, preferably with iOS and Android support.

On 01 Apr 2014, at 16:39, Fabian Tollenaar notifications@github.com wrote:

Okay, let's turn this around. Which services are you guys on already - apart from Github and e-mail.
I'm on:

Jabber (google talk), but poor support for groups
e-mail (but that get's cluttered to quickly)
IRC
Skype

Reply to this email directly or view it on GitHub.

@fabdrol

This comment has been minimized.

Show comment
Hide comment
@fabdrol

fabdrol Apr 1, 2014

I did see something interesting a while a go, a chat-based project manager with integration for Github and other services. Can't remember the name though, and if it was free. I'll check it out.

fabdrol commented Apr 1, 2014

I did see something interesting a while a go, a chat-based project manager with integration for Github and other services. Can't remember the name though, and if it was free. I'll check it out.

@fabdrol

This comment has been minimized.

Show comment
Hide comment
@fabdrol

This comment has been minimized.

Show comment
Hide comment
@fabdrol

fabdrol Apr 1, 2014

or, google hangout

fabdrol commented Apr 1, 2014

or, google hangout

@fabdrol

This comment has been minimized.

Show comment
Hide comment
@fabdrol

fabdrol Apr 1, 2014

I've set up a Trello board to get stuff started. I'm aware that not everyone pitched in yet, if we do decide to use something else it can easily be deleted again!

Join here - or send me an e-mail https://trello.com/b/1ApFRVqT

Fabian

fabdrol commented Apr 1, 2014

I've set up a Trello board to get stuff started. I'm aware that not everyone pitched in yet, if we do decide to use something else it can easily be deleted again!

Join here - or send me an e-mail https://trello.com/b/1ApFRVqT

Fabian

@fabdrol

This comment has been minimized.

Show comment
Hide comment
@fabdrol

fabdrol Apr 1, 2014

@canboat could you invite me to the hangout at fabian [at] startingpoint.nl? Thanks

fabdrol commented Apr 1, 2014

@canboat could you invite me to the hangout at fabian [at] startingpoint.nl? Thanks

@tkurki

This comment has been minimized.

Show comment
Hide comment
@tkurki

tkurki Apr 1, 2014

Owner

@fabdrol: http://navgauge.herokuapp.com/treedata.html now produces nested update messages with the nested=true option in the ws url. That is something that I plan to keep available and updated to the latest spec, subject to available time. N2k timestamp I still need to fix, now it is direct analyzer output.

I still haven't given up on the {id:'navigation.courseOverGround', value:'164.2', src:...} though. I find that a lot easier to work with on the consuming side. Navgauge now supports both.

Hmm, quite a lot of choice for team communications. I don't have a good solution for constant IRC access and as we are a global team. Persistence of the communication and access to older discussions matter for people joining later, so IRC and Skype are less suitable. A mailing list with archive/Google group would be my choice right now, but I've already felt the need for chatting/asking immediate/short questions as well.

Definitely open, if I had just emailed back and forth with Rob this thread would not have happened. Email in case we need privacy.

Owner

tkurki commented Apr 1, 2014

@fabdrol: http://navgauge.herokuapp.com/treedata.html now produces nested update messages with the nested=true option in the ws url. That is something that I plan to keep available and updated to the latest spec, subject to available time. N2k timestamp I still need to fix, now it is direct analyzer output.

I still haven't given up on the {id:'navigation.courseOverGround', value:'164.2', src:...} though. I find that a lot easier to work with on the consuming side. Navgauge now supports both.

Hmm, quite a lot of choice for team communications. I don't have a good solution for constant IRC access and as we are a global team. Persistence of the communication and access to older discussions matter for people joining later, so IRC and Skype are less suitable. A mailing list with archive/Google group would be my choice right now, but I've already felt the need for chatting/asking immediate/short questions as well.

Definitely open, if I had just emailed back and forth with Rob this thread would not have happened. Email in case we need privacy.

@fabdrol

This comment has been minimized.

Show comment
Hide comment
@fabdrol

fabdrol Apr 1, 2014

@tkurki hmm.. I must say I agree with what some others posted here before; 'navigation.courseOverGround' seems "un-native" to me. As a JSON-based format, the nested form seems much more natural.

Btw, are you simulating a dataset or is that real data?

Cheers

fabdrol commented Apr 1, 2014

@tkurki hmm.. I must say I agree with what some others posted here before; 'navigation.courseOverGround' seems "un-native" to me. As a JSON-based format, the nested form seems much more natural.

Btw, are you simulating a dataset or is that real data?

Cheers

@tkurki

This comment has been minimized.

Show comment
Hide comment
@tkurki

tkurki Apr 1, 2014

Owner

Replay of captured data, same as what you get at
http://navgauge.herokuapp.com/.

The dataset is in Navgauge internal format from last summer:
https://github.com/tkurki/navgauge/blob/master/samples/cassiopeia.json.gz

tiistai 1. huhtikuuta 2014 Fabian Tollenaar notifications@github.com
kirjoitti:

@tkurki https://github.com/tkurki hmm.. I must say I agree with what
some others posted here before; 'navigation.courseOverGround' seems
"un-native" to me. As a JSON-based format, the nested form seems much more
natural.

Btw, are you simulating a dataset or is that real data?

Cheers

Reply to this email directly or view it on GitHubhttps://github.com/tkurki/navgauge/issues/4#issuecomment-39225158
.

Owner

tkurki commented Apr 1, 2014

Replay of captured data, same as what you get at
http://navgauge.herokuapp.com/.

The dataset is in Navgauge internal format from last summer:
https://github.com/tkurki/navgauge/blob/master/samples/cassiopeia.json.gz

tiistai 1. huhtikuuta 2014 Fabian Tollenaar notifications@github.com
kirjoitti:

@tkurki https://github.com/tkurki hmm.. I must say I agree with what
some others posted here before; 'navigation.courseOverGround' seems
"un-native" to me. As a JSON-based format, the nested form seems much more
natural.

Btw, are you simulating a dataset or is that real data?

Cheers

Reply to this email directly or view it on GitHubhttps://github.com/tkurki/navgauge/issues/4#issuecomment-39225158
.

@timmathews

This comment has been minimized.

Show comment
Hide comment
@timmathews

timmathews Apr 1, 2014

IMHO, for long running discussions (like this one) nothing beats a good old-fashioned mailing list.

So, here's one: https://groups.google.com/forum/#!forum/signalk

The mailing list address is signalk@googlegroups.com. Just send an email to signalk+subscribe@googlegroups.com to join. For now anyone can join, but if it gets spammy we can switch to an apply-for-membership option.

I also have an irc server at irc.openseasproject.org if you like IRC. I'll be in #signalk

IMHO, for long running discussions (like this one) nothing beats a good old-fashioned mailing list.

So, here's one: https://groups.google.com/forum/#!forum/signalk

The mailing list address is signalk@googlegroups.com. Just send an email to signalk+subscribe@googlegroups.com to join. For now anyone can join, but if it gets spammy we can switch to an apply-for-membership option.

I also have an irc server at irc.openseasproject.org if you like IRC. I'll be in #signalk

@mweehuizen

This comment has been minimized.

Show comment
Hide comment
@mweehuizen

mweehuizen Apr 1, 2014

Hi all, I just wanted to show my face. robert pointed me at the thread
shortly after it started. So far I have been lurking rather than
feeling the need to actively participate. I'm a Dutch yachtie / techie
/ software guy based in NZ and just wanted to say I am extremely
interested the direction this effort is heading. Other commitment
means I'll probably stay relatively quiet for now but I'm very keen to
be an active part of this, and the possibilities have me very excited.
My votes for what its worth:

  • Primary discussion method that can be followed using an email
    client.
  • I like the "Signal K" name
  • Common JSON based data format would be a very positive step

Marijn Weehuizen

On Wednesday 02/04/2014 at 6:07 am, Tim Mathews wrote:

IMHO, for long running discussions (like this one) nothing beats a
good old-fashioned mailing list.
So, here's one: https://groups.google.com/forum/#!forum/signalk
The mailing list address is signalk@googlegroups.com. Just send an
email to signalk+subscribe@googlegroups.com to join. For now anyone
can join, but if it gets spammy we can switch to an
apply-for-membership option.
I also have an irc server at irc.openseasproject.org if you like IRC.
I'll be in #signalk

Reply to this email directly or view it on GitHub.

Hi all, I just wanted to show my face. robert pointed me at the thread
shortly after it started. So far I have been lurking rather than
feeling the need to actively participate. I'm a Dutch yachtie / techie
/ software guy based in NZ and just wanted to say I am extremely
interested the direction this effort is heading. Other commitment
means I'll probably stay relatively quiet for now but I'm very keen to
be an active part of this, and the possibilities have me very excited.
My votes for what its worth:

  • Primary discussion method that can be followed using an email
    client.
  • I like the "Signal K" name
  • Common JSON based data format would be a very positive step

Marijn Weehuizen

On Wednesday 02/04/2014 at 6:07 am, Tim Mathews wrote:

IMHO, for long running discussions (like this one) nothing beats a
good old-fashioned mailing list.
So, here's one: https://groups.google.com/forum/#!forum/signalk
The mailing list address is signalk@googlegroups.com. Just send an
email to signalk+subscribe@googlegroups.com to join. For now anyone
can join, but if it gets spammy we can switch to an
apply-for-membership option.
I also have an irc server at irc.openseasproject.org if you like IRC.
I'll be in #signalk

Reply to this email directly or view it on GitHub.

@ktuukkan

This comment has been minimized.

Show comment
Hide comment
@ktuukkan

ktuukkan Apr 4, 2014

Didn't remember this earlier; I was contacted by www.marssa.org around 2011. They had quite similar goals as Signal K and were building a reference architecture. Haven't heard about them since and don't know if they are still active, but maybe worth checking out.

ktuukkan commented Apr 4, 2014

Didn't remember this earlier; I was contacted by www.marssa.org around 2011. They had quite similar goals as Signal K and were building a reference architecture. Haven't heard about them since and don't know if they are still active, but maybe worth checking out.

@keesverruijt

This comment has been minimized.

Show comment
Hide comment
@keesverruijt

keesverruijt Apr 4, 2014

On 04 Apr 2014, at 16:35, Kimmo Tuukkanen notifications@github.com wrote:

Didn't remember this earlier; I was contacted by www.marssa.org around 2011. They had quite similar goals as Signal K and were building a reference architecture. Haven't heard about them since and don't know if they are still active, but maybe worth checking out.

At the time I thought it was a ploy to qualify for EU subsidies, but I see there is recent activity on github.com/marssa. Still after 2 years it only interfaces with NMEA 0183, and only a GPS and wind wand?

On 04 Apr 2014, at 16:35, Kimmo Tuukkanen notifications@github.com wrote:

Didn't remember this earlier; I was contacted by www.marssa.org around 2011. They had quite similar goals as Signal K and were building a reference architecture. Haven't heard about them since and don't know if they are still active, but maybe worth checking out.

At the time I thought it was a ploy to qualify for EU subsidies, but I see there is recent activity on github.com/marssa. Still after 2 years it only interfaces with NMEA 0183, and only a GPS and wind wand?

@schwehr

This comment has been minimized.

Show comment
Hide comment
@schwehr

schwehr Apr 5, 2014

Personal opinions only in what follows...

Kees Verruijt pinged me discussion and suggested I should add my $0.02 on requirements. Interesting discussion. For me, JSON is nice to have on the side, but I can't really consider a text based message as the primary format. I've been thinking about how to replace things like Generic Sensor Format (GSF) and the zillion vender binary formats for logging multibeam and sidescan sonars, LIDAR (e.g. tons of work in the PulseWave project), high rate intertial navigaton systems (IMU/INS), the full working set of what can be expressed for smaller sensors with NMEA 0183 messages, AIS, Radars, imagery, carrier phase GPS (aka RTK, Rinex format,...), acoustic recoders (hydrophones and geophones), e&m, an open version of MISLE ship incidents, whale sitings, data license, software versions used, the whole metadata world (ISO paywalled disaster), etc. And then think about all this combined for a very large number of platforms.... e.g. all AIS receivers in a network, all buoys, AUV, ASV, etc in a fleet. On some systems, either because high resolution timing is key or because they are underwater for long periods of time without GPS (AUVs or dataloggers that may sit on the sea floor for months), multiple time sources may be essential. And I would love to see more at the OpenROV, OpenCTD, Arduino/Raspberry Pi level for cheaper and smaller sensors. We then as a community need the ability to conflate data over long time series. e.g. Maritime Rule X changed on July 1st, 2007... how did the env change in the years before and after? Did that increase or decrease regional fuel use, ship incidents, whale strikes, ship noise, economic activity at local ports, etc? I'm already see streams that are GB to TB per day and need to scale up from there. For the browser, JSON is great for small things, but the overhead of strings to binary for rendering of say just 10M points in Chrome, forced a switch to binary over the wire.

Even just the ARGOS float project has the potential to explode... they only have a few sensors right now, but what happens when they get tons more sensors and multiple routes for data to make it to a usable datastore.

I've been worrying about this for a long time now. I saw the issue with the number of sensors on the Healy all trying to spew NMEA 0183-ish stuff to NAIS with the USCG to working with sonars. I tried to start writing some of my ideas here: http://schwehr.org/blog/archives/2014-02.html#e2014-02-18T09_06_12.txt (and other blog posts). However, I haven't gotten very far. I tried making a message definition language of my own to support AIS, but did not get very far in the RTCM community. My current thought is that the best core definition place to start is with Protocol Buffers (ProtoBufs) and RecordIO or similar. That would allow for other serializations, but start off with a well defined message content that has a build-in binary serialization combined with the ability to add other serializations (e.g. JSON, XML, proto ascii, etc) to the mix fairly easily. I'm also realizing that there is a need for a journalling style system (append only) that could allow for index messages to be written to log files that would make access to logs much faster. This becomes especially important if you are later trying to join chunks of logs that might come anywhere from near realtime to months after the fact. There are plenty of other issues in there... e.g. nothing seems to talk about recording the calibration process in sensor log streams or things like time from which source and how accurate is it? And what license is the data released under? The bathymetry attributed grid (BAG) format contains ISO XML metadata, but they only have "classified" / "non-classified" for data. What about public domain, proprietary, creative commons license X, etc. ?

I think any spec has got to be totally open and it would be best to be machine readable and usable for all for any purpose. e.g. paywalled or GPL are a real problem. Something like the Apache 2 license seem like the only way to go if we really want to open up data interchange.

I've spent a lot of time trying to survive the existing disaster of standards and want to be moving towards a world in which data is easier and more fun to work with from small devices to global scale fleets of sensors. Hopefully some of these thoughts are useful to the discussion you all are having here.

Some links to material that might be interesting for you all:

schwehr commented Apr 5, 2014

Personal opinions only in what follows...

Kees Verruijt pinged me discussion and suggested I should add my $0.02 on requirements. Interesting discussion. For me, JSON is nice to have on the side, but I can't really consider a text based message as the primary format. I've been thinking about how to replace things like Generic Sensor Format (GSF) and the zillion vender binary formats for logging multibeam and sidescan sonars, LIDAR (e.g. tons of work in the PulseWave project), high rate intertial navigaton systems (IMU/INS), the full working set of what can be expressed for smaller sensors with NMEA 0183 messages, AIS, Radars, imagery, carrier phase GPS (aka RTK, Rinex format,...), acoustic recoders (hydrophones and geophones), e&m, an open version of MISLE ship incidents, whale sitings, data license, software versions used, the whole metadata world (ISO paywalled disaster), etc. And then think about all this combined for a very large number of platforms.... e.g. all AIS receivers in a network, all buoys, AUV, ASV, etc in a fleet. On some systems, either because high resolution timing is key or because they are underwater for long periods of time without GPS (AUVs or dataloggers that may sit on the sea floor for months), multiple time sources may be essential. And I would love to see more at the OpenROV, OpenCTD, Arduino/Raspberry Pi level for cheaper and smaller sensors. We then as a community need the ability to conflate data over long time series. e.g. Maritime Rule X changed on July 1st, 2007... how did the env change in the years before and after? Did that increase or decrease regional fuel use, ship incidents, whale strikes, ship noise, economic activity at local ports, etc? I'm already see streams that are GB to TB per day and need to scale up from there. For the browser, JSON is great for small things, but the overhead of strings to binary for rendering of say just 10M points in Chrome, forced a switch to binary over the wire.

Even just the ARGOS float project has the potential to explode... they only have a few sensors right now, but what happens when they get tons more sensors and multiple routes for data to make it to a usable datastore.

I've been worrying about this for a long time now. I saw the issue with the number of sensors on the Healy all trying to spew NMEA 0183-ish stuff to NAIS with the USCG to working with sonars. I tried to start writing some of my ideas here: http://schwehr.org/blog/archives/2014-02.html#e2014-02-18T09_06_12.txt (and other blog posts). However, I haven't gotten very far. I tried making a message definition language of my own to support AIS, but did not get very far in the RTCM community. My current thought is that the best core definition place to start is with Protocol Buffers (ProtoBufs) and RecordIO or similar. That would allow for other serializations, but start off with a well defined message content that has a build-in binary serialization combined with the ability to add other serializations (e.g. JSON, XML, proto ascii, etc) to the mix fairly easily. I'm also realizing that there is a need for a journalling style system (append only) that could allow for index messages to be written to log files that would make access to logs much faster. This becomes especially important if you are later trying to join chunks of logs that might come anywhere from near realtime to months after the fact. There are plenty of other issues in there... e.g. nothing seems to talk about recording the calibration process in sensor log streams or things like time from which source and how accurate is it? And what license is the data released under? The bathymetry attributed grid (BAG) format contains ISO XML metadata, but they only have "classified" / "non-classified" for data. What about public domain, proprietary, creative commons license X, etc. ?

I think any spec has got to be totally open and it would be best to be machine readable and usable for all for any purpose. e.g. paywalled or GPL are a real problem. Something like the Apache 2 license seem like the only way to go if we really want to open up data interchange.

I've spent a lot of time trying to survive the existing disaster of standards and want to be moving towards a world in which data is easier and more fun to work with from small devices to global scale fleets of sensors. Hopefully some of these thoughts are useful to the discussion you all are having here.

Some links to material that might be interesting for you all:

@rob42

This comment has been minimized.

Show comment
Hide comment
@rob42

rob42 Apr 5, 2014

@schwehr - welcome, and thanks for the feedback. Wow - that really opens up some questions! You sound as impressed with these formats as I am with the average corporate IT 'architecture'!. Actually thats probably because many came from the same people...

But a valid point none the less, that raises questions:

  • What is the scope of Signal K?
  • If it includes your extra use cases, how would we handle them?

Some quick thoughts - we need to start somewhere and not get bogged in the 'whole universe' problem. But even on a single boat we will eventually see massive quantities of data - so we should have a plan to deal with it.

The basic Signal K format for boats is rooted in the vessels[] array. There is nothing to stop that growing for additional onboard data, and nothing to stop adding new arrays, eg argos[], or whatever. I have no idea what LIDAR data looks like - does it even convert to JSON? - or do we need a binary type.

As to wire format, and Tb data - I think we are in a reasonable position there. While our format is string JSON, mainly because its good for javascript/browsers, there is nothing to stop that being compressed, or using BSON, and JSON is also easy to move into protobufs - I already use a similar method to convert into java objects(using the Jackson project).

I quite like the BSON idea - mainly because its native to MongoDb, which Ive worked with and found to be very good for storing high rate data. Also good for replication - which is a nice way to offload data. Ive been wondering if I should use it to store history.

Probably the answer is to add a way to signal the available wire formats/supported wire formats during connection. So we start with JSON, then request BSON/protobufs/etc, and switch.

You thoughts?
Rob

rob42 commented Apr 5, 2014

@schwehr - welcome, and thanks for the feedback. Wow - that really opens up some questions! You sound as impressed with these formats as I am with the average corporate IT 'architecture'!. Actually thats probably because many came from the same people...

But a valid point none the less, that raises questions:

  • What is the scope of Signal K?
  • If it includes your extra use cases, how would we handle them?

Some quick thoughts - we need to start somewhere and not get bogged in the 'whole universe' problem. But even on a single boat we will eventually see massive quantities of data - so we should have a plan to deal with it.

The basic Signal K format for boats is rooted in the vessels[] array. There is nothing to stop that growing for additional onboard data, and nothing to stop adding new arrays, eg argos[], or whatever. I have no idea what LIDAR data looks like - does it even convert to JSON? - or do we need a binary type.

As to wire format, and Tb data - I think we are in a reasonable position there. While our format is string JSON, mainly because its good for javascript/browsers, there is nothing to stop that being compressed, or using BSON, and JSON is also easy to move into protobufs - I already use a similar method to convert into java objects(using the Jackson project).

I quite like the BSON idea - mainly because its native to MongoDb, which Ive worked with and found to be very good for storing high rate data. Also good for replication - which is a nice way to offload data. Ive been wondering if I should use it to store history.

Probably the answer is to add a way to signal the available wire formats/supported wire formats during connection. So we start with JSON, then request BSON/protobufs/etc, and switch.

You thoughts?
Rob

@tkurki

This comment has been minimized.

Show comment
Hide comment
@tkurki

tkurki Apr 22, 2014

Owner

Update here as well: this discussion has continued on SignalK Google Group (https://groups.google.com/forum/#!forum/signalk).

There is now a SignalK data model spec at http://signalk.github.io/ and conversion tools from NMEA 0183 and NMEA 2000 to SignalK.

Owner

tkurki commented Apr 22, 2014

Update here as well: this discussion has continued on SignalK Google Group (https://groups.google.com/forum/#!forum/signalk).

There is now a SignalK data model spec at http://signalk.github.io/ and conversion tools from NMEA 0183 and NMEA 2000 to SignalK.

@patuty

This comment has been minimized.

Show comment
Hide comment
@patuty

patuty Jul 14, 2014

Hello........
Sorry for asking such a basic question, but I'm sailing instructor. As I set up or run navgauge to use nmea data in real time??. I have tested and works software and gpsd. I only need to know how to make navgauge take the data from the corresponding port

patuty commented Jul 14, 2014

Hello........
Sorry for asking such a basic question, but I'm sailing instructor. As I set up or run navgauge to use nmea data in real time??. I have tested and works software and gpsd. I only need to know how to make navgauge take the data from the corresponding port

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment