Open universal json format #4
Comments
|
I am able & willing to change the Navgauge server-to-UI json format, but would really appreciate input & insight from others. Current format is pretty readable in the boat config files under https://github.com/tkurki/navgauge/tree/master/lib/boats |
|
Hi Tkurki, A good idea to get a more formal and better JSON format amongst the nautical apps. I can adapt CANboat as needed. |
|
Hey all, I too would like to work on formalizing the data format. Have a look at what I'm working on http://pyxis.openseasproject.org and https://github.com/timmathews/pyxis-design-documents. My goal with this is something akin to Maretron's N2KView. The timing of this is rather fortuitous, as I have been thinking a bit about message specifications this weekend. At a high level, I don't want the client to have to know beforehand how data is formatted or what is available. Instead, the server should provide an API to query for this information. The only thing the client needs to know is how to actually make that query. So once the client makes a request to the server (standard HTTP GET request to a published URL) it will get back a list of all of the individual messages that the server knows how to send. Using this information, our user can build their display pages in the browser and save that configuration to the server. Then when one of those user created pages is displayed, another request is made via a WebSockets connection this time subscribing to the specific data that the client wants (essentially, whatever is needed to drive the gauges, charts, graphs, etc. displayed on the screen). We call this the 'control' channel. The server responds to that request with a URL to a WebSockets connection custom built for that client. The client connects to this new WebSockets URL (the 'data' channel) and begins displaying the data received. I've pretty much worked all of this out and will be continuing to document it at http://openseasproject.org and on my GitHub page, but what I don't have yet is a good way to format the data going to the client. I know that I want to be able to break down the data packets which come in to the server (n2k, 0183, SeaTalk, whatever) and reconstitute them in a more friendly manner. Right now, what I'm doing is using my Go port of CANboat to take data from an Actisense NGT-1, pack up the formatted PGNs (using the CANboat format) with MessagePack, send them via ZeroMQ to a very simple web server (again written in Go) which serves the data (this time JSON encoded) via WebSockets to whoever wants to connect. This is all kind of a hack, but it should be a lot better soon. |
|
Yes, Im all for a standard data format, and since the web and HTML5/websockets is my target I'm all for using JSON. If you look at freeboard (https://github.com/rob42/freeboard-server/blob/master/src/main/java/nz/co/fortytwo/freeboard/server/util/Constants.java) you will see Ive used a 3-char key and associated value. I just create an array of them and convert it to JSON. Its a basic format, and could be improved by sub-grouping by use (eg position, engine, etc) to more efficiently handle long arrays. The protocol is also easily used by the arduino micro-controller as a non JSON serial stream. I just chop it [0-2][3-EOL] and I have key/value for minimal CPU. The consistent 3-char key and CR between each make it easy to send or process at the arduino end. Even on the RPi or BBB there is a minimum of spare CPU to parse data, so it needs to be easy to handle. Also in my protocol if the label starts with # its a command, which I also required. Plus its still reasonably readable for dev and debug. Probably I should add some sort of message checksum too. So I think the protocol should have:
At the client end I just pass the keys past the various gui objects and they use them if they are interested - that would work nicely for assembling custom displays. Maybe the display should also register to receive interesting keys? Same with sending keys, a given device or gui component just issues a key=value whenever necessary, its passed to the webserver (via websockets) and routed accordingly to others. Keep in mind there may be more than one 'server', we should think peer-2-peer. I dont think there is much need to create a complex formal spec at this stage, just that we all work on a similar basis and co-ordinate our keys. I expect we will find new needs and opportunities, so things will change. And I like the idea (TimMatthews) that the device can be queried as to its capabilities - need a key for that. Robert (freeboard) |
|
The key is inter-operability. While we remain unique and isolated our projects will never get critical mass ...we need to get OpenCPN into this too. |
|
Totally agree we need to get OpenCPN onboard and also OpenSkipper if they're still actively developing (it's been a year since their last update). There is another project that I've been watching at http://sailboatinstruments.blogspot.com which is pretty interesting. I'll reach out to him and see if he wants to be involved. |
|
Whoa, seems like there is some demand for this sort of thing. Making things interoperable is in my interest as well. Some thoughts on the discussion so far: What are we actually talking about when we want common json? CANboat and gpsd for example already produce json. In fact @canboat's comment about changing the json format sounded to me more like the tail wagging the dog - why not pick the N2K json format for everybody? Navgauge is not using it for the fact that half of my data is NMEA 0183. Furthermore when I got hold of datasets for more than one boat I realised that my first interest, a SailSteer clone gauge, needs some basic data and where they are coming from is different on every boat: Freya has several N2K sources for some of the data, Cassiopeia has both NMEA and N2K and Plaka just NMEA. All this lead me to create a basic stream of data that has just the needed fields and nothing extra. Then again Navgauge server can serve also the raw data in json format - see http://navgauge.herokuapp.com/swipe/bus. For N2K this is just CANboat format. For NMEA it is the dead simple format in node-nmea. Node-nmea suited my purposes most easily, so I just went with that. I need to add support for some more sentences to get my N2K tridata available as NMEA for iNavX and iRegatta. For me the use case is really that if somebody comes up with a great display/gauge/overview I would like to use that on my boat with the minimum of hassle. Also the reverse is true: @rob42 said kind words about my multiuse gauge - if Freeboard would provide the same basic data in the same format over websocket he could just copypaste some Javascript & css and use the gauge on his system. Client side browser component infrastructure is developing quickly right now, which means that there is no one way to package a component, but I am confident something will emerge. With a little bit of extra work we could extract the gauge component and use it as part of Freeboard and Navgauge. Maybe somebody with more skills for graphics could make it prettier, now it's more functional the esthetic. For this we need standard json format for the basic nav data or the component needs configuration options or intelligent fallback with defined precedence for the different formats. I've also thought about providing a "list capabilities" type of call. BTW the data set is not static - if you had yesterday a certain PGN available it may not be there today. Or querying for capabilities right now doesn't give you the PGNs from the equipment that is not on. As for the configurability evident in Pyxis plans I've decided not to go that way at all: so far Javascript/HTML/CSS is the way I configure my display. No doubt this will change, but I assume by just using javascript to instantiate more high level widgets and connect them to certain selected data from the server. Combining data items is imho pretty straightforward as well. I am not sure about the "simple key=value structure, rather than NMEA0183 style complex combinations of values" - some items have structure and losing that structure might not be a good idea, for example yaw-pitch-roll and rates of turn for example. Also handling structure in json is dead simple. "Human readable, Checksum field, fixed length short field labels" sound conflicting - the greatest joy of working with json as opposed to xml or some other format is that it really is human readable. I am definitely not fond of three letter abbreviations and it sounds like NMEA all over again. I plan to rely on tcp/ip instead of checking checksums and Mr Moore will take of the extra effort in parsing. Keys on the level of "these data items I'm interested in" make sense, as in a subscription request. Something that I've run against is that the UI might be interested in a time series of some data - say STW and SOG (see me doing the three-letter-acronym thing ;-) or a single item from two different sensor - to create a plot. So having access to history data is something I'll be looking at in the client 2 server communications. I suppose my case is a bit different, since the UI code runs on the browser and the tablet or whatever might be switched off for periods and I would still like to get history graphs. I have been thinking about in terms of client and server, not peer to peer, but can imagine scenarios for that. I suppose the json could be the same, but something other than websockets for reliable communications with queuing & retry. |
|
@timmathews I'm curious:
|
|
What would be the use case for this for OpenCPN? As it is a single application with direct access to the different data streams I don't see the need, but is there something I'm missing? |
|
@timmathews Navgauge server has the possibility to subscribe to
|
|
Does anybody have access to GoFree data in json format? I think we should consider being compatible with commercial stuff as well. And btw I would be really interested in looking at other data captures. At least for me adding configurations for different boats changed the software quite a bit. |
|
Yes. Good questions :-) On 17 Mar 2014, at 18:13, tkurki notifications@github.com wrote:
My own solution uses (as you guys probably know) uses actisense-serial | analyser -json | n2kd which is usable directly by a HTML web client. My code predates web sockets. Web sockets is pretty cool tech, but I am glad I didn’t use it at the time as the spec kept changing. I am still happy with json as a format, and I am also happy with choosing C for the analysis — it is not a trivial bit of code and the amount of processing power on my server is limited. One other thing that we may want to discuss is whether we can’t come up with a common server side infrastructure. Even with this small group we already have Go, PHP, node.js and Java if my info is correct. Boy. Sure sounds like a lot of wasted effort. I am happy to move from PHP to something else as long as it is reasonably light weight and will run in a few megabytes. One thing that kept me from publishing my PHP code is that (a) there is a lot in there that doesn’t work with other vessels, unless you want to invest in a Wago PLC running Linux. (b) You need to configure a lot in json config files, and I didn’t want to field so many questions. What I would like is to reuse those beautiful gauges and SailSteer pages that you guys have been developing. Kees |
|
Another idea might be to make a minimal generic translator from NMEA 0183 on a wifi to an open json format. Use case is that some existing solution puts out NMEA over wifi and one would like to have a browser app connect to it. An easy to install translator that connects to the NMEA 0183 feed and provides json over http and/or websocket would make the json format more approachable. |
|
I went for Node because I wanted to know what the fuss was about. It turned out to be a really nice experience, @canboat I urge you to try it. And like it says on the Github page, it is CANboat and gpsd/gpspipe underneath on the data acquisition side. I'm still not sure about parsing NMEA on the server. Should probably drop it in favor of the gpsd json format. Need NMEA encoding though for N2K to NMEA conversion. |
|
For OpenCPN the benefit of this is not to have to reinvent the wheel - we must implement N2K (and SeaTalk) in some way. And even though I was looking at extending @canboat conversion to NMEA-0183 for now, implementing a format adopted by all of the other projects seems to be a better idea long term. But of course, we will stay away from WebSockets and alike as they bring nothing in our case... |
|
@canboat: speaking of peer to peer: If you replace the line that forks the actisense-serial | analyzer -json process with something that reads from your n2kd you should be able to use Navgauge with very little effort. See freya.js for an example on routing data from PGNs to the 'normalized' format that the UI expects. Which all makes me think of running CANboat like you do and just have Navgauge server connect to it. Do you happen to have an init script for CANboat? I think this reflects the main idea of this thread: there is already a de facto open source standard for N2K data and several for NMEA in json format. Where should translation from these formats to the representation usable by a UI component should take place and what format should communication between these components use? |
|
Some more comments (random ramblings):
I will try to run navgauge soon. (So many things to do...) |
|
The use case for peer2peer is multiple producers, multiple consumers.
This is a much better model than a single central server that does all. It means that actions on any one client or device is propagated across all the devices, and things still work when others break. key=value ... yes we should use complex objects in the JSON. (I already use this for AIS data). By key/value pairs I meant that each value should be a separate key, not comma separated bundles like NMEA0183. short keys...as a unix geek 3 letters is enough:-) but I guess that we can have normal names for keys, it does make it easy to read. Maybe we could have a 3 letter abbrev for a key for special cases? Rob |
|
If you are going to do that you will probably want a sender (origin) in every message, and timestamps to check that the data is current, as well as something to have multiple instances of the same data type. Hey, we’re re-inventing N2K! Maybe the 0183 JSON format should be more like a N2K message: - have an origin (= N2K src) and timestamp added by the device that translates it to JSON. One solution is that we develop a 0183 -> N2K converter and do away with all 0183, and/or the other way around as well. Note that n2kd has the beginning of some N2K -> NMEA0183 conversion. It only needs more work :-) Rob, For the peer-to-peer part you may want to look into multicast. It dramatically simplifies network configuration (aka there isn’t any.) Unfortunately our browsers won’t be able to use multicast. Kees On 17 Mar 2014, at 20:33, rob42 notifications@github.com wrote
|
|
@tkurki So basically I reimplemented CANboat in Go (CANboatGo anyone?) in order to learn Go. I like the language and the concurrency patterns baked in make it a good fit for this sort of application. My convoluted data path from device to browser is a result of having a working application which could read N2k and spit out MsgPack and having a simple fake data generator that could speak JSON over WebSockets. It took about 10 lines of code to convert the data generator into a MsgPack/ZeroMQ to JSON/Websockets bridge, so that's what I did. My two-cents on peer-to-peer: there's nothing in current browsers or in any W3C draft that I'm aware of that allows a browser to make any p2p connections, they're still strictly client-server. P2P between servers or devices which collect and propagate data is a good thing IMO though and should be investigated further. My recommendation is to use a message queuing system there (either brokered like RabbitMQ or brokerless like ZeroMQ) instead of multicast/anycast. |
|
Yeah re-inventing NMEA200 would be great - its problem is its closed proprietary nature. If we implemented it we would be forever caught up with conflicting PGM's (theirs/ours) and susceptible to them changing spec on us. So we should 'embrace and extend' it :-) Multi-cast isnt really necessary IMO although its quite interesting, I use Apache Camel to do smart processing/routing/conversion which is pretty easy and powerful. The browser will never be a full p2p client - its always going to have to be pushed by the webserver although websockets makes that pretty easy. But the server implementation should be agnostic - we all have our preferences there. Also the transport, be it wifi, wired network, CANbus, or serial should not matter if we send the same JSON string. Rob |
|
So peer pressure has made me clean up my server a bit and push it to GitHub. I called it argo. It's really pretty simple stuff, running it on a machine which has an NGT-1 connected, it will take n2k data from that and publish it to a WebSockets stream at /ws/v1/data. You can check it out now at http://pyxis.openseasproject.org/n2k.html That's live data from a Maretron WSO100 in my basement with a fan blowing on it. It needs to be cleaned up a bit and I need to do some housekeeping like adding a README and licensing and a nice thank you letter to Kees for doing all the hard work of figuring out just what the heck the NGT-1 was saying. |
|
@timmathews - nice - I can kind of see what its saying, but there is a lot of misc data in there. So in freeboard I would send something like 'WSA=20[CR]WDA=295[CR]' in the simple serial version, and in JSON it would ideally become something like: Then any interested listener can look at a JSON message and if it has a 'wind' object, get the 'wind.apparentSpeed' I do this for AIS data, using an array of aisinfo objects in an ais header object. each aisinfo is one vessel, and holds speed dir, lon. lat etc. Rob |
|
@rob42 So yeah, there is a lot of info in each packet. The structure is something like this for wind (130306): {"Header": {
"Timestamp": "2014-03-17T17:34:16.978571418-04:00", // ISO-8601 Format timestamp
"Priority": 2, // Message priority on the wire as defined by NMEA
"Source": 128, // Device ID claimed by the WSO100 at bootup
"Destination": 255, // Every listener on the bus
"Pgn": 130306, // PGN ID
"Length": 8, // Number of bytes on the wire
"Data":"7LcAzUz6//8=" // Raw data un-decoded
},
"Index": 159, // The index of the definition of the PGN in my map of PGN definitions
"Data": {
"0":236, // Field 0, in this case the SID
"1":1.83, // Field 1, wind speed
"2":112.64932725148729, // Field 2, wind direction
"3":"Apparent" // Field 3, measurement type
}
}One of the features that I intended to implement originally was a message definition query API, so if you saw this data come in, you would make an HTTP GET request to /api/v1/pgn/:index: where :index: is the number from the packet above and you would get JSON back describing the data: field names, precision, range, units, etc. So that's why Index is there. As for the rest, well I already had that data from the n2k bus, so until I know what I need and what I don't, lets send it along. |
|
Your example shows what @canboat mentioned: it is often easier to send everything. Bandwidth on a local wifi hotspot is a nonissue and browsers running on tablets and phones have ample cpu for handling the data. In the interest of a common json format this doesn't sound too promising though: you've come up with an alternate json structure to N2K data that is different from the one CANboat analyzer uses. Getting data flowing and visible on the screen gives you the moment of satisfaction (at least it did for me!). When you start dealing with multiple sources and multiple formats it makes you start thinking about interoperability. A concrete example would be that if I wanted to put your data as a sample data set for Navgauge I would have to deal with one more format. If you were using N2K json format I would just try freya.js and it would probably work with a parameter change or two. After that I would be a bit wiser, extract the common logic for pure N2K type boats and make it easier to adapt for such boats. And your data would be available to the wind history plot in Navgauge should you choose to try it out. I don't see the need for separate metadata api, as I don't see why the format needs to be so verbose in other aspects but the data fields are just "0", "1" instead of properly named fields like they are in the CANboat format. Btw an array would more idiomatic for a list of fields. As for the 'simple' json Rob mentioned: this is what Navgauge has for 'gaugeData' (as opposed to raw data): A separate 'type' field like this has works well for switch type handling logic, but Rob's proposed typed subelement would allow one top level message to contain several different types. For now I don't need timestamp - whatever is received is displayed immediately. Once I delve into fetching history from the server this will change. Several inconsistencies that need to be fixed: boat speed item named 'knots', but wind speed is just speed without unit, depth depth..(sheesh who did this stuff..). |
|
Ah yes units of measure...lets just chose one consistent set for marine and convert when needed, then we dont have to mess with 'units' tags and extra code to deal with it. So we could use knots for SOG and wind speeds, decimal degrees for angles, meters for distance and depth, UTC for time. Assume numbers may have decimals, eg floats Maybe even do meters/sec for speeds - would that be an advantage for complex analysis/comparison later? Rob |
|
Hmm. “most convenient” for one is “hate” for others. I happen to agree with your “most convenient” units, but our set is just one of many possibilities. There is one way out of that conundrum: use scientific units, e.g. SI. The seven basic units in SI are: metre, kilogram, second, ampere, kelvin, mole, candela. If the format does not use units then the implied units should always be an SI unit. In other words: m/s, m, liters, degrees K, radians, radians/s. Floats is a good idea, it gets us out of the silly units that N2K uses for some datagram components. Time always expressed as UTC, and maybe always as seconds since 1970 (UNIX epoch) with an optional fractional part? Oh, and we define that the locale is “POSIX", e.g. numbers are expressed as nnn.ddd, e.g. with a decimal point and not any other character, like my native comma (in Dutch we swap . and , in numbers compared to English.) And we define that the character set is always UTF-8. On 18 Mar 2014, at 08:53, rob42 notifications@github.com wrote:
|
I'm not set on this format, in fact I pretty much like @canboat's format better. On the other hand, I think we should think long and hard before fixing our JSON representation to n2k PGN definitions. NMEA2000 is essentially an extension of SAE J1939 and the engineers who designed those standards had to make certain decisions informed by their transmission medium and available bandwidth. Our answers to those same questions will be different because we're working in a different medium with different rules.
Having a separate metadata API allows me to communicate a lot of static info to the client one time. Things like a nice display name for each field, an abbreviated version of the display name, the units that the field reports (or in the case of what lookup fields (enums) the possible values for the field), min and max values and what values are reported for errors. The field IDs in my current code are just numbers because I was being lazy at the time.
As far as units go, consider this my vote for SI units, ISO-8601 timestamps, POSIX locale, and UTF-8. |
|
Here's the metadata API in action: http://pyxis.openseasproject.org/api/v1/messages for the whole list and http://pyxis.openseasproject.org/api/v1/messages/130310 for a specific PGN. It doesn't work for all PGNs currently because of a limitation with Go's JSON encoder. Hopefully I'll have a chance to work that out tonight. It also only has the data that @canboat included in the original array of PGNs. I haven't added all of the other data that I'd like to see (min, max, and error values for example) mostly because I don't know what they are. |
|
@fabdrol Take a look at http://navgauge.herokuapp.com/gauges.html . Ideally you should be able to enter your own server's url in the box, press connect and the gauges would start showing data from your server. (the box is now a placeholder, no functionality) Navgauge has the ability to play back raw NMEA captures, Canboat analyzer output and Navgauge's own json capture. Handling raw Canboat actisense output and adding support for several NMEA sources is pretty straightforward. Whatever you do I suggest getting some data captures from real sailing, captures from a boat sitting still get boring pretty quick ;-). If you don't mind Node you can just start with cloning Navgauge and playing around with it. Browserify, AMD, Bower etc +1, haven't done it because it's been just me as the user and not sure what to pick. Web Components seems to be still in the future, but maybe you have better info on that. |
|
@tkurki Node is my language of choice as well, so that is a good start. Here's what I'm thinking: More practical: I think we need to fork a good (NMEA) parser and start work on the implementation of our new protocol - evolving the parsers as the protocol evolves. My thinking is, if we have a few basic parsers available (SeaTalk, NMEA 0183 & 2000 to start with?) of which we make sure they follow the specification of the protocol with each new version, new contributors have a good place to get started - either as contributor to the parser or building something on top of a parser. That said; maybe we should vote for a name quickly, set up the new organisation on github and set up a few basic things for collaboration (e.g. an IRC channel for communicating, a Trello board to manage what needs to be done etc). Then we can decide on what to do next and start making versions of the parsers and the specification, testing the spec with the parsers in our individual projects as we go along. @rob42 my vote's for Signal K for the project/organisation and "KJSON" as the name for the format (mainly to distinguish between JSON and KJSON when we're talking about stuff). |
|
@tkurki btw, a few notes on client side:
But, this is all a bit off-topic - as @ktuukkan pointed out. The Consumer implementation is entirely up to the developer's preferences, I suppose. |
|
Consider this my vote for SignalK as well! |
|
+1 for SignalK. 2014-04-01 14:06 GMT+02:00 Tim Mathews notifications@github.com:
|
|
In the interest of getting sh*t done, github.com/SignalK is now a project. You should have all gotten notification in one form or another. EDIT: I meant organization of course Now, @rob42 since you came up with this name and mentioned a logo, we need a logo. 😄 |
|
One more thing before I'm off to work. When do we ask Ben at Panbo to blog about us? The resultant conversation should be, erm, interesting. |
|
Get something that actually interoperates done first? Or something that has Speaking of communication: this issue is getting a bit longwinded. Ideas on tiistai 1. huhtikuuta 2014 Tim Mathews notifications@github.com kirjoitti:
|
|
How about IRC + Trello for managing tasks/features/whatever? |
|
Perhaps a google group as well? |
|
I don't mind IRC but I don't have a reader open for that normally, so I'd Do we want privacy? 2014-04-01 16:11 GMT+02:00 Tim Mathews notifications@github.com:
|
|
BTW, +1 for Google Group |
|
Okay, let's turn this around. Which services are you guys on already - apart from Github and e-mail.
|
|
Same here except for IRC. BTW I am looking for a new cross service chat client anyway, preferably with iOS and Android support. On 01 Apr 2014, at 16:39, Fabian Tollenaar notifications@github.com wrote:
|
|
I did see something interesting a while a go, a chat-based project manager with integration for Github and other services. Can't remember the name though, and if it was free. I'll check it out. |
|
or, google hangout |
|
I've set up a Trello board to get stuff started. I'm aware that not everyone pitched in yet, if we do decide to use something else it can easily be deleted again! Join here - or send me an e-mail https://trello.com/b/1ApFRVqT Fabian |
|
@canboat could you invite me to the hangout at fabian [at] startingpoint.nl? Thanks |
|
@fabdrol: http://navgauge.herokuapp.com/treedata.html now produces nested update messages with the nested=true option in the ws url. That is something that I plan to keep available and updated to the latest spec, subject to available time. N2k timestamp I still need to fix, now it is direct analyzer output. I still haven't given up on the {id:'navigation.courseOverGround', value:'164.2', src:...} though. I find that a lot easier to work with on the consuming side. Navgauge now supports both. Hmm, quite a lot of choice for team communications. I don't have a good solution for constant IRC access and as we are a global team. Persistence of the communication and access to older discussions matter for people joining later, so IRC and Skype are less suitable. A mailing list with archive/Google group would be my choice right now, but I've already felt the need for chatting/asking immediate/short questions as well. Definitely open, if I had just emailed back and forth with Rob this thread would not have happened. Email in case we need privacy. |
|
@tkurki hmm.. I must say I agree with what some others posted here before; 'navigation.courseOverGround' seems "un-native" to me. As a JSON-based format, the nested form seems much more natural. Btw, are you simulating a dataset or is that real data? Cheers |
|
Replay of captured data, same as what you get at The dataset is in Navgauge internal format from last summer: tiistai 1. huhtikuuta 2014 Fabian Tollenaar notifications@github.com
|
|
IMHO, for long running discussions (like this one) nothing beats a good old-fashioned mailing list. So, here's one: https://groups.google.com/forum/#!forum/signalk The mailing list address is signalk@googlegroups.com. Just send an email to signalk+subscribe@googlegroups.com to join. For now anyone can join, but if it gets spammy we can switch to an apply-for-membership option. I also have an irc server at irc.openseasproject.org if you like IRC. I'll be in #signalk |
|
Hi all, I just wanted to show my face. robert pointed me at the thread
Marijn Weehuizen On Wednesday 02/04/2014 at 6:07 am, Tim Mathews wrote:
|
|
Didn't remember this earlier; I was contacted by www.marssa.org around 2011. They had quite similar goals as Signal K and were building a reference architecture. Haven't heard about them since and don't know if they are still active, but maybe worth checking out. |
|
On 04 Apr 2014, at 16:35, Kimmo Tuukkanen notifications@github.com wrote:
|
|
Personal opinions only in what follows... Kees Verruijt pinged me discussion and suggested I should add my $0.02 on requirements. Interesting discussion. For me, JSON is nice to have on the side, but I can't really consider a text based message as the primary format. I've been thinking about how to replace things like Generic Sensor Format (GSF) and the zillion vender binary formats for logging multibeam and sidescan sonars, LIDAR (e.g. tons of work in the PulseWave project), high rate intertial navigaton systems (IMU/INS), the full working set of what can be expressed for smaller sensors with NMEA 0183 messages, AIS, Radars, imagery, carrier phase GPS (aka RTK, Rinex format,...), acoustic recoders (hydrophones and geophones), e&m, an open version of MISLE ship incidents, whale sitings, data license, software versions used, the whole metadata world (ISO paywalled disaster), etc. And then think about all this combined for a very large number of platforms.... e.g. all AIS receivers in a network, all buoys, AUV, ASV, etc in a fleet. On some systems, either because high resolution timing is key or because they are underwater for long periods of time without GPS (AUVs or dataloggers that may sit on the sea floor for months), multiple time sources may be essential. And I would love to see more at the OpenROV, OpenCTD, Arduino/Raspberry Pi level for cheaper and smaller sensors. We then as a community need the ability to conflate data over long time series. e.g. Maritime Rule X changed on July 1st, 2007... how did the env change in the years before and after? Did that increase or decrease regional fuel use, ship incidents, whale strikes, ship noise, economic activity at local ports, etc? I'm already see streams that are GB to TB per day and need to scale up from there. For the browser, JSON is great for small things, but the overhead of strings to binary for rendering of say just 10M points in Chrome, forced a switch to binary over the wire. Even just the ARGOS float project has the potential to explode... they only have a few sensors right now, but what happens when they get tons more sensors and multiple routes for data to make it to a usable datastore. I've been worrying about this for a long time now. I saw the issue with the number of sensors on the Healy all trying to spew NMEA 0183-ish stuff to NAIS with the USCG to working with sonars. I tried to start writing some of my ideas here: http://schwehr.org/blog/archives/2014-02.html#e2014-02-18T09_06_12.txt (and other blog posts). However, I haven't gotten very far. I tried making a message definition language of my own to support AIS, but did not get very far in the RTCM community. My current thought is that the best core definition place to start is with Protocol Buffers (ProtoBufs) and RecordIO or similar. That would allow for other serializations, but start off with a well defined message content that has a build-in binary serialization combined with the ability to add other serializations (e.g. JSON, XML, proto ascii, etc) to the mix fairly easily. I'm also realizing that there is a need for a journalling style system (append only) that could allow for index messages to be written to log files that would make access to logs much faster. This becomes especially important if you are later trying to join chunks of logs that might come anywhere from near realtime to months after the fact. There are plenty of other issues in there... e.g. nothing seems to talk about recording the calibration process in sensor log streams or things like time from which source and how accurate is it? And what license is the data released under? The bathymetry attributed grid (BAG) format contains ISO XML metadata, but they only have "classified" / "non-classified" for data. What about public domain, proprietary, creative commons license X, etc. ? I think any spec has got to be totally open and it would be best to be machine readable and usable for all for any purpose. e.g. paywalled or GPL are a real problem. Something like the Apache 2 license seem like the only way to go if we really want to open up data interchange. I've spent a lot of time trying to survive the existing disaster of standards and want to be moving towards a world in which data is easier and more fun to work with from small devices to global scale fleets of sensors. Hopefully some of these thoughts are useful to the discussion you all are having here. Some links to material that might be interesting for you all:
|
|
@schwehr - welcome, and thanks for the feedback. Wow - that really opens up some questions! You sound as impressed with these formats as I am with the average corporate IT 'architecture'!. Actually thats probably because many came from the same people... But a valid point none the less, that raises questions:
Some quick thoughts - we need to start somewhere and not get bogged in the 'whole universe' problem. But even on a single boat we will eventually see massive quantities of data - so we should have a plan to deal with it. The basic Signal K format for boats is rooted in the vessels[] array. There is nothing to stop that growing for additional onboard data, and nothing to stop adding new arrays, eg argos[], or whatever. I have no idea what LIDAR data looks like - does it even convert to JSON? - or do we need a binary type. As to wire format, and Tb data - I think we are in a reasonable position there. While our format is string JSON, mainly because its good for javascript/browsers, there is nothing to stop that being compressed, or using BSON, and JSON is also easy to move into protobufs - I already use a similar method to convert into java objects(using the Jackson project). I quite like the BSON idea - mainly because its native to MongoDb, which Ive worked with and found to be very good for storing high rate data. Also good for replication - which is a nice way to offload data. Ive been wondering if I should use it to store history. Probably the answer is to add a way to signal the available wire formats/supported wire formats during connection. So we start with JSON, then request BSON/protobufs/etc, and switch. You thoughts? |
|
Update here as well: this discussion has continued on SignalK Google Group (https://groups.google.com/forum/#!forum/signalk). There is now a SignalK data model spec at http://signalk.github.io/ and conversion tools from NMEA 0183 and NMEA 2000 to SignalK. |
|
Hello........ |
Navgauge uses its own 'universal' format for sending basic navigation data to the browser. 'universal' and 'basic' mean that data from different sources (NMEA 0183, N2K, different sources within a N2K network) are combined on the server and sent in the same format and same fields so that the UI doesn't need to know anything about where the data is coming from. The combining is done in a boat specific configuration file in Navgauge.
This 'universal' way is something I just came up with when making the UI work universally with the data sets from three different boats that I had available. For example units are a mixture of metric & marine unit.
Creating a more universal and standard format would support sharing innovation & functionality between different efforts.
The text was updated successfully, but these errors were encountered: