Multiple sources for the same value #48
Comments
|
Thanks @rob42 for kicking this off. I'll pitch in with my TDD hat on and commit a test for the new format. I modified your example a little:
I think that the way the identifer is constructed should be specified:
BTW we have neglected to include nmea0183 talker id, it should be added to the schema as well. The next step would be to fix the schema so that the test passes for the sample. You can run the test after PS. The test code assumes that the input is without the |
|
I added Travis configuration for this and opened pull request #49 for this so now we have CI integration as well. |
|
Also need the extra rule to make it clear that we have options object: |
|
Where should the additional documentation like rules 1-4 go? |
|
I would put it as a comment in relevant code, and I will write up a page for the web site under 'developers'. |
|
The test I mentioned earlier: https://github.com/SignalK/specification/blob/multiple-values/test/multiple_values.js |
|
The prefered source of data probably belongs in a manifest (seperate structure defining the resources available from a vessle). Such a manifest would list the readings available from a vessel, a list of devices and contain the preferred mapping between them. If a date stamp was also included that would remove the need to litter the basic data with souce and timing data By declairing the devices (with readings available) in a manifest and specifying a suitable URI structure for SignalK you would give the user the choice to query the individual device for data using the existing SignalK schema with out the need to add additional "option" tags. |
|
We have previously discussed the need for a mapping layer in case there is need for server side processing of for example derived values. This would be part of the server configuration. But I think that the simplest case should be supported with minimun a priori knowledge, eg. the server should just need minimum configuration to connect the producers on board (n2k bus, nmea sources, other sources) and then be able to pass that data on. With the current implementations there is a de facto registry of all available data in the REST api contents. There are some ideas about this type of metadata in https://github.com/SignalK/specification/blob/master/schemas/groups/sensors.json. About timestamp: I think it belongs right next to the value as it is metadata about that particular value. It is also part of the streaming data: 'new value is x with timestamp y'. |
|
More stuff available at SignalK/signalk.github.io#17 |
|
Appologies if this is going over old ground. I suspect I'm coming at this from a different angle.. Because this moves SignalK towards being essentially just a super transport of other messaging a user is going to need considerable prior knowlage of the exact configuration of the vessel -- including knowlage of whcih components are reliably calibrated and the way it's going the structure of the none SignalK messaging its self -- to use SignalK sucessfully. Coming to the vessel from the outside with none of that knowlage a menu of possable readings for each value is going to be confusing at best. So either some work is done on the 'server' to pick the right source or there's a need for a manifest to tell me which is the best reading to use from the range provided. Achitecture wise it's also pushing SignalK towards a particular and more expensive implementation model (all the work done at the end applicaiton) that doesn't allow for work to be distributed over the network. Where as a good scheme, or one that it's hoped will have a wide take up, should leave these architecture questions open. Otherwise there's a danger SignalK is going to end up tied to the implementation found here and go no further. My interest is in using SignalK end-2-end from sensor to display via processing. Then from the boat to a nework of boats and to the internet. With a large number of derived values being generated in the chain, some of which go back in the opposit dirrection. With the right SignalK schema I'd be quite happy to translate to SignalK at the device where it's not implemented natively. I'd like to see a single data structure to represent all the data available from a single location-device i.e. a single data structure that can be used to pass data from a device to intra location processing AND once processed out to the outside world. As start point the sensor data object should have the same structure as the vessel object, rather than the vessel object (which may not be my frame of reference for makign a query) containing all sensor values. Intelegent multiplexing (with an explaination of how it was done and the alternative sources that can be seperatly RESTed if wanted) of the data enables me to expose my knowlage of the local system set-up in the data. The way you're taking this the use of SignalK from the device is prety unatractive and if I have to pay for N2K from device to multiplexer then I might as well stick with it e2e. Time stamp For me the timestamp should be abstracted to the sensor object with all the values coming from that sensior referencing the sensor (which I'd do in the manifest as the mapping is prety static). That way all the data can be left in the vessel object rather than being seperated out into different sensor payloads that I have to process back together again. Timestamping for delta data belongs in the packet header no? |
|
@pod909, I can't really make out what you are saying. On an abstract level, and on a detailed technical level. Maybe it's just me though. |
|
@pod909 Maybe a real life practical example would help clarify your viewpoint? As for "a menu of possable readings for each value is going to be confusing at best": on my boat the menu would contain items like
Often there is only one source for each data item. All this can be gleaned from the delta messages in a minute or so or fetched from the server when the display starts. I fail to see that list as confusing to pretty much anybody with a little navigation background. |
|
no problem with that list if the data was at that level but the there's alread value, source, timestamp below that level and the propsal is to add multiple option values as well isn't it? |
|
@pod909 The user interface does not need to show all the information that it has. For example https://github.com/SignalK/instrumentpanel/ populates the grid as it receives new data items, creating cells when it sees new paths it hasn't seen before. Even if the information it receives has timestamp and source it only shows the name of data item, for exaple navigation.courseOverGround, and the value. Please take also a look at Delta messages at http://signalk.org/dev/messageFormat.html This addition to the spec is about having multiple values for a given data item, say navigation.courseOverGround, in the REST data model. Current delta message format already allows updates to one data item from multiple sources, for example from multiple GPS's. |
|
The handling of multiple values is superceeded by the method detailed in https://github.com/SignalK/specification/wiki/Multiple-Values-Handling, so closing this one |
|
Adding multiple values as described at https://github.com/SignalK/specification/wiki/Multiple-Values-Handling doesn’t seem to work: here’s a section: The trouble is the value 102.29 in the “source” isn’t linked to the key “courseOverGroundTrue”. If that source also provided us with other values, I’d have no way of distinguishing them. I presume this is a solved problem and that page just hasn’t been updated (and if by chance it’s not, changing the “value” in the source to a tree of values (eg : (this is an edited repost of my comments on Slack here, for reference) |
|
I noticed the same thing a few days ago when I resurrected my work on this. The problem isn't really solved, but imho the solution presented at the beginning of this issue is the best if not perfect option and I'm going to move in that direction in the node ref implementation. The wiki needs updating, but I would like to verify the thinking with some working software before a real update. I'll throw in a label that the wiki version is not normative and has problems. |
|
Just a thought; we could approach this differently:
Example: "01010101": {
"navigation": {
"courseOverGroundTrue": {
"timestamp": "2015-08-25T08:12:17.824Z",
"value": 102.29,
"source": {
"label": "aft compass",
"type": "compass",
"id": "sensor1"
},
"alternatives": {
"sensor2": {
"timestamp": "2015-08-25T08:12:17.824Z",
"value": 100.11,
"source": {
"label": "center compass",
"type": "compass",
"id": "sensor2"
}
},
"sensor2": {
"timestamp": "2015-08-25T08:12:17.824Z",
"value": 101.2,
"source": {
"label": "center compass",
"type": "compass",
"id": "sensor2"
}
}
}
}
}
}A consumer could choose to simply use the primary value, or present a list of options to the user to switch to another value. The second |
|
JavaScript maps are unordered :-) But of course if you needed to order them you could add a priority field, or you could leave them unprioritised - realistically determining priority beyond the primary might not be much use: I have 5 GPS sources on my boat, I know which one I want to use, if that's not working, well I suppose it's whichever one has a signal. So long as it's consistent. Personally, I quite like having the alternatives bundled off in a separate section of the datamodel. There are two reasons for this.
I can think of three broad ways to handle this from a datamodel point of view, I'll sketch them out here. All the models below present the same information - course and speed from two different local sources (I'm assuming that sources for data can come from other vessels - kind of the point of SignalK after all - so sources are under Option 1First, the "primary values only stored in the main datamodel, alternatives kept with their sources" option. This is the one I'd advocate. It's also only a slight variation on the model described on the wiki. {
"navigation": {
"courseOverGround": {
"value": 109.2,
"timestamp": "2015-08-25T08:12:17.824Z",
"source": "vessels.self.dev1"
},
"speedOverGround": {
"value": 4.2,
"timestamp": "2015-08-25T08:12:17.824Z",
"source": "vessels.self.dev1"
}
},
"sources": {
"vessels": {
"self": {
"dev1": {
"label": "Masthead GPS",
"timestamp": "2015-08-25T08:12:17.824Z",
"bus": "n2k:/dev/actisense",
"values": {
"navigation": {
"courseOverGround": 109.2,
"speedOverGround": 4.2
}
}
},
"dev2": {
"label": "VHF Internal GPS",
"timestamp": "2015-08-25T08:12:17.824Z",
"bus": "nmea1083:/dev/ttyUSB4",
"values": {
"navigation": {
"courseOverGround": 108.7,
"speedOverGround": 4.3
}
}
}
}
}
}
}Advantages: primary data (eg navigation, environment) is kept concise (which should simplify subscribing to just the values and not the alternatives). It's easy to determine all data from a particular source should you wish to. Disadvantages, there's no easy way to reflect two updates from an individual source with different timestamps (e.g if GPS position was updated once a second, but course/speed only every five seconds). I'm unsure if this matters. Option 2The model describe by fabdroi above, where the options are included inline with the primary datamodel. I've added a {
"navigation": {
"courseOverGround": {
"value": 109.2,
"sources": {
"vessels": {
"self": {
"dev1": {
"label": "Masthead GPS",
"timestamp": "2015-08-25T08:12:17.824Z",
"bus": "n2k:/dev/actisense",
"value": 109.2,
"primary": true
},
"dev2": {
"label": "VHF Internal GPS",
"timestamp": "2015-08-25T08:12:17.824Z",
"bus": "nmea0183:/dev/ttyUSB4",
"value": 108.7
}
}
}
}
},
"speedOverGround": {
"value": 4.2,
"options": {
"vessels": {
"self": {
"dev1": {
"label": "Masthead GPS",
"timestamp": "2015-08-25T08:12:17.824Z",
"bus": "n2k:/dev/actisense",
"value": 4.2,
"primary": true
},
"dev2": {
"label": "VHF Internal GPS",
"timestamp": "2015-08-25T08:12:17.824Z",
"bus": "nmea0183:/dev/ttyUSB4",
"value": 4.3
}
}
}
}
}
}
}Advantages: there is no separate Option 3A halfway-house, where the option values are stored in the primary datamodel, but the details on the various sources are stored elsewhere: {
"navigation": {
"courseOverGround": {
"value": 109.2,
"source": "vessels.self.dev1",
"timestamp": "2015-08-25T08:12:17.824Z",
"options": {
"vessels": {
"self": {
"dev1": {
"value": 109.2,
"timestamp": "2015-08-25T08:12:17.824Z"
},
"dev2": {
"value": 108.7,
"timestamp": "2015-08-25T08:12:17.824Z"
}
}
}
}
},
"speedOverGround": {
"value": 4.2,
"source": "vessels.self.dev1",
"options": {
"vessels": {
"self": {
"dev1": {
"value": 4.2,
"timestamp": "2015-08-25T08:12:17.824Z"
},
"dev2": {
"value": 4.3,
"timestamp": "2015-08-25T08:12:17.824Z"
}
}
}
}
}
},
"sources": {
"vessels": {
"self": {
"dev1": {
"label": "Masthead GPS",
"timestamp": "2015-08-25T08:12:17.824Z",
"bus": "n2k:/dev/actisense"
},
"dev2": {
"label": "VHF Internal GPS",
"timestamp": "2015-08-25T08:12:17.824Z",
"bus": "nmea1083:/dev/ttyUSB4"
}
}
}
}
}Advantages: data is not repeated, all alternative values are in the main section of the datamodel. and one source can set data values with different timestamps. Disadvantages: not as brief as option 1, syntax for subscribing to just the primary values is not obvious, and you still can't identify all values from a particular source. |
|
Thanks for the input. You raised an issue at least I haven't thought about before: there is currently no way to subscribe to values from a specific source. |
|
Imho the values should be available logically near each other. In your first example: how you do discover the alternative values for a data item without going through all the sources - and I assume that the sources would have all the sources on the vessel, so you would essentially have a list of all the values in a rather awkward structure. In option 2/3 I don't get the point for the The sources section in 3 is not very informative - what does the timestamp stand for there, since there is no value present? Last update - but what update? |
|
I think we should also think about this as a REST api and the urls.
Both responses should point to other resources, like sensors and say their related calibration values, if it is not logical or convenient to represent that data as part of this resource. |
|
|
The assumption that there is always one primary value for each data item is not valid. A Signal K consuming client, like some kind of electronic dashboard, must be able to provide a list of the alternative values. After that it may know exactly what it wants, but discovery and exploration of the available items should be easy. For example in a case of malfunction (for example a stuck log wheel) you may want to switch to another source. Imho data from another entity should not be present under vessels.self. We need to support the use case of a gauge that wants to subscribe to updates from the port hull sounder. See http://signalk.org/developers/subscription_protocol.html |
|
If we drop the assumption that there always is a primary data source, we could simply drop the idea of having to specify a "primary" or a priority. That would make the schema simpler: 1. one sensor, one value, indicated by using {
"navigation": {
"speedThroughWater": {
"timestamp": "2015-08-25T08:12:17.824Z",
"value": 9.5,
"source": {
"label": "Airmar DST-800",
"uuid": "22E63B84-749D-489B-A872-8B0D62507C54",
"type": "NMEA0183",
"capabilities": ["DBT", "DPT", "VHW", "VLW", "MTW"]
}
}
}
}2. Multiple sensors, multiple values. Consumers are responsible for choosing one, either by UI or (if a simpler consumer) simply selecting the first one: {
"environmental": {
"depth": {
"belowTransducer": {
"values": {
"starboard-hull": {
"timestamp": "2015-08-25T08:12:17.824Z",
"value": 9.5,
"source": {
"label": "Starboard DST-800",
"uuid": "9801FBF0-76A3-47DD-ACA6-D76DC405F95C",
"type": "NMEA2000",
"capabilities": [59392, 600928, 126208, 126464, 126464, 126996, 128259, 128267, 128275, 130310, 130311, 130312]
}
},
"port-hull": {
"timestamp": "2015-08-25T08:12:17.824Z",
"value": 6.5,
"source": {
"label": "Port DST-800",
"uuid": "C7B2F2B7-2D49-475A-B0A1-F1E7E840607D",
"type": "NMEA0183",
"capabilities": ["DBT", "DPT", "VHW", "VLW", "MTW"]
}
}
}
}
}
}
}3. One could even get rid of the two different notations and simply have a {
"environmental": {
"depth": {
"belowTransducer": {
"values": {
"dst800": {
"timestamp": "2015-08-25T08:12:17.824Z",
"value": 9.5,
"source": {
"label": "Starboard DST-800",
"uuid": "9801FBF0-76A3-47DD-ACA6-D76DC405F95C",
"type": "NMEA2000",
"capabilities": [59392, 600928, 126208, 126464, 126464, 126996, 128259, 128267, 128275, 130310, 130311, 130312]
}
}
}
}
}
}
}In case of option 2/3 and subscriptions, if a user subscribes to {
"values": {
"starboard-hull": {
"timestamp": "2015-08-25T08:12:17.824Z",
"value": 9.5,
"source": {
"label": "Airmar DST-800",
"uuid": "22E63B84-749D-489B-A872-8B0D62507C54",
"type": "NMEA0183",
"capabilities": ["DBT", "DPT", "VHW", "VLW", "MTW"]
}
},
"port-hull": {
"timestamp": "2015-08-25T08:12:17.824Z",
"value": 6.5,
"source": {
"label": "Airmar DST-800",
"uuid": "22E63B84-749D-489B-A872-8B0D62507C54",
"type": "NMEA0183",
"capabilities": ["DBT", "DPT", "VHW", "VLW", "MTW"]
}
}
}
}I did include timestamps everywhere as a consumer could use that information for determining what value to use. For instance, a research vessel might have many identical sensors, each transmitting at 1hz, running a consumer that logs this into a database. The consumers will simply pick the latest value in order to get more data points than once a second. |
|
SInce a single source (like GPS RMC message) can have several diverse values, we should support the idea of the That also means we can store config and lots of misc data like version, manufacturer etc without polluting the model. And its still easily available when needed. In practice I think that multiple values will be the minority, but still important. So I like the idea of a simple For the multple values I think we can then use
|
|
|
A few thoughts:
|
Identifiers for
|
|
SI this level of source information really necessary for a v1 spec? On 14 November 2015 at 07:21, Tim Mathews notifications@github.com wrote:
|
|
Two separate issues to pgn/sentence:
Imho it needs to go into the model, distinguishing is debatable. |
|
I think the model should support also passive capture playback eg situation where we have n2k query capability, just a dump of n2k traffic. |
|
Id scheme must support also non-nmea sources. |
|
Imho sentence/pgn should be there also for debugging purposes. When I see the value I want to be able to trace back where it came from. |
|
For N2K and NMEA0183 we need to support multiple inputs/buses. In the current node server this is generated from configuration file, eg. it is user configurable per source. Non-serial line NMEA0183 sources exits, for example udp and tcp, so referring to the input as We must support multiple N2K buses and NMEA0183 inputs and non-nmea sources. Current schema has |
|
This is what node server (multiple-values branch) outputs now with simulated multiple sources, eg. the same n2k and nmea0183 files configured twice: |
|
Talker id is used in NMEA0183 multiplexers to identify the originating device / system, see for example http://www.navstore.com/1120-shipmodul-miniplex-2s-advance-nmea-0183-multiplexer.html and should be included in the identifier. Nmea0183 parse doesn't yet support it: SignalK/nmea0183-signalk#37 |
|
Reading the manual for the Actisense device, it sounds like they handle this differently. It sounds like they inject a proprietary sentence prior to the multiplexed sentence to identify the source. Maybe Phil could donate one so we can make sure it works correctly? 😄 It sounds to me like changing the talker ID the way the ShipModul device does, i.e. GP -> G1, G2, etc, technically breaks the NMEA 0183 spec since there is a defined list of standard IDs. We need to support it, but we also need to support what Actisense is doing. What other 0183 multiplexers are out there? |
|
1 x 8-port USB hub and 8 x FTDI RS232->USB boards :-) And it's a bit cheaper than that Actisense box too. Personally, I think so long as the URN is unique, I can't visualise many situations where it's going to be broken down and parsed. I kind of agree with pod909 here - while you obviously need a URN for a sensor, it should be designed with more than just this issue in mind and this thread might not be the best place for it. Having said that, I can't resist :-) First, I'd suggest that trying to cater for the specific Actisense box is the wrong approach, as you can guarantee if you design their method into the URL, you'll find something that takes a different one tomorrow. HTTP URLs have the format protocol:host/path?options, which covers everything you need here. Some hypothetical examples if we followed that sort of model:
Explanations:
I'm not wedded to the exact syntax, but if this is going to extend I think you need to allow for: Final point for this post. I'd question the need for anything more than the URN for the device in the values list above: for example: Why do we have the "source" map here? All the information we need is in the URN already - this is a very verbose way of saying the same thing twice. And even if the URN doesn't contain everything there is to know about the device, I'm not sure all this static, unchanging information should be broadcast with every property that the device sets. A GPS could set lat/lon/vmg/cog/sog plus umpteen properties about satellites. Here, the source map above would be included with each property, vastly inflating the size of the model. If additional information about a sensor is required, it could be placed in a supplementary map and keyed on the same URL. I would propose instead: Note I have dropped the "label" concept here. I think it's redundant - the URN is unique by definition, if a label is to be added it could be done as part of that, eg n2k:///160/128260?label=Masthead+Sensor, and the UI could look for that value if required. The two main disadvantages to this that I see are:
(edit: Sorry, that one was longer than I expected it to be when I started writing it...) |
|
Excellent suggestion faceless On 14 November 2015 at 22:31, faceless2 notifications@github.com wrote:
|
|
I like this proposal and it is inline with what we have been discussing internally during the iKommunicate development. I think n183 is the most commonly used abbreviated version of NMEA0183. Well done faceless |
|
Ah, very good. I like this idea. Apart from the source format being much less verbose than the current map, to me the major improvement is the fact that it is so much more flexible. Re labels and other meta data: if a source needs this, I'd argue for an extended object with the same name under |
|
See: #84 |
|
Yes to fabdroi's idea on the sources! And yes for n183 :-) |
|
The verbose Remember that the keys in JSON objects act as paths in the REST API urls, so stuff we use as property keys should make sense there as well. |
|
I have now node code that will generate a structure like https://gist.github.com/tkurki/84cd71d2329f3ffae9f0. There is now The value data has just references to the sources. The references are paths starting at the With this structure the data is referencable like and access metadata about that source with Here the only thing really coming from manually entered configuration is the identity of the different inputs (the two nmea0183 and n2k inputs). For a software-only server like node server this makes sense and for a hardware server this would be static data derived from the hardware configuration. |
|
There are two things in Teppos solution: The underlying concept here is that However we can combine this with the embedded model, as we have discussed before. We just need simple rules for
That means we can send data outside the server with either the reference to source (leaving the client to retrieve it), or embedded. Hence the source should follow some format that allows it to be externally resolved and retrieved- doesnt matter what, so long as its consistent. We should formalise the 2) - its a generally useful feature. |
|
I prefer the sentence (or PGN) to be sent with the value, as the source in most cases will be sending out multiple sentences or PGNs. This information is not critical but nice to have for traceability. |
|
Cant find the original idea, but its discussed above, near the top. For clarity what I mean is that this 'referenced' version is funtionally eqivalent to this 'embedded' version But with the referenced version several keys can point to the same source (eg n0183 RMC source for efficiency, and if I send out the value, I might not include the source object, just the reference, making a smaller and cleaner message |
|
A bit of clarification: most of this multiple values/sources is about full tree and REST api. The delta stream has no trouble representing multiple values for the same Signal K key, as each individual message is independent. Not so with the full tree: so far it has been able to hold just one value per key. The aim here is to change the schema of the full tree so that we can manage multiple values for a single key. The reason that this issue has delved into the sources issue is that we need persistent keys in the |
|
@rob42 from your examples I see that we intrerpret things a bit differently. I've used the current This way The node server uses the ids from the configuration file as input labels. Label, type, talker and src are duplicated as keys and properties to make them easier to use from the properties (if you access just ../source/nmea1 you get an object with label property). |
|
If the name I agree that human readable information like "Starboard DST-800" belongs under the Furthermore I would like to concentrate on the multiple values issue here and not source/sensor details, as I feel they are bit different things. |
|
I think values and sources are so intrinsically linked that to try and consider them separately is not the right approach. We keep using depth or GPS as examples but in reality there are hundreds of examples on a large boat installation. Maybe we should think of a source as a "thing" which creates values "data". |
|
Fixed by #118 |
It is quite possible for a key value to come from more than one device. eg position (lat/lon) could come from several gps enabled devices, and multiple depth sounders are not uncommon. We need a consistent way to handle this.
All the incoming values may well be valid in their own context, and it is feasible that all of them may be wanted, for instance, dispalying depth under each hull on a catamaran.
Hence discarding or averaging is not a solution, and since signalk is unable to derive the best way to handle multiple values it must always fall to a default action, with human over-ride when needed.
I propose we should simply store all the options in the tree, and have the main 'source' reference the options?
Then simple rules can apply:
The text was updated successfully, but these errors were encountered: