-
Notifications
You must be signed in to change notification settings - Fork 68
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Magnetic Deviation #242
Comments
Deviation should be stored wherever we define static properties of a sensor. Its naturally a table of heading ranges and deviation angles. It should probably be added to the What is the NMEA convention here? In the HDG sentence, if deviation is present should it be added to the heading value or is purely informational and assumed to be already accounted for by the instrument? A deviation table schema might look like:
|
@timmathews further to our discussions last night, I think we should have a dynamic deviation value in Signal K that we take from the NMEA 0183 HDG sentence and NMEA2000 PGN 127250 if present that is the current deviation value that is being used by the heading system. I am not sure how many systems output this data but even if they do not, if we have deviation defined in Signal K then sensors can use it or if we also support a Deviation table, a Signal K server or client could interpolate the current deviation and populate the current deviation key. The deviation table should be a table with deviation values every 15degs, so 24 values which then allows for a very accurate table with all values entered or a less accurate table with values for every 30degs or even 45degs. |
A server could also add value by working out magnetic variation using this code from NOAA... |
This has more to do with correcting data than the real value of the heading or any other key. This same adjustment is made by various sensors internally (calibration) and needs to be applied to some data by the server, eg wind vane offset. There was discussion on this recently (slack?) and I proposed a 'correctIncomingSensorData' functionality, but I think its an implementation choice. Not sure we can have a std format for the data we might need to store. Definitely a bad idea to start sending data plus corrections (that the receiver must apply anyway). Easiest is to add "local adjustments to correct sensor data MUST(SHOULD?) be applied before data is updated in signalk. How this is done is implementation specific" for v1 Then we can do more after v1? |
Changed to after v1. My point in this is that if we want this as part of the spec it needs to be in the data model, and I agree with Tim's point about this being part of the sensor. Server correcting the incoming heading with this is beyond SK spec and an implementation detail. |
Agree and I am dubious as to how many heading sensors actually output this data. They will send deviation corrected values out. |
Probably still a good idea to be clear about it even for V1 |
During some work on iKommunicate, it became clear that we never did support magnetic deviation in the SIgnal K V1 spec. Think at the very least we should have the following key so that we have somewhere to store NMEA0183 and NMEA2000 Deviation values... /vessels//navigation/magneticDeviation Description: The magnetic deviation at the current heading that must be added to the magnetic heading to correct it for local magnetic deviation. Easterly deviations are positive and Westerly deviations are negative (in Radians). |
"Data in signalk is assumed to already be corrected for any known sensor adjustment" . The gateway or sensor should apply the correction before sending the data. The actual deviation varies with heading, its traditional to have a 'card' of values around 360deg so you can adjust any given heading for the correct amount. Its much simpler to fix this at the sensor or gateway than propagating the error around the network and having every subscriber compensate. |
I agree @rob42 and if we were only dealing with new SIgnal K sensors then this would be fine, but we have to work with legacy, current and future 3rd party NMEA0183 and NMEA2000 based products and these output Deviation and at the moment there is nowhere to put this data in Signal K, so that is why I am proposing having a key for deviation and then developers can decide how they want to use it - No Key, No Option ;-) |
Even if data is adjusted so that deviation is not needed a user may want to display it, so having it in the schema is useful. @sumps I think Rob's comment was more about your description of the value: _ that must be added to the magnetic heading to correct it_. This is not in line with "all values being corrected beforehand" thinking. Nor is local appropriate here, or talk about world magnetic model, as they relate to magnetic variation/declination - the diff between true & magnetic north in a certain location. As I see it there are
(1) could be navigation/magneticDeviation/current and then the source is the key to associate that with the navigation/headingMagnetic that it is "current" for. (2) could be a structure such as Tim's above, only where should that go? navigation/magneticDeviation/deviceId/ (where deviceId would be $sourceId?) |
I don't think it is correct for developers and users to make assumptions about the data and whether it has been adjusted for variation and/or deviation. Basically NMEA devices that transmit heading send the raw measured magnetic heading and if known the variation and deviation values for the "local" situation i.e. the local variation taken from the electronic chart or calculated from the world variation model and the local deviation table for that vessel. Often a compass sensor will have been calibrated to remove the local deviation effects in which case its heading will include deviation and it will either output 0 deviation or "Null" data not available. The heading, variation and deviation always have to be linked to the source of data, just like the Position, Datum, Satellite Status, etc. from a GPS but this is handled by the source metadata. Variation can also change, albeit more slowly than deviation, so I think having some abstract concept of a current key is over complicating things. Basically the majority of heading sensors with be communicating by NMEA and I just want a Signal K key in the schema to put this data in. |
I never saw a magnetic compass with the ability to add the vessels deviation to get a deviation corrected HDM as output. |
I don't understand this concept of "current" by its very nature the deviation value will be the one applicable to the heading the compass is outputting. You need another "static" deviation table stored in another part of the schema applicable to a particular compass sensor which an app can load at start up and apply to any values it receives from the applicable compass sensor. As a general principle, I think we should always have a Signal K key for all NMEA data so that there is clear and accurate conversion of the data, with no further ambiguity or introduction of errors. Then if we think there is benefit in improving or expanding on that data, we extend the schema, create new derived values, implement new ways of doing things, etc. |
Just for clarity here - I was talking about compass deviation as compass inaccuracy eg most vessels (especially steel) have local magnetic fields that make the compass read wrongly in various directions. Then we have the external magnetic deviation due to the earths magnetic poles not being aligned with geographic poles. We have Which magnetic deviation are we talking about here? @inspirity true, most current compasses dont, but in signalk we should fix sensor errors at the edge of the signalk network, not propagate the error info too. Incidentally thats different from Early on we went to SI units throughout signalk, which dropped the need to send |
Magentic Variation or Declination is not the same than Magnetic Deviation. I need the deviation AND the variation to navigate without GPS. So if my electronic compass fails and my gps fails i could still have an app calculating my compass course to navigate to the next waypoint. Without the deviation data i simply can't. |
This theoretically perfect world where we have Signal K sensors that output perfect data is great but we still need to work with imperfect NMEA based sensors and displays and if we wish to receive and transmit imperfect data then we need to have keys to store them in. |
Any data can be wrong. Thats why eyeballs and brains are still the primary navigation instrument :-) But when we know errors we should correct then as near to source as possible. That said I understand better what @inspirity is after now. If you have a compass that simply sends its heading and has no capacity for holding/correcting for deviation, then storing it and applying it elsewhere makes sense. I was thinking of @sumps example of an N2K packet that sent both heading and deviation, that should be corrected in the gateway. There was some discussion elsewhere about applying corrections to signalk data but I cant find it. The general idea was that incoming data would be parsed for keys that were known to need correction and the corrections applied. It was a server implementation issue. I think we need a generic way to add a correction factor into any signalk key. the The process would need to differentiate by source, and possibly a method to ensure it didnt correct the same value twice (eg on echo, or mirrored servers). Not simple. |
@rob42 i don't see deviation as an error but similar to the polars as a data table connected to the boat. |
If i want to build a display that shows the current deviation next to my
analog compass I want that in the sk data model.
If I want show a deviation graph as in the previous post I want that in the
sk data model.
Neither of these have anything to do with whether or not heading* values
are corrected. I think that is a separete issue.
pe 14.12.2018 klo 23.25 Olivier Hendrikx <notifications@github.com>
kirjoitti:
… @rob42 <https://github.com/rob42> i don't see deviation as an error but
similar to the polars as a data table connected to the boat.
Calibrations in meta.correction for a deviation table would have to be a
function since the deviation depends from the headingMagnetic.
We already have the headingTrue Key in the definitions wich should contain
the Variation AND the deviation.
The Deviation is usually a table. Question is do we store this data in SK
or do we keep this external.
Just to clarify what we are talking about:
[image: deviationtable]
<https://user-images.githubusercontent.com/6239660/50028164-d2199480-ff92-11e8-9a4f-9ab7455aa8c6.jpg>
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#242 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABAETnWch3-x2zecg5AnkLTTinRmmMr3ks5u5Bc3gaJpZM4JsY5A>
.
|
I support tkurki's point of view. Maybe we should just clarify in the key's Description:
|
I also sense from this thread that no one knows whether any systems populate the deviation fields in NMEA 0183 and NMEA 2000 so having a key that iKommunicate, CANBoat, etc can populate will give us, for the first time, the visibility to start understanding what data is available and from which systems. |
@sumps |
The more we describe these marine values and terminologies, the more value we add to the schema and work that everyone is doing. HeadingTrue used to be only generated by expensive gyro systems on commercial vessels and now with GPS Compass and MEMs sensors it will become a much more common data type. In these circumstances having a Signal K Server that can efficiently and effectively manage the steering compasses deviation table is a desirable goal. |
@rob42 I understand absolutely what you talking about. This also means that most of the sources on boats wouldn't be able to deliver a corrected headingMagnetic value for SK. I'm navigating and sailing since more than 40 years and since 10 years full time around the globe. I never met a boat with a deviation calibrated electronic compass. I guess i have to investigate and get one. This is no big issue, we could define headingMagnetic as the 'error free' value and let the user or source responsable to get the correct value. This is the actual situation on most boats anyway since most captains use gps's and don't care about deviation. SK doesn't need to do any special calculation. It is just a definition aspect what headingMagnetic in SK means, with or without deviation correction. That all a developer needs to handle it from there. Sk is not responsable for the quality of the sources, as long the definitions are clear. Till now i didn't know which value headingMagnetic expected. Do I understand your proposition is:
I would totally agree with this solution. |
In an ideal system you would make sure that all data was corrected/adjusted at source but I fear that on your average boat this is not the case or at least you have no way of knowing if this is the case or not. This is where a Signal K Server could add value providing extra functionality, beyond what your average MFD or Instrument system does, to apply corrections to the received data or derive new values from the data before passing it on to other SK devices/apps. It would be worth considering having a metadata field that says if the value is raw, calibrated or derived and even perhaps a quality rating so that different sources can be rated by the user/installer. I am a bit worried about this comment you made... “I also added magneticVariation to signalk, but on consensus it was changed to magneticDeclination, so I used that term now.“ I can find no Issue or PR for this change from navigation.Variation to navigationDerivation and this would break V1 backward compatibility. Can you clarify what you meant? |
@sumps you are quite right the spec has |
@inspirity - exactly, We can never have error free sensors ( https://en.wikipedia.org/wiki/Observer_effect_(physics) ), and never really know how precise they are. But if we have a compass we know is out 4deg at E, then we should fix that value in signalk (if we can). From a consumers point of view we know that the value may not be correct, but we also know its the best we will get. @sumps additional metadata on data quality is an interesting idea but I fear it will create a quagmire of quality definitions and estimation methods. In the end you get a data value (as a consumer) and you choose to trust it or not. If I wanted to fake data, then I would also fake the quality flag, so as a consumer you are no better off. Something to leave for V2+ So are we in agreement:
Rob |
So how does a Signal K Compass Sensor communicate its data to the server, would we have to define headingMagneticRaw ? I don't quite see how the fake data argument has made its way in to this issue. This is a subject all of its own and I assume is being tackled under Signal K Security. Having a field that says whether data is Raw, Calibrated, Derived, etc. seems to make a lot of sense to me, allowing you to have one key and then a standard way for any data key to be categorised. It does not break V1 and could be added tomorrow as V1.x Perhaps the use of a data Quality flag is a step too far, put a Priority flag in metadata could be used across all Data types as a way for the user/installer to select what they know to be the best or most reliable data of that type. The Priority field could also be auto-populated in NMEA2000 systems based on the CAN address - lowest being highest priority. Again a Priority flag could be added without breaking V1. |
@sumps without need to dig into metadata, flags etc. It is possible to add a key headingMagenticRaw which would hold a value without Deviation and Variation. We would have all the possibilities covered. There are only Deviation and Variation between Magnetic and True. everything else is an sensor error and as @rob42 explained (even if rob42 sees the deviation as an error), this should be corrected at the source or where ever. Once we have the definitions set we can figure out if, where and how the correction are made. This would be a different thread and applicable to all the errors and corrections. The same applies to priority lists - defining which source is the best and in which order. |
Fake data was only demonstrating that any sort of data quality measurement provided by the server is just as suspect as the data, so of dubious value. Raw, Calibrated, Derived, etc. could be added but I dont see the use case for it. We already have a definition of the 'default' data value, its |
My suggestion of headingMagneticRaw was not meant to be taken literally, as there are many other data sets that have raw and calibrated data and I do not want to increase the number of keys in this way. The key thing here is that there is no way to automatically tell from an NMEA compass sensor whether it is calibrated for deviation, whether it is a derived value say from a GPS Compass and calculated variation or any other combination. We are dealing with an imperfect situation, as you and Rob have both already stated/agreed and so I would stick to the keys that we can populate automatically and accept/document that they may not be accurate and then consider a way through metadata to allow users/installers to categorise and prioritise the data.
|
@sumps I see. The question is also how to implement a correction/calibration handling in SK. Metadata is one possibility but since some corrections like deviation are not static but changing with the heading I dont see how to do this in the metadata object. I mention once in slack about having a delta value and a factor value in the source meta data to be able to do some basic corrections but as @rob42 illustrated this could become a nightmare to handle on a installation with several servers. We would need something like a values preprocessor once at the Input of the SK Server (similar to the InputHandler in the NodeRed plugin) and forget about errors in the data tree. We just need to know where which data goes. This could be an external addition and would keep any SK Server clean. @sumps the problem with the quality of the data source is not only compass related and should be handled in a general approach. |
@inspirity the value of deviation is data that is why I am proposing it has a key navigation.headingDeviation. The value of magnetic heading is also data and it will have a key navigation.headingMagnetic but it could potentially also have some metatdata fields to tell whether the value is raw, calibrated, derived, etc. and what priority it has in the system. I already said that these metadata fields could be applied to any data object and that they could be introduced without breaking V1. I think we need more opinions on this, otherwise we will start to repeat ourselves and never agree a solution. |
@sumps I understand your point. This would imply that I would have to check the metadata of every path or key to know with what kind of data I deal with before I can use it. Why then not just have heading as path and in the meta data define if it's true, magnetic raw, high priority etc? Where should we draw the line? I'm just trying to understand the practical advantages of this kind of metadata. We could end up with a headingMagnetic value the same than headingTrue just becaus they would both be raw uncalibraded etc. One thing is to know the quality of the data. To start calculate corrections and calculating source choices is a completely different decision. So I agree to have the information about the quality like raw, uncalibraded, reliable or what ever... If this implies calculations then we could end up where @rob42 warned us. I'm not arguing this is a very important matter and I'm thankful for the dialogue. |
From my theory days, once you have corrected magnetic heading for variance and deviation, you have true heading. Same way a gyro heading is corrected to get true heading. Some sensors or systems push out a current deviation, but I would not trust these, as I do not understand how they are made or trust the auto creation of internal deviation tables without a true reference. The prudent approach is to add a deviation table. But I don't understand the need for such a long discussion to decide adding the deviation path. |
You have compass heading, magnetic heading and true heading. Auto deviation tables are to correct compass heading to magnetic heading. You still need a |
You are right, except that the earth's magnetic field correction is called variation. The NMEA sources that have been used in defining SK paths have used magnetic heading instead of compass heading. It is basically an uncorrected magnetic heading using a compass as source. See http://www.catb.org/gpsd/NMEA.html#_hdg_heading_deviation_amp_variation |
Anyone interested in this subject should read this article... https://en.wikipedia.org/wiki/Magnetic_declination Aside from the fact that they use the term declination more than variation (they are the same), it covers the subject pretty well. The reason for why Signal K are using variation, is that this term is more commonly used in the marine world and throughout the NMEA specifications. It is important that we are all "singing from the same hymn sheet" if we are to resolve this issue. |
Yes sorry, I meant variation :) |
@JonRowe you have nicely identified an important distinction. In signalk |
Adding my opinion to this very interesting discussion ;) First looking at the standards and what we do today, then trying to pick what I like best in the thread and summarize what I would like to see and finally taking a look at my equipment on my boat. 1 - Standards and code we have todayAccording to catb.org, NMEA We currently ignore deviation in the NMEA parser and in the NMEA2000 parser. This seems to be a bug :
Also note that the NMEA2000 conversion code already sends 2 - Picking what I like best from the thread@sumps said (and I agree):
I support the idea of adding So when using So to be clear we would have:
In terms of code change:
And I also agree that the table discussion can be held separately, in a different PR. 3 - Data from my boatExtra info: I just checked my logs and my B&G ZG100 GPS+Heading sensor does not include deviation:
Which via canboatjs is: |
I agree with your methodology @sarfata but I would change navigation.headingCompass to navigation.headingRaw to make it clearer that the heading is "raw" with no variation and deviation. Perhaps we can apply this "raw" naming convention to other values in the future as more SK sensors are developed and we need a method of transmitting raw and calibrated values. |
|
@JonRowe no I think Current NMEA conversions should be corrected to match this. Lets close on that, before my head |
The value the compass sending out is usually already corrected for deviation. |
In that case (and most cases) the magneticDeviation will be unknown, hence 0. So |
So, are we going to add "Raw" for every single key in the schema? Is this the only key that might need adjustment? (like a wind vane that's not right?) I'm not convinced that this is the correct solution. |
See my comment in the PR. |
I prefer Since Lets call it |
The original issue was about deviation and where to put it and we do seem to have been dragged off in another direction. I actually think headingCompass is worse and more confusing than headingRaw and I would prefer to close this issue with the creation of a new key magneticDeviation and leave it at that. I think we should stick with headingMagnetic and headingTrue which are clear data types and then handle the nature of the data in the metadata i.e. Raw, Damped, Calibrated, etc. |
Dragging this one up from the past... From what I can see, the schema was updated to add magneticDeviation and headingCompass, but the code was never updated to populate them correctly. If I send the following sentence navigation.magneticDeviation is not set (should be -5) Incidentally, the NMEA spec clearly states that the first value in the HDG sentence is the magnetic sensor reading before any deviation correction is applied. |
For example HDG NMEA 0183 has heading, I believe we are missing it.
A bit tricky, as it is associated with the sensor and current heading, so where should it go?
The text was updated successfully, but these errors were encountered: