Meteorological and hydrographic data from AIS VDM binary message 8

[KEGustafsson]

Meteorological and hydrographic data from AIS VDM binary message 8 mapped to Signal K paths with contextPrefix = 'atons.' .
Resolves issue #244

!AIVDM,1,1,1,B,8>h8nkP0Glr=<hFI0D6??wvlFR06EuOwgwl?wnSwe7wvlOw?sAwwnSGmwvh0,0*17

[
  {
    "context": "atons.urn:mrn:imo:mmsi:990000846",
    "updates": [
      {
        "source": {
          "sentence": "VDM",
          "talker": "AI",
          "type": "NMEA0183"
        },
        "values": [
          {
            "path": "",
            "value": {
              "mmsi": "990000846"
            }
          },
          {
            "path": "navigation.position",
            "value": {
              "longitude": 171.5985,
              "latitude": 12.2283
            }
          },
          {
            "path": "sensors.ais.designatedAreaCode",
            "value": 1
          },
          {
            "path": "sensors.ais.functionalId",
            "value": 31
          },
          {
            "path": "environment.averageWindSpeed",
            "value": 127
          },
          {
            "path": "environment.windGust",
            "value": 127
          },
          {
            "path": "environment.windDirection",
            "value": 360
          },
          {
            "path": "environment.windGustDirection",
            "value": 360
          },
          {
            "path": "environment.airTemperature",
            "value": -1024
          },
          {
            "path": "environment.relativeHumidity",
            "value": 101
          },
          {
            "path": "environment.dewPoint",
            "value": 501
          },
          {
            "path": "environment.airPressure",
            "value": 511
          },
          {
            "path": "environment.airPressureTendency",
            "value": 3
          },
          {
            "path": "environment.horizontalVisibility",
            "value": 127
          },
          {
            "path": "environment.waterLevelTrend",
            "value": 3
          },
          {
            "path": "environment.waterLevel",
            "value": 4001
          },
          {
            "path": "environment.surfaceCurrentSpeed",
            "value": 255
          },
          {
            "path": "environment.surfaceCurrentDirection",
            "value": 360
          },
          {
            "path": "environment.significantWaveHeight",
            "value": 252
          },
          {
            "path": "environment.wavePeriod",
            "value": 63
          },
          {
            "path": "environment.waveDirection",
            "value": 360
          },
          {
            "path": "environment.swellHeight",
            "value": 255
          },
          {
            "path": "environment.swellPeriod",
            "value": 63
          },
          {
            "path": "environment.swellDirection",
            "value": 360
          },
          {
            "path": "environment.seaState",
            "value": 13
          },
          {
            "path": "environment.waterTemperature",
            "value": 501
          },
          {
            "path": "environment.precipitation",
            "value": 7
          },
          {
            "path": "environment.salinity",
            "value": 510
          },
          {
            "path": "environment.ice",
            "value": 3
          }
        ]
      }
    ]
  }
]

[KEGustafsson]

Meteo data functions are now re-checked and some fixes implemented. Should be ready for merge.

[tkurki]

@Hakansv here's the work in progress of implementing AIS meteo data in 0183-signalk.

[tkurki]

See also OpenCPN/OpenCPN#3541

[Hakansv]

Very good. Some comments:

I miss the time stamps. Needed to info about issued time.

horizontalVisibility: The most significant bit is an indication for a static value or a greater than or equal value. Do you handle that? If MSB is high we may need a tag for that?

Position and e.g. air temperature are as per 2's complement to get + or - signs. How do you handle that?

MMSI numbers:
Do you deal with the fact that many countries use an equal MMSI number, like 002655619, for all there stations? Other countries use a AtoN alike number but very often the same number as for a nearby AtoN.
In OCPN we give every meteo station a new unique mmsi-ID and separate them by the position. So if SK returns the MMSI as it is we will try to handle it in the same way as for N0183 data.

Håkan

[KEGustafsson]

Bug fixes for ggencoder of VDM msg8 are still pending release to npmjs. Otherwise fixes are merged to code.

https://github.com/fulup-bzh/GeoGate/commits/master/

Other part is is this update to VDM code. If it looks ok, then we are very close to have VDM msg8 support in SK.

[KEGustafsson]

@Hakansv

I miss the time stamps. Needed to info about issued time.

This can be added.

horizontalVisibility: The most significant bit is an indication for a static value or a greater than or equal value. Do you handle that? If MSB is high we may need a tag for that?

I'll take a look to this.

Position and e.g. air temperature are as per 2's complement to get + or - signs. How do you handle that?

Encoding done in ggencoder and it has support for msg8 now. Lat&Lon calculations have still issue and fix not released to Npmjs. Merged to code.

MMSI numbers:
Do you deal with the fact that many countries use an equal MMSI number, like 002655619, for all there stations? Other countries use a AtoN alike number but very often the same number as for a nearby AtoN.
In OCPN we give every meteo station a new unique mmsi-ID and separate them by the position. So if SK returns the MMSI as it is we will try to handle it in the same way as for N0183 data.

This part requires a bit refactoring if AtoNs reuse MMSI code. Need to add check for position too.
Could you point your code where you give a new unique MMSI ID. It would be good to reuse same style to generate the new ID. Then OCPN and SK would make similar ouput.

[Hakansv]

For the MMSI number it's not only reuse of single AtoN numbers. For Sweden about 80 stations have the same MMSI number. UK, Finland and many others use the same system.
The implementation of Meteo data in OCPN is here and here The latter is about the US standard for Ais8_Dac367_Fi33. Some small adjustments are done after that but the base encoding is there.
I'm not sure SK would handle the mmsi number? OCPN does use the mmsi series 199xxxxxx not used by any N0183 AIS messages. But that would be done locally to not interfere with other targets. My comment was more about any use of SK meteo data in other system.

[Hakansv]

Btw: The horizontal visibility ">" sign is here

[KEGustafsson]

Is OCPN decoding Msg8 DAC 1 FI 11 messages? These messages should be deprecated 10y ago, but still those seems to be in use, mainly U.S.

Msg 8 DAC 1 FI 31 are currently added to ggencoder. I'm not familiar U.S. addons. My code is based on IALA data
https://academy.iala-aism.org/asm/meteorological-hydrographic-data/

Msg8 DAC 1 FI 11
https://academy.iala-aism.org/asm/metreorological-hydrological-data-2/

[Hakansv]

Is OCPN decoding Msg8 DAC 1 FI 11 messages? These messages should be deprecated 10y ago, but still those seems to be in use, mainly U.S.

No.

Msg 8 DAC 1 FI 31 are currently added to ggencoder. I'm not familiar U.S. addons. My code is based on IALA data https://academy.iala-aism.org/asm/meteorological-hydrographic-data/

There are some US data in the air. But can be discussed later. Let's start with dac 1 fi 31.

Msg8 DAC 1 FI 11 https://academy.iala-aism.org/asm/metreorological-hydrological-data-2/

Hmm. A quick test of dac 001 Fi 11 and there are some data in the air now. According to mmsi 002300061 probably Finnish. I'll look further into this and consider implementation. It may be the same data is in dac 001 Fi 31? Not less since the same mmsi is present also there.

Thanks for the links.

[tkurki]

In principle I am not fond of making up mmsi numbers. I think on the SK side we don’t need that, as we could use uuid’s as context, the id that is used for identifying the entity that the data is for.

Another option would be to create a new ”mmsi like” context type: use some other prefix than urn:mrn:imo:mmsi to indicate that this is not a real mmsi, buy can be treated as one. Like O could do that, if the internal logic is strictly dependent on mmsi being there.

[KEGustafsson]

OCPN is using 1994+ running number. AtoN ID might then change from time to time.

Could we use geofencing for generating rest of the ID? Then AtoN could most probably have "permanent" new ID based on location. That would not change at boot or restat of the software.

[Hakansv]

OCPN use an own data base for meteo stations. Every received message is compared by it's position to detect if that station is received before. If so the message mmsi ID is reused. If not found a new mmsi-ID is assigned for the station at that position.
When comparing positions we use not more than three decimals, about 50 meters, to secure a single station can move a little, if e.g. a buoy. All stations seems to be separated more than 50 meters. No clashes seen. The created "fake" mmsi-ID is consequently not repeated between O sessions.
A unique repeatable name (ship name) is though created for each station. Meteo XXX where XXX is calculated from position. This calculation produce not more than about two identical XXX for about 100 stations. :)

See attached example. Origin and "fake" mmsi are both shown.

double id1, id2;
wxString slat = wxString::Format("%0.3f", lat_tentative);
wxString slon = wxString::Format("%0.3f", lon_tentative);
slat.ToDouble(&id1);
slon.ToDouble(&id2);
wxString nameID = "METEO ";
nameID << wxString::Format("%0.3f", abs(id1) + abs(id2)).Right(3);
strncpy(ptd->ShipName, nameID, SHIP_NAME_LEN - 1);

[KEGustafsson]

Would this work? What is probability that other AtoN has same first 3 decimals for lat & lon?
This would generate unique but repeatable ID without database.

lat = xxx.123xxx
lon = xxx.456xxx

const mmsiPre = 199;
const latPart = parseInt(("00" + (lat % 1).toFixed(3).slice(-3)).slice(-3), 10);
const lonPart = parseInt(("00" + (lon % 1).toFixed(3).slice(-3)).slice(-3), 10);
const mmsi = mmsiPre * 1000000 + latPart * 1000 + lonPart;

console.log(mmsi); // Output: 199123456

[Hakansv]

The construction would probably work as such. But not for SignalK as a distributed system.
When we internally use that mmsi-id syntax it's no risk for clashes. But if you use it global it's not safe.
A normal MMSI is unique and can be used global since it's assigned by some authority.
The local use of 199xxxxxx is safe since the 199-series is INMARSAT-A MID and should not occur ever in a received AIS stream.
So in a local OCPN we use it for different needs. 1991xxxxx is used for local ARPA targets, 1992 and 1993 for others and now 1994xxxxx for Meteo targets. In the local system no risk for a clash.
But if you use the same in a distributed system it can and probably will be local clashes.

[KEGustafsson]

If SK data is proxied and distributed outside private network, then fake ID is a big show stopper. But situation is as bad if same AtoN MMSI contains data from multiple places and that is distributed outside private networks too.

I see no good solution...

[afischerdev]

Bug fixes for ggencoder of VDM msg8 are still pending release to npmjs. Otherwise fixes are merged to code.

https://github.com/fulup-bzh/GeoGate/commits/master/

New lib version 1.0.7 should be online

[KEGustafsson]

@Hakansv do you have test/real life data available that you could share with us for development & verify purposes?

[Hakansv]

@KEGustafsson links to my test server below.
For data to OCPN: Protocol: Signal K Address: signalk.stupan.se port 3000
Protokol: TCP N0183: Address: signalk.stupan.se port 10112
Json pointing to a specific Meteo target: http://signalk.stupan.se:3000/signalk/v1/api/vessels/urn:mrn:imo:mmsi:002655619

Welcome..

[Hakansv]

If SK data is proxied and distributed outside private network, then fake ID is a big show stopper. But situation is as bad if same AtoN MMSI contains data from multiple places and that is distributed outside private networks too.

I see no good solution...

Agree. What you can do is to forward the mmsi logic you receive from the Meteo issuer. As the VHF AIS system does now.
The receiving system e.g. OCPN need to take care of it. That will be no big difference to make adaptations as we do for the N0183 VDM AIS message 8 Dac 001 Fi 31.

One possible adding: We, OCPN, has introduced a new "sensors.ais.class" called "METEO". You could do the same and add that value for all Ais8_Dac001:Fi31 targets? (And further Ais8_Dac367_Fi33)

[tkurki]

Another viewpoint: normal AIS messages are for a vessel/aircraft/something with an identity and a position. These messages are different in that the identity is the sender, not the "thing" that the data is for, and the position is not the position of the "thing" but the observations.

So we have two things to play with in SK: the identity and the position. One obvious discrepancy that I assume you addressed in O is that a track for these observations makes no sense as the position is not navigation.position but the position of the observations. What if we switch from outputing individual paths and instead output a single object, including position data, under environment.observations.<stationid> where station id is generated permanently from the position data? Then we could use mmsi as is as the context.

[Hakansv]

@tkurki
I think you have a point here.

It's correct a Meteo station is a stable point not moving and thus all trace info in OCPN are abandoned.
For data originated from Ai18_Dac001_Fi 31 there are no further identification for a point than the position and the reused mmsi. So for these stations we make our own mmsi alike ID locally in OCPN. (1994xxxxx) The logic for that is already present.

This is to compare with Meteo data from Ais8_Dac367_Fi33 described here where each station also has a more or less common MMSI but also a Site-ID constructed so that the combination MMSI and Site-ID are always unique. That logic is also reused in OCPN when we assign such a station to our local mmsi-ID 1994-series.

It could be SK can do something similar? Make a "MeteoPoint-ID".
For Ais8_Dac001_Fi31 data it has to be based on position as discussed in above posts.
For future Ais8_Dac367_Fi33 implementation the combination MMSI and Site-ID can be used.

Edit: And if I understand you correct; when using "environment.observations" the Class A confusion may be automatically solved since such a data collection would not be mixed with other target "updates"?

[KEGustafsson]

Did I understood correctly? Would it be something like this?
"context": "atons.urn:mrn:imo:mmsi:<re-used_mmsi>"
and below this context there would be bunch of environment.observations.<stationid>
stationid = lat + lon with e.g. 3 decimal accuracy

E.g. Hankon.p.ylem
lat = 059.82383
lon= 022.91966
stationid = 059823022919
environment.observations.059823022919

Data taken from https://meri.digitraffic.fi/api/sse/v1/measurements
Officially these AtoN's are refered with siteNumber and siteName, not mmsi.
Need to investigate how the number is formed.

[tkurki]

I think we can start tying up the loose ends here:

• this thing should have tests. if somebody can provide representative input samples I can contribute with writing the assertions
• observations or not? Now that I've mulled over it I think we could do without that path part, when there is nothing in the path to indicate the observation station and that is in context
• horizontalVisibility.overRange should imho be a boolean. @afischerdev any thoughts on that?

[Hakansv]

@tkurki
For samples Ive run these:
Live VDM stream Sweden around are included in this source: TCP address: signalk.stupan.se Port: 10114
This file includes N0183 VDM the world around. Download link
Note: if you run the live stream and the file simultaneously they may disturb each others since the same station may occur on both and the issued date/time will be strange.

[panaaj]

I'd also like to put on the table for re-consideration the following paths which have been assigned object values.
I am proposing that we don't use object values and rather just use paths with string or number values
e.g.
pressureTendency: "steady"
pressureTendencyValue: 0
levelTendency: "increasing"
levelTendencyValue: 2

Similarly for horizontalVisibility, precipitationType,ice, seaState.

The justification being that VDM messages are not the only likely source of weather data and it should be straight forward to align values.
Weather services such as OpenWeather will provide similar information without the use of an indexed value (enum) e.g. pressureTendency = "steady", similarly with horizontalVisibilty will just be a number.

OpenWeather observation data example:

[afischerdev]

horizontalVisibility.overRange should imho be a boolean. @afischerdev any thoughts on that?

This is already done by @KEGustafsson
I'm either waiting for new examples on DAC=367 to appear
or for the library to be released as is - to go on later.

[KEGustafsson]

observations or not?

I would say not.

I'd also like to put on the table for re-consideration the following paths which have been assigned object values. I am proposing that we don't use object values and rather just use paths with string or number values e.g. pressureTendency: "steady" pressureTendencyValue: 0 levelTendency: "increasing" levelTendencyValue: 2

Similarly for horizontalVisibility, precipitationType,ice, seaState.

The justification being that VDM messages are not the only likely source of weather data and it should be straight forward to align values. Weather services such as OpenWeather will provide similar information without the use of an indexed value (enum) e.g. pressureTendency = "steady", similarly with horizontalVisibilty will just be a number.

Good reasoning, fully agree
What about current set & drift? Why to leave this one as an object?

[panaaj]

Yes also to current set and drift.
(Missed that one.)

[Hakansv]

Last fix tested successfully.

[KEGustafsson]

@tkurki @panaaj What is the status with this PR? Any more modification required?

@afischerdev Is ggencoder ready to release a new version to npmjs? This one depends on that one.

[afischerdev]

@KEGustafsson

Is ggencoder ready to release a new version to npmjs? This one depends on that one

Yes, I could do that. When you say the current status is alright for your needs.
We could do more for DAC=367 later on when there are more examples.

[panaaj]

@tkurki @panaaj What is the status with this PR? Any more modification required?

Upon resolution of the following points I am OK with it:

1. environment.horizontalVisibility, just confirming this is the correct path and not environment.outside.horizontalVisibility? I don't have a preference but just calling it out.
2. Meta deltas. It would be good if the first time VDM.js is called that the meta deltas for the paths are emitted.

[panaaj]

FYI Freeboard-SK v2.5.0 release has support for this PR.

[Hakansv]

OpenCPN will also have support as soon this PR is implemented and validated there.

[KEGustafsson]

There is investigation going on about meta if feasible to add here. Because this is not normal plugin, there is no plugin methods to add functionalities.
E.g. to find if data is coming first time (path already available) and method to push meta without losing payload data and might be something more too.
My proposal is to merge this as it is and make needed meta changes to specification side.

[tkurki]

I started adding some tests. The first line of Håkan's test data file !AIVDM,1,1,,A,8@2R5Ph0GhF8MAb6ankrgwvlFR06EuOwiOl?wnSwe7wvlOwwsAwwnSGmwvwt,0*13 outputs environment.date of 2024-02-22T15:42:00.000Z, that seems to be in the future, so is there something off? Should it be January 22nd?

[tkurki]

The test naturally fails, because there is no ggencoder with these features published yet.

[Hakansv]

@tkurki
Test file date values:
The VDM sentence has only day, hour and minute. The rest of the ISO date string is from "now". So running a recorded test file may produce strange values?
If I look at the live stream from my SK server the date is correct.

[Hakansv]

But producing a "home made" full ISO string from date, hour, minute can of course give strange value also live. If for example the sensor's time in the AIVDM sentence is 31-23:59 and "now"-time is two minutes later the fictional ISO date format will be in future.
Now, OCPN don't care since we obviously use only day, hour and minute from this path.
(See also here )

[KEGustafsson]

@afischerdev Can you release & publish ggencoder? All fine from this point of view.

@tkurki Date is partially parsed & generated as it is lacking year details as @Hakansv commented earlier. Can we omit year in the date test?

[afischerdev]

Can you release & publish ggencoder? All fine from this point of view

New version ggencoder@1.0.8 is online,, please see

[KEGustafsson]

Minor updates to VDM tests and now those are passed too.

[KEGustafsson]

Dual sentence VDM meteo test case added to increase test coverage.

[Hakansv]

@tkurki
Is there a schedule for next SK server common update including this PR?
I won't PR my changes to OpenCPN core code until verified using a updated SK server. OCPN is rather close to next beta test of a new release and it would be nice if this change can be included.