Add UTC timestamp to decoded message when consuming stream

[doncqueurs]

Include the UTC timestamp in the output when they are fetched from a UDP/TCP stream.

[M0r13n]

Hey there,

when you are talking about a UTC timestamp, you mean the time when the decoder received the message. Am I right? Because the AIS message itself does not contain such information.

The question is, where to add such additional information. Pyais distinguishes between NMEA messages and AIS messages. Both messages currently represent the NMEA or AIS protocol as accurately as possible. Therefore I am a little bit reluctant to write additional information into these two messages, if they are not part of the official protocol. I could imagine an extra category for decoder specific metadata. But it could be somewhat tricky to add such a feature without breaking backwards compatibility.

[M0r13n]

I played around a bit and came up with the idea, of a meta field inside the NMEAMessage class. A decoded message could then look like this:

{
    "nmea": {
        "ais_id": 37,
        "raw": "!AIVDM,1,1,,A,U31<0OOP000CshrMdl600?wP00SL,0*43",
        "talker": "AI",
        "msg_type": "VDM",
        "count": 1,
        "index": 1,
        "seq_id": "",
        "channel": "A",
        "data": "U31<0OOP000CshrMdl600?wP00SL",
        "checksum": 67,
        "bit_array": "100101000011000001001100000000011111011111100000000000000000000000010011111011110000111010011101101100110100000110000000000000001111111111100000000000000000100011011100",
        "meta": {
            "received_at": 1616934612,
            "decoder": "pyais-v1"
        }
    },
    "decoded": {}
}

[doncqueurs]

I think added it as meta data is the right approach.

[Nambarc]

I agree with you @M0r13n. NMEA messages don't contain a time stamp, so neither should this library. In my opinion it's up to the user of the library to timestamp their own data. This is made easy because the library allows the messages to be converted into a dictionary.

[Inrixia]

I also agree, this should be on the user and not part of the library. There is no point in including it.

[anttinousiainen]

when you are talking about a UTC timestamp, you mean the time when the decoder received the message. Am I right? Because the AIS message itself does not contain such information

Actually AIS messages do have the UTC timestamp embedded, sort of.

Coarse UTC timestamp (hours and minutes) is embedded into the SOTDMA "radio state" fields (19 bits) in eg. message types 1, 2, 4, 9, 11 and 18 among others - thought not every single transmitted message carries it, it is interleaved with slot numbers and such - they clearly ran out of bits when specifying the protocol.

Would be nice for the library to decode these radio fields also, as they could be used for example to detect missed, but expected transmissions from other stations and just out of general interest.

They also show among other things interesting info like how many other stations each station currently observes. My vote is to include these fields in the decoding, the data is there in the messages.

[anttinousiainen]

And observing this coarse UTC timestamp and the slot number in question (1 - 2250), a pretty accurate UTC timestamp can be deduced from the AIS messages.

In fact, this is one of the failover modes for timesync in the "real" AIS receivers and transmitters, if I understand correctly.

[anttinousiainen]

I am not really a python person, but a really quick and dirty minimal test code I wrote in few minutes:

mask2 = 0x03
mask3 = 0x07
mask5 = 0x1f
mask7 = 0x7f
mask14 = 0x3fff

if (decoded.msg_type == 1 or decoded.msg_type == 2):

                              # rad2 here indicated station sync state, 0 = utc direct, 1 = utc indirect and so on:
                              rad2  = (decoded.radio >> 17) & mask2

                              rad3  = (decoded.radio >> 14) & mask3
                              rad14 = (decoded.radio >> 0 ) & mask14

                              if (rad3 == 0):
                                                            slot_offset = rad14
                              if (rad3 == 1):
                                                            utc_hours =(rad14 >> 9) & mask5
                                                            utc_mins = (rad14 >> 2) & mask7

                              if (rad3 == 2 or rad3 == 4 or rad3 == 6):
                                                            slot_current = rad14

                              if (rad3 == 3 or rad3 == 5 or rad3 == 7):
                                                            current_stations_heard = rad14
                                

Tested with real on the air transmissions here in Kiel Canal with ships passing by, seems to work.

edit. some fields seem to give "interesting" values. Maybe there is still some finetuning with the bitmasks etc, but generally the UTC timestamp seems to be correct, at least for the stations indication "utc direct" sync state.

[Inrixia]

They also show among other things interesting info like how many other stations each station currently observes. My vote is to include these fields in the decoding, the data is there in the messages.

Are these fields not already present? Afik the decoder currently provides hours/minutes etc.

[anttinousiainen]

Are these fields not already present? Afik the decoder currently provides hours/minutes etc.

For the AIS messages, say message type 1?

Not the version I have at least does not, it only decodes a raw radio-status field as a number, not what that number represents. Do I have an older version maybe then and need to update?

msg = "!AIVDM,1,1,,A,13HOI:0P0000VOHLCnHQKwvL05Ip,0*23"
decoded = msg.decode()
print(decoded)

MessageType1(msg_type=1, repeat=0, mmsi=227006760, status=<NavigationStatus.UnderWayUsingEngine: 0>, turn=None, speed=0.0, accuracy=False, lon=0.13138, lat=49.475577, course=36.7, heading=511, second=14, maneuver=0, spare_1=b'\x00', raim=False, radio=22136)

The above 19 bit field (decimal 22136) btw decodes to:

00 001 01011001111000

meaning it is:

• direct UTC synced (00)
• by pure luck is indeed a message containing UTC timestamp (001)
• which can be decoded from the last part (01011001111000) to be UTC 11:30 if I calculated it correctly

[M0r13n]

Hey @anttinousiainen,

I believe that the original author had something else in mind. He wanted a timestamp to be added as soon as the library received a message via a socket. Instead, you are talking about the UTC timestamp of SOTDMA messages.

Never the less I think that your suggestion it is a nice feature. I decided to add support to get the SOTDMA communication state from the raw radio field.Therefore, I added a new method in utils.py called get_sotdma_comm_state. This method takes a single integer (the raw radio field) and returns a dictionary with the following keys:

• slot_number
• utc_hour
• utc_minute
• slot_offset
• slot_timeout
• sync_state

Each key has an integer value. If we take your message "!AIVDM,1,1,,A,13HOI:0P0000VOHLCnHQKwvL05Ip,0*23" as an example, the output looks like this:

msg = "!AIVDM,1,1,,A,13HOI:0P0000VOHLCnHQKwvL05Ip,0*23"
decoded = decode(msg)
actual = get_sotdma_comm_state(decoded.radio)
# {'received_stations': 0, 'slot_number': 0, 'utc_hour': 11, 'utc_minute': 30, 'slot_offset': 0, 'slot_timeout': 1, 'sync_state': <SyncState.UTC_DIRECT: 0>}

@anttinousiainen Would you take a look at my implementation here ?

Greetings from Kiel and thanks for your input!

[anttinousiainen]

Hey @M0r13n,

Never the less I think that your suggestion it is a nice feature. I decided to add support to get the SOTDMA communication state from the raw radio field.Therefore, I added a new method in utils.py called get_sotdma_comm_state. This method takes a single integer (the raw radio field) and returns a dictionary with the following keys:

Great, thanks, looks good and is a welcome addition to observing real-time AIS-traffic, like for example as mentioned at spotting missed transmissions.

However, worth noting, not all radio states are always SOTDMA. For example message_type 18 has a "communication state selector flag" bit, that describes whether radio_state following is SOTDMA or ITDMA, with different structure and meaning.

Greetings from Schilksee and thanks!