IndexError: bitarray index out of range

[void4main]

Hi,

I have this message:

ais_print = message.decode()
and I get this:

{'type': 5, 'repeat': 0, 'mmsi': 248659000, 'ais_version': 1, 'imo': 9745500, 'callsign': '9HA4748', 'shipname': 'CMA CGM PREGOLIA', 'shiptype': <ShipType.Cargo_HazardousCategory_A: 71>, 'to_bow': 183, 'to_stern': 12, 'to_port': 30, 'to_starboard': 2, 'epfd': <EpfdType.GPS: 1>, 'month': 7, 'day': 20, 'hour': 22, 'minute': 30, 'draught': 10.5, 'destination': 'DEHAM', 'dte': False}

then

ais_content = message.decode().to_json()
and I get this:

{'nmea': {'ais_id': 5, 'raw': '!AIVDM,2,1,2,B,53e8t>42De5kTP7COCP<l60<Ln118DLthT400017Fp<N25rFNJA1B0CH,0*18', 'talker': 'AI', 'msg_type': 'VDM', 'count': 2, 'index': 1, 'seq_id': '2', 'channel': 'B', 'data': '53e8t>42De5kTP7COCP<l60<Ln118DLthT400017Fp<N25rFNJA1B0CH', 'checksum': 24, 'bit_array': '000101000011101101001000111100001110000100000010010100101101000101110011100100100000000111010011011111010011100000001100110100000110000000001100011100110110000001000001001000010100011100111100110000100100000100000000000000000000000001000111010110111000001100011110000010000101111010010110011110011010010001000001010010000000010011011000'}, 'decoded': {'type': 5, 'repeat': 0, 'mmsi': 248659000, 'ais_version': 1, 'imo': 9745500, 'callsign': '9HA4748', 'shipname': 'CMA CGM PREGOLIA', 'shiptype': 71, 'to_bow': 183, 'to_stern': 12, 'to_port': 30, 'to_starboard': 2, 'epfd': 1, 'month': 7, 'day': 20, 'hour': 22, 'minute': 30, 'draught': 10.5, 'destination': 'DEHAM', 'dte': False}, 'date': '2020-07-20 23:17:14.338438'}

The 'date' is added later on to the dict.

And with the next message pyais crushes, not sure if it is a coincident with the messages above:

Traceback (most recent call last):
File "./get2.py", line 31, in
ais_print = message.decode()
File "/usr/local/lib/python3.7/dist-packages/pyais/messages.py", line 139, in decode
return AISMessage(self)
File "/usr/local/lib/python3.7/dist-packages/pyais/messages.py", line 151, in init
self.content = decode(self.nmea)
File "/usr/local/lib/python3.7/dist-packages/pyais/decode.py", line 677, in decode
return DECODE_MSGmsg.ais_id
File "/usr/local/lib/python3.7/dist-packages/pyais/decode.py", line 38, in decode_msg_1
'raim': bit_arr[148],
IndexError: bitarray index out of range

Not sure how to debug this, hope it helps?!

Best
Frank

[M0r13n]

Hey,

thanks your issue. Could you provide the exact raw message that leads to this crash?

Thank you

[void4main]

Yes, I try to get these ...

[void4main]

I have two examples for now.
The decoding of part 1 is alwasy ok but part 2 leads to a crash:

1)

b'!AIVDM,2,1,1,A,538CQ>02A;h?D9QC800pu8@T>0P4l9E8L0000017Ah:;;5r50Ahm5;C0,0*07'
b'!AIVDM,2,2,1,A,F@V@00000000000,2*35'

Traceback (most recent call last):
File "./get2.py", line 34, in
ais_print = message.decode()
File "/usr/local/lib/python3.7/dist-packages/pyais/messages.py", line 139, in decode
return AISMessage(self)
File "/usr/local/lib/python3.7/dist-packages/pyais/messages.py", line 151, in init
self.content = decode(self.nmea)
File "/usr/local/lib/python3.7/dist-packages/pyais/decode.py", line 677, in decode
return DECODE_MSGmsg.ais_id
File "/usr/local/lib/python3.7/dist-packages/pyais/decode.py", line 451, in decode_msg_22
'addressed': bit_arr[139],
IndexError: bitarray index out of range

2)

b'!AIVDM,2,1,9,A,538CQ>02A;h?D9QC800pu8@T>0P4l9E8L0000017Ah:;;5r50Ahm5;C0,0*0F'
b'!AIVDM,2,2,9,A,F@V@00000000000,2*3D'

Traceback (most recent call last):
File "./get2.py", line 34, in
ais_print = message.decode()
File "/usr/local/lib/python3.7/dist-packages/pyais/messages.py", line 139, in decode
return AISMessage(self)
File "/usr/local/lib/python3.7/dist-packages/pyais/messages.py", line 151, in init
self.content = decode(self.nmea)
File "/usr/local/lib/python3.7/dist-packages/pyais/decode.py", line 677, in decode
return DECODE_MSGmsg.ais_id
File "/usr/local/lib/python3.7/dist-packages/pyais/decode.py", line 451, in decode_msg_22
'addressed': bit_arr[139],
IndexError: bitarray index out of range

Looks like a problem of multipart-messages?

[M0r13n]

Thank you again for the messages. I will dig into it.

[M0r13n]

So. I took a look at your issue. There wasn't actually a bug in the decoding itself. Instead, you slightly misused the interface.

When working with multi-line messages you need to assemble them before you can decode them.

So you either have to do it like this:

msg_1_part_0 = b'!AIVDM,2,1,1,A,538CQ>02A;h?D9QC800pu8@T>0P4l9E8L0000017Ah:;;5r50Ahm5;C0,0*07'
msg_1_part_1 = b'!AIVDM,2,2,1,A,F@V@00000000000,2*35'

NMEAMessage.assemble_from_iterable(
        messages=[
            NMEAMessage(msg_1_part_0),
            NMEAMessage(msg_1_part_1)
        ]
    ).decode().to_json()

msg_2_part_0 = b'!AIVDM,2,1,9,A,538CQ>02A;h?D9QC800pu8@T>0P4l9E8L0000017Ah:;;5r50Ahm5;C0,0*0F'
msg_2_part_1 = b'!AIVDM,2,2,9,A,F@V@00000000000,2*3D'

NMEAMessage.assemble_from_iterable(
        messages=[
            NMEAMessage(msg_2_part_0),
            NMEAMessage(msg_2_part_1)
        ]
    ).decode().to_json()

or with the newest release it is also possible to do soimething like this:

 messages = [
            b'!AIVDM,2,1,1,A,538CQ>02A;h?D9QC800pu8@T>0P4l9E8L0000017Ah:;;5r50Ahm5;C0,0*07',
            b'!AIVDM,2,2,1,A,F@V@00000000000,2*35',
            b'!AIVDM,2,1,9,A,538CQ>02A;h?D9QC800pu8@T>0P4l9E8L0000017Ah:;;5r50Ahm5;C0,0*0F',
            b'!AIVDM,2,2,9,A,F@V@00000000000,2*3D',
        ]
for msg in ByteStream(messages):
        decoded = msg.decode()

But depending on the source of your messages you might want to take a look at the FileReaderStream o TCPStream.

I hope this helps. If you have further questions, feel free to ask. But I mark this issue as resolved.

[void4main]

Thank you, it helps!

(I get the data via antenna and udp broadcast, so it might get tricky to assemble the messages in the correct order but let's see .. :)

[M0r13n]

Interesting. I thought about adding out-of-order message support. I might add this feature in a future release together with an UDPStream.

[M0r13n]

Hey,

I integrated a native UDPStream in the newest release. It includes out-of-order message assembly. Usage is straight forward:

from pyais.stream import UDPStream

host = "127.0.0.1" # change this
port = 55555 # and change that

for msg in UDPStream(host, port):
    msg.decode()
    # do something 

@void4main Would you like to check it out and give me some feedback on how it works for you?

Greetings

[void4main]

Hey, that's great. I'll test it asap ...

[void4main]

I had some time to test it with live AIS data and it worked perfectly fine for me.
Since I get broadcasted UDPs I had to set an emtpy host address:
host = "" port = 10110

It crashed once in the beginning but only inside the json conversion:
message = msg.decode().to_json() and I couldn't reproduce the problem.
With the UDPStream I inserted ~300.000 entries to my mongodb by now.

I had one more crash adding data to the db:
b'!AIVDM,2,1,1,A,14S1fp00002E5k8A5Ejv1c>200Rfr>l00000000s1rA<<5kQ0ClmAFP0,0*78!AIVDM,2,2,1,A,000000000000000,2*25'

Traceback (most recent call last):
  File "./get3.py", line 39, in <module>
    result = posts.insert_one(message)
  File "/usr/local/lib/python3.7/dist-packages/pymongo/collection.py", line 657, in insert_one
    bypass_doc_val=bypass_document_validation),
  File "/usr/local/lib/python3.7/dist-packages/pymongo/collection.py", line 562, in _insert
    check_keys, manipulate, write_concern, op_id, bypass_doc_val)
  File "/usr/local/lib/python3.7/dist-packages/pymongo/collection.py", line 551, in _insert_one
    self.__write_response_codec_options)
  File "/usr/local/lib/python3.7/dist-packages/pymongo/collection.py", line 480, in _legacy_write
    rqst_id, msg, max_size = func(*args)
OverflowError: MongoDB can only handle up to 8-byte ints

but haven't looked into it and probably hasn't got anthing to do with your code.
Thank you for the implementation.

[M0r13n]

I am happy to hear, that it works for you! :-)

If you observe any new issues or if you need another feature, feel free to open a new issue.