Improve handling of variable length strings, especially in JSON and XML descriptions

[ttlappalainen]

Definition for 126998 is wrong. It has only fields:
1 Installation Description, Field 1
2 Installation Description, Field 2
3 Manufacturer Information, Field 3
All fields are variable length strings so that there is:
<0x01>, where len is length()+2. So this PGN message can be:
<0x02><0x01><0x03><0x01><"C"><0x05><0x01><"CAT">

Also on PGN 129802 field 7 "Safety Related Text" is constructed as above - not fixed length field. And possibly on other PGNs containing text data.

[keesverruijt]

Can you upload some raw PGN strings, in a format that analyzer understands?
I have no sources providing these PGNs and since this is all produced by reverse-engineering ...

There are (at least) four ways of representing strings in N2K, so with the 'have never seen' definitions I had to guess, and I guess I guessed wrong here :-)

[ttlappalainen]

Have you ever checked my NMEA Simulator (http://www.kave.fi/Apps/NMEA-Simulator/NMEA-Simulator.7z - unfortunately only Windows version available). It sends all data to selected port and also configuration information (126998) in Actisense format. I normally test and play with simulator either by null modem cable (com0com driver or physical) or through real N2k bus by using my NMEA 2000 library sample ActisenseSender.

[ttlappalainen]

I have actually wondered how many designer they have needed to create mesh like N2k. But could you point me other ways for strings. I know only these two with fixed field like on product information (126996) and variable field on some others like 126998 and 129802.

[keesverruijt]

The CANBoat analyzer has:

1. RES_ASCII (fixed field length, no byte count)
2. RES_STRING (starts with 1 byte length byte)
3. RES_STRINGLZ (starts with 1 byte length byte and terminated by zero byte)
4. RES_STRINGLAU (starts with 1 byte length byte and 1 byte ASCII/UNICODE byte)

The RES_STRINGLZ variation is used by Fusion for the old company specific entertainment (mainly audio) PGNs.

The RES_STRINGLAU variation is used by the new N2K PGNs since 2015 for DSC and entertainment. My CANBoat analyzer doesn't yet properly handle the UTF16 content variation. (Why in the world you would choose UTF16 instead of UTF8 when coming up with this in 2014-2015 is really beyond me, but okay.)

[ttlappalainen]

Maybe 126998 and 129802 uses RES_STRINGLAU, since it has length byte and then 0x01, which meaning I have not understood. So could 0x01 mean ASCII. What would be the value for UNICODE 0x02?

You seem to have length for RES_STRINGLAU field often 2 ( BYTES(2) ). Do you still handle length of whole message right by checking string length from first byte? In 129041 it seem to have length 34, why?

[keesverruijt]

Maybe 126998 and 129802 uses RES_STRINGLAU, since it has length byte and then 0x01, which meaning I have not understood.
That does sound like RES_STRINGLAU.

So could 0x01 mean ASCII. What would be the value for UNICODE 0x02?
No, 0.
See https://www.nmea.org/Assets/20160715%20corrigenda%20entertainment%20pgns%20.pdf for example.

You seem to have length for RES_STRINGLAU field often 2 ( BYTES(2) ). Do you still handle length of whole message right by checking string length from first byte? In 129041 it seem to have length 34, why?

Yes, when the length is up to 255 bytes I generally coded the minimal (no value = 2 bytes). The analyzer will consume the correct # of bytes from the PGN, the field length for these fields is only really used for 'description' use. In 12041 the source is not N2K directly but the radio format for AIS, so the length will be less, and I coded the maximum length. I agree that this is confusing. Maybe I should set the length data to 255 for the others.

[keesverruijt]

Note that for RES_STRINGLAU the length field contains the length including the length and type fields, so an empty string is 0x02 0x01 or 0x02 0x00.

[ttlappalainen]

That is how it is on PGN 126998 and 129802. So definition for PGN 126998 should be:
{ "Configuration Information", 126998, false, 0x2a, 0,
{ { "Installation Description #1", BYTES(2), RES_STRINGLAU, false, 0, "" }
, { "Installation Description #2", BYTES(2),RES_STRINGLAU, false, 0, "" }
, { "Manufacturer", BYTES(2), RES_STRINGLAU, false, 0, "" }
, { 0 }
}
}

[ttlappalainen]

Could length be 0 ( BYTES(0) ) to inform that it is unknown?

[keesverruijt]

I'd rather encode the maximum length, but I don't know what the maximum is.

[ttlappalainen]

It is limited to 70 as far as I know. But if you use max. length, then handler has to allways also check the type, when calculating the length. Also the size on the pgn struct should be fixed somehow. As shortest it can be 6 and longest 216.

[larsujensen]

PGN 129802 covers AIS message 14, "Safety Related Broadcast Message". This message can have 1 - 161 characters of text. I.e. as mentioned it is of variable length, at least in the ais message.
The pgn is marked as not complete, and with a bit length of 288 this sounds reasonable, 288 bits can not be expected to convey reasonable safety messages.

But PGN 129801 is marked as complete. This pgn covers AIS message 12, "AIS Addressed Safety Related Message". This message also has a variable length text field, but in this case 1-156 characters. The pgn has a bit length of 2040. Now, the reason for 156 in stead of 161 characters in message 12 is to keep it within 5 slots, the same as message 14, both having a maximum length of 1008 bits (a char occupies 6 bits in ais messages, not 8). To the best of my understanding the safety related text fields are identical in message 12 and 14, except for maximum length. It seems plausible that the same should hold for PGN 129801 and PGN 129802.

[dbeckwith01]

From the looks of this trace for PGN 126998 the sequence $0200 is UTF16.

2021-01-30-20:43:21.684,6,126998,1,255,8,00,50,02,00,02,00,4c,00
2021-01-30-20:43:21.684,6,126998,1,255,8,01,52,00,6f,00,73,00,65
2021-01-30-20:43:21.685,6,126998,1,255,8,02,00,20,00,50,00,6f,00
2021-01-30-20:43:21.685,6,126998,1,255,8,03,69,00,6e,00,74,00,20
2021-01-30-20:43:21.686,6,126998,1,255,8,04,00,72,00,6f,00,73,00
2021-01-30-20:43:21.687,6,126998,1,255,8,05,65,00,70,00,6f,00,69
2021-01-30-20:43:21.687,6,126998,1,255,8,06,00,6e,00,74,00,2e,00
2021-01-30-20:43:21.688,6,126998,1,255,8,07,63,00,6f,00,6d,00,20
2021-01-30-20:43:21.688,6,126998,1,255,8,08,00,34,00,32,00,35,00
2021-01-30-20:43:21.689,6,126998,1,255,8,09,2d,00,36,00,30,00,35
2021-01-30-20:43:21.689,6,126998,1,255,8,0a,00,2d,00,30,00,39,00
2021-01-30-20:43:21.690,6,126998,1,255,8,0b,38,00,35,00,ff,ff,ff

The above decodes to:

1. Install Description#1 = ""
2. Install Description#2 = ""
3. Manufacturer Information = "Rose Point rosepoint.com 425-605-0985"

With this detail, are you able to update the handling of PGN 126998 and remove the unhanded UNICODE error generation message?

[keesverruijt]

Thanks, this is exactly the detail I needed.

[keesverruijt]

Fixed in master as merged #57