Skip to content

Ublox NEO M8N deep dive

dave griffiths edited this page Sep 4, 2019 · 24 revisions

Running the sonic kayaks workshop last week the GPS on all boats (running on Raspberry Pis) stopped working at once. Although configured to only produce text readable NMEA messages they were all spouting this gibberish:

So we need to figure out why these all broke at once... two options I can think of both based on corruption from unplugging the power repeatedly to get bluetooth working (which is another issue entirely):

  • they switched to binary mode
  • the baud rate changed on the NMEA messages

Either way, power cycling them did not work - but they were back to normal after a weekend of being switched off. I think this discounts some corruption to the internal eeprom configuration, as it would have stayed broke. Perhaps some hot/cold reset thing - need to look into that.

Looking at normal messages I noticed this burst of binary mixed with the NMEA data:

դ@�
��*�i)SI
            �S	�b��b��b��b���b��b��b��b���bb��bb���bb�R�դ��
��*��)�i)Si���i�I�)ӓ���b��b���bb�R��j�D���ԕ))		��SjRD:uje�)b�b��b�R��jDD:�u�������	S�ʊ�r��b��r���j�D�ʊE�	�b��b��b
9QMQQUM�=-���j��$GNRMC,090911.00,V,,,,,,,030919,,,N,V*1B

^[[?62;c$GNVTG,,,,,,,,,N*2E

$GNGGA,090911.00,,,,,0,04,5.44,,,,,,*49

$GNGSA,A,1,16,10,20,21,,,,,,,,,10.12,5.44,8.53,1*3D

$GNGSA,A,1,,,,,,,,,,,,,10.12,5.44,8.53,2*39

$GNGSA,A,1,,,,,,,,,,,,,10.12,5.44,8.53,3*38

$GPGSV,4,1,15,07,21,296,27,08,44,290,,10,45,120,29,11,07,256,,0*65

The chunk at the top looks very much like the screenshot of the crap coming out of the GPS on the Pi - so this indeed looks like UBX format binary was turned on.

So - how to tweak the GPS unit from Linux without the proprietary tool, and hopefully Raspbian - so we can write some scripts that reset them?

gpsctl

Seems like you can use gpsctl to send byte strings to the GPS receiver - but the spec is massive and this is a confusing way to approach things....

ubxtool to the rescue?

This seems like it should be part of gps-clients package, but isn't. Luckily it's just a python script with minimal dependancies so it seems to run ok on Ubuntu - getting serial write timeouts on Pi via UART though.

Let's try "Poll Enabled Constellations" from the manpage (on ubuntu laptop with gps plugged into USB):

$ ./ubxtool -p CFG-GNSS -f /dev/ttyUSB0 

returns:

ubxtool: poll CFG-GNSS
sent:
UBX-CFG-GNSS:
Poll request

UBX-CFG-GNSS:
 msgVer 0  numTrkChHw 32 numTrkChUse 32 numConfigBlocks 7
  gnssId 0 TrkCh  8 maxTrCh 16 reserved 0 Flags x01010001
   GPS L1C/A enabled
  gnssId 1 TrkCh  1 maxTrCh  3 reserved 0 Flags x01010001
   SBAS L1C/A enabled
  gnssId 2 TrkCh  4 maxTrCh  8 reserved 0 Flags x01010001
   Galileo E1 enabled
  gnssId 3 TrkCh  8 maxTrCh 16 reserved 0 Flags x01010000
   BeiDou B1I 
  gnssId 4 TrkCh  0 maxTrCh  8 reserved 0 Flags x03010000
   IMES L1 
  gnssId 5 TrkCh  0 maxTrCh  3 reserved 0 Flags x05010001
   QZSS L1C/A enabled
  gnssId 6 TrkCh  8 maxTrCh 14 reserved 0 Flags x01010001
   GLONASS L1 enabled

UBX-ACK-ACK:
 ACK to Class x6 (CFG) ID x3e (GNSS)`

Lets try breaking them in a similar manner...

Switch it to binary

$ ./ubxtool -e BINARY -f /dev/ttyUSB0 


ubxtool: enable BINARY
sent:
UBX-CFG-MSG:
 Rate set: Class x1 (NAV) ID x4 (DOP) Rate:1

sent:
UBX-CFG-MSG:
 Rate set: Class x1 (NAV) ID x6 (SOL) Rate:1
...

and a ton of other config...

If we cat the GPS device now, it terminates the stream immediately - as it is presumably sending a byte sequence interpreted as end of file...? This happens after any command sent... it seems the state is modified but needs saving as power cycling resets everything...

Ah ok - so you set up your config, e.g:

$ ./ubxtool -d NMEA -f /dev/ttyUSB0
$ ./ubxtool -e BINARY -f /dev/ttyUSB0
$ ./ubxtool -p SAVE -f /dev/ttyUSB0 

Turn off NMEA, turn on binary messages and save the config - power cycle then:

cat /dev/ttyUSB0 
u�b�b9��b��b)�b�L�b0��Ji

                                 �
>*

$ cat /dev/ttyUSB0 
vܵb�Ri
      ���C
�
 �MK����\�����������bpVi
                            ���C
�ϵbXZi
       ���C
��bC��b���b��b�ii
                      ���D
	*�b�mi
               ���D
v�b0��qi

           �
>*

$ cat /dev/ttyUSB0 
���b���b>��b����Y�����
Q/�b>2�b噵b0�5j

                   �

Not quite the same behavior but the byte sequences seem to have a similar pattern... lets try the reverse - turning off all binary messages:

$ ./ubxtool -e NMEA -f /dev/ttyUSB0
$ ./ubxtool -d BINARY -f /dev/ttyUSB0
$ ./ubxtool -p SAVE -f /dev/ttyUSB0 

That seems to work, no longer any binary mixed in. We could do this to repair the GPS from a laptop during an event at least.

Detective work

What are the binary messages we were getting that also appear sporadically with the mixed NMEA/binary data? It seemed that we were only getting these repeatedly and no NMEA at all. The string "jRD:" appears a lot in these, so must represent something we can look for - or rather the bytes in hexadecimal these characters represent: "6a 52 44 3a".

Not getting much on what these could mean - but reading up more on the UBX message format:

Header ("sync chars") Message class Message ID Length Payload Checksum
0xb5 0x62 1 byte 1 byte 2 bytes N bytes 2 bytes

0x62 is 'b' in ascii - so all the "�b" bits are probably the headers of individual UBX messages.

Wrote a python script to scan and search the UBX stream, left it running for a few hours and nothing - then finally caught a jRD (0x6a 0x52 0x44) in the wild! This happened as the GPS was booting up (similar to when I saw it earlier today) it's in the middle of this dump:

0x6a 0x65 0xc5 0x89 0x62 0x8a 0x62 0x8a 0x9a 0x62 0x82 0x8a 0x62 0x9a 0x8a 0x62 0x8a 0x92 0xba 0x62 0x62 0x82 0x9a 
0x62 0xb2 0xaa 0x62 0x82 0xb2 0xaa 0x62 0x62 0x82 0xb2 0x62 0x9a 0xaa 0x62 0x9a 0x82 0x92 0x62 0x62 0x82 0xca 0x62 
0x9a 0x92 0x62 0x8a 0xca 0xb2 0x62 0x62 0x82 0x52 0xb2 0x8a 0x6a 0xa4 0x44 0xe9 0x0a 0xea 0xd4 0x95 0x15 0x89 0x13 
0x49 0x13 0x29 0xd3 0x13 0x29 0x53 0x13 0x29 0x53 0x13 0x29 0xab 0x53 0x13 0x89 0x29 0x93 0x13 0x09 0xeb 0x13 0x69 
0xd3 0x13 0x26 0x89 0x29 0x93 0x13 0x09 0x53 0x13 0x09 0xd3 0xd3 0x13 0x89 0x29 0xd3 0x13 0x89 0x53 0x62 0x92 0xaa 
0x82 0x62 0x62 0x82 0x52 0xb2 0x1a 0xd5 0xa4 0x88 0xe9 0x14 0xd5 0xd4 0x95 0x15 0x89 0x13 0x69 0x13 0x29 0xd3 0x13 
0x29 0x13 0x62 0x82 0xba 0x62 0x8a 0x92 0xca 0x62 0x62 0x8a 0xca 0x62 0xa2 0xc2 0x62 0x92 0xba 0xb2 0x62 0x62 0x92 
0x92 0x62 0xa2 0xa2 0x62 0x82 0xba 0xba 0x62 0x62 0x92 0x9a 0x62 0xb2 0x92 0x62 0x8a 0xaa 0xba 0x62 0x62 0x82 0x52 
0xb2 0xaa 0x6a 0x52 0x44 0x3a 0x05 0x75 0x6a 0x65 0xc5 0x89 0x62 0xa2 0x62 0x8a 0x9a 0x62 0x9a 0x8a 0x62 0x8a 0xb2 
0x62 0x82 0xa2 0xba 0x62 0x62 0x82 0x52 0xaa 0x8a 0x6a 0xa4 0x44 0xe9 0x8a 0xea 0xd4 0x95 0x15 0x69 0x13 0x29 0x13 
0x29 0x13 0x13 0xc9 0x53 0x62 0x82 0xca 0x62 0x82 0xa2 0xc2 0x62 0x62 0xba 0x82 0x62 0xb2 0xba 0x62 0x82 0x9a 0xb2 
0x62 0x62 0xba 0x8a 0x62 0xaa 0xaa 0x62 0x92 0xa2 0x82 0x62 0x62 0xba 0x92 0x62 0x82 0xba 0x62 0x92 0x9a 0x92 0x62 
0x62 0x82 0x52 0xba 0x92 0x6a 0xa4 0x44 0x74 0x8a 0xea 0xd4 0x95 0x15 0x69 0x13 0x49 0x13 0x29 0x82 0x62 0xba 0xc2 
0x62 0x82 0xca 0x62 0x9a 0xaa 0xc2 0x62 0x62 0xba 0xca 0x62 0x8a 0x92 0x62 0x82 0xa2 0xba 0x62 0x62 0xc2 0x82 0x62 
0x82 0x82 0x62 0x82 0xca 0xaa 0x62 0x62 0xc2 0xaa 0x62 0xa2 0xca 0x62 0x8a 0xb2 0xca 0x62 0x62 0x82 0x52 0xba 0xaa 
0x6a 0xa4 0x44 0xe9 0x8a 0xea 0xd4 0x95 0x15 0x69 0x83 0x69 0x13 0x29 0x13 0x13 0x09 0xb2 0x62 0xb2 0xa2 0x62 0x9a 
0x82 0xb2 0x62 0x62 0xc2 0xba 0x62 0x82 0xc2 0x62 0x9a 0x92 0xca 0x62 0x62 0x82 0x52 0xba 0x2a 0xd5 0xa4 0x44 0xe9 
0x2a 0xd4 0xd4 0x2a 0x15 0x49 0x93 0x29 0x13 0x09 0x13 0x62 0x82 0xa2 0x62 0x82 0xb2 0x62 0x92 0x92 0x9a 0x62 0x62 
0x82 0xba 0x62 0x8a 0xa2 0x62 0x82 0xaa 0x82 0x62 0x62 0x8a 0x8a 0x62 0x8a 0x9a 0x62 0x9a 0x8a 0xa2 0x62 0x62

or in ASCII form

jeʼnb�b��b��b��b���bb��b��b���bb��b��b���bb��b��b�ʲbb�R��j�D�
�ԕ�I)�)S)S)�S�)�	�i�&�)�	S	���)��Sb���bb�R�դ���ԕ�i)�)
b��b���bb��b��b���bb��b��b���bb��b��b���bb�R��jRD:ujeʼnb�b��b��b��b���bb�R��j�D���ԕi))
�Sb��b���bb��b��b���bb��b��b���bb��b��b���bb�R��j�Dt��ԕiI)�b��b��b���bb��b��b���bb‚b��b�ʪbbªb��b���bb�R��j�D���ԕ
i�i)	�b��b���bbºb��b���bb�R�*դD�*��*I�)	b��b��b���bb��b��b���bb��b��b���bb

And crucially there are no 0xb5 0x62 headers anywhere here so these do not seem to be usual UBX packets...

another one (just putting here for storage)

0x62 0xa2 0x82 0x62 0x92 0x9a 0xa2 0x62 0x92 0x9a 0x62 0x82 0x52 0xb2 0x82 0x6a 0xa4 0x44 0xe9 0x0a 0xd5 0xd4 0x2a 0x15 0x89 0x13 0x69 0x13 0x29 0xd3 0x13 0x29 0xca 0x62 0xa2 0xaa 0x62 0x92 0xaa 0xaa 0x62 0x62 0x92 0x92 0x62 0x9a 0x92 0x62 0x82 0xc2 0xa2 0x62 0x92 0xb2 0x62 0x92 0x9a 0x62 0xba 0x8a 0x62 0x8a 0x92 0xc2 0x62 0x92 0xba 0x62 0x92 0xaa 0x62 0x82 0x92 0x62 0x9a 0xaa 0xb2 0x62 0x62 0x82 0x52 0xb2 0x2a 0xd5 0xa4 0x88 0xe9 0x14 0xd5 0xd4 0x2a 0x15 0x89 0x13 0x89 0x13 0x29 0x9a 0x62 0x9a 0x8a 0x62 0x8a 0x9a 0x62 0x82 0x9a 0xaa 0x62 0x92 0xca 0x62 0x82 0x52 0xaa 0x0a 0xd5 0xa4 0x88 0xe9 0x14 0xea 0xd4 0x95 0x15 0x69 0x83 0x29 0x13 0x09 0x53 0x62 0xba 0x82 0x62 0xaa 0x8a 0x62 0x82 0xa2 0xa2 0x62 0x62 0xba 0x8a 0x62 0xb2 0xca 0x62 0x92 0xb2 0xa2 0x62 0x62 0xba 0x92 0x62 0x8a 0xca 0x62 0x92 0xa2 0x92 0x62 0x62 0xba 0xc2 0x62 0x82 0x9a 0x62 0x9a 0xa2 0xb2 0x62 0x62 0x82 0x52 0xba 0x1a 0xd5 0xa4 0x88 0xe9 0x14 0xea 0x6a 0x95 0x15 0x69 0x13 0x49 0x13 0x09 0x53 0x62 0xba 0xca 0x62 0x8a 0xb2 0x62 0x82 0x9a 0x9a 0x62 0x92 0xba 0x62 0xc2 0x82 0x62 0x8a 0x8a 0x62 0x82 0xc2 0xaa 0x62 0x92 0x92 0x62 0xc2 0xaa 0x62 0x9a 0x8a 0x62 0x8a 0xba 0x9a 0x62 0x92 0xca 0x62 0xc2 0xb2 0x62 0xba 0x8a 0x62 0x92 0xb2 0xa2 0x62 0x62 0x82 0x52 0xba 0x8a 0x6a 0x52 0x44 0x3a 0xc5 0x75 0x6a 0x65 0xc5 0x69 0x62 0x9a 0x62 0x82 0xca 0x62 0xc2 0xba 0x62 0x92 0x92 0x62 0x9a 0x92 0xca 0x62 0x92 0x9a 0x62 0x82 0x52 0xa2 0xb2 0x6a 0xa4 0x44 0xe9 0x2a 0xd4 0xd4 0x2a 0x15 0x49 0x13 0x29 0x13 0x09 0xeb 0x13 0x09 0x13 0x13 0x29 0x93 0x13 0x49 0x92 0xc2 0x62 0x62 0x82 0xba 0x62 0x82 0xaa 0x62 0x82 0xaa 0x9a 0x62 0x62 0x8a 0x8a 0x62 0x92 0x92 0x62 0x9a 0x8a 0xa2 0x62 0x92 0x8a 0x62 0x8a 0x92 0x62 0xb2 0xca 0x62 0x92 0xb2 0xba 0x62 0x62 0x82 0x52 0xba 0xaa 0x6a 0xa4 0x44 0xe9 0x2a 0xd4 0xd4 0xaa 0x15 0x49 0x13 0x49 0x13 0x09 0xd3 0x13 0x29 0xca 0x62 0xba 0x92 0x62 0x92 0x82 0xba 0x62 0x62 0x9a 0x82 0x62 0x82 0xaa 0x62 0x82 0x9a 0xca 0x62 0x62 0x9a 0x9a 0x62 0xa2 0x9a 0x62 0x8a 0xb2 0xa2 0x62 0x9a 0x9a 0x62 0x82 0x52 0xa2 0x9a 0x6a 0x64 0x44 0x3a 0xe5 0x75 0x8a 0x8a 0x8a 0xa9 0x13 0x53 0x26 0x26 0x09 0xd3 0x53 0x26 0x26 0x4d 0x4b 0xa7 0x96 0x30 0x30 0x35 0x30 0x35 0x2e 0x38 0x37 0x31 0x33 0x34 0x2c 0x57 0x2c 0x31 0x35 0x33 0x30 0x30 0x38 0x2e 0x30 0x30 0x2c 0x41 0x2c 0x41 0x2a 0x36 0x35 0x0a 0x0a 0x24 0x47 0x4e 0x47 0x53 0x54 0x2c 0x31 0x35 0x33 0x30 0x30 0x38 0x2e 0x30 0x30 0x2c 0x33 0x32 0x2c 0x2c 0x2c 0x2c 0x31 0x34 0x2c 0x32 0x30 0x2c 0x33 0x32 0x2a 0x36 0x46 0x0a 0x0a 0x24 0x47 0x4e 0x5a 0x44

This one happened when binary mode was turned off!!! This means resetting them to NMEA won't work during an event, and we are looking at something more serious.

0xba 0xba 0x62 0x62 0xc2 0xaa 0x62 0x92 0x8a 0x62 0x8a 0xba 0xa2 0x62 0x62 0xc2 0xb2 0x62 0xb2 0xba 0x62 0x92 0x9a 0x9a 0x62 0x62 0xc2 0xba 0x62 0x9a 0x8a 0x62 0x9a 0x92 0xb2 0x62 0x92 0x9a 0x62 0x82 0x52 0xba 0xaa 0x6a 0xa4 0x44 0xe9 0x2a 0xea 0xd4 0x95 0x15 0x49 0x13 0x29 0x13 0x09 0xd3 0x13 0x09 0x13 0x13 0x49 0x93 0x13 0x49 0xd3 0x93 0x26 0x89 0x29 0x53 0x13 0x49 0x09 0x62 0x9a 0x8a 0x9a 0x62 0x62 0x8a 0x92 0x62 0xb2 0xc2 0x62 0x92 0xa2 0xb2 0x62 0x62 0x8a 0xca 0x62 0xc2 0x82 0x62 0x92 0x82 0xaa 0x62 0x62 0x82 0x52 0xba 0xb2 0x6a 0xa4 0x44 0xe9 0x2a 0xd4 0xd4 0x9a 0x15 0x49 0x13 0x49 0x93 0x09 0xd3 0x13 0x49 0xd3 0x13 0x09 0xd3 0x13 0x09 0x13 0x82 0x62 0x62 0x9a 0x82 0x62 0x82 0xba 0x62 0x82 0x9a 0x9a 0x62 0x62 0x9a 0x9a 0x62 0x9a 0xb2 0x62 0x8a 0xb2 0xb2 0x62 0x92 0xca 0x62 0x82 0x52 0xa2 0xb2 0x6a 0xa4 0x44 0xe9 0xca 0xea 0x14 0x8a 0x8a 0xa9 0x82 0x8a 0x82 0x72 0x82 0xba 0x82 0xb2 0x82 0x62 0x72 0xc5 0x82 0x82 0xaa 0x82 0xaa 0x72 0xc2 0xb2 0x8a 0x82 0xba 0x62 0xba 0xc5 0x8a 0xaa 0xa2 0xca 0xa2 0xca 0x72 0x82 0x82 0x62 0x0a 0xb1 0x0a 0xa9 0xb2 0x1a 0xd5 0xa4 0x88 0xe9 0x94 0xea 0x6a 0x15 0x15 0x29 0x53 0x13 0x53 0x27 0x53 0x27 0x09 0x82 0x62 0xa2 0x82 0x62 0x62 0x62 0x62 0x92 0xca 0x62 0xaa 0x82 0x62 0xa2 0xca 0x52 0xb2 0xa2 0x6a 0x52 0x44 0x3a 0xe5 0xa5 0x45 0x2a 0x8a 0x29 0x53 0x13 0x53 0x27 0x53 0x27 0x09 0x13 0x13 0x09 0xd3 0x13 0x09 0x53 0x62 0x92 0x82 0x8a 0xca 0x62 0x82 0x82 0x62 0x82 0x82 0x52 0xba 0x1a 0xd5 0xa4 0x88 0xe9 0x94 0xea 0x94 0xd4 0x2a 0x29 0x53 0x13 0x53 0x27 0x53 0x27 0x09 0x13 0x13 0x49 0x53 0x72 0x9a 0x62 0xaa 0x82 0x72 0x9a 0x62 0xa2 0xca 0x72 0x9a 0x62 0x62 0x62 0x62 0x62 0x62 0x52 0xb2 0x12 0x6b 0xa4 0x44 0xe9 0xca 0x8a 0x15 0x45 0xc5 0x09 0x8a 0x62 0x82 0x8a 0x62 0x82 0x92 0x62 0x0a 0xe5 0x45 0x35 0x45 0x15 0x8a 0x55 0x35 0xd5 0x7a 0xb5 0xa5 0x49 0xaa 0x6a 0xa4 0x44 0xe9 0xca 0x4a 0x55 0x6a 0x14 0x29 0x53 0x13 0x53 0xa7 0x13 0x26 0x09 0x13 0x13 0x29 0x14 0xa9 0x13 0x53 0x26 0x26 0x09 0xd3 0x53 0x26 0x26 0x13 0xc9 0x8a 0x09 0x13 0x53 0x13 0xa6 0x93 0x09 0xb2 0xaa 0x9a 0x82 0x62 0xba 0xc5 0x82 0x72 0xaa 0xaa 0xa2 0x62 0x62 0x82 0x9a 0x82 0xca 0x8a 0xca 0x62 0x62 0x62 0x0a 0xc5 0xb2 0xa9 0x82 0x82 0x6a 0xa4 0x44 0xe9 0xca 0xca 0x15 0xd5 0x14 0x89 0x89 0x15 0x89 0xa9 0x8a 0x09 0x93 0xa9 0xaa 0xa2 0x62 0x72 0xc5 0x8a 0x72 0x82 0x92 0xb2 0x62 0x5a 0x65 0x0a 0xa5 0x9a 0x1a 0x35 0x52 0x22 0x3a 0xe5 0x75 0xea 0x2a 0x14 0x29 0x53 0x13 0x53 0x53 0x13 0x26 0x09 0x13 0x13 0xa9 0x13 0x53 0x26 0x26 0x09 0xd3 0x53 0x26 0x26 0x13 0xc9 0x8a 
��bbªb��b���bb²b��b���bbºb��b���b��b�R��j�D�*�ԕI)	�	I�Iӓ&�)SI	b���bb��b��b���bb��b‚b���bb�R��j�D�*�ԚII�	�I�	�	�bb��b��b���bb��b��b���b��b�R��j�D���������r�����brł����r²���b�Ŋ��ʢ�r��b
�
��դ����j)SS'S'	�b��bbbb��b��b��R��jRD:��E*�)SS'S'		�	Sb����b��b��R�դ������*)SS'S'	ISr�b��r�b��r�bbbbbbR�k�D�ʊE�	�b��b��b
�E5E�U5�z��I�j�D��JUj)SS�&	)�S&&	�S&&Ɋ	S��	����b�łr���bb���ʊ�bbb
Ų���j�D���������	����brŊr���bZe
��5R":�u�*)SSS&	�S&&	�S&&Ɋ

As these are not UBX packets I'm back to thinking this is just junk data. As it is appearing just after startup, another idea is that this could be due to the battery slowly discharging. I've tried undervolting the whole system in controlled conditions, but no joy yet on triggering this.

Next step now we have collected some evidence, ask on the forum.

You can’t perform that action at this time.