Character Encoding Error in Collector PacketBatchInserter.py

[shackrat]

I have this setup and largely working well in a Ubuntu 20.04 container. Having one issue where it seems the collector is unable to process the UTF-8 characters coming from the aprsc server.

The error is as follows:

root@aprs:~# ~/trackdirect/server/scripts/collector.sh trackdirect.ini 0
/usr/local/lib/python2.7/dist-packages/autobahn/wamp/auth.py:43: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release.
  from cryptography.hazmat.primitives.kdf.pbkdf2 import PBKDF2HMAC
2022-08-16 14:43:19,682 - trackdirect - WARNING - Starting (Collecting from 192.168.9.90:10152 using N2RWE-S and 10887)
2022-08-16 14:43:19,686 - aprslib.IS - INFO - Attempting connection to 192.168.9.90:10152
2022-08-16 14:43:19,686 - aprslib.IS - INFO - Connected to ('192.168.9.90', 10152)
2022-08-16 14:43:19,711 - aprslib.IS - INFO - Sending login information
2022-08-16 14:43:19,711 - aprslib.IS - INFO - Login successful
2022-08-16 14:43:20,536 - trackdirect.collector.PacketBatchInserter - ERROR - 'ascii' codec can't encode character u'\xe9' in position 85: ordinal not in range(128)
Traceback (most recent call last):
  File "/root/trackdirect/server/trackdirect/collector/PacketBatchInserter.py", line 184, in _insertIntoPacketTable
    "(%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)", x) for x in datePacketTuples)
  File "/root/trackdirect/server/trackdirect/collector/PacketBatchInserter.py", line 184, in <genexpr>
    "(%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)", x) for x in datePacketTuples)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 85: ordinal not in range(128)

Locales appear to be set properly for UTF8.

root@aprs:~# locale
LANG=en_US.UTF-8
LANGUAGE=C.UTF-8
LC_CTYPE="C.UTF-8"
LC_NUMERIC="C.UTF-8"
LC_TIME="C.UTF-8"
LC_COLLATE="C.UTF-8"
LC_MONETARY="C.UTF-8"
LC_MESSAGES="C.UTF-8"
LC_PAPER="C.UTF-8"
LC_NAME="C.UTF-8"
LC_ADDRESS="C.UTF-8"
LC_TELEPHONE="C.UTF-8"
LC_MEASUREMENT="C.UTF-8"
LC_IDENTIFICATION="C.UTF-8"
LC_ALL=C.UTF-8

It doesn't matter if I use my own aprsc server or one of the T2 servers. I get the same error so I can safely rule out the server as being the issue.

Unfortunately I am not a Python expert. Could it be that something is be missing in the Python2 setup instructions related to character set?

[shackrat]

This was solved. In a normal installation of Postgres the default encoding is set to ASCII. This was raising issues with mogrify which is tied to the character set of the database.

I had to drop the database then alter the CREATE DATABASE statement in the instructions to add ENCODING 'UTF8'

The final statement was:

CREATE DATABASE trackdirect ENCODING 'UTF8';

After that the collectors work.

[nathantcampbell]

Hi, Does this now need to be done now on the main branch or just on the python3 port? If on main, can you please modify README.md to reflect the new step of including the encoding scheme during database creation? Nathan Campbell

…

On Tue, Aug 16, 2022 at 8:46 AM shackrat ***@***.***> wrote: This was solved. In a normal installation of Postgres the default encoding is set to ASCII. This was raising issues with mogrify which is tied to the character set of the database. I had to drop the database then alter the CREATE DATABASE statement in the instructions to add ENCODING 'UTF8' The final statement was: CREATE DATABASE trackdirect ENCODING 'UTF8'; After that the collectors work. — Reply to this email directly, view it on GitHub <#26 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AF44PQKWGGR4H4RLQ2IH6F3VZOZV5ANCNFSM56WFCVPQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[shackrat]

It's the main branch. It appears that a fresh install of Postgres using the instructions will default to ASCII as the character set. It's necessary to specify encoding when creating the database to get around this. I've created a PR to update the readme as you asked.

Thanks!

[nathantcampbell]

Perfect, thanks so much! Nathan

…

On Wed, Aug 17, 2022 at 9:57 PM shackrat ***@***.***> wrote: It's the main branch. It appears that a fresh install of Postgres using the instructions will default to ASCII as the character set. It's necessary to specify encoding when creating the database to get around this. I've created a PR to update the readme as you asked. Thanks! — Reply to this email directly, view it on GitHub <#26 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AF44PQNHKEKVPGDVEES2I2DVZW7CVANCNFSM56WFCVPQ> . You are receiving this because you commented.Message ID: ***@***.***>