Xcelium: Fatal Python error: Py_Initialize: Unable to get the locale encoding

[martijnbastiaan]

I'm currently contracted at a client that uses Xcelium to simulate their designs. After the design got fairly sizable, we started running into an issue where libcocotbvpi_ius.so:vlog_startup_routines_bootstrap fail run. I've "solved" the issue now, so I'm documenting it here such that it's googlable. Hopefully that prevents the next poor soul having to dig through Cython code 😄.

The full error message was:

WORKLIB.XXXXXXXXXX:STRUCT (architecture):
	streams: 10, words: 133925
     -.--ns INFO     cocotb.gpi                         ..mbed/gpi_embed.cpp:110  in set_program_name_in_venv        Using Python virtual environment interpreter at /data/home/martijn.bastiaan/code/xxxxxx/cocotb/env36/bin/python
Fatal Python error: Py_Initialize: Unable to get the locale encoding
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007f1248321f00 (most recent call first):
ImportError: No module named site
xmelab: *F,INTERR: INTERNAL EXCEPTION
-----------------------------------------------------------------
The tool has encountered an unexpected condition and must exit.
Contact Cadence Design Systems customer support about this
problem and provide enough information to help us reproduce it,
including the logfile that contains this error message.
  TOOL:	xmelab(64)	21.03-s008
  HOSTNAME: cn01
  OPERATING SYSTEM: Linux 3.10.0-1160.25.1.el7.x86_64 #1 SMP Wed Apr 28 21:49:45 UTC 2021 x86_64
  MESSAGE: Unexpected signal #6, while trying to run bootstrap function vlog_startup_routines_bootstrap
-----------------------------------------------------------------
csi-xmelab - CSI: Cadence Support Investigation, sending details to /data/home/martijn.bastiaan/code/xxxxxxxxxxx/cocotb/xmelab_682.err
csi-xmelab - CSI: investigation complete, send /data/home/martijn.bastiaan/code/xxxxxxxxxxxxx/cocotb/xmelab_682.err to Cadence Support
xrun: *E,ELBERR: Error during elaboration (status 253), exiting.
TOOL:	xrun(64)	21.03-s008: Exiting on Oct 04, 2021 at 15:31:20 CEST  (total: 00:00:05)
make[1]: *** [results.xml] Error 1
make[1]: Leaving directory `/data/home/martijn.bastiaan/code/xxxxxxxxx/cocotb'
make: *** [sim] Error 2

Curiously, the newest Python containing the Unable to get the locale encoding error message was Python 3.6. Apparently, >3.6 changed or even removed this logic - turns out upgrading Python solved our issue!

[martijnbastiaan]

So in order to mark this thread as solved:

Upgrading our Python installation from 3.6 to 3.8 mitigated this error message. I'm still not entirely sure what the actual underlying issue is, unfortunately.

[ktbarrett]

I sincerely doubt this has anything to do with limitations of Python version, but is in fact an environment problem. Upgrading your Python worked-around the problem by installing cocotb into a version of Python equivalent to the version that ships with Xcelium, so that setting PYTHONHOME to the external Python where you have cocotb installed doesn't fail (though it very much works by accident).

There are a bunch more questions to ask about why it started to fail (did you upgrade Xcelium along the way?), and why it's failing at all. We added a bunch of additional logic to prevent the incorrect loading of libpython in these simulators. It might be that your environment isn't set up correctly and it's using the default behavior of loading whatever libpython it can find (it's usually the one bundled with the simulator) instead of the one specified using LIBPYTHON_LOC (You can check this by running cocotb-config --libpython). Or if Xcelium has finally done what I have feared and loaded libpython before cocotb does... There isn't a way to combat that.

We have had a bunch of problems with Xcelium and environment configuration (they just can't get it right). And I'm going to recommend users start installing cocotb into the Python included in their Xcelium installs, as it seems to be including relatively modern versions of Python.

[martijnbastiaan]

The error is back, sometimes. Python 3.8 yields some more useful debugging info:

Python path configuration:
  PYTHONHOME = '/usr'
  PYTHONPATH = '../SOFTWARE/XXXXXX:'
  program name = 'python3'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = '/data/home/XXXXXXX/work/workspace/XXXXX/cocotb/env/bin/python3'
  sys.base_prefix = '/usr'
  sys.base_exec_prefix = '/usr'
  sys.executable = '/data/home/XXXXXXXXXX/work/workspace/XXXXXX/cocotb/env/bin/python3'
  sys.prefix = '/usr'
  sys.exec_prefix = '/usr'
  sys.path = [
    '../SOFTWARE/XXXXXXX',
    '',
    '/usr/lib64/python38.zip',
    '/usr/lib64/python3.8',
    '/usr/lib64/python3.8/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'
[..]
mkdir -p sim_build

set -o pipefail; \

MODULE=simple_test TESTCASE= TOPLEVEL=XXXXXXX GPI_EXTRA=/data/home/XXXXX/.local/lib/python3.6/site-packages/cocotb/libs/libcocotbvhpi_ius.so:cocotbvhpi_entry_point TOPLEVEL_LANG=verilog \

/cadappl/ictools/cadence_xcelium/21.03.008/tools/bin/xrun -timescale 1ns/1ps \

-v93 -V200X -INCDIR ../RTL/include  -licqueue -64 -xmlibdirpath sim_build -plinowarn  -top XXXXXX -loadvpi /data/home/XXXXXX/.local/lib/python3.6/site-packages/cocotb/libs/libcocotbvpi_ius.so:vlog_startup_routines_bootstrap  -access +rwc

[martijnbastiaan]

Oh wait a minute, this doesn't make all that much sense.

• The error message is clearly generated by 3.8, as 3.6 does not include the environment info.
• It is using locally installed stuff (i.e., from $HOME/.local/lib)
• /usr/lib64/python38.zip does not exist. The environment was made with /opt/rh/rh-python38/root/usr/bin/python3.8 -m venv env.

[martijnbastiaan]

Alright, this particular user had installed packages in their "user" environment. I don't see how that could affect things in a virtual env :|, but removing it worked.

[ktbarrett]

I'm pretty sure I've had issues with that as well. I never figured out why it messed it up. Glad to hear you figured it out.

[martijnbastiaan]

Thanks. Python 3.6 still doesn't work on any account, so I'm hoping the error will be back on 3.8 - given that it yields much nicer debugging info. I'll report back if it ever happens 🤞.