Segmentation faults

[benenaes]

On:
PyArmor 3.9.9 and PyArmor 4.3.3
Ubuntu 16.04
x86 architecture

What happens:

• We obfuscate our entire codebase
• When we run integration (end-to-end) tests against it, we get a segmentation fault (not often that I have seen this in Python !)
• When a simple log line (or a print statement) is added in the right module, the segmentation fault is gone
• Removing the log line, results in a segmentation fault again
• Without code obfuscation, no log lines need to be added: the integration tests run smoothly

This has happened before multiple times. Actually, when code changes over time in the specific modules, the log lines can be removed again.

I cannot reproduce this behaviour on example modules, unfortunately. I have the impression that the generation of some specific byte codes, results in segfaults. Changing or adding an arbitrary line in the module seems to fix the issue, but I fear that this way we need to have very (almost over-the-top) testing.

Sorry that this ticket is a bit vague and I cannot provide example files, but I just wonder if other people have seen this behaviour as well ?

[jondy]

I have been obfuscated test module of Python self, and run it, refer to

https://github.com/dashingsoft/pyarmor/blob/master/tests/python-test.sh

All the test cases about settrace/setprofile will be failed, because pyarmor don't allow it.

There are some failed test but not fixed yet

• Python26
  Hangup: test_urllib2_localnet (linux_x86_64)

• Python27
  Hangup: test_argparse (win32)

• Python32 (linux_x86_64)
  Hangup: test_urllib2_localnet

• Python36 (linux_x86_64)
  Segmentation Fault:
  test_logging.test_compute_rollover_weekly_attime
  test_platform.test_sys_version
  test_struct.test_half_float
  test_time.test_FromSecondsObject

[benenaes]

We don't have issues with the test module itself (test scripts are not obfuscated), but with the obfuscated code that is getting tested. We just run our end-to-end tests to see if everything is still working after obfuscation. Or maybe I misunderstood what you explained ?

[drewm1980]

I also just hit a segfault in obfuscated code:

(gdb) run camera_server.py
Starting program: /home/tulip3d/miniconda3/bin/python camera_server.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
0x00005555556f688e in PyEval_EvalFrameEx ()
(gdb) bt
#0 0x00005555556f688e in PyEval_EvalFrameEx ()
#1 0x000055555570103d in PyEval_EvalCodeEx ()
#2 0x0000555555701f5c in PyEval_EvalCode ()
#3 0x000055555575dd3b in exec_code_in_module ()
#4 0x000055555562854c in PyImport_ExecCodeModuleWithPathnames ()
#5 0x00007ffff449cc81 in ?? ()
from /home/tulip3d/tulip3d_release_beta-10/_pytransform.so
#6 0x000055555569f4bf in PyCFunction_Call ()
#7 0x00005555556f67cc in PyEval_EvalFrameEx ()
#8 0x000055555570103d in PyEval_EvalCodeEx ()
#9 0x0000555555701f5c in PyEval_EvalCode ()
#10 0x000055555575e454 in run_mod ()
#11 0x000055555575fac1 in PyRun_FileExFlags ()
#12 0x000055555575fcde in PyRun_SimpleFileExFlags ()
#13 0x00005555557603c2 in Py_Main ()
#14 0x000055555562b871 in main ()

It's in pytransform.so

Running in strace to get more clues about where it happened:
...
open("product.key", O_RDONLY) = 4
fstat(4, {st_mode=S_IFREG|0664, st_size=140, ...}) = 0
fstat(4, {st_mode=S_IFREG|0664, st_size=140, ...}) = 0
lseek(4, 0, SEEK_SET) = 0
read(4, "j\332\315\36\316\244\4\344\7\215\230eB\2020\234\345\26`\31\376\274L\374\236\271\3\256V\v%~"..., 140) = 140
lseek(4, 140, SEEK_SET) = 140
close(4) = 0
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_ACCERR, si_addr=0x7f08df270ba0} ---
+++ killed by SIGSEGV (core dumped) +++
Segmentation fault (core dumped)

So it looks like it happens shortly after successfully loading "product.key".

I did not observe segfaults until today. The timing between me and @benenaes is not a coincidence; we're on different teams at the same company and had a conversation about pyarmor.

I do not observe the segfaults when running the code on the same machine that performed the obfuscation, but when running the obfuscated files on another machine.

Obfuscation machine:
$ cat /etc/issue
Ubuntu 16.04.3 LTS \n \l
$ pyarmor --version
Pyarmor Version 4.4.3

Remote machine observing segfaults:
tulip3d@tars:~/tulip3d_release_beta-10$ cat /etc/issue
Ubuntu 16.04.4 LTS \n \l
$ pyarmor --version
Pyarmor Version 4.5.1
(Repeated with pyarmor uninstalled from remote machine; no change)

Another remote machine that does ~not segfault:
$ cat /etc/issue
Ubuntu 18.04.1 LTS \n \l
$ pyarmor --version
Pyarmor Version 4.1.4

[drewm1980]

We're on linux x86-64. If you're not able to reproduce this there, do you have a build of the closed source .so file with debug symbols, so we can try to get a better stack trace for you?

[drewm1980]

So I tried a bunch of old versions of pyarmor from pypi with no luck:

4.5.1 obfuscated code segfaults on tars
4.5.0 no _pytransform
4.4.3 obfuscated code segfaults on tars
4.4.2 obfuscated code segfaults on tars
4.4.1 obfuscated code segfaults on tars
4.4.0 obfuscated code segfaults on tars
4.3.4
4.3.3
4.3.2 MemoryError
4.3.1
4.3.0
4.2.3
4.2.2 MemoryError
4.2.1
4.1.4
4.1.3
4.1.2
4.1.1
4.0.3
4.0.2
4.0.1 MemoryError
3.9.9 MemoryError

Then realized that the machine that was segfaulting (tars) was on python 3.5.5, whereas the obfuscation machine, and the other working machine, were both on 3.6.6. Upgrading the segfaulting machine to 3.6.6 appears to have cleared it up (fingers crossed).

Are you obfuscating and running with different versions of python @benenaes ?

[jondy]

We don't have issues with the test module itself (test scripts are not obfuscated), but with the obfuscated code that is getting tested. We just run our end-to-end tests to see if everything is still working after obfuscation. Or maybe I misunderstood what you explained ?

What I want to explain is that all the features of Python in the test module are tested pass, except Python 3.6 on linux_x86_64, there is a little trouble, there are four test cases failed. I have not fixed it.

[jondy]

Yes, the python version used to obfuscate script must be same as the running.

@drewm1980 If @benenaes use the same python version and still raise segfaulting, I'll build _pytransform.so with debug information for you. Thanks.

[jondy]

In pyarmor v4.6.0, I fix an issue which will cause obfuscated scripts crash in Python 3.6 if the code object includes a special instruction "EXTENDED_ARG 0". Maybe it's the root cause for your case.