Advanced mode 1 Failing with Pyenv environment on Pyarmor(6.2.1)

[e0lithic]

Hi Again,

Environment

Python - 3.6.9
OS - Ubuntu 20.04
Pyarmor - 6.2.0

Project structure

mypkg
├── bar.py
├── foo.py
└── init.py

Configuration files

I am using the child project structure to demonstrate a use-case for a large package. Here we are just obfuscating everything else with restrict mode 4 and using restrict mode 1 for __init__.py. As is recommended for a package.

# Content .pyarmor_config
{ 
  "version": "2.0",
  "name": "mypkg",
  "title": "mypkg",
  "src": ".",
  "is_package": 1,
  "manifest": "include __init__.py",
  "entry": "__init__.py",
  "output": "dist",
  "runtime_path": null,
  "restrict_mode": 1,
  "obf_code": 2,
  "obf_mod": 1,
  "wrap_mode": 1,
  "advanced_mode": 0,
  "bootstrap_code": 1,
  "cross_protection": 1,
  "plugins": null,
  "platform": null,
  "package_runtime": 1,
  "enable_suffix": 0,
  "license_file": null,
  "build_time": 1589994512.6738808
}

# Content .pyarmor_config_pkg
{
  "version": "2.0",
  "name": "mypkg",
  "title": "mypkg",
  "src": ".",
  "is_package": 1,
  "manifest": "global-include *.py, exclude __init__.py",
  "entry": "",
  "output": "dist",
  "runtime_path": null,
  "restrict_mode": 4,
  "obf_code": 2,
  "obf_mod": 1,
  "wrap_mode": 1,
  "advanced_mode": 0,
  "bootstrap_code": 1,
  "cross_protection": 1,
  "plugins": null,
  "platform": null,
  "package_runtime": 1,
  "enable_suffix": 0,
  "license_file": null,
  "build_time": 1589994515.6668053
}

Command :

pyarmor build -B
pyarmor build --no-runtime -B .pyarmor_config_pkg

The build completes successfully.

Behaviour/Expectation

If we try to import the obfuscated mykg then we get the following error

Python 3.6.9 (default, May 20 2020, 20:21:06) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import mypkg
XXX lineno: 8, opcode: 228
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "</media/e0lithic/D/GameX/pyarmor_test/mypkg/dist/mypkg/__init__.py>", line 3, in <module>
  File "<frozen mypkg>", line 8, in <module>
SystemError: unknown opcode

If we just turn advanced_mode 0 in both configuration files and rerun the build. I am able to import the package.

I read the documentation but the python 3.6 limitation was only mentioned for advanced_mode 2 / a.k.a super mode. Is the support for advanced mode 1 on python 3.6 also depreciated? Kindly let me know if I am missing something.

Thanks

[jondy]

Does it work just obfuscating one simple script with advanced 1? It should work in Ubuntu /Python 3.6.

[e0lithic]

I tried it on a single file and the issue persists.

[jondy]

Could you upload the file libpython3.6m.so in your Ubuntu20.04 to "https://tmpfiles.org/" and send me a link? I need check it.

[e0lithic]

Here is the download link for the same. Just for additional information I am using pyenv and instead of a clean installation, I have upgraded to Ubuntu 20.04 from 18.04

[e0lithic]

Since you mentioned the involvemenet of libpython for the .so files. I think this could very well be the case of pyenv not being able to link the libpython library as mentioned in the issue. I will try enabling shared libraries and then give it a go.

[e0lithic]

Tried it , but the problem still persists.

[e0lithic]

Ok, I tried to create a virtualenv which is using the python from the root folders instead of pyenv's version. And in such a case, I am able to run the tool perfectly. It seems that when I using using a python installed by pyenv, that the library stops working. Is there a recommended way to use pyarmor with pyenv?

[jondy]

I know a little about pyenv. But if pyenv don't use CPython library, it's difficult to work with Pyarmor. If pyenv can be configured with CPython python library, it should be OK.

[e0lithic]

I have tried using env PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.6.9. Which should have ideally allowed .so to reference the cpython libraries as mentioned in the referenced issue above, but am still unable to get it running. Is there a workaround or am I missing something?

[jondy]

What's the error message to run pyarmor? Or it's building OK, only advanced mode 1 failed as before?

[e0lithic]

Command for build

pyarmor obfuscate --advanced 1 --exact foo.py

Build happens successfully.

Output

On running python foo.py in dist

XXX lineno: 1, opcode: 228
Traceback (most recent call last):
  File "<foo.py>", line 3, in <module>
  File "<frozen foo>", line 1, in <module>
SystemError: unknown opcode

Tried this with both versions of python installs on pyenv

• pyenv install 3.6.9
• env PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.6.9

[jondy]

It seems that the python library used by pyenv isn't patched by pyarmor correctly. What the output with extra "-d" to run the obfuscated foo.py?

cd dist/
python -d foo.py

If it complains of the library patched failed, find the dynamic library used by pyenv, and upload it to the temporary site.

[e0lithic]

This is the output :

Initialize python patcher failed (1519)
Traceback (most recent call last):
  File "<foo.py>", line 3, in <module>
RuntimeError: The script is obfuscated in super mode but this library doesn't support

However, I have double checked that I have used advanced mode 1 (not 2)

[jondy]

The key problem is patch failed. I think pyenv uses different python dynamic library from the host python. You can start python interpreter in pyenv and check the dynamic libraries used by this process, I need check it.

[e0lithic]

Since I am not explicitly setting the LD_LIBRARY_PATH, I presume the libraries would be picked from the sys.path which :

>>> import sys
>>> sys.path
['', '/home/e0lithic/.pyenv/versions/3.6.9/lib/python36.zip', '/home/e0lithic/.pyenv/versions/3.6.9/lib/python3.6', '/home/e0lithic/.pyenv/versions/3.6.9/lib/python3.6/lib-dynload', '/home/e0lithic/.local/lib/python3.6/site-packages', '/home/e0lithic/.pyenv/versions/3.6.9/lib/python3.6/site-packages']

Do you want me to upload the /home/e0lithic/.pyenv/versions/3.6.9 folder?

[jondy]

No. What I care about only dynamic library. You can got it by this way:

1. Start python interpreter from pyenv
2. Get the pid of this interpreter by ps
3. Check the dynamic library used by this process

cat /proc/${pid}/maps | grep libpython*.so

Just upload this .so file.

[e0lithic]

Thanks for the detailed instruction. Link to the .so file

[jondy]

@e0lithic Could you upload it again? I try to download it just now, but it's gone.

[e0lithic]

@jondy I have uploaded it again. Here's the link. Let me know if you are able to download it otherwise I will share a google drive link.

[jondy]

I have downloaded this one, It's OK, thanks.

[e0lithic]

@jondy Do you see a fix/workaround in near time frame for this. Or should I just temporary remove pyenv from my poetry env to make a couple of releases needed for my team.

[jondy]

@e0lithic I'll check it and feedback you tomorrow.

[jondy]

@e0lithic I build a temporary one here, just replace the old one with it and check it work or not.

If it still doesn't work, run the obfuscated scripts with -d, and show the output message here.

[e0lithic]

Tried it . Got the following result :

Initialize python patcher failed (66889)
Traceback (most recent call last):
  File "<foo.py>", line 3, in <module>
RuntimeError: The script is obfuscated in advanced mode but the runtime library doesn't support

[jondy]

Try this one
md5sum: 0cc7b9f15dc7c4f0b442760f00a96f6f

[e0lithic]

Here is the output:

Initialize python patcher failed (66926)
Traceback (most recent call last):
  File "<foo.py>", line 3, in <module>
RuntimeError: The script is obfuscated in advanced mode but the runtime library doesn't support

[jondy]

Could you upload this file /lib/x86_64-linux-gnu/libm.so.6 in your Ubuntu? It uses GLIBC-2.29 which missing in my ubuntu.

[e0lithic]

Heres the link

[jondy]

Try this one
md5sum: 7bb8bbedc3f2bb2ea1e64ea4be5fa2d3

Hope it works

[e0lithic]

Got the following error now :

Initialize python patcher failed (66932)
Traceback (most recent call last):
  File "<foo.py>", line 3, in <module>
RuntimeError: The script is obfuscated in advanced mode but the runtime library doesn't support

[jondy]

Try this one
md5sum: 9fe5857ffe3b88d34b73b1179bbd8771

[e0lithic]

It worked. 👍

[jondy]

Nice, it'll be released with next version v6.2.2

[e0lithic]

Closing the issue as this has been fixed in the 6.2.2 release. Thanks for the great support. I will also update the issue title to better reflect the issue.