Hiding python code inside the native image

[der-eugen]

Dear friends!

I am looking for a possibility to "hide" python code inside the native image. I have a python app, which is contained of several *.py files, which shouldn't be seen by user.

Are there any settings / options / approaches to make it possible?

I am happy for every hint and link! Thank you a lot in advance! :-)

[msimacek]

Yes, our standalone module can pack the python code into a single executable file.

[der-eugen]

Hi again! Thank you very much for you reply! It sounds like a really cool feature!

I've read that documentation and tried to execute:

graalpy -m standalone binary --module myscript.py --output mybinary

and I got this:
/home/xyz/.sdkman/candidates/java/current/bin/graalpy: No module named standalone⏎

graalpy --version
says: GraalVM Python 3.8.5 (GraalVM CE Native 22.3.0)

java -version
openjdk version "17.0.5" 2022-10-18 LTS
OpenJDK Runtime Environment GraalVM 22.3.0 (build 17.0.5+8-LTS)
OpenJDK 64-Bit Server VM GraalVM 22.3.0 (build 17.0.5+8-LTS, mixed mode, sharing)

I'v tried different ways to install standalone module, but I wasn't able to do it right... I was able to install numpy for a test using ginstall as numpy is listed by ginstall as a known package... but standalone wasn't listed.

Please pardon me for my poor python knowledge... I seem to be in the dark...

Is standalone some standard python module? or is it something graalpy specific? Should I use pip or venv or something to install standalone module? Or may be just another graalvm or graalpy version? Could you please hint me to a right direction?

Thank you very much for every hint and link!!!

Have a nice day! :-)

[msimacek]

Hi. It's a new feature in version 23.0, you need to update to get it. It's a graalpy-specific module that we ship as a part of our standard library.

[syan-cn]

Hi. It's a new feature in version 23.0, you need to update to get it. It's a graalpy-specific module that we ship as a part of our standard library.

I run this command in my container which has GraalVM 23.0.1 and GraalPy 3.10.8. I got the binary file successfully, but it also raised an error in step [8/8]: .

sh-5.1# graalpy -m standalone binary --module py_binary_test.py --output py_binary_test
WARNING: package com.oracle.truffle.api.impl.asm.tree.analysis not in org.graalvm.truffle
======================================================================
GraalVM Native Image: Generating 'py_binary_test' (executable)...
====================================================================== 
[1/8] Initializing...            (21.4s @ 0.29GB)
 Java version: 20.0.2+9, vendor version: GraalVM CE 20.0.2+9.1
 Graal compiler: optimization level: 2, target machine: x86-64-v3
 C compiler: gcc (redhat, x86_64, 11.3.1) 
 Garbage collector: Serial GC (max heap size: 80% of RAM) 
 5 user-specific feature(s) 
 - com.oracle.graal.python.BouncyCastleFeature 
 - com.oracle.svm.truffle.ICU4JFeature: Provides support for ICU4J library.
 - com.oracle.svm.truffle.TruffleBaseFeature: Provides base support for Truffle 
 - com.oracle.svm.truffle.TruffleFeature: Enables compilation of Truffle ASTs to machine code
 - org.graalvm.home.HomeFinderFeature: Finds GraalVM paths and its version number
[2/8] Performing analysis...  [*******]                    (259.7s @ 3.37GB)
  39,191 (97.07%) of 40,376 types reachable
  58,021 (63.39%) of 91,533 fields reachable
 178,366 (73.08%) of 244,082 methods reachable
  27,479 (11.26%) of 244,082 methods included for runtime compilation
  11,147 types,   105 fields, and 3,357 methods registered for reflection
      64 types,    68 fields, and    94 methods registered for JNI access
       4 native libraries: dl, pthread, rt, z 
[3/8] Building universe...                      (12.0s @ 3.59GB)
[4/8] Parsing methods...      [******]                        (41.6s @ 2.44GB) 
[5/8] Inlining methods...     [****]                         (20.9s @ 2.34GB)
[6/8] Compiling methods...    [******************]                (356.7s @ 4.01GB)
[7/8] Layouting methods...    [[7/8] Layouting methods...    [******]                 (40.7s @ 4.66GB) 
[8/8] Creating image...       [*******Error: Image build request for 'py_binary_test' (pid: 1204, path: /app) failed with exit status 137
Traceback (most recent call last): 
  File "<string>", line 1, in <module>
  File "/opt/graalvm-community-java20/languages/python/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/graalvm-community-java20/languages/python/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/opt/graalvm-community-java20/languages/python/lib/graalpy23.0/modules/standalone/__main__.py", line 332, in <module>
    main(sys.argv[1:])
  File "/opt/graalvm-community-java20/languages/python/lib/graalpy23.0/modules/standalone/__main__.py", line 325, in main
    build_binary(targetdir, jc, java_file, ni, parsed_args)
  File "/opt/graalvm-community-java20/languages/python/lib/graalpy23.0/modules/standalone/__main__.py", line 226, in build_binary
    subprocess.check_call(cmd)
  File "/opt/graalvm-community-java20/languages/python/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/opt/graalvm-community-java20/bin/native-image', '--language:python', '-H:-CopyLanguageResources', '-o', '/app/py_binary_test', 'Py2BinLauncher']' returned non-zero exit status 137. 

And when I did the same process on my Mac, it always used more memory and failed with the information "out of memory" in step [7/8].

The worst problem that I couldn't understand was the time was longer when I run the binary file compared with the .py file. For the sample code:

sh-5.1# cat py_binary_test.py
import time
if __name__ == '__main__':
    print('graalpy23.0.1')
    start_time = time.time()
    s = 0
    loop = 500
    for i in range(loop):
        for j in range(loop):
            for k in range(loop):
                s += 1
    print(f"total amount: {s}")
    print(f"elaspsed time: {time.time() - start_time}s")

The results were:

sh-5.1# graalpy py_binary_test.py
graalpy23.0.1
total amount: 125000000
elaspsed time: 2.948000192642212s

sh-5.1# ./py_binary_test
graalpy23.0.1
total amount: 125000000
elaspsed time: 5.148000001907349s

Those two time might fluctuate when I tested, but it seemed that the binary file was always slower.

[syan-cn]

These problems seem to be dealt with in the lastest dev version 23.1.0. See #350 (comment)