Packaging mlc-llm

[kmn1024]

• Nuitka version, full Python version, flavor, OS, etc. as output by this exact command.

python -m nuitka --version
2.1.5
Commercial: None
Python: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]
Flavor: Debian Python
Executable: /usr/bin/python3
OS: Linux
Arch: aarch64
Distribution: Ubuntu (based on Debian) 22.04.3
Version C compiler: /usr/bin/gcc (gcc 11).

• How did you install Nuitka and Python

Within a conda environment, I did "python -m pip install nuitka"

• The specific PyPI names and versions

I am trying to package code that uses: https://github.com/mlc-ai/mlc-llm/. The setup of mlc-llm is quite elaborate: https://llm.mlc.ai/docs/install/mlc_llm.html. However, after setup, the code I'm trying to package is bare minimal (see below).

• Also supply a Short, Self Contained, Correct, Example

import time
from mlc_chat import ChatModule

cm = ChatModule(model="/home/ubuntu/new-mlc/mlc-llm/dist/mistral-mali-q4f16_1/",
                model_lib_path="/home/ubuntu/new-mlc/mlc-llm/dist/mistral-mali-q4f16_1/lib.so")
start_time = time.time()
cm._prefill("The meaning of life", decode_next_token=True)
print(time.time() - start_time)

The code is failing with:

...
[2024-04-18 21:12:29] ERROR auto_device.py:79: GPU device detection failed. Please report this issue with the output of command: ./mlc_test.bin -m mlc_chat.cli.check_device opencl
[2024-04-18 21:12:29] INFO auto_device.py:85: Not found device: opencl:0
[2024-04-18 21:12:29] INFO auto_device.py:31: Not found: No available device detected
Traceback (most recent call last):
  File "/home/ubuntu/theplatform/theplatform/mlc_test.py", line 4, in <module>
    cm = ChatModule(model="/home/ubuntu/new-mlc/mlc-llm/dist/mistral-mali-q4f16_1/",
  File "/home/ubuntu/new-mlc/mlc-llm/python/mlc_chat/chat_module.py", line 716, in __init__
    device_type = self.device.device_type
AttributeError: 'NoneType' object has no attribute 'device_type'

which comes from: https://github.com/mlc-ai/mlc-llm/blob/7d3f34e686ee64ffd207595043656ff88360d51f/python/mlc_llm/support/auto_device.py#L54-L59

So I suspect the sys.executable call is incompatible with Nuitka, but I don't know how to fix this.

When it succeeds, the output looks like:

python mlc_test.py 
[2024-04-18 16:24:02] INFO auto_device.py:85: Not found device: cuda:0
...
[2024-04-18 16:24:05] INFO auto_device.py:76: Found device: opencl:0
[2024-04-18 16:24:05] INFO auto_device.py:76: Found device: opencl:1
[2024-04-18 16:24:05] INFO auto_device.py:33: Using device: opencl:0
[2024-04-18 16:24:05] INFO chat_module.py:370: Using model folder: /home/ubuntu/new-mlc/mlc-llm/dist/mistral-mali-q4f16_1
[2024-04-18 16:24:05] INFO chat_module.py:371: Using mlc chat config: /home/ubuntu/new-mlc/mlc-llm/dist/mistral-mali-q4f16_1/mlc-chat-config.json
...
arm_release_ver: g13p0-01eac0, rk_so_ver: 3
arm_release_ver of this libmali is 'g6p0-01eac0', rk_so_ver is '7'.
0.8314714431762695

• Provide in your issue the Nuitka options used

python -m nuitka --follow-imports mlc_test.py

Nuitka-Options: Used command line options: --follow-imports mlc_test.py
Nuitka: Starting Python compilation with Nuitka '2.1.5' on Python '3.10' commercial grade 'not installed'.
Nuitka-Plugins:WARNING: anti-bloat: Undesirable import of 'IPython' in 'tvm.script.highlight' (at '/home/ubuntu/new-mlc/relax/python/tvm/script/highlight.py:85') encountered. It may slow down compilation.
Nuitka-Plugins:WARNING:     Complex topic! More information can be found at https://nuitka.net/info/unwanted-module.html
Nuitka-Plugins:WARNING: anti-bloat: Undesirable import of 'IPython' in 'tvm.script.highlight' (at '/home/ubuntu/new-mlc/relax/python/tvm/script/highlight.py:85') encountered. It may slow down compilation.
Nuitka-Plugins:WARNING:     Complex topic! More information can be found at https://nuitka.net/info/unwanted-module.html
Nuitka-Plugins:WARNING: anti-bloat: Undesirable import of 'numba' in 'torch.testing._internal.common_cuda' (at '/home/ubuntu/.local/lib/python3.10/site-packages/torch/testing/_internal/common_cuda.py:33') encountered. It may slow down compilation.
Nuitka-Plugins:WARNING:     Complex topic! More information can be found at https://nuitka.net/info/unwanted-module.html
Nuitka-Plugins:anti-bloat: Not including 'PIL.ImageQt' automatically in order to avoid bloat, but this may cause: PIL will not be able to create Qt image objects.
Nuitka: Completed Python level compilation and optimization.
Nuitka: Generating source code for C backend compiler.
Nuitka: Running data composer tool for optimal constant value handling.                                                                                            
Nuitka: Running C compilation via Scons.
Nuitka-Scons: Backend C compiler: gcc (gcc 11).
Nuitka-Scons: Slow C compilation detected, used 360s so far, scalability problem.
Nuitka-Scons: Running gcc -o module.rdflib.namespace._SDO.o -c -std=c11 -fvisibility=hidden -fwrapv -pipe -fpartial-inlining -ftrack-macro-expansion=0 -Wno-deprecated-declarations -fno-var-tracking -Wno-misleading-indentation -fcompare-debug-second -fno-lto -O2
Nuitka-Scons: -D__NUITKA_NO_ASSERT__ -DPy_NO_ENABLE_SHARED -D_NUITKA_STATIC_LIBPYTHON -D_NUITKA_USE_UNEXPOSED_API -D_NUITKA_CONSTANTS_FROM_INCBIN -D_NUITKA_FROZEN=0 -D_NUITKA_EXE -D_NUITKA_FILE_REFERENCE_ORIGINAL_MODE -D_NUITKA_PLUGIN_MULTIPROCESSING_ENABLED=1
Nuitka-Scons: -I~/.local/lib/python3.10/site-packages/nuitka/build/inline_copy/zlib -I/usr/include/python3.10 -I. -I~/.local/lib/python3.10/site-packages/nuitka/build/include -I~/.local/lib/python3.10/site-packages/nuitka/build/static_src
Nuitka-Scons: -I~/.local/lib/python3.10/site-packages/nuitka/build/inline_copy/libbacktrace module.rdflib.namespace._SDO.c took 747.71 seconds
Nuitka-Scons: Slow C compilation detected, used 360s so far, scalability problem.
Nuitka-Scons: Running gcc -o module.sympy.polys.polyquinticconst.o -c -std=c11 -fvisibility=hidden -fwrapv -pipe -fpartial-inlining -ftrack-macro-expansion=0 -Wno-deprecated-declarations -fno-var-tracking -Wno-misleading-indentation -fcompare-debug-second -fno-lto -O2
Nuitka-Scons: -D__NUITKA_NO_ASSERT__ -DPy_NO_ENABLE_SHARED -D_NUITKA_STATIC_LIBPYTHON -D_NUITKA_USE_UNEXPOSED_API -D_NUITKA_CONSTANTS_FROM_INCBIN -D_NUITKA_FROZEN=0 -D_NUITKA_EXE -D_NUITKA_FILE_REFERENCE_ORIGINAL_MODE -D_NUITKA_PLUGIN_MULTIPROCESSING_ENABLED=1
Nuitka-Scons: -I~/.local/lib/python3.10/site-packages/nuitka/build/inline_copy/zlib -I/usr/include/python3.10 -I. -I~/.local/lib/python3.10/site-packages/nuitka/build/include -I~/.local/lib/python3.10/site-packages/nuitka/build/static_src
Nuitka-Scons: -I~/.local/lib/python3.10/site-packages/nuitka/build/inline_copy/libbacktrace module.sympy.polys.polyquinticconst.c took 379.58 seconds
Nuitka-Scons: Backend linking program with 7434 files (no progress information available for this stage).
Nuitka-Scons:WARNING: You are not using ccache, re-compilation of identical code will be slower than necessary. Use your OS package manager to install it.
Nuitka: Keeping build directory 'mlc_test.build'.
Nuitka: Successfully created 'mlc_test.bin'.

• Consider getting commercial support

Yes! I am developing on an edge device (Orange Pi 5), and using multiple AI inference platforms (MLC above, ONNX, and RKNPU). I am certainly considering the "Full Care package", if I can know more about the process, probability of success, and price.

[kayhayen]

@KRRT7 currently support for -m is hard coded for a few packages in Nuitka, we may either replace that code with something that just imports into the running binary and runs it as if it was a main call, or we can make something that allows plugins to sort of contribute these kinds of codes:

[kayhayen]

@kmn1024 What is open for this one to be happy?

[kmn1024]

That code path turned out to be unnecessary, so I am completely unblocked now.