Cannot use transformers package #59

Forsworns · 2023-07-09T08:50:58Z

Thanks for your great work on this plugin.

My env:

MacOS 12.5 (x86), Python 3.11, Ghidrathon master branch, Ghidra 10.4

Background

Recently I'm trying to write a script like Gepetto to analyze bytecodes with llm. The langchain pacakge seems work well with Ghidrathon. But I want to use a local llm (after all, it's free, lol). So I played with Hugging Face and try to use transformers in Ghidrathon.

Problem

But I cannot import transformers, just try this in Ghidrathon:

from transformers import AutoTokenizer

and I got a CPython error

File "/usr/local/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/transformers/data/__init__.py", line 26, in <module>
    from .metrics import glue_compute_metrics, xnli_compute_metrics
  File "/usr/local/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/transformers/data/metrics/__init__.py", line 19, in <module>
    from scipy.stats import pearsonr, spearmanr
  File "/usr/local/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/scipy/stats/__init__.py", line 608, in <module>
    from ._stats_py import *
  File "/usr/local/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/scipy/stats/_stats_py.py", line 37, in <module>
    from numpy.testing import suppress_warnings
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "/usr/local/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/jep/shared_modules_hook.py", line 134, in exec_module
    self.sharedImporter.sharedImport(fullname)
TypeError: CalledProcessError.__init__() missing 1 required positional argument: 'cmd'

I guess it was due to the lazy module loading in transformers pakage. And I don't know if I need to place some package under PYTHON_SHARED_MODULES or PYTHON_INCLUDE_PATHS. Or if this related to a known limitation of JEP.

The text was updated successfully, but these errors were encountered:

mike-hunhoff · 2023-07-10T20:37:52Z

Hi @Forsworns thank you for reporting. I'm trying to recreate on my end and received the following error:

None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.

To better help us troubleshoot this issue can you please provide your exact setup?

Forsworns · 2023-07-11T01:19:26Z

Yep, you have to make sure you can use transformer with the native python interpreter at first. For me, I use Pytorch.

But I cann't provide an exact requirement.txt, since I don't use a virtual env, and there may be some packages not related to the transformers as following:

https://gist.github.com/Forsworns/0a80a60cb8c362bb36a9ea4de4398370#file-requirements-txt

The pytorch is installed by pip3 install torch torchvision torchaudio on my Mac. And you can find that there is not cuda support installed, on a platform with cuda, don't know if the problem for JEP compatibility will get worse.

mike-hunhoff · 2023-07-21T20:33:27Z

Hi @Forsworns - based on feedback from the Jep developers we just pushed a PR (#61) to switch Ghidrathon from using sub-interpreters to shared interpreters. This should enable the use of (C)Python modules that do not support sub-interpreters. Could you test out the referenced PR to see if your issue is resolved?

mike-hunhoff · 2023-07-31T20:41:41Z

The latest release fixes this issue in my local tests. Please reopen if you continue to experience issues with the latest release.

Forsworns · 2023-08-01T01:09:03Z

@mike-hunhoff Thanks! Sorry for the late response, the last comments was buried in the mailbox list.

Do we still need the xml configuration file to tell the Ghidrathon which package should be treated as shared?
I import another package

from transformers import pipeline

and get the following error, it seems my platform problem:

Traceback (most recent call last):
  File "/usr/local/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1086, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/local/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/transformers/pipelines/__init__.py", line 60, in <module>
    from .document_question_answering import DocumentQuestionAnsweringPipeline
  File "/usr/local/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/transformers/pipelines/document_question_answering.py", line 29, in <module>
    from .question_answering import select_starts_ends
  File "/usr/local/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/transformers/pipelines/question_answering.py", line 8, in <module>
    from ..data import SquadExample, SquadFeatures, squad_convert_examples_to_features
  File "/usr/local/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/transformers/data/__init__.py", line 26, in <module>
    from .metrics import glue_compute_metrics, xnli_compute_metrics
  File "/usr/local/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/transformers/data/metrics/__init__.py", line 19, in <module>
    from scipy.stats import pearsonr, spearmanr
  File "/usr/local/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/scipy/stats/__init__.py", line 608, in <module>
    from ._stats_py import *
  File "/usr/local/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/scipy/stats/_stats_py.py", line 37, in <module>
    from numpy.testing import suppress_warnings
  File "/usr/local/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/numpy/testing/__init__.py", line 11, in <module>
    from ._private.utils import *
  File "/usr/local/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/numpy/testing/_private/utils.py", line 64, in <module>
    _tags = list(sys_tags())
            ^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/packaging/tags.py", line 536, in sys_tags
    yield from cpython_tags(warn=warn)
  File "/usr/local/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/packaging/tags.py", line 211, in cpython_tags
    platforms = list(platforms or platform_tags())
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/packaging/tags.py", line 399, in mac_platforms
    version_str = subprocess.run(
                  ^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/Library/Java/JavaVirtualMachines/temurin-17.jdk/Contents/Home/bin/java', '-sS', '-c', 'import platform; print(platform.mac_ver()[0])']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/kyrie/.ghidra/.ghidra_10.4_DEV/Extensions/Ghidrathon-2.2.0/data/python/jepeval.py", line 66, in jepeval
    more_input_needed = _jepeval(line)
                        ^^^^^^^^^^^^^^
  File "/Users/kyrie/.ghidra/.ghidra_10.4_DEV/Extensions/Ghidrathon-2.2.0/data/python/jepeval.py", line 49, in _jepeval
    exec(compile(line, "<string>", "single"), globals(), globals())
  File "<string>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 1229, in _handle_fromlist
  File "/usr/local/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1076, in __getattr__
    module = self._get_module(self._class_to_module[name])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1088, in _get_module
    raise RuntimeError(
RuntimeError: Failed to import transformers.pipelines because of the following error (look up to see its traceback):
Command '['/Library/Java/JavaVirtualMachines/temurin-17.jdk/Contents/Home/bin/java', '-sS', '-c', 'import platform; print(platform.mac_ver()[0])']' returned non-zero exit status 1.

Forsworns · 2023-08-01T01:35:51Z

@mike-hunhoff seems it should call python -sS -c 'import platform; print(platform.mac_ver()[0])' instead of java -sS -c 'import platform; print(platform.mac_ver()[0])' in the error above, rather confusing ...

mike-hunhoff · 2023-08-01T18:09:59Z

@Forsworns oh this is an interesting one. Based on the output that you provided I tracked the offending lines of code to packaging.types: https://github.com/pypa/packaging/blob/3030822b0b06a43fbbb2710da7b0846d1bebd2ba/src/packaging/tags.py#L399-L410

The code calls subprocess.run where the target executable is pulled from sys.executable. sys.executable is set correctly in my Linux test environment:

>>> import sys
>>> sys.executable
'/home/spring/dragon/.env/bin/python3`

Can you run the code listed above from your Ghidrathon interpreter? I'm guessing sys.executable is set to the Java executable used to run Ghidra in your Mac environment...

mike-hunhoff · 2023-08-01T18:28:05Z

It appears the Jep developers know of this issue: https://github.com/ninia/jep/blob/056ce9907f5ecbf2364df1ec55755404b2e8a947/commands/test.py#L73-L76 which they avoid by setting the environment variable PYTHONEXECUTABLE. Ghidrathon probably needs to set this as well...or manually configure sys.executable during the init phase if running in a Mac environment. Let's open a new issue to track this work

mike-hunhoff added the jep Issue related to jep label Jul 10, 2023

mike-hunhoff mentioned this issue Jul 21, 2023

switching GhidrathonInterpreter to use Jep SharedInterpreter #61

Merged

mike-hunhoff closed this as completed in #61 Jul 31, 2023

mike-hunhoff mentioned this issue Aug 1, 2023

Ensure sys.executable set correctly #62

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot use transformers package #59

Cannot use transformers package #59

Forsworns commented Jul 9, 2023 •

edited

Loading

mike-hunhoff commented Jul 10, 2023

Forsworns commented Jul 11, 2023 •

edited

Loading

mike-hunhoff commented Jul 21, 2023

mike-hunhoff commented Jul 31, 2023

Forsworns commented Aug 1, 2023

Forsworns commented Aug 1, 2023

mike-hunhoff commented Aug 1, 2023 •

edited

Loading

mike-hunhoff commented Aug 1, 2023

Cannot use transformers package #59

Cannot use transformers package #59

Comments

Forsworns commented Jul 9, 2023 • edited Loading

My env:

Background

Problem

mike-hunhoff commented Jul 10, 2023

Forsworns commented Jul 11, 2023 • edited Loading

mike-hunhoff commented Jul 21, 2023

mike-hunhoff commented Jul 31, 2023

Forsworns commented Aug 1, 2023

Forsworns commented Aug 1, 2023

mike-hunhoff commented Aug 1, 2023 • edited Loading

mike-hunhoff commented Aug 1, 2023

Forsworns commented Jul 9, 2023 •

edited

Loading

Forsworns commented Jul 11, 2023 •

edited

Loading

mike-hunhoff commented Aug 1, 2023 •

edited

Loading