Skip to content
This repository has been archived by the owner on Nov 26, 2023. It is now read-only.

Error running pycg. #29

Closed
michelescarlato opened this issue May 23, 2022 · 8 comments
Closed

Error running pycg. #29

michelescarlato opened this issue May 23, 2022 · 8 comments

Comments

@michelescarlato
Copy link

Hi there,

writing the Fasten-PyPI-plugin, we would need to run pycg locally to analyze missing call graphs on the FASTEN server.

Trying out the tool, I got with several packages this error.

michelescarlato@splinter:~/call-graphs-creation/scipy-1.8.1$ pycg --fasten  --package scipy $(find scipy -type f -name "*.py") --product "scipy" --version "1.8.1" --forge "PyPI" --max-iter -1 -o cg.json
Traceback (most recent call last):
  File "/home/michelescarlato/.local/bin/pycg", line 8, in <module>
    sys.exit(main())
  File "/home/michelescarlato/.local/lib/python3.10/site-packages/pycg/__main__.py", line 79, in main
    cg.analyze()
  File "/home/michelescarlato/.local/lib/python3.10/site-packages/pycg/pycg.py", line 155, in analyze
    self.do_pass(PreProcessor, True,
  File "/home/michelescarlato/.local/lib/python3.10/site-packages/pycg/pycg.py", line 144, in do_pass
    self.import_manager.install_hooks()
  File "/home/michelescarlato/.local/lib/python3.10/site-packages/pycg/machinery/imports.py", line 203, in install_hooks
    loader = get_custom_loader(self)
  File "/home/michelescarlato/.local/lib/python3.10/site-packages/pycg/machinery/imports.py", line 34, in get_custom_loader
    class CustomLoader(importlib.abc.SourceLoader):
AttributeError: module 'importlib' has no attribute 'abc'. Did you mean: '_abc'?

Am I doing something wrong?
@vitsalis
Copy link
Owner

What version of Python are you running?

@michelescarlato
Copy link
Author

3.10.4

@vitsalis
Copy link
Owner

I see. It might be due to some updates to the class on version 3.10. Can you try running it with version 3.9?

@michelescarlato
Copy link
Author

Hi there,

@gdrosos tried it with the Python version 3.9.13.
He used this command:
pycg --fasten --package scipy $(find scipy -type f -name "*.py") --product "scipy" --version "1.8.1" --forge "PyPI" --max-iter -1 -o cg.json

And this is the output:

File "/Users/gdrosos/test/PyCG/pycg/processing/preprocessor.py", line 65, in analyze_submodule
    super().analyze_submodule(PreProcessor, modname,
  File "/Users/gdrosos/test/PyCG/pycg/processing/base.py", line 484, in analyze_submodule
    visitor = cls(fname, imp, *args, **kwargs)
  File "/Users/gdrosos/test/PyCG/pycg/processing/preprocessor.py", line 33, in __init__
    super().__init__(filename, modname, modules_analyzed)
  File "/Users/gdrosos/test/PyCG/pycg/processing/base.py", line 37, in __init__
    self.contents = f.read()
  File "/opt/homebrew/Cellar/python@3.9/3.9.13/Frameworks/Python.framework/Versions/3.9/lib/python3.9/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xcf in position 0: invalid continuation byte

He got the same output using numpy version 1.22.4.

@amphioxis
Copy link

amphioxis commented Jun 1, 2022 •
edited

Hey @gdrosos and @vitsalis ,

I am trying to help Michele with some problems running pycg but I am facing problems by myself:

Downloading https://pypi.org/project/setuptools/#files (the tar.gz file) untar it and running
pycg --package setuptools --fasten --product "setuptools" --forge "PyPI" --version "0.1" --timestamp 42 $(find setuptools -type f -name "*.py") -o setuptools.json inside the folder setuptools-62.3.2

results in

Traceback (most recent call last):
  File "/mnt/stuff/projects/work/pypi-Plugin-venv/bin/pycg", line 11, in <module>
    load_entry_point('pycg==0.0.5', 'console_scripts', 'pycg')()
  File "/mnt/stuff/projects/work/pypi-Plugin-venv/lib/python3.9/site-packages/pycg-0.0.5-py3.9.egg/pycg/__main__.py", line 79, in main
    cg.analyze()
  File "/mnt/stuff/projects/work/pypi-Plugin-venv/lib/python3.9/site-packages/pycg-0.0.5-py3.9.egg/pycg/pycg.py", line 155, in analyze
    self.do_pass(PreProcessor, True,
  File "/mnt/stuff/projects/work/pypi-Plugin-venv/lib/python3.9/site-packages/pycg-0.0.5-py3.9.egg/pycg/pycg.py", line 148, in do_pass
    processor.analyze()
  File "/mnt/stuff/projects/work/pypi-Plugin-venv/lib/python3.9/site-packages/pycg-0.0.5-py3.9.egg/pycg/processing/preprocessor.py", line 375, in analyze
    self.visit(ast.parse(self.contents, self.filename))
  File "/usr/lib/python3.9/ast.py", line 50, in parse
    return compile(source, filename, mode, flags,
  File "/mnt/stuff/projects/work/packagesToBeAnalyzed/setuptools-62.3.2/setuptools/tests/script-with-bom.py", line 1
    # -*- coding: utf-8 -*-
    ^
SyntaxError: invalid non-printable character U+FEFF

Running it one folder above results in an empty call Graph:

{"product": "setuptools", "forge": "PyPI", "generator": "PyCG", "depset": [], "version": "0.1", "timestamp": "42", "modules": {"internal": {}, "external": {}}, "graph": {"internalCalls": [], "externalCalls": [], "resolvedCalls": []}, "nodes": 0}?

What is the Problem?

@michelescarlato
Copy link
Author

Hi there,

I report different errors, running different python versions:

michele.scarlato@pycg-tester:~/setuptools-62.3.2$ pycg --package setuptools --fasten --product "setuptools" --forge "PyPI" --version "0.1" --timestamp 42 $(find setuptools -type f -name "*.py") -o setuptools.json
Traceback (most recent call last):
  File "/home/michele.scarlato/.local/bin/pycg", line 8, in <module>
    sys.exit(main())
  File "/home/michele.scarlato/.local/lib/python3.8/site-packages/pycg/__main__.py", line 79, in main
    cg.analyze()
  File "/home/michele.scarlato/.local/lib/python3.8/site-packages/pycg/pycg.py", line 155, in analyze
    self.do_pass(PreProcessor, True,
  File "/home/michele.scarlato/.local/lib/python3.8/site-packages/pycg/pycg.py", line 148, in do_pass
    processor.analyze()
  File "/home/michele.scarlato/.local/lib/python3.8/site-packages/pycg/processing/preprocessor.py", line 375, in analyze
    self.visit(ast.parse(self.contents, self.filename))
  File "/usr/lib/python3.8/ast.py", line 47, in parse
    return compile(source, filename, mode, flags,
  File "/home/michele.scarlato/setuptools-62.3.2/setuptools/tests/script-with-bom.py", line 1
    # -*- coding: utf-8 -*-
    ^
SyntaxError: invalid character in identifier
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 57, in apport_excepthook
    apt_pkg.DATE
AttributeError: module 'apt_pkg' has no attribute 'DATE'

Original exception was:
Traceback (most recent call last):
  File "/home/michele.scarlato/.local/bin/pycg", line 8, in <module>
    sys.exit(main())
  File "/home/michele.scarlato/.local/lib/python3.8/site-packages/pycg/__main__.py", line 79, in main
    cg.analyze()
  File "/home/michele.scarlato/.local/lib/python3.8/site-packages/pycg/pycg.py", line 155, in analyze
    self.do_pass(PreProcessor, True,
  File "/home/michele.scarlato/.local/lib/python3.8/site-packages/pycg/pycg.py", line 148, in do_pass
    processor.analyze()
  File "/home/michele.scarlato/.local/lib/python3.8/site-packages/pycg/processing/preprocessor.py", line 375, in analyze
    self.visit(ast.parse(self.contents, self.filename))
  File "/usr/lib/python3.8/ast.py", line 47, in parse
    return compile(source, filename, mode, flags,
  File "/home/michele.scarlato/setuptools-62.3.2/setuptools/tests/script-with-bom.py", line 1
    # -*- coding: utf-8 -*-
    ^
SyntaxError: invalid character in identifier
michele.scarlato@pycg-tester:~/setuptools-62.3.2$ python3
python3           python3-config    python3.8         python3.8-config  python3.9         
michele.scarlato@pycg-tester:~/setuptools-62.3.2$ python3 --version
Python 3.8.10

michele.scarlato@pycg-tester-instance-3:~/setuptools$ pycg --package setuptools --fasten --product "setuptools" --forge "PyPI" --version "0.1" --timestamp 42 $(find setuptools -type f -name "*.py") -o setuptools.json
Traceback (most recent call last):
  File "/home/michele.scarlato/.local/bin/pycg", line 8, in <module>
    sys.exit(main())
  File "/home/michele.scarlato/.local/lib/python3.9/site-packages/pycg/__main__.py", line 79, in main
    cg.analyze()
  File "/home/michele.scarlato/.local/lib/python3.9/site-packages/pycg/pycg.py", line 155, in analyze
    self.do_pass(PreProcessor, True,
  File "/home/michele.scarlato/.local/lib/python3.9/site-packages/pycg/pycg.py", line 146, in do_pass
    processor = cls(input_file, input_mod,
  File "/home/michele.scarlato/.local/lib/python3.9/site-packages/pycg/processing/preprocessor.py", line 33, in __init__
    super().__init__(filename, modname, modules_analyzed)
  File "/home/michele.scarlato/.local/lib/python3.9/site-packages/pycg/processing/base.py", line 36, in __init__
    with open(filename, "rt") as f:
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 982, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 925, in _find_spec
  File "<frozen importlib._bootstrap_external>", line 1349, in find_spec
  File "<frozen importlib._bootstrap_external>", line 1321, in _get_spec
  File "<frozen importlib._bootstrap_external>", line 1477, in find_spec
  File "<frozen importlib._bootstrap_external>", line 1434, in _get_spec
  File "/home/michele.scarlato/.local/lib/python3.9/site-packages/pycg/machinery/imports.py", line 39, in __init__
    ig_obj.create_edge(self.fullname)
  File "/home/michele.scarlato/.local/lib/python3.9/site-packages/pycg/machinery/imports.py", line 87, in create_edge
    raise ImportManagerError("Can't add edge to a non existing node")
pycg.machinery.imports.ImportManagerError: Can't add edge to a non existing node
michele.scarlato@pycg-tester-instance-3:~/setuptools$ python3 --version
Python 3.9.2

michele.scarlato@pycg-tester-instance-2:~/setuptools-62.3.2$ pycg --package setuptools --fasten --product "setuptools" --forge "PyPI" --version "0.1" --timestamp 42 $(find setuptools -type f -name "*.py") -o setuptools.json
Traceback (most recent call last):
  File "/home/michele.scarlato/.local/bin/pycg", line 8, in <module>
    sys.exit(main())
  File "/home/michele.scarlato/.local/lib/python3.9/site-packages/pycg/__main__.py", line 79, in main
    cg.analyze()
  File "/home/michele.scarlato/.local/lib/python3.9/site-packages/pycg/pycg.py", line 155, in analyze
    self.do_pass(PreProcessor, True,
  File "/home/michele.scarlato/.local/lib/python3.9/site-packages/pycg/pycg.py", line 148, in do_pass
    processor.analyze()
  File "/home/michele.scarlato/.local/lib/python3.9/site-packages/pycg/processing/preprocessor.py", line 375, in analyze
    self.visit(ast.parse(self.contents, self.filename))
  File "/usr/lib/python3.9/ast.py", line 50, in parse
    return compile(source, filename, mode, flags,
  File "/home/michele.scarlato/setuptools-62.3.2/setuptools/tests/script-with-bom.py", line 1
    # -*- coding: utf-8 -*-
    ^
SyntaxError: invalid non-printable character U+FEFF
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 57, in apport_excepthook
    apt_pkg.DATE
AttributeError: module 'apt_pkg' has no attribute 'DATE'

Original exception was:
Traceback (most recent call last):
  File "/home/michele.scarlato/.local/bin/pycg", line 8, in <module>
    sys.exit(main())
  File "/home/michele.scarlato/.local/lib/python3.9/site-packages/pycg/__main__.py", line 79, in main
    cg.analyze()
  File "/home/michele.scarlato/.local/lib/python3.9/site-packages/pycg/pycg.py", line 155, in analyze
    self.do_pass(PreProcessor, True,
  File "/home/michele.scarlato/.local/lib/python3.9/site-packages/pycg/pycg.py", line 148, in do_pass
    processor.analyze()
  File "/home/michele.scarlato/.local/lib/python3.9/site-packages/pycg/processing/preprocessor.py", line 375, in analyze
    self.visit(ast.parse(self.contents, self.filename))
  File "/usr/lib/python3.9/ast.py", line 50, in parse
    return compile(source, filename, mode, flags,
  File "/home/michele.scarlato/setuptools-62.3.2/setuptools/tests/script-with-bom.py", line 1
    # -*- coding: utf-8 -*-
    ^
SyntaxError: invalid non-printable character U+FEFF
michele.scarlato@pycg-tester-instance-2:~/setuptools-62.3.2$ python3 --version
Python 3.9.7

On each machine I run these commands:

pip install pycg
wget https://files.pythonhosted.org/packages/4a/25/ec29a23ef38b9456f9965c57a9e1221e6c246d87abbf2a31158799bca201/setuptools-62.3.2.tar.gz
tar -xvzf setuptools-62.3.2.tar.gz
cd setuptools-62.3.2/
pycg --package setuptools --fasten --product "setuptools" --forge "PyPI" --version "0.1" --timestamp 42 $(find setuptools -type f -name "*.py") -o setuptools.json

@gdrosos gdrosos mentioned this issue Jun 4, 2022
Merged
@alaalial
Copy link

alaalial commented Jun 11, 2022 •
edited

Downloading https://pypi.org/project/setuptools/#files (the tar.gz file) untar it and running
pycg --fasten --package SETUPTOOLS $(find SETUPTOOLS -type f -name "exerxises.py") --product "SETUPTOOLS" --version "1.8.1" --forge "PyPI" --max-iter -1 -o cg.json
**give me the error

format parameter not correct**

how I can get the call graph for exercises.py?

@vitsalis
Copy link
Owner

vitsalis commented Nov 2, 2022

Hi @alaalial , please create a new issue for that.

@vitsalis vitsalis closed this as completed Nov 2, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@vitsalis @michelescarlato @alaalial @amphioxis