New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

scan_module(): ValueError: bad marshal data (unknown type code) #1257

Closed
jwilk opened this Issue Jan 23, 2018 · 24 comments

Comments

Projects
None yet
5 participants
@jwilk

jwilk commented Jan 23, 2018

This happens with CPython from git master (e768c86ef442ef89004089a8a34ce5909ffb90f2):

Traceback (most recent call last):
  File "setup.py", line 16, in <module>
    install_requires=["appdirs"],
  File ".../python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File ".../python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File ".../python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File ".../python3.7/site-packages/setuptools/command/install.py", line 67, in run
    self.do_egg_install()
  File ".../python3.7/site-packages/setuptools/command/install.py", line 109, in do_egg_install
    self.run_command('bdist_egg')
  File ".../python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File ".../python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File ".../python3.7/site-packages/setuptools/command/bdist_egg.py", line 209, in run
    os.path.join(archive_root, 'EGG-INFO'), self.zip_safe()
  File ".../python3.7/site-packages/setuptools/command/bdist_egg.py", line 245, in zip_safe
    return analyze_egg(self.bdist_dir, self.stubs)
  File ".../python3.7/site-packages/setuptools/command/bdist_egg.py", line 355, in analyze_egg
    safe = scan_module(egg_dir, base, name, stubs) and safe
  File ".../python3.7/site-packages/setuptools/command/bdist_egg.py", line 392, in scan_module
    code = marshal.load(f)
ValueError: bad marshal data (unknown type code)

I believe this is because PEP 552 has been implemented, which adds extra 4 bytes to the .pyc header; but setuptools is not aware of that yet.

@benoit-pierre

This comment has been minimized.

Member

benoit-pierre commented Jan 25, 2018

See #1263.

lucc added a commit to lucc/khard that referenced this issue Feb 2, 2018

Allow tests to fail with nightly python version
We are currently getting errors from the "nightly" python version on
travis.  This is a known bug:
pypa/setuptools#1257

We accept the travis build even if the "nightly" job errors or fails.

lucc added a commit to lucc/khard that referenced this issue Feb 2, 2018

Allow tests to fail with nightly python version
We are currently getting errors from the "nightly" python version on
travis.  This is a known bug:
pypa/setuptools#1257

We accept the travis build even if the "nightly" job errors or fails.
@jaraco

This comment has been minimized.

Member

jaraco commented Feb 3, 2018

Release on v38.4.1 pending...

@jaraco jaraco closed this Feb 3, 2018

kryskool added a commit to mgasystem/giraumon that referenced this issue Feb 5, 2018

Update .travis.yml
Fix nigthly failing see pypa/setuptools#1257
@jenisys

This comment has been minimized.

jenisys commented Feb 5, 2018

Same problem occurred w/ Python 3.7.0a4+ on Travis CI (reported as: travis-ci/travis-ci#9181), probably w/o the fix.

@smagafurov

This comment has been minimized.

@benoit-pierre

This comment has been minimized.

Member

benoit-pierre commented Feb 14, 2018

@smagafurov : did you update setuptools?

@smagafurov

This comment has been minimized.

smagafurov commented Feb 14, 2018

setuptools (38.5.1)

@benoit-pierre

This comment has been minimized.

Member

benoit-pierre commented Feb 14, 2018

@smagafurov: I'm sorry but I don't see the setuptools' update in your Travis log and the traceback line numbers don't match 38.5.1. Can you add a call to pip freeze --all before to make sure the right version is used? Just to be clear, you're not relying on setup_requires to update setuptools, right?

@smagafurov

This comment has been minimized.

smagafurov commented Feb 14, 2018

@benoit-pierre: I repeat this problem locally

$ python setup.py install
running install
running bdist_egg
running egg_info
writing aiopg.egg-info/PKG-INFO
writing dependency_links to aiopg.egg-info/dependency_links.txt
writing requirements to aiopg.egg-info/requires.txt
writing top-level names to aiopg.egg-info/top_level.txt
reading manifest file 'aiopg.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files found matching 'tests/**'
writing manifest file 'aiopg.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-10.13-x86_64/egg
running install_lib
running build_py
copying aiopg.egg-info/PKG-INFO -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
copying aiopg.egg-info/SOURCES.txt -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
copying aiopg.egg-info/dependency_links.txt -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
copying aiopg.egg-info/requires.txt -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
copying aiopg.egg-info/top_level.txt -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
Traceback (most recent call last):
  File "setup.py", line 63, in <module>
    include_package_data=True)
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/__init__.py", line 129, in setup
    return distutils.core.setup(**attrs)
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/install.py", line 67, in run
    self.do_egg_install()
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/install.py", line 109, in do_egg_install
    self.run_command('bdist_egg')
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 220, in run
    os.path.join(archive_root, 'EGG-INFO'), self.zip_safe()
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 273, in zip_safe
    return analyze_egg(self.bdist_dir, self.stubs)
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 383, in analyze_egg
    safe = scan_module(egg_dir, base, name, stubs) and safe
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 422, in scan_module
    code = marshal.load(f)
ValueError: bad marshal data (unknown type code)
(venv) sergey@magafurov:~/projects/aiopg [master]
$ pip list | grep tool
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
prompt-toolkit (1.0.15)
setuptools (38.5.1)
@smagafurov

This comment has been minimized.

smagafurov commented Feb 14, 2018

@benoit-pierre: How to repeat:

git clone git@github.com:aio-libs/aiopg.git
cd aiopg
python3.7 -m venv ./venv
source ./venv/bin/activate
sudo apt-get install libpq-dev
pip install -U setuptools
python setup.py install
@benoit-pierre

This comment has been minimized.

Member

benoit-pierre commented Feb 14, 2018

I can't reproduce the issue:

> git clone https://github.com/aio-libs/aiopg.git
Cloning into 'aiopg'...
remote: Counting objects: 3267, done.
remote: Total 3267 (delta 0), reused 0 (delta 0), pack-reused 3267
Receiving objects: 100% (3267/3267), 666.68 KiB | 2.22 MiB/s, done.
Resolving deltas: 100% (2151/2151), done.
> cd aiopg
> python3.7 -m venv venv
> ./venv/bin/python --version
Python 3.7.0a4
> ./venv/bin/pip install -U pip wheel setuptools
Requirement already up-to-date: pip in ./venv/lib/python3.7/site-packages
Collecting wheel
  Using cached wheel-0.30.0-py2.py3-none-any.whl
Collecting setuptools
  Using cached setuptools-38.5.1-py2.py3-none-any.whl
Installing collected packages: wheel, setuptools
  Found existing installation: setuptools 28.8.0
    Uninstalling setuptools-28.8.0:
      Successfully uninstalled setuptools-28.8.0
Successfully installed setuptools-38.5.1 wheel-0.30.0
> ./venv/bin/python setup.py -q bdist_egg                                                        
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files found matching 'tests/**'
  File "build/bdist.linux-x86_64/egg/aiopg/utils.py", line 18
    ensure_future = asyncio.async
                                ^
SyntaxError: invalid syntax

  File "build/bdist.linux-x86_64/egg/aiopg/connection.py", line 109
    self._conn = psycopg2.connect(dsn, async=True, **kwargs)
                                           ^
SyntaxError: invalid syntax

zip_safe flag not set; analyzing archive contents...
@benoit-pierre

This comment has been minimized.

Member

benoit-pierre commented Feb 14, 2018

And setuptools' master is still ok on Travis with nightly.

@smagafurov

This comment has been minimized.

smagafurov commented Feb 14, 2018

@benoit-pierre: |'m sorry. Please clone this instead:

git clone git@github.com:smagafurov/aiopg.git

I am trying to add python3.7 support to 'aio-libs/aiopg.git'

@benoit-pierre

This comment has been minimized.

Member

benoit-pierre commented Feb 14, 2018

I don't think the clone URL matters (works fine with either one). Can you patch setuptools.commands.bdist_egg to show what's being used?

 setuptools/command/bdist_egg.py | 1 +
 1 file changed, 1 insertion(+)

diff --git i/setuptools/command/bdist_egg.py w/setuptools/command/bdist_egg.py
index 423b8187..a66b03c7 100644
--- i/setuptools/command/bdist_egg.py
+++ w/setuptools/command/bdist_egg.py
@@ -417,6 +417,7 @@ def scan_module(egg_dir, base, name, stubs):
         skip = 12  # skip magic & date & file size
     else:
         skip = 16  # skip magic & reserved? & date & file size
+    print(sys.version_info, skip)
     f = open(filename, 'rb')
     f.read(skip)
     code = marshal.load(f)
@smagafurov

This comment has been minimized.

smagafurov commented Feb 14, 2018

sys.version_info(major=3, minor=7, micro=0, releaselevel='alpha', serial=3) 16

@smagafurov

This comment has been minimized.

smagafurov commented Feb 14, 2018

$ python setup.py install
running install
running bdist_egg
running egg_info
writing aiopg.egg-info/PKG-INFO
writing dependency_links to aiopg.egg-info/dependency_links.txt
writing requirements to aiopg.egg-info/requires.txt
writing top-level names to aiopg.egg-info/top_level.txt
reading manifest file 'aiopg.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files found matching 'tests/**'
writing manifest file 'aiopg.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-10.13-x86_64/egg
running install_lib
running build_py
copying aiopg.egg-info/PKG-INFO -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
copying aiopg.egg-info/SOURCES.txt -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
copying aiopg.egg-info/dependency_links.txt -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
copying aiopg.egg-info/requires.txt -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
copying aiopg.egg-info/top_level.txt -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
sys.version_info(major=3, minor=7, micro=0, releaselevel='alpha', serial=3) 16
Traceback (most recent call last):
  File "setup.py", line 63, in <module>
    include_package_data=True)
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/__init__.py", line 129, in setup
    return distutils.core.setup(**attrs)
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/install.py", line 67, in run
    self.do_egg_install()
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/install.py", line 109, in do_egg_install
    self.run_command('bdist_egg')
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 220, in run
    os.path.join(archive_root, 'EGG-INFO'), self.zip_safe()
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 273, in zip_safe
    return analyze_egg(self.bdist_dir, self.stubs)
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 383, in analyze_egg
    safe = scan_module(egg_dir, base, name, stubs) and safe
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 423, in scan_module
    code = marshal.load(f)
ValueError: bad marshal data (unknown type code)
@smagafurov

This comment has been minimized.

smagafurov commented Feb 14, 2018

clone URL matters because I fix python3.7 syntax error here 'smagafurov/aiopg.git'

@benoit-pierre

This comment has been minimized.

Member

benoit-pierre commented Feb 14, 2018

That's another issue, irelevant to the setuptools issue.

@jwilk

This comment has been minimized.

jwilk commented Feb 14, 2018

sys.version_info(major=3, minor=7, micro=0, releaselevel='alpha', serial=3) 16

The first version that implemented PEP 552 was v3.7.0a4.
But setuptools currently does:

elif sys.version_info < (3, 7):
@benoit-pierre

This comment has been minimized.

Member

benoit-pierre commented Feb 14, 2018

That's what I was thinking, thanks for confirming.

@smagafurov

This comment has been minimized.

smagafurov commented Feb 14, 2018

this works for me

elif sys.version_info < (3, 7, 0, 'alpha', 4):
@benoit-pierre

This comment has been minimized.

Member

benoit-pierre commented Feb 14, 2018

I'm not sure it make sense to explicitely test for 3.7.0a4: if you're running with nightly, then problems are to be expected, and you're also expected to keep current, no?

What I don't understand is why it fails on Travis, or is it another hole in our testsuite?

@benoit-pierre

This comment has been minimized.

Member

benoit-pierre commented Feb 14, 2018

@smagafurov: just to be clear, to fix your issues you should:

  • locally: update Python to >=3.7.0a4
  • Travis CI: update setuptools first before installing other requirements (IMHO adding back nightly testing is a good idea)
@benoit-pierre

This comment has been minimized.

Member

benoit-pierre commented Feb 14, 2018

@smagafurov: https://travis-ci.org/benoit-pierre/aiopg/jobs/341442846

I'm also confused by the fact that python setup.py install is called before pip install -Ur requirements.txt.

@smagafurov

This comment has been minimized.

smagafurov commented Feb 15, 2018

@benoit-pierre: Thank! Updating setuptools in Travis fixes the problem.

miso-belica added a commit to miso-belica/sumy that referenced this issue Mar 24, 2018

peterjc added a commit to peterjc/backports.lzma that referenced this issue Jul 11, 2018

update setuptools on Travis
setuptools on Travis doesn't support Python 3.7 yet
see pypa/setuptools#1257

white-gecko added a commit to AKSW/QuitStore that referenced this issue Sep 25, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment