scan_module(): ValueError: bad marshal data (unknown type code)

[jwilk]

This happens with CPython from git master (e768c86ef442ef89004089a8a34ce5909ffb90f2):

Traceback (most recent call last):
  File "setup.py", line 16, in <module>
    install_requires=["appdirs"],
  File ".../python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File ".../python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File ".../python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File ".../python3.7/site-packages/setuptools/command/install.py", line 67, in run
    self.do_egg_install()
  File ".../python3.7/site-packages/setuptools/command/install.py", line 109, in do_egg_install
    self.run_command('bdist_egg')
  File ".../python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File ".../python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File ".../python3.7/site-packages/setuptools/command/bdist_egg.py", line 209, in run
    os.path.join(archive_root, 'EGG-INFO'), self.zip_safe()
  File ".../python3.7/site-packages/setuptools/command/bdist_egg.py", line 245, in zip_safe
    return analyze_egg(self.bdist_dir, self.stubs)
  File ".../python3.7/site-packages/setuptools/command/bdist_egg.py", line 355, in analyze_egg
    safe = scan_module(egg_dir, base, name, stubs) and safe
  File ".../python3.7/site-packages/setuptools/command/bdist_egg.py", line 392, in scan_module
    code = marshal.load(f)
ValueError: bad marshal data (unknown type code)

I believe this is because PEP 552 has been implemented, which adds extra 4 bytes to the .pyc header; but setuptools is not aware of that yet.

[benoit-pierre]

See #1263.

[jaraco]

Release on v38.4.1 pending...

[jenisys]

Same problem occurred w/ Python 3.7.0a4+ on Travis CI (reported as: travis-ci/travis-ci#9181), probably w/o the fix.

[smagafurov]

@jaraco Same problem
https://travis-ci.org/aio-libs/aiopg/jobs/341326946

[benoit-pierre]

@smagafurov : did you update setuptools?

[smagafurov]

setuptools (38.5.1)

[benoit-pierre]

@smagafurov: I'm sorry but I don't see the setuptools' update in your Travis log and the traceback line numbers don't match 38.5.1. Can you add a call to pip freeze --all before to make sure the right version is used? Just to be clear, you're not relying on setup_requires to update setuptools, right?

[smagafurov]

@benoit-pierre: I repeat this problem locally

$ python setup.py install
running install
running bdist_egg
running egg_info
writing aiopg.egg-info/PKG-INFO
writing dependency_links to aiopg.egg-info/dependency_links.txt
writing requirements to aiopg.egg-info/requires.txt
writing top-level names to aiopg.egg-info/top_level.txt
reading manifest file 'aiopg.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files found matching 'tests/**'
writing manifest file 'aiopg.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-10.13-x86_64/egg
running install_lib
running build_py
copying aiopg.egg-info/PKG-INFO -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
copying aiopg.egg-info/SOURCES.txt -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
copying aiopg.egg-info/dependency_links.txt -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
copying aiopg.egg-info/requires.txt -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
copying aiopg.egg-info/top_level.txt -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
Traceback (most recent call last):
  File "setup.py", line 63, in <module>
    include_package_data=True)
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/__init__.py", line 129, in setup
    return distutils.core.setup(**attrs)
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/install.py", line 67, in run
    self.do_egg_install()
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/install.py", line 109, in do_egg_install
    self.run_command('bdist_egg')
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 220, in run
    os.path.join(archive_root, 'EGG-INFO'), self.zip_safe()
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 273, in zip_safe
    return analyze_egg(self.bdist_dir, self.stubs)
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 383, in analyze_egg
    safe = scan_module(egg_dir, base, name, stubs) and safe
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 422, in scan_module
    code = marshal.load(f)
ValueError: bad marshal data (unknown type code)
(venv) sergey@magafurov:~/projects/aiopg [master]
$ pip list | grep tool
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
prompt-toolkit (1.0.15)
setuptools (38.5.1)

[smagafurov]

@benoit-pierre: How to repeat:

git clone git@github.com:aio-libs/aiopg.git
cd aiopg
python3.7 -m venv ./venv
source ./venv/bin/activate
sudo apt-get install libpq-dev
pip install -U setuptools
python setup.py install

[benoit-pierre]

I can't reproduce the issue:

> git clone https://github.com/aio-libs/aiopg.git
Cloning into 'aiopg'...
remote: Counting objects: 3267, done.
remote: Total 3267 (delta 0), reused 0 (delta 0), pack-reused 3267
Receiving objects: 100% (3267/3267), 666.68 KiB | 2.22 MiB/s, done.
Resolving deltas: 100% (2151/2151), done.
> cd aiopg
> python3.7 -m venv venv
> ./venv/bin/python --version
Python 3.7.0a4
> ./venv/bin/pip install -U pip wheel setuptools
Requirement already up-to-date: pip in ./venv/lib/python3.7/site-packages
Collecting wheel
  Using cached wheel-0.30.0-py2.py3-none-any.whl
Collecting setuptools
  Using cached setuptools-38.5.1-py2.py3-none-any.whl
Installing collected packages: wheel, setuptools
  Found existing installation: setuptools 28.8.0
    Uninstalling setuptools-28.8.0:
      Successfully uninstalled setuptools-28.8.0
Successfully installed setuptools-38.5.1 wheel-0.30.0
> ./venv/bin/python setup.py -q bdist_egg                                                        
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files found matching 'tests/**'
  File "build/bdist.linux-x86_64/egg/aiopg/utils.py", line 18
    ensure_future = asyncio.async
                                ^
SyntaxError: invalid syntax

  File "build/bdist.linux-x86_64/egg/aiopg/connection.py", line 109
    self._conn = psycopg2.connect(dsn, async=True, **kwargs)
                                           ^
SyntaxError: invalid syntax

zip_safe flag not set; analyzing archive contents...

[benoit-pierre]

And setuptools' master is still ok on Travis with nightly.

[smagafurov]

@benoit-pierre: |'m sorry. Please clone this instead:

git clone git@github.com:smagafurov/aiopg.git

I am trying to add python3.7 support to 'aio-libs/aiopg.git'

[benoit-pierre]

I don't think the clone URL matters (works fine with either one). Can you patch setuptools.commands.bdist_egg to show what's being used?

 setuptools/command/bdist_egg.py | 1 +
 1 file changed, 1 insertion(+)

diff --git i/setuptools/command/bdist_egg.py w/setuptools/command/bdist_egg.py
index 423b8187..a66b03c7 100644
--- i/setuptools/command/bdist_egg.py
+++ w/setuptools/command/bdist_egg.py
@@ -417,6 +417,7 @@ def scan_module(egg_dir, base, name, stubs):
         skip = 12  # skip magic & date & file size
     else:
         skip = 16  # skip magic & reserved? & date & file size
+    print(sys.version_info, skip)
     f = open(filename, 'rb')
     f.read(skip)
     code = marshal.load(f)

[smagafurov]

sys.version_info(major=3, minor=7, micro=0, releaselevel='alpha', serial=3) 16

[smagafurov]

$ python setup.py install
running install
running bdist_egg
running egg_info
writing aiopg.egg-info/PKG-INFO
writing dependency_links to aiopg.egg-info/dependency_links.txt
writing requirements to aiopg.egg-info/requires.txt
writing top-level names to aiopg.egg-info/top_level.txt
reading manifest file 'aiopg.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files found matching 'tests/**'
writing manifest file 'aiopg.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-10.13-x86_64/egg
running install_lib
running build_py
copying aiopg.egg-info/PKG-INFO -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
copying aiopg.egg-info/SOURCES.txt -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
copying aiopg.egg-info/dependency_links.txt -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
copying aiopg.egg-info/requires.txt -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
copying aiopg.egg-info/top_level.txt -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
sys.version_info(major=3, minor=7, micro=0, releaselevel='alpha', serial=3) 16
Traceback (most recent call last):
  File "setup.py", line 63, in <module>
    include_package_data=True)
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/__init__.py", line 129, in setup
    return distutils.core.setup(**attrs)
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/install.py", line 67, in run
    self.do_egg_install()
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/install.py", line 109, in do_egg_install
    self.run_command('bdist_egg')
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 220, in run
    os.path.join(archive_root, 'EGG-INFO'), self.zip_safe()
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 273, in zip_safe
    return analyze_egg(self.bdist_dir, self.stubs)
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 383, in analyze_egg
    safe = scan_module(egg_dir, base, name, stubs) and safe
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 423, in scan_module
    code = marshal.load(f)
ValueError: bad marshal data (unknown type code)

[smagafurov]

clone URL matters because I fix python3.7 syntax error here 'smagafurov/aiopg.git'

[benoit-pierre]

That's another issue, irelevant to the setuptools issue.

[jwilk]

sys.version_info(major=3, minor=7, micro=0, releaselevel='alpha', serial=3) 16

The first version that implemented PEP 552 was v3.7.0a4.
But setuptools currently does:

elif sys.version_info < (3, 7):

[benoit-pierre]

That's what I was thinking, thanks for confirming.

[smagafurov]

this works for me

elif sys.version_info < (3, 7, 0, 'alpha', 4):

[benoit-pierre]

I'm not sure it make sense to explicitely test for 3.7.0a4: if you're running with nightly, then problems are to be expected, and you're also expected to keep current, no?

What I don't understand is why it fails on Travis, or is it another hole in our testsuite?

[benoit-pierre]

@smagafurov: just to be clear, to fix your issues you should:

• locally: update Python to >=3.7.0a4
• Travis CI: update setuptools first before installing other requirements (IMHO adding back nightly testing is a good idea)

[benoit-pierre]

@smagafurov: https://travis-ci.org/benoit-pierre/aiopg/jobs/341442846

I'm also confused by the fact that python setup.py install is called before pip install -Ur requirements.txt.

[smagafurov]

@benoit-pierre: Thank! Updating setuptools in Travis fixes the problem.