scan_module(): ValueError: bad marshal data (unknown type code) #1257

jwilk · 2018-01-23T20:27:25Z

This happens with CPython from git master (e768c86ef442ef89004089a8a34ce5909ffb90f2):

Traceback (most recent call last):
  File "setup.py", line 16, in <module>
    install_requires=["appdirs"],
  File ".../python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File ".../python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File ".../python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File ".../python3.7/site-packages/setuptools/command/install.py", line 67, in run
    self.do_egg_install()
  File ".../python3.7/site-packages/setuptools/command/install.py", line 109, in do_egg_install
    self.run_command('bdist_egg')
  File ".../python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File ".../python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File ".../python3.7/site-packages/setuptools/command/bdist_egg.py", line 209, in run
    os.path.join(archive_root, 'EGG-INFO'), self.zip_safe()
  File ".../python3.7/site-packages/setuptools/command/bdist_egg.py", line 245, in zip_safe
    return analyze_egg(self.bdist_dir, self.stubs)
  File ".../python3.7/site-packages/setuptools/command/bdist_egg.py", line 355, in analyze_egg
    safe = scan_module(egg_dir, base, name, stubs) and safe
  File ".../python3.7/site-packages/setuptools/command/bdist_egg.py", line 392, in scan_module
    code = marshal.load(f)
ValueError: bad marshal data (unknown type code)

I believe this is because PEP 552 has been implemented, which adds extra 4 bytes to the .pyc header; but setuptools is not aware of that yet.

benoit-pierre · 2018-01-25T21:10:56Z

See #1263.

We are currently getting errors from the "nightly" python version on travis. This is a known bug: pypa/setuptools#1257 We accept the travis build even if the "nightly" job errors or fails.

jaraco · 2018-02-03T14:26:10Z

Release on v38.4.1 pending...

Fix nigthly failing see pypa/setuptools#1257

jenisys · 2018-02-05T22:30:00Z

Same problem occurred w/ Python 3.7.0a4+ on Travis CI (reported as: travis-ci/travis-ci#9181), probably w/o the fix.

smagafurov · 2018-02-14T08:53:13Z

@jaraco Same problem
https://travis-ci.org/aio-libs/aiopg/jobs/341326946

benoit-pierre · 2018-02-14T09:43:01Z

@smagafurov : did you update setuptools?

smagafurov · 2018-02-14T10:04:03Z

setuptools (38.5.1)

benoit-pierre · 2018-02-14T10:12:15Z

@smagafurov: I'm sorry but I don't see the setuptools' update in your Travis log and the traceback line numbers don't match 38.5.1. Can you add a call to pip freeze --all before to make sure the right version is used? Just to be clear, you're not relying on setup_requires to update setuptools, right?

smagafurov · 2018-02-14T10:44:13Z

@benoit-pierre: I repeat this problem locally

$ python setup.py install
running install
running bdist_egg
running egg_info
writing aiopg.egg-info/PKG-INFO
writing dependency_links to aiopg.egg-info/dependency_links.txt
writing requirements to aiopg.egg-info/requires.txt
writing top-level names to aiopg.egg-info/top_level.txt
reading manifest file 'aiopg.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files found matching 'tests/**'
writing manifest file 'aiopg.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-10.13-x86_64/egg
running install_lib
running build_py
copying aiopg.egg-info/PKG-INFO -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
copying aiopg.egg-info/SOURCES.txt -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
copying aiopg.egg-info/dependency_links.txt -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
copying aiopg.egg-info/requires.txt -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
copying aiopg.egg-info/top_level.txt -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
Traceback (most recent call last):
  File "setup.py", line 63, in <module>
    include_package_data=True)
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/__init__.py", line 129, in setup
    return distutils.core.setup(**attrs)
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/install.py", line 67, in run
    self.do_egg_install()
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/install.py", line 109, in do_egg_install
    self.run_command('bdist_egg')
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 220, in run
    os.path.join(archive_root, 'EGG-INFO'), self.zip_safe()
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 273, in zip_safe
    return analyze_egg(self.bdist_dir, self.stubs)
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 383, in analyze_egg
    safe = scan_module(egg_dir, base, name, stubs) and safe
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 422, in scan_module
    code = marshal.load(f)
ValueError: bad marshal data (unknown type code)
(venv) sergey@magafurov:~/projects/aiopg [master]
$ pip list | grep tool
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
prompt-toolkit (1.0.15)
setuptools (38.5.1)

smagafurov · 2018-02-14T10:50:36Z

@benoit-pierre: How to repeat:

git clone git@github.com:aio-libs/aiopg.git
cd aiopg
python3.7 -m venv ./venv
source ./venv/bin/activate
sudo apt-get install libpq-dev
pip install -U setuptools
python setup.py install

benoit-pierre · 2018-02-14T11:07:41Z

I can't reproduce the issue:

> git clone https://github.com/aio-libs/aiopg.git
Cloning into 'aiopg'...
remote: Counting objects: 3267, done.
remote: Total 3267 (delta 0), reused 0 (delta 0), pack-reused 3267
Receiving objects: 100% (3267/3267), 666.68 KiB | 2.22 MiB/s, done.
Resolving deltas: 100% (2151/2151), done.
> cd aiopg
> python3.7 -m venv venv
> ./venv/bin/python --version
Python 3.7.0a4
> ./venv/bin/pip install -U pip wheel setuptools
Requirement already up-to-date: pip in ./venv/lib/python3.7/site-packages
Collecting wheel
  Using cached wheel-0.30.0-py2.py3-none-any.whl
Collecting setuptools
  Using cached setuptools-38.5.1-py2.py3-none-any.whl
Installing collected packages: wheel, setuptools
  Found existing installation: setuptools 28.8.0
    Uninstalling setuptools-28.8.0:
      Successfully uninstalled setuptools-28.8.0
Successfully installed setuptools-38.5.1 wheel-0.30.0
> ./venv/bin/python setup.py -q bdist_egg                                                        
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files found matching 'tests/**'
  File "build/bdist.linux-x86_64/egg/aiopg/utils.py", line 18
    ensure_future = asyncio.async
                                ^
SyntaxError: invalid syntax

  File "build/bdist.linux-x86_64/egg/aiopg/connection.py", line 109
    self._conn = psycopg2.connect(dsn, async=True, **kwargs)
                                           ^
SyntaxError: invalid syntax

zip_safe flag not set; analyzing archive contents...

benoit-pierre · 2018-02-14T11:12:20Z

And setuptools' master is still ok on Travis with nightly.

smagafurov · 2018-02-14T11:12:58Z

@benoit-pierre: |'m sorry. Please clone this instead:

git clone git@github.com:smagafurov/aiopg.git

I am trying to add python3.7 support to 'aio-libs/aiopg.git'

benoit-pierre · 2018-02-14T11:18:59Z

I don't think the clone URL matters (works fine with either one). Can you patch setuptools.commands.bdist_egg to show what's being used?

 setuptools/command/bdist_egg.py | 1 +
 1 file changed, 1 insertion(+)

diff --git i/setuptools/command/bdist_egg.py w/setuptools/command/bdist_egg.py
index 423b8187..a66b03c7 100644
--- i/setuptools/command/bdist_egg.py
+++ w/setuptools/command/bdist_egg.py
@@ -417,6 +417,7 @@ def scan_module(egg_dir, base, name, stubs):
         skip = 12  # skip magic & date & file size
     else:
         skip = 16  # skip magic & reserved? & date & file size
+    print(sys.version_info, skip)
     f = open(filename, 'rb')
     f.read(skip)
     code = marshal.load(f)

smagafurov · 2018-02-14T11:28:51Z

sys.version_info(major=3, minor=7, micro=0, releaselevel='alpha', serial=3) 16

smagafurov · 2018-02-14T11:29:26Z

$ python setup.py install
running install
running bdist_egg
running egg_info
writing aiopg.egg-info/PKG-INFO
writing dependency_links to aiopg.egg-info/dependency_links.txt
writing requirements to aiopg.egg-info/requires.txt
writing top-level names to aiopg.egg-info/top_level.txt
reading manifest file 'aiopg.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files found matching 'tests/**'
writing manifest file 'aiopg.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-10.13-x86_64/egg
running install_lib
running build_py
copying aiopg.egg-info/PKG-INFO -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
copying aiopg.egg-info/SOURCES.txt -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
copying aiopg.egg-info/dependency_links.txt -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
copying aiopg.egg-info/requires.txt -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
copying aiopg.egg-info/top_level.txt -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
sys.version_info(major=3, minor=7, micro=0, releaselevel='alpha', serial=3) 16
Traceback (most recent call last):
  File "setup.py", line 63, in <module>
    include_package_data=True)
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/__init__.py", line 129, in setup
    return distutils.core.setup(**attrs)
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/install.py", line 67, in run
    self.do_egg_install()
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/install.py", line 109, in do_egg_install
    self.run_command('bdist_egg')
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 220, in run
    os.path.join(archive_root, 'EGG-INFO'), self.zip_safe()
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 273, in zip_safe
    return analyze_egg(self.bdist_dir, self.stubs)
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 383, in analyze_egg
    safe = scan_module(egg_dir, base, name, stubs) and safe
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 423, in scan_module
    code = marshal.load(f)
ValueError: bad marshal data (unknown type code)

smagafurov · 2018-02-14T11:30:59Z

clone URL matters because I fix python3.7 syntax error here 'smagafurov/aiopg.git'

benoit-pierre · 2018-02-14T11:31:51Z

That's another issue, irelevant to the setuptools issue.

jwilk · 2018-02-14T11:45:55Z

sys.version_info(major=3, minor=7, micro=0, releaselevel='alpha', serial=3) 16

The first version that implemented PEP 552 was v3.7.0a4.
But setuptools currently does:

elif sys.version_info < (3, 7):

benoit-pierre · 2018-02-14T11:49:38Z

That's what I was thinking, thanks for confirming.

smagafurov · 2018-02-14T11:49:59Z

this works for me

elif sys.version_info < (3, 7, 0, 'alpha', 4):

benoit-pierre · 2018-02-14T11:51:20Z

I'm not sure it make sense to explicitely test for 3.7.0a4: if you're running with nightly, then problems are to be expected, and you're also expected to keep current, no?

What I don't understand is why it fails on Travis, or is it another hole in our testsuite?

benoit-pierre · 2018-02-14T13:48:01Z

@smagafurov: just to be clear, to fix your issues you should:

locally: update Python to >=3.7.0a4
Travis CI: update setuptools first before installing other requirements (IMHO adding back nightly testing is a good idea)

benoit-pierre · 2018-02-14T14:04:29Z

@smagafurov: https://travis-ci.org/benoit-pierre/aiopg/jobs/341442846

I'm also confused by the fact that python setup.py install is called before pip install -Ur requirements.txt.

smagafurov · 2018-02-15T15:28:58Z

@benoit-pierre: Thank! Updating setuptools in Travis fixes the problem.

pypa/setuptools#1257

setuptools on Travis doesn't support Python 3.7 yet see pypa/setuptools#1257

ref: https://travis-ci.org/pygridtools/drmaa-python/jobs/409174795 xref: pypa/setuptools#1257

Because of travis-ci/travis-ci#9181 resp. pypa/setuptools#1257

ref: https://travis-ci.org/pygridtools/drmaa-python/jobs/409174795 xref: pypa/setuptools#1257

golyalpha · 2020-02-15T07:18:02Z

Having exact same issue.
Python: 3.7.6
Setuptools: 45.2.0 (just tried to update, no luck)

Nevermind, it was an issue with the source release I downloaded containing pre-existing .pyc files.

jwilk mentioned this issue Jan 23, 2018

Python nightly and 3.7-dev not updating travis-ci/travis-ci#9119

Closed

benoit-pierre mentioned this issue Jan 25, 2018

fix Python 3.7 support #1263

Merged

jaraco closed this Feb 3, 2018

oremanj mentioned this issue Feb 8, 2018

Stop using 'async' as an identifier (it's a keyword in 3.7) python-trio/trio-asyncio#5

Merged

smagafurov mentioned this issue Feb 14, 2018

python3.7 support aio-libs/aiopg#437

Merged

This was referenced Feb 19, 2018

Bug/fix circle off by one error CTPUG/pygame_cffi#96

Merged

Force later setuptools version for nightly build CTPUG/pygame_cffi#97

Merged

miso-belica added a commit to miso-belica/sumy that referenced this issue Mar 24, 2018

Install newest setuptools to pass tests on Py3.7

Loading status checks…

7b54e2c

pypa/setuptools#1257

wiggin15 mentioned this issue Jul 10, 2018

Use setuptools instead of distutils if available peterjc/backports.lzma#34

Merged

peterjc added a commit to peterjc/backports.lzma that referenced this issue Jul 11, 2018

update setuptools on Travis

Loading status checks…

b3f5be8

setuptools on Travis doesn't support Python 3.7 yet see pypa/setuptools#1257

jakirkham added a commit to jakirkham/drmaa-python that referenced this issue Jul 28, 2018

Need newer version of setuptools for Python 3.7

0095a33

ref: https://travis-ci.org/pygridtools/drmaa-python/jobs/409174795 xref: pypa/setuptools#1257

white-gecko added a commit to AKSW/QuitStore that referenced this issue Sep 25, 2018

Update setuptools on travis.

Loading status checks…

2c34753

Because of travis-ci/travis-ci#9181 resp. pypa/setuptools#1257

jakirkham added a commit to jakirkham/drmaa-python that referenced this issue Sep 25, 2018

Need newer version of setuptools for Python 3.7

7d9d5a2

ref: https://travis-ci.org/pygridtools/drmaa-python/jobs/409174795 xref: pypa/setuptools#1257

Hanaasagi mentioned this issue Mar 18, 2019

Fix tests under pytest 4 PyCQA/redbaron#179

Open

gautamkrishnar mentioned this issue May 19, 2019

install not working in python 3.7 gautamkrishnar/socli#152

Closed

diricxbart mentioned this issue Jan 5, 2020

HomeAssistant crash - ValueError: bad marshal data home-assistant/supervisor#1415

Closed

LeoniePhiline mentioned this issue Mar 15, 2020

Fails to install: "ValueError: bad marshal data (unknown type code)" rodgomesc/avell-unofficial-control-center#41

Closed

andy-maier mentioned this issue Apr 13, 2020

Installation test with setup.py fails on Python 3.7 with bad marshaling data pywbem/pywbem#2167

Closed

pypa / setuptools

scan_module(): ValueError: bad marshal data (unknown type code) #1257

scan_module(): ValueError: bad marshal data (unknown type code) #1257

jwilk commented Jan 23, 2018

benoit-pierre commented Jan 25, 2018

jaraco commented Feb 3, 2018

jenisys commented Feb 5, 2018

smagafurov commented Feb 14, 2018

benoit-pierre commented Feb 14, 2018

smagafurov commented Feb 14, 2018

benoit-pierre commented Feb 14, 2018

smagafurov commented Feb 14, 2018

smagafurov commented Feb 14, 2018 •

edited

benoit-pierre commented Feb 14, 2018

benoit-pierre commented Feb 14, 2018

smagafurov commented Feb 14, 2018

benoit-pierre commented Feb 14, 2018

smagafurov commented Feb 14, 2018

smagafurov commented Feb 14, 2018

smagafurov commented Feb 14, 2018 •

edited

benoit-pierre commented Feb 14, 2018

jwilk commented Feb 14, 2018

benoit-pierre commented Feb 14, 2018

smagafurov commented Feb 14, 2018 •

edited

benoit-pierre commented Feb 14, 2018

benoit-pierre commented Feb 14, 2018

benoit-pierre commented Feb 14, 2018 •

edited

smagafurov commented Feb 15, 2018

golyalpha commented Feb 15, 2020 •

edited

pypa / setuptools

Sponsor pypa/setuptools

scan_module(): ValueError: bad marshal data (unknown type code) #1257

scan_module(): ValueError: bad marshal data (unknown type code) #1257

Comments

jwilk commented Jan 23, 2018

benoit-pierre commented Jan 25, 2018

jaraco commented Feb 3, 2018

jenisys commented Feb 5, 2018

smagafurov commented Feb 14, 2018

benoit-pierre commented Feb 14, 2018

smagafurov commented Feb 14, 2018

benoit-pierre commented Feb 14, 2018

smagafurov commented Feb 14, 2018

smagafurov commented Feb 14, 2018 • edited

benoit-pierre commented Feb 14, 2018

benoit-pierre commented Feb 14, 2018

smagafurov commented Feb 14, 2018

benoit-pierre commented Feb 14, 2018

smagafurov commented Feb 14, 2018

smagafurov commented Feb 14, 2018

smagafurov commented Feb 14, 2018 • edited

benoit-pierre commented Feb 14, 2018

jwilk commented Feb 14, 2018

benoit-pierre commented Feb 14, 2018

smagafurov commented Feb 14, 2018 • edited

benoit-pierre commented Feb 14, 2018

benoit-pierre commented Feb 14, 2018

benoit-pierre commented Feb 14, 2018 • edited

smagafurov commented Feb 15, 2018

golyalpha commented Feb 15, 2020 • edited

smagafurov commented Feb 14, 2018 •

edited

smagafurov commented Feb 14, 2018 •

edited

smagafurov commented Feb 14, 2018 •

edited

benoit-pierre commented Feb 14, 2018 •

edited

golyalpha commented Feb 15, 2020 •

edited