RuntimeError: original source dist cannot contain .pyc files

[netrounds-oleg-b]

Trying to convert PIP packages into Debian packages using python-stdeb (for maintainability purposes). Having scripts and mechanism to do it, which works perfectly fine. It converts any PIP package into debian except for flasgger version 0.6.0+ (so 0.5.14 and every version below getting converted just fine):

$ ./build.sh
Collecting flasgger==0.6.5
  Downloading flasgger-0.6.5.tar.gz (1.7MB)
  Saved /tmp/pip2deb_flasgger==0.6.5_mJFDnt/flasgger-0.6.5.tar.gz
Successfully downloaded flasgger
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
running the following command in directory: /tmp/pip2deb_flasgger==0.6.5_mJFDnt/flasgger-0.6.5/tmp_py2dsc/flasgger-0.6.5
/usr/bin/python setup.py --command-packages stdeb.command sdist_dsc --dist-dir=/tmp/pip2deb_flasgger==0.6.5_mJFDnt/flasgger-0.6.5 --use-premade-distfile=/tmp/pip2deb_flasgger==0.6.5_mJFDnt/flasgger-0.6.5.tar.gz --extra-cfg-file=/shared/pip2deb.conf bdist_deb
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
running sdist_dsc
running egg_info
writing requirements to flasgger.egg-info/requires.txt
writing flasgger.egg-info/PKG-INFO
writing top-level names to flasgger.egg-info/top_level.txt
writing dependency_links to flasgger.egg-info/dependency_links.txt
reading manifest file 'flasgger.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*' under directory 'flasgger/ui3/static/css'
warning: no files found matching '*' under directory 'flasgger/ui3/static/fonts'
warning: no files found matching '*' under directory 'flasgger/ui3/static/images'
warning: no files found matching '*' under directory 'flasgger/ui3/static/lang'
warning: no files found matching '*' under directory 'flasgger/ui3/static/lib'
writing manifest file 'flasgger.egg-info/SOURCES.txt'
configuration files were specified, but no options were found in "flasgger" or "DEFAULT" sections.
Traceback (most recent call last):
  File "setup.py", line 52, in <module>
    'six>=1.10.0'
  File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/usr/local/lib/python2.7/dist-packages/stdeb/command/sdist_dsc.py", line 123, in run
    raise RuntimeError('original source dist cannot '
RuntimeError: original source dist cannot contain .pyc files
ERROR running: /usr/bin/python setup.py --command-packages stdeb.command sdist_dsc --dist-dir=/tmp/pip2deb_flasgger==0.6.5_mJFDnt/flasgger-0.6.5 --use-premade-distfile=/tmp/pip2deb_flasgger==0.6.5_mJFDnt/flasgger-0.6.5.tar.gz --extra-cfg-file=/shared/pip2deb.conf bdist_deb
ERROR in /tmp/pip2deb_flasgger==0.6.5_mJFDnt/flasgger-0.6.5/tmp_py2dsc/flasgger-0.6.5
Traceback (most recent call last):
  File "pip2deb.py", line 129, in <module>
    main()
  File "pip2deb.py", line 124, in main
    extra_cfg_file=args.extra_cfg_file,
  File "pip2deb.py", line 84, in download_and_build
    extra_cfg_file=extra_cfg_file,
  File "pip2deb.py", line 66, in build
    subprocess.check_call(cmd, env=env)
  File "/usr/lib/python2.7/subprocess.py", line 541, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['py2dsc-deb', '--dist-dir', '/tmp/pip2deb_flasgger==0.6.5_mJFDnt/flasgger-0.6.5', '--extra-cfg-file', 'pip2deb.conf', '/tmp/pip2deb_flasgger==0.6.5_mJFDnt/flasgger-0.6.5.tar.gz']' returned non-zero exit status 1

Again this happens only of Flasgger version >= 0.6.0.

Was something changed severely between 0.5.14 and 0.6.0 that could cause the problem with PYC files?

[rochacbruno]

between 0.5.14 and 0.6.0 many things changed, we instroduced python 3.x support and it comes with six package also we splitted flasgger in different modules.

I cannot find any explicit reason for the .pyc being included, but maybe the problem is the use of * in MANIFEST.in

Can you try editing your https://github.com/rochacbruno/flasgger/blob/master/MANIFEST.in#L2 remove that line and try again.

We may need to include all static files include one by one or use specific wildcard to get them instead of *

[netrounds-oleg-b]

@rochacbruno thank you for a response.

Can you try editing your https://github.com/rochacbruno/flasgger/blob/master/MANIFEST.in#L2 remove that line and try again.

Have cloned flasgger, modified the MANIFEST.in as you suggested and ran my build script. This time it would build all dependency-modules:

In the end it failed for the same reason. By the way, reverting MINIFEST.in back didn't affect anything. So looks like, unfortunately, it is not the case.

[JoakimSoderberg]

The *.pyc files included are these:

# find /tmp/ -type f -name '*.pyc'
pip2deb_flasgger==0.6.6_2Kpip0/ tmpFXJFhK/
root@2ea58c377e15:/shared# find /tmp/pip2deb_flasgger\=\=0.6.6_2Kpip0/ -type f -name '*.pyc'
/tmp/pip2deb_flasgger==0.6.6_2Kpip0/flasgger-0.6.6/tmp_sdist_dsc/flasgger-0.6.6/flasgger/__pycache__/utils.cpython-36.pyc
/tmp/pip2deb_flasgger==0.6.6_2Kpip0/flasgger-0.6.6/tmp_sdist_dsc/flasgger-0.6.6/flasgger/__pycache__/marshmallow_apispec.cpython-36.pyc
/tmp/pip2deb_flasgger==0.6.6_2Kpip0/flasgger-0.6.6/tmp_sdist_dsc/flasgger-0.6.6/flasgger/__pycache__/__init__.cpython-36.pyc
/tmp/pip2deb_flasgger==0.6.6_2Kpip0/flasgger-0.6.6/tmp_sdist_dsc/flasgger-0.6.6/flasgger/__pycache__/constants.cpython-36.pyc/tmp/pip2deb_flasgger==0.6.6_2Kpip0/flasgger-0.6.6/tmp_sdist_dsc/flasgger-0.6.6/flasgger/__pycache__/base.cpython-36.pyc
/tmp/pip2deb_flasgger==0.6.6_2Kpip0/flasgger-0.6.6/flasgger-0.6.6/flasgger/__pycache__/utils.cpython-36.pyc
/tmp/pip2deb_flasgger==0.6.6_2Kpip0/flasgger-0.6.6/flasgger-0.6.6/flasgger/__pycache__/marshmallow_apispec.cpython-36.pyc
/tmp/pip2deb_flasgger==0.6.6_2Kpip0/flasgger-0.6.6/flasgger-0.6.6/flasgger/__pycache__/__init__.cpython-36.pyc
/tmp/pip2deb_flasgger==0.6.6_2Kpip0/flasgger-0.6.6/flasgger-0.6.6/flasgger/__pycache__/constants.cpython-36.pyc
/tmp/pip2deb_flasgger==0.6.6_2Kpip0/flasgger-0.6.6/flasgger-0.6.6/flasgger/__pycache__/base.cpython-36.pyc
/tmp/pip2deb_flasgger==0.6.6_2Kpip0/flasgger-0.6.6/tmp_py2dsc/flasgger-0.6.6/flasgger/__pycache__/utils.cpython-36.pyc
/tmp/pip2deb_flasgger==0.6.6_2Kpip0/flasgger-0.6.6/tmp_py2dsc/flasgger-0.6.6/flasgger/__pycache__/marshmallow_apispec.cpython-36.pyc
/tmp/pip2deb_flasgger==0.6.6_2Kpip0/flasgger-0.6.6/tmp_py2dsc/flasgger-0.6.6/flasgger/__pycache__/__init__.cpython-36.pyc
/tmp/pip2deb_flasgger==0.6.6_2Kpip0/flasgger-0.6.6/tmp_py2dsc/flasgger-0.6.6/flasgger/__pycache__/constants.cpython-36.pyc
/tmp/pip2deb_flasgger==0.6.6_2Kpip0/flasgger-0.6.6/tmp_py2dsc/flasgger-0.6.6/flasgger/__pycache__/base.cpython-36.pyc

[JoakimSoderberg]

They don't get included if I use the git URL in requirements.txt which results in a .zip instead of .tar.gz gets downloaded... And in this case py2dsc-deb is happy and does not find any of these pyc files. Not sure why.

But of course it would be nice to be able to download these as a normal pip package as well.

[rochacbruno]

looks like __pycache__ folder is being distributed with the pip release.

[rochacbruno]

And the problem must be the https://github.com/rochacbruno/flasgger/blob/master/MANIFEST.in

[rochacbruno]

or maybe the problem is https://github.com/rochacbruno/flasgger/blob/fe53c1dbed31e0ecd0752f068f03b09f4183458f/setup.py#L44

[JoakimSoderberg]

https://docs.python.org/2/distutils/sourcedist.html#commands

Maybe this can be used in the MANIFEST file:

global-exclude pat1 pat2 ...	exclude all files anywhere in the source tree matching — & any of the listed patterns

something like (if I understand it correctly):

global-exclude __pycache__