Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bdist_egg --exclude-source-files does not work with Python3 #844

Open
dmiyakawa opened this issue Nov 9, 2016 · 6 comments
Open

bdist_egg --exclude-source-files does not work with Python3 #844

dmiyakawa opened this issue Nov 9, 2016 · 6 comments

Comments

@dmiyakawa
Copy link

@dmiyakawa dmiyakawa commented Nov 9, 2016

(Coming from here: https://bugs.python.org/issue28630)

I'm trying to prepare an egg file without source code.
I had used python setup.py bdist_egg --exclude-source-files, which does not work with Python3.

After digging into the issue more, I found PEP-3147 is related to the problem.
https://www.python.org/dev/peps/pep-3147/

$ git clone https://github.com/dmiyakawa/python_greeting.git
..
$ cd python_greeting
$ python --version
Python 3.5.2
$ python setup.py bdist_egg --exclude-source-files
..
$ unzip -l dist/greeting-1.0.0-py3.5.egg
Archive:  dist/greeting-1.0.0-py3.5.egg
  Length      Date    Time    Name
---------  ---------- -----   ----
        1  11-09-2016 16:31   EGG-INFO/dependency_links.txt
       57  11-09-2016 16:31   EGG-INFO/entry_points.txt
      319  11-09-2016 16:31   EGG-INFO/PKG-INFO
      213  11-09-2016 16:31   EGG-INFO/SOURCES.txt
        9  11-09-2016 16:31   EGG-INFO/top_level.txt
        1  11-09-2016 16:31   EGG-INFO/zip-safe
      139  11-09-2016 16:31   greeting/__pycache__/__init__.cpython-35.pyc
      392  11-09-2016 16:31   greeting/__pycache__/greeting.cpython-35.pyc
---------                     -------
     1131                     8 files

Here, greeting/__pycache__/__init__.cpython-35.pyc and greeting/__pycache__/greeting.cpython-35.pyc are not where they should be.
See the following results (with Python 2.7.12)

$ python --version
Python 2.7.12
$ python setup.py bdist_egg --exclude-source-files
..
$ unzip -l dist/greeting-1.0.0-py2.7.egg
Archive:  dist/greeting-1.0.0-py2.7.egg
  Length      Date    Time    Name
---------  ---------- -----   ----
        1  11-09-2016 16:32   EGG-INFO/dependency_links.txt
       57  11-09-2016 16:32   EGG-INFO/entry_points.txt
      319  11-09-2016 16:32   EGG-INFO/PKG-INFO
      213  11-09-2016 16:32   EGG-INFO/SOURCES.txt
        9  11-09-2016 16:32   EGG-INFO/top_level.txt
        1  11-09-2016 16:32   EGG-INFO/zip-safe
      143  11-09-2016 16:32   greeting/__init__.pyc
      482  11-09-2016 16:32   greeting/greeting.pyc
---------                     -------
     1225                     8 files

According to the chart in PEP-3147, original py file must also exist to make pyc cache in __pycache__ effective.

To let bdist_egg's --exclude-sourcce-files option work as it worked in Python2 era, pyc files in __pycache__ must be at where legacy Python2 placed. I tried tweaking the Py3's egg content with the following code, and the tweaked egg worked nicely.

RE_PYCACHE_FILE = re.compile('(.+/)__pycache__/(.+)\.cpython-3\d(.pyc)$')

with tempfile.TemporaryDirectory() as tmpdir:
    names_lst = []
    tmpdir_abs_path = os.path.abspath(tmpdir)
    with ZipFile(egg_path, 'r') as zf:
        for name in zf.namelist():
            m = RE_PYCACHE_FILE.match(name)
            if m:
                w_name = m.group(1) + m.group(2) + m.group(3)
            else:
                w_name = name
            w_path = os.path.join(tmpdir_abs_path, w_name)
            dir_path = os.path.dirname(w_path)
            if not os.path.exists(dir_path):
                os.makedirs(dir_path)
            with open(w_path, 'wb') as file_to_write:
                with zf.open(name) as file_to_read:
                    file_to_write.write(file_to_read.read())
            names_lst.append(w_name)
    with ZipFile(egg_path, 'w', compression=ZIP_DEFLATED) as zf:
        for name in names_lst:
            r_path = os.path.join(tmpdir_abs_path, name)
            zf.write(r_path, arcname=name)
@gaw19
Copy link

@gaw19 gaw19 commented Jun 4, 2017

may i ask how would you use this code ? in a separate script ? or maybe contribute back into setuptools as a separate flag to enable it ?

@dmiyakawa
Copy link
Author

@dmiyakawa dmiyakawa commented Jun 4, 2017

Use it as a separate script toward the egg file that is discussed as problematic here.

@ghost
Copy link

@ghost ghost commented Sep 12, 2017

Could you tell me where exactly, we need to keep the above script? Below is the structure of my folder.
root_folder/
--------app_folder
--------setup.py
--------README.rst
--------MANIFEST.in

@haobibo
Copy link
Contributor

@haobibo haobibo commented Sep 13, 2017

@dmiyakawa I created a PR (#1145) to mend this issue.
The PR directly rename files in the bdist_egg command.
Your comments or suggestions are welcome.

@dmiyakawa
Copy link
Author

@dmiyakawa dmiyakawa commented Sep 14, 2017

Sorry I had been satisfied with my workaround and haven't seen this issue for a while.

Seems there are people who want the functionality, while I am still not sure if mainstream developers think the feature itself is still important enough, or, should be deprecated and removed in the future (I hope the former of course). #1145 looks quite nice.

Maybe wheel should have same functionality?

@bRsatUoh Try it after building the egg. dist/ dir and compiled egg will be needed.

@haobibo Looks very nice but CI complains something.

@laggardkernel
Copy link

@laggardkernel laggardkernel commented Aug 11, 2020

Today, I encountered the same problem when build .egg package for Scrapy deployment. (.egg packages are required by scrapyd for deployment).

Even there's no __pycache__ in my source files, __pycache__ folders are added within the .egg pacakge, which doesn't happen with .wheel packages.

Seems a bug.

Environment:

  • Python 3.7.3
  • pip 19.0.3
  • setuptools 40.8.0
  • wheel 0.34.2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants