Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bdist_egg --exclude-source-files does not work with Python3 #844

Open
dmiyakawa opened this Issue Nov 9, 2016 · 5 comments

Comments

Projects
None yet
4 participants
@dmiyakawa
Copy link

dmiyakawa commented Nov 9, 2016

(Coming from here: https://bugs.python.org/issue28630)

I'm trying to prepare an egg file without source code.
I had used python setup.py bdist_egg --exclude-source-files, which does not work with Python3.

After digging into the issue more, I found PEP-3147 is related to the problem.
https://www.python.org/dev/peps/pep-3147/

$ git clone https://github.com/dmiyakawa/python_greeting.git
..
$ cd python_greeting
$ python --version
Python 3.5.2
$ python setup.py bdist_egg --exclude-source-files
..
$ unzip -l dist/greeting-1.0.0-py3.5.egg
Archive:  dist/greeting-1.0.0-py3.5.egg
  Length      Date    Time    Name
---------  ---------- -----   ----
        1  11-09-2016 16:31   EGG-INFO/dependency_links.txt
       57  11-09-2016 16:31   EGG-INFO/entry_points.txt
      319  11-09-2016 16:31   EGG-INFO/PKG-INFO
      213  11-09-2016 16:31   EGG-INFO/SOURCES.txt
        9  11-09-2016 16:31   EGG-INFO/top_level.txt
        1  11-09-2016 16:31   EGG-INFO/zip-safe
      139  11-09-2016 16:31   greeting/__pycache__/__init__.cpython-35.pyc
      392  11-09-2016 16:31   greeting/__pycache__/greeting.cpython-35.pyc
---------                     -------
     1131                     8 files

Here, greeting/__pycache__/__init__.cpython-35.pyc and greeting/__pycache__/greeting.cpython-35.pyc are not where they should be.
See the following results (with Python 2.7.12)

$ python --version
Python 2.7.12
$ python setup.py bdist_egg --exclude-source-files
..
$ unzip -l dist/greeting-1.0.0-py2.7.egg
Archive:  dist/greeting-1.0.0-py2.7.egg
  Length      Date    Time    Name
---------  ---------- -----   ----
        1  11-09-2016 16:32   EGG-INFO/dependency_links.txt
       57  11-09-2016 16:32   EGG-INFO/entry_points.txt
      319  11-09-2016 16:32   EGG-INFO/PKG-INFO
      213  11-09-2016 16:32   EGG-INFO/SOURCES.txt
        9  11-09-2016 16:32   EGG-INFO/top_level.txt
        1  11-09-2016 16:32   EGG-INFO/zip-safe
      143  11-09-2016 16:32   greeting/__init__.pyc
      482  11-09-2016 16:32   greeting/greeting.pyc
---------                     -------
     1225                     8 files

According to the chart in PEP-3147, original py file must also exist to make pyc cache in __pycache__ effective.

To let bdist_egg's --exclude-sourcce-files option work as it worked in Python2 era, pyc files in __pycache__ must be at where legacy Python2 placed. I tried tweaking the Py3's egg content with the following code, and the tweaked egg worked nicely.

RE_PYCACHE_FILE = re.compile('(.+/)__pycache__/(.+)\.cpython-3\d(.pyc)$')

with tempfile.TemporaryDirectory() as tmpdir:
    names_lst = []
    tmpdir_abs_path = os.path.abspath(tmpdir)
    with ZipFile(egg_path, 'r') as zf:
        for name in zf.namelist():
            m = RE_PYCACHE_FILE.match(name)
            if m:
                w_name = m.group(1) + m.group(2) + m.group(3)
            else:
                w_name = name
            w_path = os.path.join(tmpdir_abs_path, w_name)
            dir_path = os.path.dirname(w_path)
            if not os.path.exists(dir_path):
                os.makedirs(dir_path)
            with open(w_path, 'wb') as file_to_write:
                with zf.open(name) as file_to_read:
                    file_to_write.write(file_to_read.read())
            names_lst.append(w_name)
    with ZipFile(egg_path, 'w', compression=ZIP_DEFLATED) as zf:
        for name in names_lst:
            r_path = os.path.join(tmpdir_abs_path, name)
            zf.write(r_path, arcname=name)
@gaw19

This comment has been minimized.

Copy link

gaw19 commented Jun 4, 2017

may i ask how would you use this code ? in a separate script ? or maybe contribute back into setuptools as a separate flag to enable it ?

@dmiyakawa

This comment has been minimized.

Copy link
Author

dmiyakawa commented Jun 4, 2017

Use it as a separate script toward the egg file that is discussed as problematic here.

@ghost

This comment has been minimized.

Copy link

ghost commented Sep 12, 2017

Could you tell me where exactly, we need to keep the above script? Below is the structure of my folder.
root_folder/
--------app_folder
--------setup.py
--------README.rst
--------MANIFEST.in

@haobibo

This comment has been minimized.

Copy link
Contributor

haobibo commented Sep 13, 2017

@dmiyakawa I created a PR (#1145) to mend this issue.
The PR directly rename files in the bdist_egg command.
Your comments or suggestions are welcome.

@dmiyakawa

This comment has been minimized.

Copy link
Author

dmiyakawa commented Sep 14, 2017

Sorry I had been satisfied with my workaround and haven't seen this issue for a while.

Seems there are people who want the functionality, while I am still not sure if mainstream developers think the feature itself is still important enough, or, should be deprecated and removed in the future (I hope the former of course). #1145 looks quite nice.

Maybe wheel should have same functionality?

@bRsatUoh Try it after building the egg. dist/ dir and compiled egg will be needed.

@haobibo Looks very nice but CI complains something.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.