New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Eggs/Wheels have broken shebangs #168

Closed
snordhausen opened this Issue Aug 26, 2015 · 8 comments

Comments

Projects
None yet
3 participants
@snordhausen

snordhausen commented Aug 26, 2015

For distribution on Pypi, I wanted to create both Eggs and Wheels with

    project.set_property("distutils_commands", ["sdist", "bdist_egg", "bdist_wheel"])

The project includes a script in src/main/scripts with a shebang of #!/usr/bin/env python. In the "sdist", this shebang remains intact and it works fine when installed from Pypi.

In the Egg and the Wheel, there is a modified shebang that points to the Python interpreter in the virtual environment that I use for building. Consequently, these files do not work when installed via Pypi. The same thing happens when I call the auto-generated setup.py myself.

I was able to work around this issue (after some inspiration) by manually adding

options = {'build_scripts': {'executable': '/usr/bin/env python'}},

into the setup.py, deleting the temporary files and then calling setup.py myself.

@mriehl

This comment has been minimized.

Show comment
Hide comment
@mriehl

mriehl Aug 26, 2015

Member

Hi, thanks for opening an issue. I actually already ran into the same problem, investigated, and decided that I will not use wheels for the time being.

In my opinion this is a bug in distutils. The PEP recommends to write #!python in your shebang which I think is stupid beyond belief since the script will not run unless you use some magic installer like pip which replaces the shebang.
Note that I'm not making this up

I guess we can hardcode the executable in the setup script but I'm not sure how this might affect existing users.

Member

mriehl commented Aug 26, 2015

Hi, thanks for opening an issue. I actually already ran into the same problem, investigated, and decided that I will not use wheels for the time being.

In my opinion this is a bug in distutils. The PEP recommends to write #!python in your shebang which I think is stupid beyond belief since the script will not run unless you use some magic installer like pip which replaces the shebang.
Note that I'm not making this up

I guess we can hardcode the executable in the setup script but I'm not sure how this might affect existing users.

@mriehl

This comment has been minimized.

Show comment
Hide comment
@mriehl

mriehl Aug 26, 2015

Member

Sorry for the rant... but what do you think? Did you know about the #!python part? (do you still want to use wheels? :-P). Maybe using #!python would be a possibility.. honestly I really like the simplicity of a tarball so apart from huge projects like numpy I never bother with wheels.

Also I'm not able to find documentation on executable.. what does it do? Does it always replace the shebang? e.G. if I have a script with #!/usr/bin/python3 will it rewrite it to #!/usr/bin/env python?

Member

mriehl commented Aug 26, 2015

Sorry for the rant... but what do you think? Did you know about the #!python part? (do you still want to use wheels? :-P). Maybe using #!python would be a possibility.. honestly I really like the simplicity of a tarball so apart from huge projects like numpy I never bother with wheels.

Also I'm not able to find documentation on executable.. what does it do? Does it always replace the shebang? e.G. if I have a script with #!/usr/bin/python3 will it rewrite it to #!/usr/bin/env python?

@mriehl

This comment has been minimized.

Show comment
Hide comment
@mriehl

mriehl Sep 25, 2015

Member

@snordhausen any update on this? IMHO the wheel spec sucks and should be changed so I'd rather not work around it. What do you think?

Member

mriehl commented Sep 25, 2015

@snordhausen any update on this? IMHO the wheel spec sucks and should be changed so I'd rather not work around it. What do you think?

@arcivanov

This comment has been minimized.

Show comment
Hide comment
@arcivanov

arcivanov Nov 13, 2015

Contributor

@mriehl question for you based on this:

which I think is stupid beyond belief since the script will not run unless you use some magic installer like pip which replaces the shebang.

It does assume the use of pip or a similar installer but here's the deal - you can't install a wheel without pip or a similar tool to begin with. And pip already does a proper rewrite, I believe.

Setuptools, however, does rewrite the shebang for wheel only.

We have a bug how we package as the order of invocation of setup tools matters.
If you do in pyb build.py clean package and then run ./setup.py bdist_wheel sdist you'll notice that the shebang in the pyb in wheel will be #!python and in sdist it'll be untouched.

However if you clean again and run ./build.py and then go and run ./setup.py bdist_wheel, the build/scripts-2.7/pyb already has a rewritten shebang and wheel picks that crap up.

Contributor

arcivanov commented Nov 13, 2015

@mriehl question for you based on this:

which I think is stupid beyond belief since the script will not run unless you use some magic installer like pip which replaces the shebang.

It does assume the use of pip or a similar installer but here's the deal - you can't install a wheel without pip or a similar tool to begin with. And pip already does a proper rewrite, I believe.

Setuptools, however, does rewrite the shebang for wheel only.

We have a bug how we package as the order of invocation of setup tools matters.
If you do in pyb build.py clean package and then run ./setup.py bdist_wheel sdist you'll notice that the shebang in the pyb in wheel will be #!python and in sdist it'll be untouched.

However if you clean again and run ./build.py and then go and run ./setup.py bdist_wheel, the build/scripts-2.7/pyb already has a rewritten shebang and wheel picks that crap up.

@arcivanov

This comment has been minimized.

Show comment
Hide comment
@arcivanov

arcivanov Nov 13, 2015

Contributor

So either we have to blow away $dist_dir/build or stop generating useless bdist_dumb that rewrites shebang right in the $dist_dir/build/scripts-2.7/pyb.

Contributor

arcivanov commented Nov 13, 2015

So either we have to blow away $dist_dir/build or stop generating useless bdist_dumb that rewrites shebang right in the $dist_dir/build/scripts-2.7/pyb.

@arcivanov

This comment has been minimized.

Show comment
Hide comment
@arcivanov

arcivanov Nov 13, 2015

Contributor

I'm going to implement the following solution:

  • Change defaults to ['sdist', 'bdist_wheel'].
  • Run clean -all before every command to ensure to insulate the individual builds from each other.
Contributor

arcivanov commented Nov 13, 2015

I'm going to implement the following solution:

  • Change defaults to ['sdist', 'bdist_wheel'].
  • Run clean -all before every command to ensure to insulate the individual builds from each other.
@mriehl

This comment has been minimized.

Show comment
Hide comment
@mriehl

mriehl Nov 13, 2015

Member

You're right, bdist_dump is usually not what you want.

It does assume the use of pip or a similar installer but here's the deal - you can't install a wheel without pip or a similar tool to begin with. And pip already does a proper rewrite, I believe.

That's true, but my point is that if I get the project sources and try to run the file (which should always work, IMHO) it won't run because the shebang is broken. So the app is broken unless I actually install it.
I like the "have the cake and eat it too" approach from being able to run the sources or install as a package, which is no longer easy with wheels.

Member

mriehl commented Nov 13, 2015

You're right, bdist_dump is usually not what you want.

It does assume the use of pip or a similar installer but here's the deal - you can't install a wheel without pip or a similar tool to begin with. And pip already does a proper rewrite, I believe.

That's true, but my point is that if I get the project sources and try to run the file (which should always work, IMHO) it won't run because the shebang is broken. So the app is broken unless I actually install it.
I like the "have the cake and eat it too" approach from being able to run the sources or install as a package, which is no longer easy with wheels.

@mriehl

This comment has been minimized.

Show comment
Hide comment
@mriehl

mriehl Nov 13, 2015

Member
  • Change defaults to ['sdist', 'bdist_wheel'].
  • Run clean -all before every command to ensure to insulate the individual builds from each other.

Sounds good.

Member

mriehl commented Nov 13, 2015

  • Change defaults to ['sdist', 'bdist_wheel'].
  • Run clean -all before every command to ensure to insulate the individual builds from each other.

Sounds good.

arcivanov added a commit to arcivanov/pybuilder that referenced this issue Nov 14, 2015

arcivanov added a commit to arcivanov/pybuilder that referenced this issue Nov 14, 2015

arcivanov added a commit to arcivanov/pybuilder that referenced this issue Nov 14, 2015

@arcivanov arcivanov closed this in #253 Nov 14, 2015

@arcivanov arcivanov removed the in progress label Nov 14, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment