Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Automatically use Python 3 if available #11783

Closed
wants to merge 1 commit into from
Closed

Automatically use Python 3 if available #11783

wants to merge 1 commit into from

Conversation

michaelkuhn
Copy link
Member

As discussed during today's BoF, some people would like Spack to use Python 3 if available. Since we cannot depend on any version of Python being available on all systems, this needs a slightly complex approach: The spack binary is moved to spack-real and replaced by a shell script that checks for available versions of Python (preferring Python 3) and invokes spack-real accordingly.

This should also take care of the situation where no python binary is available (as will be on RHEL 8 by default).

Not sure if this is really the best way to go but I have been meaning to take a stab at this for a while now. (Only tested on Linux.)
@tgamblin @alalazo @becker33 @adamjstewart

@michaelkuhn
Automatically use Python 3 if available
8c90048 
As discussed during today's BoF, some people would like Spack to use
Python 3 if available. Since we cannot depend on any version of Python
being available on all systems, this needs a slightly complex approach:
The spack binary is moved to spack-real and replaced by a shell script
that checks for available versions of Python (preferring Python 3) and
invokes spack-real accordingly.

This should also take care of the situation where no python binary is
available (as will be on RHEL 8 by default).
@scheibelp
Copy link
Member

Also @citibeth who has worked on related Python3 concerns in Spack (although generally more-related to building Spack packages with Python3 rather than using Python3 to run Spack):

See also:

@citibeth
Copy link
Member

Adding an extra shellscript in front sounds like a Java nightmare to me. Why not just use whatever Python the system provides? In my experience, Spack works fine with either.

@adamjstewart
Copy link
Member

I agree with @citibeth, if users really want to test python3, they can alias python to python3.

@citibeth
Copy link
Member

Another simpler way to do it would be to just rewrite the #/usr/bin/env line at the top of bin/spack when you install. Why don't we just do that? A simple "switcher" script could be made to do that, similar to the way MacPorts and CentOS allow you to "select" certain versions. Then the 99% of users who don't need this functionality won't need to deal with the cruft it brings every time the program is run.

@michaelkuhn
Copy link
Member Author

Another simpler way to do it would be to just rewrite the #/usr/bin/env line at the top of bin/spack when you install. Why don't we just do that? A simple "switcher" script could be made to do that, similar to the way MacPorts and CentOS allow you to "select" certain versions. Then the 99% of users who don't need this functionality won't need to deal with the cruft it brings every time the program is run.

I like that idea (will give it a go once ISC is over). It might still make sense to default to python3 at some point since at least a few Linux distributions are dropping the python binary soon (Fedora, RHEL/CentOS).

@citibeth
Copy link
Member

citibeth commented Jun 19, 2019 via email

tgamblin added a commit that referenced this pull request Sep 29, 2019
@tgamblin
prefer python3 to python2 for running Spack.
e5d2f0b 
The Python landscape is going to be changing in 2020, and Python 2 will
be end of life. Spack should to *prefer* Python 3 to Python 2 by default.

This is trickier than it sounds, as on some systems, the `python` command
is python2; on others it's python3, and RHEL8 doesn't even have the
`python` command.  Instead, it makes you choose python3 or python2. You
can thus no longer make a simple shebang to handle all the cases.

This commit makes the spack script bilingual.  It is still valid Python,
but its shebang is `#!/bin/sh`, and it has a tiny bit of shell code at
the beginning to pick the right python and execute itself with what it
finds.

This has some advantages.  You can still execute the spack script with a
python interpreter to force your own version of python, or to run
coverage and profiling tools on the spack script itself.  i.e., this
works if you really wan to force python2, or if you want to run
pyinstrument on spack:

```console
$ python2 $(which spack) <args>
$ coverage run $(which spack) <args>
$ pyinstrument $(which spack) <args>
```

These would not work if we split `spack` into a python file and a shell
script (see #11783).  So, this gives us the best of both worlds.  We get
to control our interpreter *and* remain a mostly pure python executable.
@tgamblin tgamblin mentioned this pull request Sep 29, 2019
Merged
tgamblin added a commit that referenced this pull request Sep 29, 2019
@tgamblin
prefer Python 3 to Python 2 for running Spack
79b1594 
The Python landscape is going to be changing in 2020, and Python 2 will
be end of life. Spack should *prefer* Python 3 to Python 2 by default,
but we still need to run on systems that only have Python2 available.

This is trickier than it sounds, as on some systems, the `python` command
is `python2`; on others it's `python3`, and RHEL8 doesn't even have the
`python` command.  Instead, it makes you choose `python3` or
`python2`. You can thus no longer make a simple shebang to handle all the
cases.

This commit makes the `spack` script bilingual.  It is still valid
Python, but its shebang is `#!/bin/sh`, and it has a tiny bit of shell
code at the beginning to pick the right python and execute itself with
what it finds.a

This has a lot of advantages.  I think this will help ensure that Spack
works well in Python3 -- there are cases where we've missed things
because Python2 is still the default `python` on most systems.  Also,
with this change, you do not lose the ability to execute the `spack`
script directly with a python interpreter.  This is useful for forcing
your own version of python, running coverage tools, and running profiling
tools.  i.e., these will not break with this change:

```console
$ python2 $(which spack) <args>
$ coverage run $(which spack) <args>
$ pyinstrument $(which spack) <args>
```

These would not work if we split `spack` into a python file and a shell
script (see #11783).  So, this gives us the best of both worlds.  We get
to control our interpreter *and* remain a mostly pure python executable.
tgamblin added a commit that referenced this pull request Sep 29, 2019
@tgamblin
prefer Python 3 to Python 2 for running Spack
23268eb 
The Python landscape is going to be changing in 2020, and Python 2 will
be end of life. Spack should *prefer* Python 3 to Python 2 by default,
but we still need to run on systems that only have Python2 available.

This is trickier than it sounds, as on some systems, the `python` command
is `python2`; on others it's `python3`, and RHEL8 doesn't even have the
`python` command.  Instead, it makes you choose `python3` or
`python2`. You can thus no longer make a simple shebang to handle all the
cases.

This commit makes the `spack` script bilingual.  It is still valid
Python, but its shebang is `#!/bin/sh`, and it has a tiny bit of shell
code at the beginning to pick the right python and execute itself with
what it finds.

This has a lot of advantages.  I think this will help ensure that Spack
works well in Python3 -- there are cases where we've missed things
because Python2 is still the default `python` on most systems.  Also,
with this change, you do not lose the ability to execute the `spack`
script directly with a python interpreter.  This is useful for forcing
your own version of python, running coverage tools, and running profiling
tools.  i.e., these will not break with this change:

```console
$ python2 $(which spack) <args>
$ coverage run $(which spack) <args>
$ pyinstrument $(which spack) <args>
```

These would not work if we split `spack` into a python file and a shell
script (see #11783).  So, this gives us the best of both worlds.  We get
to control our interpreter *and* remain a mostly pure python executable.
@tgamblin
Copy link
Member

Hey folks, see #12972 -- a little black magic allows this to be accomplished with none of the very annoying downsides @citibeth mentioned above.

@citibeth
Copy link
Member

citibeth commented Sep 29, 2019 via email

tgamblin added a commit that referenced this pull request Sep 29, 2019
@tgamblin
prefer Python 3 to Python 2 for running Spack
d0ce203 
The Python landscape is going to be changing in 2020, and Python 2 will
be end of life. Spack should *prefer* Python 3 to Python 2 by default,
but we still need to run on systems that only have Python2 available.

This is trickier than it sounds, as on some systems, the `python` command
is `python2`; on others it's `python3`, and RHEL8 doesn't even have the
`python` command.  Instead, it makes you choose `python3` or
`python2`. You can thus no longer make a simple shebang to handle all the
cases.

This commit makes the `spack` script bilingual.  It is still valid
Python, but its shebang is `#!/bin/sh`, and it has a tiny bit of shell
code at the beginning to pick the right python and execute itself with
what it finds.

This has a lot of advantages.  I think this will help ensure that Spack
works well in Python3 -- there are cases where we've missed things
because Python2 is still the default `python` on most systems.  Also,
with this change, you do not lose the ability to execute the `spack`
script directly with a python interpreter.  This is useful for forcing
your own version of python, running coverage tools, and running profiling
tools.  i.e., these will not break with this change:

```console
$ python2 $(which spack) <args>
$ coverage run $(which spack) <args>
$ pyinstrument $(which spack) <args>
```

These would not work if we split `spack` into a python file and a shell
script (see #11783).  So, this gives us the best of both worlds.  We get
to control our interpreter *and* remain a mostly pure python executable.
tgamblin added a commit that referenced this pull request Sep 29, 2019
@tgamblin
prefer Python 3 to Python 2 for running Spack
97980a8 
The Python landscape is going to be changing in 2020, and Python 2 will
be end of life. Spack should *prefer* Python 3 to Python 2 by default,
but we still need to run on systems that only have Python2 available.

This is trickier than it sounds, as on some systems, the `python` command
is `python2`; on others it's `python3`, and RHEL8 doesn't even have the
`python` command.  Instead, it makes you choose `python3` or
`python2`. You can thus no longer make a simple shebang to handle all the
cases.

This commit makes the `spack` script bilingual.  It is still valid
Python, but its shebang is `#!/bin/sh`, and it has a tiny bit of shell
code at the beginning to pick the right python and execute itself with
what it finds.

This has a lot of advantages.  I think this will help ensure that Spack
works well in Python3 -- there are cases where we've missed things
because Python2 is still the default `python` on most systems.  Also,
with this change, you do not lose the ability to execute the `spack`
script directly with a python interpreter.  This is useful for forcing
your own version of python, running coverage tools, and running profiling
tools.  i.e., these will not break with this change:

```console
$ python2 $(which spack) <args>
$ coverage run $(which spack) <args>
$ pyinstrument $(which spack) <args>
```

These would not work if we split `spack` into a python file and a shell
script (see #11783).  So, this gives us the best of both worlds.  We get
to control our interpreter *and* remain a mostly pure python executable.
tldahlgren pushed a commit to tldahlgren/spack that referenced this pull request Oct 11, 2019
@tgamblin @tldahlgren
prefer Python 3 to Python 2 for running Spack
d16996f 
The Python landscape is going to be changing in 2020, and Python 2 will
be end of life. Spack should *prefer* Python 3 to Python 2 by default,
but we still need to run on systems that only have Python2 available.

This is trickier than it sounds, as on some systems, the `python` command
is `python2`; on others it's `python3`, and RHEL8 doesn't even have the
`python` command.  Instead, it makes you choose `python3` or
`python2`. You can thus no longer make a simple shebang to handle all the
cases.

This commit makes the `spack` script bilingual.  It is still valid
Python, but its shebang is `#!/bin/sh`, and it has a tiny bit of shell
code at the beginning to pick the right python and execute itself with
what it finds.

This has a lot of advantages.  I think this will help ensure that Spack
works well in Python3 -- there are cases where we've missed things
because Python2 is still the default `python` on most systems.  Also,
with this change, you do not lose the ability to execute the `spack`
script directly with a python interpreter.  This is useful for forcing
your own version of python, running coverage tools, and running profiling
tools.  i.e., these will not break with this change:

```console
$ python2 $(which spack) <args>
$ coverage run $(which spack) <args>
$ pyinstrument $(which spack) <args>
```

These would not work if we split `spack` into a python file and a shell
script (see spack#11783).  So, this gives us the best of both worlds.  We get
to control our interpreter *and* remain a mostly pure python executable.
jrmadsen pushed a commit to jrmadsen/spack that referenced this pull request Oct 30, 2019
@tgamblin @jrmadsen
prefer Python 3 to Python 2 for running Spack
42f1990 
The Python landscape is going to be changing in 2020, and Python 2 will
be end of life. Spack should *prefer* Python 3 to Python 2 by default,
but we still need to run on systems that only have Python2 available.

This is trickier than it sounds, as on some systems, the `python` command
is `python2`; on others it's `python3`, and RHEL8 doesn't even have the
`python` command.  Instead, it makes you choose `python3` or
`python2`. You can thus no longer make a simple shebang to handle all the
cases.

This commit makes the `spack` script bilingual.  It is still valid
Python, but its shebang is `#!/bin/sh`, and it has a tiny bit of shell
code at the beginning to pick the right python and execute itself with
what it finds.

This has a lot of advantages.  I think this will help ensure that Spack
works well in Python3 -- there are cases where we've missed things
because Python2 is still the default `python` on most systems.  Also,
with this change, you do not lose the ability to execute the `spack`
script directly with a python interpreter.  This is useful for forcing
your own version of python, running coverage tools, and running profiling
tools.  i.e., these will not break with this change:

```console
$ python2 $(which spack) <args>
$ coverage run $(which spack) <args>
$ pyinstrument $(which spack) <args>
```

These would not work if we split `spack` into a python file and a shell
script (see spack#11783).  So, this gives us the best of both worlds.  We get
to control our interpreter *and* remain a mostly pure python executable.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
python3
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@michaelkuhn @scheibelp @citibeth @adamjstewart @tgamblin