Successfully Concretize Python3-Based DAGs #7926

citibeth · 2018-04-26T19:18:19Z

This is an on-going PR to collect unmergable changes ("hacks") to Spack packages required to build Python3-based software stacks. The hacks are due to shortcomings in the concretizer, which will be solved someday. Until then, this can serve as a resource. Please add to this PR if you find more changes needed.

davydden · 2018-04-26T19:23:11Z

var/spack/repos/builtin/packages/petsc/package.py

@@ -114,7 +114,7 @@ class Petsc(Package):
    depends_on('mpi', when='+mpi')

    # Build dependencies
-    depends_on('python@2.6:2.8', type='build')


can you check if PETSc support 3.x? Maybe that should be

depends_on('python@2.6:2.8,3.abc:', type='build')

I've asked the authors. Until then, I'm assuming it's Python2 only.

so the intention is to avoid using Spack's python and hope the user has python 2 in path?

p.s. I thought there were some PRs merged on conretization of build dependencies separately (@adamjstewart @alalazo ?) that might help here to have 2 different version of python in the graph.

p.p.s. you should also adjust slepc then

@davydden Yes that is the intention. Whatever concretization fixes were merged, they don't solve this problem. If you try concretizing a software stack that involves python@3 and petsc, concretization will fail. This PR is never going to be merged, it is simply a resource for people to get their Python3 stuff working.

p.p.s. you should also adjust slepc then

Since I don't use slepc, I'm not going to bother.

This PR is never going to be merged, it is simply a resource for people to get their Python3 stuff working.

ah, i see, i missed that.

I thought there were some PRs merged on conretization of build dependencies separately

You're thinking of #2548, which still hasn't been merged ☹️

which still hasn't been merged

☹️

balay · 2018-04-26T19:30:23Z

petsc configure uses python-2 - and does not yet work with python-3

…

On Thu, 26 Apr 2018, Denis Davydov wrote: davydden commented on this pull request. > @@ -114,7 +114,7 @@ class Petsc(Package): depends_on('mpi', when='+mpi') # Build dependencies - ***@***.***:2.8', type='build') can you check if PETSc support 3.x? Maybe that should be ``` ***@***.***:2.8, 3.abc:', type='build') ```

balay · 2018-04-26T19:51:05Z

Should add: petsc-3.9 has the following change: https://bitbucket.org/petsc/petsc/commits/a3e07f7d98935102301a2849d6c5439db2167066 i.e if 'python3 configure' is invoked - it looks for python2 in path - and attempts to use it.

…

On Thu, 26 Apr 2018, Denis Davydov wrote: davydden commented on this pull request. > @@ -114,7 +114,7 @@ class Petsc(Package): depends_on('mpi', when='+mpi') # Build dependencies - ***@***.***:2.8', type='build') so the intention is to avoid using Spack's python and hope the user has python 2 in path? p.s. I thought there were some PRs merged on conretization of build dependencies separately ***@***.*** @alalazo ?) that might help here to have 2 different version of python in the graph. p.p.s. you should also adjust slepc then

adamjstewart · 2018-04-26T20:11:54Z

I'm not a huge fan of PRs that are never intended to be merged...

What about a note in http://spack.readthedocs.io/en/latest/known_issues.html that describes what the problem is and how to resolve it if you run into it?

citibeth · 2018-04-27T00:38:55Z

Having a pr there makes it easier to merge in if you need it.

…

On Thu, Apr 26, 2018 at 4:12 PM Adam J. Stewart ***@***.***> wrote: I'm not a huge fan of PRs that are never intended to be merged... What about a note in http://spack.readthedocs.io/en/latest/known_issues.html that describes what the problem is and how to resolve it if you run into it? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#7926 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB1cd07nG3J9drXa7qGWK770NeW252yCks5tsioMgaJpZM4TnCKG> .

citibeth · 2018-04-28T03:54:44Z

I realized this could be turned into a mergeable PR. Packages that need it can have a +python3 variant. It would be turned on when concretizing with Python3. I know it's a bit of a crutch compared to fixing the concretizer. But it would work, and be better than what we have now. Interested users would just have to put all: [+python3] in their packages.yaml.

citibeth · 2018-04-28T21:32:11Z

OK... I've made this PR mergeable. Please see the documentation for how it works. If we can merge this, then we will get a substantial improvement for anyone using Python3 in their software stack.

davydden · 2018-04-29T04:40:38Z

lib/spack/docs/known_issues.rst

+variant to the problematic package, with the understanding that the
+user must set ``all: ['+python3']`` in ``packages.yaml`` for software
+stacks involving Python3.
+


maybe mention #2548 as a WIP which should resolved this issue.

davydden · 2018-04-29T04:41:22Z

var/spack/repos/builtin/packages/petsc/package.py

@@ -91,6 +91,8 @@ class Petsc(Package):
            multi=False)
    variant('suite-sparse', default=False,
            description='Activates support for SuiteSparse')
+    variant('python3', default=False,


i would add a comment above

FIXME: remove once #2548 is merged and tested

Or maybe just merge #2548 and forget about this crazy PR. @tgamblin @scheibelp ping???

svenevs · 2018-05-02T02:33:32Z

Yaaaaay python3!!! 2020 is coming up pretty soon ;) If the decision is to merge this please ping me and I will PR to your PR with other affected packages I've hacked locally for python 3.

I don't know how I feel about introducing the variant, because when it disappears it'll orphan those packages. However, having a PR open even if it doesn't get merged means people can at the very least apply the patch here with two commands (fetch the pr and merge) rather than needing to find them all in their own. That works for me because I only do python3, but may have more serious implications for traditional spack usage (I'm only administering my local system, not a cluster).

svenevs · 2018-05-02T02:40:02Z

Ugh sorry I read too fast / got too excited. Given the no merge intention, I will submit the PR when I have access to that computer again tonight / tomorrow.

citibeth · 2018-05-02T11:46:26Z

Yaaaaay python3!!! 2020 is coming up pretty soon ;) If the decision is to merge this please ping me and I will PR to your PR with other affected packages I've hacked locally for python 3.

Please create a PR based on this branch (efischer/Python3Fixes), and I will have it added to this PR. If we can collect all the Python3-related fixes in one place, then that will make life easier for others who need it.

Given the no merge intention, I will submit the PR when I have access to that computer again tonight / tomorrow.

At first, this PR was no merge, because it would break Python2. Now with the +python3 variant, it can be merged; hence the "ready" flag.

I don't know how I feel about introducing the variant, because when it disappears it'll orphan those packages.

That's why I left a note on #2548 to revert this PR once it is merged. The more we can collect all our Python3 fixes in one place resulting in a single merge, the easier that reversion will be. At worst, someone (@citibeth?) will have to grep through the packages for python3, determine the commit in which each of those variants was introduced (via git blame), and revert each of those commits. It's really not hard, and Git really is amazing.

That works for me because I only do python3, but may have more serious implications for traditional spack usage (I'm only administering my local system, not a cluster).

@tgamblin Yes that's why we need to merge either this or #2548 sooner, rather than later.

python3: more

s-sajid-ali · 2018-06-01T14:05:39Z

I've made a separate PR for python-3 only version of ipython. Since ipython and matplotlib along with numpy and scipy are now python-3 only, it's strange to see python-3 support being a variant not the other way around!

citibeth · 2018-06-01T14:26:19Z

Since ipython and matplotlib along with numpy and scipy are now python-3 only, it's strange to see python-3 support being a variant not the other way around!

At some point we will almost certainly switch from supportying Python2 by default to python3. I just don't know when that point will be.

…Fixes

s-sajid-ali · 2018-06-08T20:06:10Z

I was wondering if we could split the python package into 2 packages, one for python2 and another for python3. This would be a simpler fix than trying to fix the concretizer itself.

Did spack decide against this ?

citibeth · 2018-06-08T20:41:50Z

On Fri, Jun 8, 2018 at 4:06 PM, Sajid Ali ***@***.***> wrote: I was wondering if we could split the python package into 2 packages, one for python2 and another for python3. This would be a simpler fix than trying to fix the concretizer itself. Did spack decide against this ?

The current arrangement is considered to be preferable. We will either fix the concretizer, or eventually merge something like this PR.

To overcome the limitation of current concretiser in spack. See spack#7926

See spack#7926

chuckatkins · 2019-05-22T17:17:16Z

lib/spack/docs/known_issues.rst

+
+This can happen for two reasons, both related to the concretizer:
+
+1. A package requires ``python@2`` as a build dependency only (eg:


The inverse of this happens as well. For instance, mesa required python 3 as a build-only dependency, but paraview requires python 2. Thus paraview+python fails to concretize. The workaround at the moment is to do a two stage install where you first install mesa explicitly. Then when you install par5aview, the concretizer only evaluates the link and runtime dependencies of the installed mesa.

adamjstewart · 2019-08-03T18:19:00Z

I've been using:

packages:
    python:
        version: ['3:']

in my packages.yaml for awhile now, and I have yet to run into any Python 3 concretization problems on develop. Is this PR still necessary?

hack: Python3

1b5a89a

citibeth added bug Something isn't working concretization python3 labels Apr 26, 2018

davydden requested a review from balay April 26, 2018 19:22

davydden reviewed Apr 26, 2018

View reviewed changes

citibeth added the WIP label Apr 26, 2018

citibeth changed the title ~~Fixes for Python3 Build~~ Ongoing: Fixes for Python3 Build Apr 26, 2018

More changes to make sure PETSc uses System Python2.

0d078fa

Elizabeth Fischer added 4 commits April 28, 2018 16:22

python3

71d8aa6

Implemented +python3 variant.

5e93ae3

Added python3 variant.

3e1d033

Added documentation

669b767

citibeth changed the title ~~Ongoing: Fixes for Python3 Build~~ Successfully Concretize Python3-Based DAGs Apr 28, 2018

citibeth requested a review from davydden April 28, 2018 21:31

citibeth added ready and removed WIP labels Apr 28, 2018

davydden approved these changes Apr 29, 2018

View reviewed changes

This was referenced Apr 30, 2018

qt3 and qt4/q5 causing unsatisfiable version constraint #7946

Closed

Perform concretization of build-only deps separately #2548

Closed

citibeth mentioned this pull request May 2, 2018

Python as a virtual dependency #7966

Open

fix syntax

ddd38f8

citibeth mentioned this pull request May 16, 2018

Weird Build Bug with Spack Environments #8146

Closed

python3 stuff

3b36025

python3: more

citibeth mentioned this pull request May 31, 2018

concretization error for python3 #8301

Closed

citibeth mentioned this pull request Jun 1, 2018

Ipython > 6 is python 3 only. Update pacakges and dependencies. #8309

Closed

Merge remote-tracking branch 'upstream/develop' into efischer/Python3…

34571f1

…Fixes

adamjstewart mentioned this pull request Jun 14, 2018

Dependency patch being applied to dependent #8475

Open

pramodk added a commit to BlueBrain/spack that referenced this pull request Aug 26, 2018

Allow PETSc to build with Python 3.

e9bd45b

To overcome the limitation of current concretiser in spack. See spack#7926

pramodk added a commit to BlueBrain/spack that referenced this pull request Aug 26, 2018

Allow PETSc to build with Python 3.

1c14137

To overcome the limitation of current concretiser in spack. See spack#7926

This was referenced Aug 26, 2018

Allow PETSc to build with Python 3. BlueBrain/spack#69

Merged

steps: new package BlueBrain/spack#68

Merged

pramodk added a commit to BlueBrain/spack that referenced this pull request Aug 27, 2018

Allow PETSc to build with Python 3. (#69)

1417a80

See spack#7926

pramodk added a commit to BlueBrain/spack that referenced this pull request Sep 29, 2018

Allow PETSc to build with Python 3. (#69)

4539f67

See spack#7926

pramodk added a commit to BlueBrain/spack that referenced this pull request Nov 10, 2018

Allow PETSc to build with Python 3. (#69)

81873ce

See spack#7926

pramodk added a commit to BlueBrain/spack that referenced this pull request Nov 27, 2018

Allow PETSc to build with Python 3. (#69)

ae6ebd1

See spack#7926

This was referenced Dec 12, 2018

Maintain a view for an environment #10017

Merged

emacs build error unsatisfied contraint meson with incompatible python #9979

Closed

benzwick mentioned this pull request Jan 5, 2019

Installation issue: petsc4py ^python@3: #10256

Closed

citibeth mentioned this pull request Jan 15, 2019

Default to Python 3.7 #10319

Merged

This was referenced May 13, 2019

Installation issue: py-matplotlib #11396

Closed

Python3 Migration Strategy #11468

Closed

chuckatkins reviewed May 22, 2019

View reviewed changes

scheibelp mentioned this pull request Jun 18, 2019

Automatically use Python 3 if available #11783

Closed

adamjstewart closed this in #10319 Oct 22, 2019

tgamblin deleted the efischer/Python3Fixes branch October 25, 2019 06:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Successfully Concretize Python3-Based DAGs #7926

Successfully Concretize Python3-Based DAGs #7926

citibeth commented Apr 26, 2018

davydden Apr 26, 2018 •

edited

Loading

citibeth Apr 26, 2018

davydden Apr 26, 2018

citibeth Apr 26, 2018

davydden Apr 26, 2018

adamjstewart Apr 26, 2018

davydden Apr 29, 2018

balay commented Apr 26, 2018 via email

balay commented Apr 26, 2018 via email

adamjstewart commented Apr 26, 2018

citibeth commented Apr 27, 2018 via email

citibeth commented Apr 28, 2018

citibeth commented Apr 28, 2018

davydden Apr 29, 2018

davydden Apr 29, 2018

citibeth May 1, 2018

svenevs commented May 2, 2018

svenevs commented May 2, 2018

citibeth commented May 2, 2018

s-sajid-ali commented Jun 1, 2018

citibeth commented Jun 1, 2018 via email

s-sajid-ali commented Jun 8, 2018 •

edited

Loading

citibeth commented Jun 8, 2018 via email

chuckatkins May 22, 2019

adamjstewart commented Aug 3, 2019


		This can happen for two reasons, both related to the concretizer:

		1. A package requires ``python@2`` as a build dependency only (eg:

Successfully Concretize Python3-Based DAGs #7926

Successfully Concretize Python3-Based DAGs #7926

Conversation

citibeth commented Apr 26, 2018

davydden Apr 26, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

balay commented Apr 26, 2018 via email

balay commented Apr 26, 2018 via email

adamjstewart commented Apr 26, 2018

citibeth commented Apr 27, 2018 via email

citibeth commented Apr 28, 2018

citibeth commented Apr 28, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

svenevs commented May 2, 2018

svenevs commented May 2, 2018

citibeth commented May 2, 2018

s-sajid-ali commented Jun 1, 2018

citibeth commented Jun 1, 2018 via email

s-sajid-ali commented Jun 8, 2018 • edited Loading

citibeth commented Jun 8, 2018 via email

Choose a reason for hiding this comment

adamjstewart commented Aug 3, 2019

davydden Apr 26, 2018 •

edited

Loading

s-sajid-ali commented Jun 8, 2018 •

edited

Loading