Conflict between python paths for Python and Fortran packages

[ketch]

In preparing the 5.2 release, @rjleveque and I discovered the following. If you clone and then

python setup.py install

that puts a compiled PyClaw in your site-packages. However, you don't have access to packages like geoclaw via Python, because they don't get copied to your site-packages. If you then put your clawpack clone on your PYTHONPATH, as our instructions suggest, you suddenly have access to geoclaw in Python, but you no longer have a working PyClaw (because you're not pointing to the one that got built).

[ketch]

For the moment, we have resolved this by doing an editable install on Travis.

[ahmadia]

Why don't we just install the geoclaw Python sources to site-packages? I don't understand.

[ketch]

@ahmadia I discussed that solution with @rjleveque today. This is a decision for @mandli @rjleveque and perhaps other developers of the Fortran packages.

[mandli]

I imagine the complaint with doing this might be that changing the python source in the src directory will not change the code unlike the fortran where this is the case. Doing an editable install I think implies perfectly the type of behavior that we are use to and people who are likely to muck with the Python are perhaps savvy enough to do editable installs (I actually find these easier myself). Perhaps if they are not installed in this manor we could remove the source from the GeoClaw source tree as to not cause confusion?

[ketch]

+1 for what @mandli suggests:

• Install Python parts of GeoClaw, AMRClaw, and Classic to site-packages
• Delete the corresponding directories in the cloned repo, possibly replacing them with symlinks to site-packages subdirectories

[rjleveque]

I guess that's ok and consistent with what other Python packages do, but I guess that means all users would have to do a full Python install, which would build PyClaw too? Not so bad I guess but it will be a change from what the Fortran users now do. It also seems a bit strange to me that users will end up with a repository where all the Fortran code is available to peruse and modify but the Python code has magically disappeared. But maybe I'm just old fashioned.

[ketch]

This all goes back to what we discussed last summer: Python's setuptools are meant for installing libraries (i.e., software that the user may build on but would not edit). If you consider the Python code in GeoClaw to be a library, then putting it in site-packages makes sense. If you don't consider it to be a library, then it doesn't make sense -- and using setuptools to install it (as we do!) doesn't make sense either.

I'm not arguing for a particular plan of action, since I don't use the Fortran packages. Just trying to point out the root of all this.

[ketch]

And for the record, as far as PyClaw is concerned, I view it as a library in this regard (so the current installation makes sense). The same goes for Riemann.

[ketch]

Just to be clear, there is at least one other option that would completely resolve this: PyClaw and Riemann could be promoted to top-level packages, outside the Clawpack namespace. That option has some disadvantages that we discussed last summer (and which led to the current setup).

Sorry for making so many comments so quickly.

[mandli]

I fell like for the most part that the Python in GeoClaw and it's ilk could be considered library code while the Fortran is left alone. I actually think it might be beneficial to consider the Fortran source as a library as well and compile and structure things a bit differently but that might be a topic for a broader discussion. Maybe the real question to address is how we figure users are mostly like to interface with packages like GeoClaw, do they expect to be able to modify the source of the Python?

[rjleveque]

No, most users won't be modifying the Python probably although I do sometimes find it useful to refer users to the code for examples of how to do something. As long as there still an option to do an editable install or use "python setup.py symlink-only" for users who want access to these files it may be less confusing for the average user to have all the Python code handled the same way and treated as libraries.

[ketch]

It sounds like we should make it so the python parts of all packages are installed to site-packages. We will modify the installation instructions to make it clear that if users want to modify the python code (for any package) then they should do an editable install. Any objections?

[rjleveque]

That seems ok to me. It's possible some users will want to modify the
geoclaw (or amrclaw) python codes for their own needs, but we can include
some instructions on how to copy the desired source files to their own
directory to modify, and how to set the path to find their own first.

• Randy

On Sat, Aug 9, 2014 at 12:55 PM, David Ketcheson notifications@github.com
wrote:

It sounds like we should make it so the python parts of all packages are
installed to site-packages. We will modify the installation instructions to
make it clear that if users want to modify the python code (for any
package) then they should do an editable install. Any objections?

—
Reply to this email directly or view it on GitHub
#44 (comment).

[mandli]

+1

[ketch]

Related reading: "Run-time extensibility and librarization of
simulation software" http://arxiv.org/abs/1407.2905

The first page is especially recommended.

[rjleveque]

Interesting article, thanks for the link.

If we drop the pip install -e . option, as recently discussed offline, and have all python modules installed to site-packages so we can avoid setting PYTHONPATH, what's the recommended way for people (users or developers) to modify the Python code when necessary?

[ahmadia]

What's the recommended way for people (users or developers) to modify the Python code when necessary?

My recommendation was to remove pip install -e as advice to anybody who was trying to install and run our code. If the intent is to modify any Python code, then pip install -e is still my recommendation. If the libraries are installed consistently (they are all located under the top-level clawpack install), we will stop seeing these issues.

[mandli]

I have not found a better option and have not had issues as long as you install consistently as @ahmadia suggested. If I need to switch between versions I still use virtualenvs as I have not found something that is ubiquitous and easy to use as that quite yet. Perhaps we can address clarifying the installation instructions at SIAM CSE in a week at the very least and discuss installing the other Python packages to the site-packages to correct for the inconsistency problems.

[ketch]

If we drop the pip install -e . option, as recently discussed offline, and have all python modules installed to site-packages so we can avoid setting PYTHONPATH, what's the recommended way for people (users or developers) to modify the Python code when necessary?

This should be no different from users who need to modify, say, numpy, matplotlib, or any other Python package. You download the code, modify it, and run python setup.py build followed by python setup.py install.

[mandli]

Addressed in #61.