-
Notifications
You must be signed in to change notification settings - Fork 12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adding new software to install #448
Comments
It may indeed be a permissions issue. Let me take a look. For installations we use for production, we probably do not want to generally allow modification of the conda environment for concern about upgrading other packages in the environment. This twinkles_20170208 install is "older" and as far as I know, not used for production. |
For fun I tried to install basemap, as i feared, other packages would be impacted, such as numpy and scipy. I could try pinning those modules we know are used by DM so they wouldn't be touched.. alternatively we could try setting up a separate conda environment to try this out.
|
I need it but I would not argue that it is generally useful enough for including in your future installs.
I understand the concern about the production environment. But it would be nice to for other DESC members trying to run on NERSC to utilize your work in setting up stack installs, and adding new packages/modifying numpy versions etc. if necessary. While one could argue that each member trying to run on NERSC could follow your instructions in installing the stack, I wonder if it is possible to achieve this separation by using conda environments, or In the shorter term, could you please point me to your instructions for stack installs on NERSC? |
Did not see your update before I posted my reply, but yes I think conda environments would be a great way to go ... but this does not solve the problem of requiring a new numpy version, right? So, I think the question is
|
You're right, the conda env won't solve the problem on its own... but it could be a playground as we try to figure this out. In this case, I think the nomkl versions of numpy and scipy are a hard requirement for DM and that's precisely what the basemap installation is balking at. Other than that, the versions are exactly the same. It may be that basemap really wouldn't care if it was numpy nomkl. So I'd be inclined to try basemap with the versions of the packages that DM prefers. To answer b), I believe there's no problem with installing a new package that leaves the rest of the environment untouched - it just seems there are very few such packages! I think anything that depends on numpy or scipy defaults to use the mkl version of those packages - while we really want the nomkl versions. When you do the "conda install ", conda will report the changes it will perform, so it's easy enough to take a peek before proceeding. |
Ah I see, I did not know about pinning packages. This sounds good! Thanks for trying this! |
Still struggling with pinning in this case. I opened an issue and hoping an expert will respond: |
Haven't forgotten about this.. I have a workaround to avoid the introduction of the mkl versions of numpy and scipy where we have to include nomkl in the conda install step: |
@heather999 Thanks for pushing this along! Do I understand right that when the It might be good to draw attention of @danielsf and @rhiannonlynne here. I think |
Yes, @rbiswas4 that is my understanding, if nomkl is not indicated, then it is the mkl version since last year: |
I think you're right that numexpr just came from pandas. Actually I'm surprised that conda let you use a non nomkl version, so I think it's safe to go back to nomkl. The version of numpy Ivan be important, so I'd install nomkl and pin numpy to 1.11 for now; I think that would get you the right versions of everything else. |
Just wanted to give you an update @rbiswas4, I have cloned the conda env and made you your own environment. I have played with pinning, but have run into troubles. Fortunately, it seems I'm not the only one, so I opened a new post on community: |
Learned a little more about pinning, and I got that to work. However, I continued to run into dependency problems when trying to add the basemap package. So, this is what I've done: Dump the current set of installed python packages: @rbiswas4 I haven't used basemap, nor do I know how to test it out. I'm curious to see if this worked as we might expect. To test it, you can log onto Cori and setup to use twinkles_20170208, and then do I think in the future, perhaps our best bet is to gather a list of desired python packages in addition to what DM installs by default. We would plan to install those at the time of any new DM installations. We can grow that list as needed, but care will be required to introduce them. |
@heather999 Thanks! I probably won't get a chance to test this out till the weekend. If you are looking for a piece of code which tests
I think we need to have two things: Thanks! |
Unfortunately I think any package our group is likely to be interested in will have dependencies on packages like numpy, scipy, etc - so basemap might be a very good example of what we would typically run into. That said - while my initial run of the test failed, I now have it working. I had to re-install the conda env. It seems conda does not approve of my attempt to remove and then reintroduce a conda env with the same name. So instead of rb, the env is now named ame, that was conda's doing. The test.png file was generated and I could open it via xdg-open :) |
@heather999 I tested this out and this works! A slight problem I encountered was this:
but did not see a problem getting what I needed to do. Now that we have a working solution, my question is what is the best way to do this on say the latest stack (preferably not increasing anybody's work):
Thanks! |
Hi @rbiswas4 Yes, I saw that same Fontconfig error. I found this reference http://www.stata.com/support/faqs/unix/fontconfig-error/, this may be something to ask the NERSC helpdesk about. I'm assuming it's rather innocuous, but hopefully we can fix that. Concerning how to move ahead and use this - yes, we could have a setup script that points to this slightly updated python environment. That would avoid any possibility of inadvertently updating the production environment. Is that flexible enough? |
@heather999
I was trying to use the lsst sims install you put up on Cori but wanted to install some other python packages using the conda. Specifically, I tried to install
basmap
(https://anaconda.org/anaconda/basemap) to view plots of the sky in different projections.This fails with the message:
Should I expect this to happen, and what should I do to get around this problem? Thanks!
The text was updated successfully, but these errors were encountered: