Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Install with conda #13

Closed
brey opened this issue Apr 2, 2020 · 35 comments
Closed

Install with conda #13

brey opened this issue Apr 2, 2020 · 35 comments

Comments

@brey
Copy link
Contributor

brey commented Apr 2, 2020

Dear All, may I suggest to configure also a conda based installation option, preferably on conda-forge. One can have multiple builds to meet all the configurations of SCHISM.

I think this option will provide a flexible and portable way to use SCHISM.

I have played a bit with this and I have managed to have it up and running relatively easy. You can try it out with

conda install -c gbrey pschism

It works for linux-64 and os-x.

Let me know if I could be of any help.

George
@platipodium
Copy link
Member

I would like a conda install possibility, but be aware that conda is not recommended on HPC systems (we recently removed it from ours), see e.g. https://docs.computecanada.ca/wiki/Anaconda/en.

Also, with the current strategy of most dependencies resolved in-tree, the dependency management is not (yet) too complicated (mostly netcdf). This could become a more pressing issue if we get Joseph to accept more relying on external dependencies .. :=).

@josephzhang8
Copy link
Member

I think George B. also mentioned the possibility of conda install.

Too bad conda is now blacklisted as we are moving in that direction.

@jreniel
Copy link
Member

jreniel commented Jan 5, 2021

Not being able to use conda is inconvenient, but not a game changer, as you are still allowed to use virtualenv. The main difference would be that with with virtualenv you might need to satisfy some dependencies manually, something that we still have to do anyways when we provision a machine with the SCHISM binaries.

@josephzhang8
Copy link
Member

Good to know; thx Jaime.

@brey
Copy link
Contributor Author

brey commented Jan 6, 2021

Dear All, I do agree that for a production HPC env it would make sense to have a module based installation for efficiency but, in my opinion, for development and prototyping nothing beats the simplicity of conda. Having the same configuration on the HPC and my laptop (where the dev is happening) within minutes without the need of Admin rights is very productive. I am using conda on our HPC since I couldn't get SCHISM compiled with the available modules and I am only doing test runs anyway. Your mileage may vary. The above referenced conda command installs the 4.8 tagged version and works for me (let me know if you have any problems). It would take some efforts to have all options configured though. I have compiled mine with mpich but we need to have one with OpenMP as well. Not to mention all the variants of SCHISM modules. I will make a pull request with the conda recipe files asap.

@brey
Copy link
Contributor Author

brey commented Nov 12, 2021

I am working in finalising the conda recipe. May I reiterate the possibility of having the metis/parmetis dependency covered independently from a stand alone installation.

Both packages are available in conda.

Any reason why they need to be included in the build process?

@josephzhang8
Copy link
Member

josephzhang8 commented Nov 12, 2021 via email

@brey
Copy link
Contributor Author

brey commented Nov 12, 2021

In that case, any ideas on how to mask the compilation of the local Metis folder from CMake (not my strong point)?

@josephzhang8
Copy link
Member

josephzhang8 commented Nov 12, 2021 via email

@jreniel
Copy link
Member

jreniel commented Nov 12, 2021 •
edited
Loading

Not my strong point either, but my intuition tells me that you probably still need the parmetis headers to compile schism, and of course, they have to match whatever binary version is being bootstrapped through conda. Since you still need the parmetis headers, and since the parmetis source code is bundled inside the schism source code, doing full parmetis compilation might actually be better than using a conda one. In any case, it's your choice, it depends how much you want to leverage "flexibility" against complexity.

In any case, I think it might be possible to bootstrap the conda metis lib, as long as the headers used match versions (I'm not sure if conda will distribute the headers along with the binaries though, which would technically be the safest route). I would just compile the provided metis as part of the schism-conda recipe, but that's just me...

@jreniel
Copy link
Member

jreniel commented Nov 12, 2021 •
edited
Loading

FYI, if you want to deploy SCHISM anywhere and make sure it uses with modules then you can use any of these links:

https://github.com/jreniel/schism_deploy_v5.8.0
https://github.com/jreniel/schism_deploy_v5.9.0
https://github.com/jreniel/schism_deploy_master

@josephzhang8
Copy link
Member

josephzhang8 commented Nov 12, 2021 via email

@jreniel
Copy link
Member

jreniel commented Nov 12, 2021 •
edited
Loading

is the subject of this to compile parmetis and not metis or is the subject to drop the whole library.

The subject is whether the conda recipe should use the precompiled version that exists in the conda repo, or should it compile the one included with SCHISM. Technically you can do either, the question is what you gain and lose from chose one or the other. I don't think dropping the whole library is part of the conversation (I don't think that's even possible, since parmetis obviosuly plays a critical role!)

Easy and safe is to compiled the one bundled. Using the conda one would probably require to find and match the correct version of the headers, so while doable, might be more complicated than simply compiling the one provided.

One thing we discussed in the burst of activity around moving to GitHub was the possibility of automatically downloading parmetis if its location isn't named. I believe this required a version bump. I guess it isn't a very powerful idea though while the library is housed inside the schism source code.

I remember these conversations, and while we flirted with the idea of downloading on the fly, now that I have more experience, I think it is better to just keep it in the source code (for simplicity). But there are other formal alternatives, most notably git subtrees, which I would suggest to talk about in the event that we need to change the Parmetis version in the future.

@josephzhang8
Copy link
Member

josephzhang8 commented Nov 12, 2021 via email

@jreniel
Copy link
Member

jreniel commented Nov 12, 2021

FYI This isn't Eli talking. He hasn't actually responded. Eli probably knows by memory which CMake variables need to be overridden for using conda parmetis. I would have to dig a little to find them, but through CMake variables you certainly can override which Parmetis gets used.

@brey
Copy link
Contributor Author

brey commented Feb 13, 2022

Hi All. I have sorted out the issues with CMAKE and METIS/PARMETIS. The conda recipe avoids compiling the local library and uses the conda-forge version.

You can get both pschism and pschism-wwm with

conda install -c gbrey pschism=5.9

or

conda install -c gbrey pschism-wwm=5.9

However, I am having an issue with the wwm flavor on Linux (it compiles nicely on osx). I have traced the problem to the definition of constants in OPENMP. See https://stackoverflow.com/questions/63016641/cant-support-both-openmp-v4-and-v5-in-gcc.

Apparently such an issue doesn't affect clang that is why it works on osx. Let me know if you've seen this issue and how, in your opinion, should be addressed.

@platipodium
Copy link
Member

Thanks for the update. I think that we should deprecate the use of gcc < 9 / OpenMP < 5. The current stable version is 11, with version 12 expected in April 2021.

@jamal919
Copy link
Member

However, I am having an issue with the wwm flavor on Linux (it compiles nicely on osx). I have traced the problem to the definition of constants in OPENMP. See https://stackoverflow.com/questions/63016641/cant-support-both-openmp-v4-and-v5-in-gcc.

Do you think #57 relates to this problem?

@platipodium
Copy link
Member

@gbrey what is the status of the conda deployment?

@brey
Copy link
Contributor Author

brey commented Mar 5, 2023

@platipodium I think it's done. I was wondering if there is a need to have the OLD IO option even with >5.10 but I think that having the 5.9 version should be enough. You can test both versions on my channel with

conda install -c gbrey pschism=5.10*=mpi_openmpi*
conda install -c gbrey pschism=5.10*=mpi_mpich*
conda install -c gbrey pschism=5.9*=mpi_openmpi*
conda install -c gbrey pschism=5.9*=mpi_mpich*

and

conda install -c gbrey pschism-wwm=5.10*=mpi_openmpi*
conda install -c gbrey pschism-wwm=5.10*=mpi_mpich*
conda install -c gbrey pschism-wwm=5.9*=mpi_openmpi*
conda install -c gbrey pschism-wwm=5.9*=mpi_mpich*

depending on your MPI flavour. I will do a last check and then I will push a PL to the repo and also one in the conda-forge channel if no one has an objection.

In fact someone should also be admin/maintainer on conda-forge besides me. Any volunteers?

@platipodium
Copy link
Member

@gbrey Thanks for the update.

  • Yes, I think you should submit to conda-forge.
  • If needed, I can be a secondary maintainer.
  • Yes, we need OLDIO for all versions, as we need single-processor capability. (I know you can circumvent this with --oversubscribe, but that is much slower)

@josephzhang8
Copy link
Member

josephzhang8 commented Mar 5, 2023 via email

@brey brey mentioned this issue Mar 5, 2023
Closed
@brey
Copy link
Contributor Author

brey commented Mar 5, 2023

@platipodium Keeping the OLDIO option creates a problem.

Currently there are 2 binaries (TVD-LD and WWM) and 2 MPI options (mpich and openmpi). Thus the 4 total versions/release above. However, if we include the OLDIO as an option it would create 8 versions and most importantly a naming issue. How are we going to call the OLDIO packages?

pschism, pschism-oldio
pschism-wwm, pschism-wwm-oldio

This will create some confusion. Is running on one core a long term requirement?

In fact, this is relevant to a more general question: How many conda versions we want to release (in terms of modules)? If there are many, the OLDIO option will increase the number. Ideally, the best option would be to have one binary and invoke the modules in runtime. Is this doable?

Also, maybe there is a way to speed up --oversubscribe.

@josephzhang8
Copy link
Member

josephzhang8 commented Mar 5, 2023 via email

@brey
Copy link
Contributor Author

brey commented Mar 5, 2023

Thanks Joseph. How about using only OpenMP and multithreading for running on one core without oversubscribing?

@platipodium
Copy link
Member

  1. Hmm, I think we need to find a way to deal with multiple options. OLDIO is only one of them, WWM, TVD-VL, FABM, GEN and any of the others could also be relevant. In principle conda should be able to handle this, but I am no expert on the variants.

There could be a default variant
pschism (using openmpi, OLDIO=OFF and TVD-VL=SB, or any other)

And then many variants with positive selection in any way conceivable....
pschism+FABM+OLDIO+WWM+SB
pschism+VL+WWM+GEN+ICM+MPICH

We can use CD to create all of them and the respective conda packages.

  1. Not providing MPI (in both flavours openmpi/mpich) would deter users who need that capability, not offering single-processor options would deter first-timers or people who'd like to try the system or who have simple test cases.

@josephzhang8
Copy link
Member

josephzhang8 commented Mar 6, 2023 via email

@brey
Copy link
Contributor Author

brey commented Mar 6, 2023 

not offering single-processor options would deter first-timers or people who'd like to try the system or who have simple test cases.

I am not sure I get this. Every computer now is multicore. So testing the code on a simple case, even with MPI, and at minimum 2 cores shouldn't be a problem. If you output a large number of vectors this could be an issue for, say, an 8 core laptop but this gets very specific.

Note that coda installation is good enough for prototyping but not necessarily for production (side note, @pmav99 achieved extremely good latency with a conda implementation on Azure HPC with InfiniBand) .

In addition, conda provides automatically the MPI without any issues for the user. I remember you've mentioned multiple 1 core experiments with FABM. Those become now 2+ cores experiments which is an issue but I would say poses no big constraint for a new user testing stuff. It would, however, require more resources for scaling up.

Did I miss something?

Another option would be to update the code on the 5.9 branch differentiating between OLDIO and the new one on the semantic versioning. I am not sure though if this is the right approach going forward.

@josephzhang8
Copy link
Member

Issue resolved per @brey

@brey
Copy link
Contributor Author

brey commented Mar 8, 2023

Dear All, I am trying to finalise the matrix of binaries that will be offered via conda. From the cmake files, I have:

#   Algorithm choices
# TVD_LIM must be one of SB, VL, MM or OS for Superbee, Van Leer, Minmod, or Osher.")
set (TVD_LIM VL CACHE STRING "Flux limiter")
#Turn OLDIO off to use the new scribe based I/O
set (OLDIO OFF CACHE BOOLEAN "Old nc output (each rank dumps its own data)")
set (PREC_EVAP OFF CACHE BOOLEAN "Include precipitation and evaporation calculation")
set (USE_GOTM OFF CACHE BOOLEAN "Use GOTM turbulence model. This just enables the build -- GOTM must still be selected in param.nml")
set (USE_HA OFF CACHE BOOLEAN "Enable harmonic analysis output modules")   # Not implemented yet
set( USE_MARSH OFF CACHE BOOLEAN "Use marsh module")

#   Enable/Disable Modules
set( USE_SED2D OFF CACHE BOOLEAN "Use 2D sediment module")
set( USE_WWM OFF CACHE BOOLEAN "Use wind-wave module")
set( USE_ICE OFF CACHE BOOLEAN "Use ICE module")

#Tracer models
set( USE_GEN OFF  CACHE BOOLEAN "Use generic tracer module")
set( USE_AGE OFF  CACHE BOOLEAN "Use age module")
set( USE_ECO OFF   CACHE BOOLEAN "Use ECO-SIM module")
set( USE_ICM OFF   CACHE BOOLEAN "Use ICM module")
##pH model is active only if ICM is on
set( ICM_PH OFF   CACHE BOOLEAN "Use pH module inside ICM")
set( USE_COSINE OFF   CACHE BOOLEAN "Use CoSiNE module")
set( USE_FIB OFF   CACHE BOOLEAN "Use fecal indicating bacteria module")
set( USE_SED OFF    CACHE BOOLEAN "Use sediment module")
set( USE_FABM OFF  CACHE BOOLEAN "FABM BGC model interface")

set (DEBUG OFF CACHE BOOLEAN "Enable diagnostic output")
set (USE_ANALYSIS OFF CACHE BOOLEAN "Enable (somewhat costly) derviation of derived flow/stress quantities")

Q1. What are the options for TVD_LIM? I am bit confused with OS.
Q2. Are all module options compatible between them? That is, do all possible combinations of SED2, WWM & ICE work and therefore should be offered?
Q3. Should there be only one tracer model available per binary or more tracer modules can be present?
Q4. Same as Q2 for PH, COSINE, FIB, SED (different from SED2?) and FABM.

In a nutshell, if you can provide a matrix of the combinations we should provide, I can finalise the conda recipe.

@platipodium
Copy link
Member

Hmm, I see how this could be very confusing for users who want to "Just download SCHISM". How about providing those that we (whoever) frequently use. My personal list is as matrix of all combinations of [SB, OLDIO, FABM, GEN]

@josephzhang8
Copy link
Member

josephzhang8 commented Mar 9, 2023 via email

@brey
Copy link
Contributor Author

brey commented Mar 12, 2023

I have pushed the final conda recipe. It includes instructions on how to build ones own package. At this point, I think it's better if the interested users build their own package with their own configuration. If the complexity of the modules/models improves or a way to initiate them in run time is developed we can consider conda-forge.

I don't believe there is a graceful way to build all the variants in conda-forge currently. Maybe also conda will offer some new ways going forward.

@platipodium
Copy link
Member

Out of curiosity, how do you provide your own channel -c gbrey ?

@brey
Copy link
Contributor Author

brey commented Mar 12, 2023

https://docs.anaconda.com/anacondaorg/user-guide/tasks/work-with-packages/#uploading-packages

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@platipodium @jamal919 @brey @jreniel @josephzhang8