-
Notifications
You must be signed in to change notification settings - Fork 62
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
move uberenv to new spack release #77
Comments
spack v0.10.0 was released on 2017-01-17 |
In addition to testing packages, this requires reworking our |
Thoughts + Notes on upgrading our developer TPL build process to a newer version of spack. These notes apply not only to Conduit, but also Axom and Alpine (https://github.com/Alpine-DAV/) , since they use the same spack-based TPL build strategy. Since Axom and Alpine both depend on Conduit, we expect to flush out the upgrade process in Conduit, and then propagate it to those projects. Additionally, there are other LLNL codes that have similar requirements -- even if they aren't using uberenv. Our ultimate goal is a sharable & reproducible TPL build process. Once someone has paved the way for building TPLs on a platform (eg: a specific HPC cluster or OSX) with a given set of compilers, other team members should be able to easily replicate this. Easily here means minimally selecting which compiler to use, and possibly some feature variants -- the platform should automatically be detected. Here are a few key things we want to do in support of this goal:
Towards this goal, spack does the heavy lifting for us -- however we also use a small python script named uberenv (https://github.com/LLNL/conduit/blob/master/scripts/uberenv/uberenv.py) to automate the process. uberenv is a thin veneer around spack that helps automate TPL builds for Axom, Conduit, or Alpine. Here is what uberenv currently does: -checks out spack from github (a specific hash, selected by an entry in a json file) -optionally helps setup a spack mirror in a shared location -patches spack to disable any user or system settings related to compilers -copies in a compilers.yaml file with blessed compilers -patches spack to limit max number of build jobs to 8 -copies a set of custom spack packages over the built-in spack package repo files -launches the spack build of a special "uberenv-zzz" package for a specific spack spec This special "uberenv-zzz" package specifies all of the dependencies that are needed to develop the desired software project zzz (eg: Conduit, Axom, etc) It does not build these software projects, instead it generates a file that can be used to locate the compilers and all of the TPLs that spack built. This file is called a "host-config" file b/c we use the host name in the names of these files. The host config file contents are used as CMake initial cache file that the build systems of all of these projects support. We revision control these files, so that anyone on a system with a shared TPL install can use this file to bootstrap a build. Long term, we hope to simplify uberenv and rely more on spack features to achieve the same process. The version of spack we are using is quite old, but to update we need to address a few issues: The version we are using allows us to easily craft a single compilers.yaml file that outlines details for a wide range of systems. We can do so generally (say for Linux or OSX) and provide more specific options for a known HPC cluster (we are using LLNL's SYS_TYPE var). For a concrete example, see: https://github.com/LLNL/conduit/blob/master/scripts/uberenv/compilers.yaml Newer versions of spack support different naming schemes to identify platforms. We would even be happy with host-name based solutions. But we need to understand what is supported, and how we can craft our new compilers.yaml file(s). We want to use external packages for a very small set of TPLs. The version we are using lacks support for external packages. In our automated build process for Conduit and Alpine on LLNL clusters, we rely on the proper MPI and CUDA being exposed in a user's PATH. (MPI via looking for mpicc, CUDA via looking for nvcc). Axom's TPLs don't require MPI yet. In our automated build process for Axom TPLs, we have manually created files that augment the spack generated host config to enable MPI. These manual edits files are revison controlled, and provide per platform + compiler paths to MPI. In both cases, we want to get away from our current solutions and instead use external package support via packages.yaml. With a newer version of spack, I don't know what the correct path to use packages.yaml to select a specific MPI for a given platform + compiler. We need to discuss what is supported. We could manage the platform specifics via uberenv spack tweaks (forcing a specific compilers.yaml + packages.yaml file based on SYS_TYPE), but we would like to do as much as possible using spack features. We would like spack command line options that allow us to specify a "compilers.yaml" and "packages.yaml" file. When these are passed, spack needs to use these and ignore any other user or system level spack settings. When this exists, we can remove the uberenv step that patches spack to disable user settings. We would like a spack command line option to limit the max build jobs, to replace our current patch. (Perhaps this already exists?) In some cases, errors that occur when building packages aren't captured in log files. Unfortunately -- this happens in LLNL batch jobs, which are a very important case for our automated builds. I have also seen it a CI setting where I was building a docker container inside of a docker container. This is a big issue b/c when things go wrong, we have to spawn another build by hand (outside of a batch job) to try to tackle the error. How do we move our custom packages forward? There are two issues here: a. Our current packages are tested heavily and frequently on LLNL's HPC clusters, and we had to hardened them against the build horrors we experienced (for example on BG/Q). New packages have not been exposed to the same vetting. There will be many issues when we upgrading. b. We can address this with our own packages, but we want to use as many off the shelf packages as possible. |
the output issue (5 above) is now resolved in spack develop, w/ the merge of spack/spack#5084 |
(4 above), there is a config option in new version of spack to limit the # of build tasks. |
We meet with @becker33, here are a few near term things we will work in:
|
this is complete with #225, which enhanced uberenv to fill the gaps needed for us to update to a newer spack. However it still outlines our wishlist for spack support that would simplify what uberenv requires. |
there should be a new spack release before SC16, eval if we should move uberenv to this version.
The text was updated successfully, but these errors were encountered: