Update Uberenv and introduce Gitlab CI [feature/uberenv_ci] #502

adrienbernede · 2020-02-08T00:44:46Z

In this PR I updated uberenv to the latest version (https://github.com/LLNL/uberenv/tree/master + LLNL/uberenv#20), and introduce a very basic Gitlab CI pipeline.

CI simply compiles 4 conduit targets using uberenv.

I will also include LLNL/uberenv#19, which should help preserve the current workflow.

This PR is ready for review.

cyrush · 2020-02-08T00:52:10Z

scripts/uberenv/project.json

+"package_final_phase" : "configure",
+"package_source_dir" : "../..",
+"spack_url": "https://github.com/spack/spack",
+"spack_branch": "develop",


i know we need to use a newer spack, but I am a bit worried about doing this without testing in a few places (NERSC)

I would also want it to be pinned to some version to avoid random failures when spack develop moves on.

I can create a fork in the alpine space that points to a newer spack

I am testing with v0.13.4 tag. In addition to be a fixed point in the history, it also includes the pipeline feature. It means we can keep relying on this one for a while without preventing new stuff to happen. (combination of spack pipelines with a child pipelines in gitlab will allow cool things in the next weeks/months).

v0.13.4 is in fact too old (does not include latest conduit).
So I will try using the merge commit of this branch into develop. Then we can either keep the commit in the project.json, or use a clone is ascent.

cyrush

This looks cool!

Lets see how the test go and I'll think about a path forward.
One option may be to merge to a special branch we can use to test vs spack develop continually

scripts/uberenv/packages/conduit/package.py

adrienbernede · 2020-02-08T00:54:46Z

@cyrush I re-introduced conduit package in the projeck.
However, you may prefer using the existing package in Spack.

adrienbernede · 2020-02-08T00:56:16Z

@cyrush I need to be added as "developer" (I’m not sure of the terminology) to create the mirroring in Gitlab.

cyrush · 2020-02-08T00:58:48Z

I see -- I think my latest spack package does have configure:

https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/conduit/package.py

can you see if that is compatible?

adrienbernede · 2020-02-08T01:06:05Z

Great, I’ll do the testing.
Concerning the way we integrate this, I would like to be a bit more ambitious than "a special branch".
I can point to a fixed version of Spack to ensure stability, and change in to develop in CI. What else would it take to have this merged into master? I would like to use Conduit as a kickstart to then update Uberenv in Ascent and Axom.
I hope it does not sound crazy :)

cyrush · 2020-02-10T21:10:02Z

@adrienbernede was there some change that increased the output of the spack builds? The travis jobs failed b/c they hit the max log file size.

cyrush · 2020-02-10T21:25:16Z

.gitlab-ci.yml

+##############################################################################
+# Copyright (c) 2016-19, Lawrence Livermore National Security, LLC and Serac
+# project contributors. See the COPYRIGHT file for details.
+##############################################################################


switch to (2014-2020) and Conduit project contributors

Does it mean all the sources in conduit should switch to 2020, or is it done incrementally when the given file is modified?
Serac to Conduit: OK

cyrush · 2020-02-10T21:25:27Z

.gitlab/ci/build_lassen.yml

+# Copyright (c) 2016-19, Lawrence Livermore National Security, LLC and Umpire
+# project contributors. See the COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (MIT)


again, switch to (2014-2020) and Conduit project contributors

mclarsen · 2020-02-10T21:29:58Z

We found that azure will tolerate log files of .5GB.....

adrienbernede · 2020-02-11T00:14:30Z

@adrienbernede was there some change that increased the output of the spack builds? The travis jobs failed b/c they hit the max log file size.

@cyrush The reason is because dev-build is verbose by default on dependencies build. I will make it quiet (there’s an option for it).

cyrush · 2020-02-11T18:53:13Z

is uberenv still copying the resulting host-config.cmake to the root uberenv_libs dir?

(it seems that that is not the case and that is what is causing the travis ci error)

adrienbernede · 2020-02-11T19:26:00Z

I agree that this is what is happening.

I am focusing on inconsistencies I have with the tests of Gitlab, but I will address that afterwards. I know the location changed several times when we worked on Serac. I want to remind the reasoning before taking any action.

I also have an error related to python configuration.

cyrush · 2020-02-11T19:28:43Z

sounds good, I didn't catch that that had changed in uberenv.

we do need the host-config file to exist in that location -- the primary way I configure when developing (across projects) involves using the file in that location.

adrienbernede · 2020-02-11T19:30:07Z

I understand. As far as I remember, we still had at least a link there.

cyrush · 2020-02-11T19:30:52Z

yes - symlink is fine

adrienbernede · 2020-02-11T19:42:27Z

Now I remember:
The host-config file is generated in the project install directory in spack. Then is used to be copied or symlinked to the uberenv-libs directory.
I changed that for a link to the ~~project root~~ spack-build directory because I wanted to share the same uberenv-libs between several instances of the project, (hello CI).

cyrush · 2020-02-11T19:46:52Z

in this case what is the project root directory?

Most of our existing CI infrastructure is going to look for it in the prior location. (Also true for Ascent)

adrienbernede · 2020-02-11T19:51:31Z

~~/home/travis/build/LLNL/conduit~~
/home/travis/build/LLNL/conduit/spack-build

adrienbernede · 2020-02-12T01:12:21Z

In fact, what is happening is:

in dependencies only mode, spack dev-build generates a spack-build directory at the package_source_dir location. It means the project is directly buildable with Spack dependencies there, so the host-config file was not copied or linked to uberenv-libs directory any more.
in install mode, the host-config file is sym-linked in uberenv-libs directory.

I could copy the generated host-config file from spack-build to uberenv-libs. I would like to learn more about your multi-project workflow though.

cyrush · 2020-02-12T16:54:24Z

so the dev build build in directories outside of the prefix passed to ubrerenv?

If you want to share uberenv_libs dirs between CI builds and the host-config.cmake could conflict, could your CI move the generated host config instead of changing the behavior of uberenv?

adrienbernede · 2020-02-12T17:14:00Z

Dev-build would build the dependencies in prefix, but perform the build at the source location, either in a spack-build directory for CMake or in source for Makefile. It makes sense since the idea is to build with the current version of the sources. Is that an issue? Maybe because it breaks the rule "everything generated by Spack goes in uberenv-libs"?

I will find a way to preserve the original behavior.

cyrush · 2020-02-12T20:17:06Z

I see, the host config generation process (since it is part of the final package) will happen in a directory outside of the prefix. I didn't realize the behavior was changing

If at all possible, I'd like to preserve the old semantics of uberenv:

default mode: creates a working set of dependencies and a host config file in --prefix
install mode: creates a full spack build of the package, a host config file and a symlink to the install dir in --prefix.

adrienbernede · 2020-02-12T21:34:03Z

This is what I did in the last commit, with one detail:
I created conduit/uberenv-libs/host-config.cmake as a symbolic link on conduit/spack-build/host-config.cmake. It means you need to preserve spack-build. I will change it for a copy (like I initially said).

Remaining decision: Do you want to get rid of this spack-build? I guess so.

I also have to address a limitation: Spack will refuse to dev-build a package already installed. If that’s possible in your workflow, then we may have to uninstall the package first, or ask the developer to do so.

cyrush · 2020-02-12T23:36:21Z

thanks for your efforts on this -- you are immersed in the details, but I admin am a little lost with the changes.

My hope is that we could use the newer spack features without changing really much of anything related to the workflow of uberenv. Including keeping any build artifacts limited to --prefix, etc.
Another important thing: you can run uberenv again after success and get no errors.

I don't have a good handle on the limitations of the dev-build etc. Are there any aspects of it might need to be address by spack folks before we should consider using it?

adrienbernede · 2020-02-12T23:44:46Z

It’s OK to deal with the details, and that’s no big changes we are talking about.

It’s not a Spack limitation proper. And I with it was already the case before: uberenv --install, followed by uberenv doesn’t make sense because conduit would already be installed.

coveralls · 2020-02-13T00:01:28Z

Coverage remained the same at 85.548% when pulling 203c25c on adrienbernede:feature/uberenv_ci into 96f8e53 on LLNL:master.

Make spack build of dependencies quiet Copy the host-config file in uberenv-libs in build mode Removing spack-build directory Remove existing host-config file in install mode

adrienbernede · 2020-03-02T21:26:09Z

Update and clean branch

NEXT: allow to capture output while printing it.

adrienbernede · 2020-03-05T01:18:45Z

@cyrush with the latest changes, I’m addressing two things:

In uberenv, I’m fixing the issue with already installed packages. If a spec is already installed, and one ask for a dev-build of it, uberenv with create a link to the already installed host-config file.
In CI, I am introducing new jobs testing "what if I update uberenv?".

Notes on 1.: To do so, I had do change the behavior of the command executor sexe() in order to allow capturing the output while also displaying it.

Notes on 2.: Paradoxaly, these new tests would fail since uberenv is currently newer in conduit than in uberenv repo... The next step is to add "what if I update spack". At the moment I don’t see any obstacle in introducing these tests, and my goal is to have them triggered only periodically and/or on changes in master.

adrienbernede · 2020-03-05T01:21:21Z

Also: I still need dev access to LLNL/conduit in order to setup mirroring on LC.

cyrush · 2020-03-05T23:17:09Z

@adrienbernede I added you to the Conduit Devs Team

cyrush · 2020-03-05T23:22:17Z

For the uberenv check, one way we have handled things like this (in other projects) is to record a hash for the current version in a text file. Then when you update the script you also have to update the hash or the test fails.

This is a good strategy to prevent surprise updates, but it doesn't help you test and vet a new version of uberenv -- which I think is what you are aiming for?

Like with spack, I think we will be intentional about when we update uberenv. Staying up-to-date with the latest and greatest isn't necessary, when we can always easily explicitly update and vet though our CI process.

adrienbernede · 2020-03-10T17:04:27Z

LGTM

adrienbernede · 2020-03-10T17:06:10Z

(Previous comment is a sign-off to trigger GitLab CI)

adrienbernede · 2020-03-10T17:18:53Z

The goal here is not to force the adoption of latest Uberenv or Spack, but to make it possible to test Conduit with both of them.

It would allow to ease the "test and vet" phase and "document" how to do that. But it also means I will be able to trigger such a test from Uberenv CI (or Radiuss CI), to check if the last version of Uberenv or Spack would break something in Conduit.

Even if this isn’t the best example, I think it’s an interesting demonstrator, and allows me to experience with the best way to do it in our context.

(By the way, if you feel the need for such a cross-project integration between two Radiuss projects, just let me know. I would like to promote this for projects using submodules like BLT.)

cyrush · 2020-03-10T17:20:35Z

@adrienbernede that sounds good, knowing the answer to the "what if" we use the latest version will be helpful.

adrienbernede · 2020-03-10T18:09:09Z

Close and transferred to #517.

adrienbernede requested a review from cyrush February 8, 2020 00:45

cyrush reviewed Feb 8, 2020

View reviewed changes

scripts/uberenv/packages/conduit/package.py Outdated Show resolved Hide resolved

cyrush reviewed Feb 10, 2020

View reviewed changes

adrienbernede force-pushed the feature/uberenv_ci branch from 3dffca0 to f5e684e Compare February 11, 2020 17:04

adrienbernede changed the title ~~Feature/uberenv ci~~ Update Uberenv and introduce Gitlab CI [feature/uberenv_ci] Feb 11, 2020

adrienbernede added 7 commits March 2, 2020 13:19

CI: Allowing failure for lassen builds

db451eb

Updating with uberenv master

f3e6498

FIX: Small fix in uberenv

be053e7

Update uberenv, fix issue with testing mode

fc43ea2

Allowing no package in uberenv

7a4cb5a

Copyright, timeout, spack version

9a4f3ae

Uberenv: workflow considerations

62ab9e9

Make spack build of dependencies quiet Copy the host-config file in uberenv-libs in build mode Removing spack-build directory Remove existing host-config file in install mode

adrienbernede force-pushed the feature/uberenv_ci branch from 781a8e7 to 62ab9e9 Compare March 2, 2020 21:25

adrienbernede added 8 commits March 2, 2020 15:56

Clean up in gitlab

31b4d5a

Reformat gitlab yaml file for modular scripts

349de79

Attempt to define jobs with uberenv update

4db29be

Placing update_and_build jobs in a specific stage

53c871b

Attempt to split build and install

063b2fe

CI: creating missing jobs + reformat

4adf3eb

Uberenv: allow already installed project

ee690da

NEXT: allow to capture output while printing it.

Uberenv: Improved management of external commands outputs

203c25c

adrienbernede closed this Mar 10, 2020

adrienbernede deleted the feature/uberenv_ci branch March 10, 2020 17:52

adrienbernede mentioned this pull request Mar 10, 2020

Update Uberenv and introduce Gitlab CI [feature/uberenv_ci] #517

Closed

adrienbernede mentioned this pull request Apr 22, 2020

Fixes, Improvements and CI LLNL/uberenv#21

Merged

Update Uberenv and introduce Gitlab CI [feature/uberenv_ci] #502

Update Uberenv and introduce Gitlab CI [feature/uberenv_ci] #502

Conversation

adrienbernede commented Feb 8, 2020

cyrush Feb 8, 2020 • edited

Choose a reason for hiding this comment

adrienbernede Feb 8, 2020

Choose a reason for hiding this comment

cyrush Feb 10, 2020

Choose a reason for hiding this comment

adrienbernede Feb 11, 2020 • edited

Choose a reason for hiding this comment

adrienbernede Feb 11, 2020

Choose a reason for hiding this comment

cyrush left a comment

Choose a reason for hiding this comment

adrienbernede commented Feb 8, 2020 • edited

adrienbernede commented Feb 8, 2020

cyrush commented Feb 8, 2020

adrienbernede commented Feb 8, 2020 • edited

cyrush commented Feb 10, 2020

cyrush Feb 10, 2020

Choose a reason for hiding this comment

adrienbernede Feb 11, 2020

Choose a reason for hiding this comment

cyrush Feb 10, 2020

Choose a reason for hiding this comment

mclarsen commented Feb 10, 2020

adrienbernede commented Feb 11, 2020

cyrush commented Feb 11, 2020

adrienbernede commented Feb 11, 2020

cyrush commented Feb 11, 2020

adrienbernede commented Feb 11, 2020

cyrush commented Feb 11, 2020

adrienbernede commented Feb 11, 2020 • edited

cyrush commented Feb 11, 2020

adrienbernede commented Feb 11, 2020 • edited

adrienbernede commented Feb 12, 2020

cyrush commented Feb 12, 2020

adrienbernede commented Feb 12, 2020 • edited

cyrush commented Feb 12, 2020

adrienbernede commented Feb 12, 2020 • edited

cyrush commented Feb 12, 2020 • edited

adrienbernede commented Feb 12, 2020

coveralls commented Feb 13, 2020 • edited

adrienbernede commented Mar 2, 2020

adrienbernede commented Mar 5, 2020

adrienbernede commented Mar 5, 2020

cyrush commented Mar 5, 2020

cyrush commented Mar 5, 2020

adrienbernede commented Mar 10, 2020

adrienbernede commented Mar 10, 2020

adrienbernede commented Mar 10, 2020 • edited

cyrush commented Mar 10, 2020

adrienbernede commented Mar 10, 2020

cyrush Feb 8, 2020 •

edited

adrienbernede Feb 11, 2020 •

edited

adrienbernede commented Feb 8, 2020 •

edited

adrienbernede commented Feb 8, 2020 •

edited

adrienbernede commented Feb 11, 2020 •

edited

adrienbernede commented Feb 11, 2020 •

edited

adrienbernede commented Feb 12, 2020 •

edited

adrienbernede commented Feb 12, 2020 •

edited

cyrush commented Feb 12, 2020 •

edited

coveralls commented Feb 13, 2020 •

edited

adrienbernede commented Mar 10, 2020 •

edited