Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update uberenv #584

Closed
wants to merge 9 commits into from
Closed

Update uberenv #584

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
7932301
Update uberenv
adrienbernede May 20, 2020
3d00d60
Updating spack config
adrienbernede Jun 15, 2020
0c89ce9
Updating allowed failures with new results
adrienbernede Jul 9, 2020
f440e1b
Using a specific commit in Spack instead of a branch reference
adrienbernede Jul 9, 2020
614098d
Fix mirror issue in uberenv (update)
adrienbernede Jul 10, 2020
2c76686
Merge branch 'master' into feature/update-uberenv
adrienbernede Jul 10, 2020
63f5616
Tentative fix for azure with new uberenv
adrienbernede Jul 10, 2020
7f5e06d
Revert "Tentative fix for azure with new uberenv"
adrienbernede Jul 13, 2020
ab6831d
Enforcing compatible arch with gcc5
adrienbernede Jul 13, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions .gitlab/lassen_jobs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,9 @@ clang deps on lassen:
clang install on lassen:
extends: [.install_on_lassen, .with_clang]
needs: [clang deps on lassen]
allow_failure: true

xl deps on lassen:
extends: [.build_deps_on_lassen, .with_xl]
allow_failure: true

xl install on lassen:
extends: [.install_on_lassen, .with_xl]
Expand Down
2 changes: 1 addition & 1 deletion .gitlab/quartz_jobs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,11 @@ gcc deps on quartz:
gcc install on quartz:
extends: [.install_on_quartz, .with_gcc]
needs: [gcc deps on quartz]
allow_failure: true

gcc_static deps on quartz:
extends: [.build_deps_on_quartz, .with_gcc_static]

gcc_static install on quartz:
extends: [.install_on_quartz, .with_gcc_static]
needs: [gcc_static deps on quartz]
allow_failure: true
2 changes: 1 addition & 1 deletion azure-pipelines.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -185,7 +185,7 @@ stages:
# show final spec
echo $SPACK_SPEC
# run uber to build tpls
python scripts/uberenv/uberenv.py -k --pull --spec "${SPACK_SPEC}" --spack-config-dir=scripts/uberenv/spack_configs/ci/ubuntu_16/ --mirror=https://www.ascent-dav.org/mirror/conduit/latest/
python scripts/uberenv/uberenv.py -k --pull --spec "${SPACK_SPEC} arch=ivybridge" --spack-config-dir=scripts/uberenv/spack_configs/ci/ubuntu_16/ --mirror=https://www.ascent-dav.org/mirror/conduit/latest/
displayName: 'Spack Build Tpls'

- script: |
Expand Down
87 changes: 0 additions & 87 deletions scripts/uberenv/packages/uberenv-conduit/package.py
View file
Open in desktop

This file was deleted.

Binary file removed scripts/uberenv/packages/uberenv-conduit/uberenv-conduit.tar.gz
View file
Open in desktop
Binary file not shown.
12 changes: 7 additions & 5 deletions scripts/uberenv/project.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,12 @@
{
"package_name" : "conduit",
"uberenv_package_name" : "uberenv-conduit",
"spack_url": "https://github.com/alpine-DAV/spack",
"spack_branch": "task/2019_11_update_ascent",
"spack_activate" : {"py-numpy" : ["+python"],
"py-sphinx": ["+python","+doc"],
"package_version" : "master",
"package_final_phase" : "configure",
"package_source_dir" : "../..",
"spack_url": "https://github.com/spack/spack",
"spack_commit": "9d1f71e89f184d9b05f15ff81532aec460795fd0",
"spack_activate" : {"py-numpy" : ["+python"],
"py-sphinx": ["+python","+doc"],
"py-mpi4py" : ["+python", "+mpi"],
"py-sphinx-rtd-theme": ["+python","+doc"] }
}
27 changes: 25 additions & 2 deletions scripts/uberenv/spack_configs/config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,11 +77,25 @@ config:
misc_cache: $spack/var/spack/cache


# Timeout in seconds used for downloading sources etc. This only applies
# to the connection phase and can be increased for slow connections or
# servers. 0 means no timeout.
connect_timeout: 10


# If this is false, tools like curl that use SSL will not verify
# certifiates. (e.g., curl will use use the -k option)
verify_ssl: true


# Suppress gpg warnings from binary package verification
# Only suppresses warnings, gpg failure will still fail the install
# Potential rationale to set True: users have already explicitly trusted the
# gpg key they are using, and may not want to see repeated warnings that it
# is self-signed or something of the sort.
suppress_gpg_warnings: false


# If set to true, Spack will attempt to build any compiler on the spec
# that is not already available. If set to False, Spack will only use
# compilers already configured in compilers.yaml
Expand Down Expand Up @@ -112,7 +126,7 @@ config:
# of Spack. For filesystems that don't support locking, you should set
# this to false and run one Spack at a time, but otherwise we recommend
# enabling locks.
locks: false
locks: true
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I worry about this one being true, it has caused hangs in the past

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lot of change with lock management is Spack since.
I would suggest giving it a chance. I’ve had issues with this not being set to true.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It’s is true that this has "only" be tested by me locally and in CI.
So I understand that merging this is less straight-forward than merging CI only.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, lets keep them on
I guess you may actually need it for the CI workflows that have shared spack instances?

In the past locks weren't necessary and could be a barrier (hangs) for single install. But if we are going to use shared upstreams, we need them.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

conduit/scripts/ci/build-on-lc.sh

Line 39 in 9d654df

PREFIX="--prefix=${INSTALL_ROOT}_${SYS_TYPE}_${TOOLCHAIN}"

Prefix still depends on the toolchain, I wanted to avoid concurrency at first, but it’s a potential optimization.



# The maximum number of jobs to use when running `make` in parallel,
Expand All @@ -131,7 +145,7 @@ config:
# when Spack needs to manage its own package metadata and all operations are
# expected to complete within the default time limit. The timeout should
# therefore generally be left untouched.
db_lock_timeout: 120
db_lock_timeout: 3


# How long to wait when attempting to modify a package (e.g. to install it).
Expand All @@ -140,3 +154,12 @@ config:
# anticipates that a significant delay indicates that the lock attempt will
# never succeed.
package_lock_timeout: null

# Control whether Spack embeds RPATH or RUNPATH attributes in ELF binaries.
# Has no effect on macOS. DO NOT MIX these within the same install tree.
# See the Spack documentation for details.
shared_linking: 'rpath'

# Set to 'false' to allow installation on filesystems that doesn't allow setgid bit
# manipulation by unprivileged user (e.g. AFS)
allow_sgid: true
Loading