Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

installer.py: show timers for binary install #33305

Merged
merged 2 commits into from
Oct 17, 2022
Merged

installer.py: show timers for binary install #33305

merged 2 commits into from
Oct 17, 2022

Conversation

haampie
Copy link
Member

@haampie haampie commented Oct 14, 2022 •
edited

Print a message of the form

Fetch mm:ss.  Build: mm:ss.  Total: mm:ss

when installing from buildcache. Previously this only happened for
source builds.

This way we get better insights into where Gitlab CI wastes its time
(fetching from a slow bucket or is our relocation logic the bottleneck?)

--

Currently the format "fetch: build: total:" is static, maybe slightly better to actually dynamically show the phases? that would be search fetch install for binary installs.

@haampie
installer.py: show timers for binary install
7adc776 
Print a message of the form

```
Fetch mm:ss.  Build: mm:ss.  Total: mm:ss
```

when installing from buildcache. Previously this only happened for
source builds.

This way we get better insights into where Gitlab CI wastes its time
(fetching from a slow bucket or is our relocation logic the botleneck?)
@spackbot-app spackbot-app bot added the core PR affects Spack core functionality label Oct 14, 2022
@alalazo
Copy link
Member

alalazo commented Oct 15, 2022

@spackbot run pipeline

@alalazo
Copy link
Member

alalazo commented Oct 15, 2022

@spackbot help

@spackbot-app
Copy link

spackbot-app bot commented Oct 15, 2022

You can interact with me in many ways!

  • @spackbot hello: say hello and get a friendly response back!
  • @spackbot help or @spackbot commands: see this message
  • @spackbot run pipeline or @spackbot re-run pipeline: to request a new run of the GitLab CI pipeline
  • @spackbot fix style if you have write and would like me to run spack style --fix for you.
  • @spackbot maintainers or @spackbot request review: to look for and assign reviewers for the pull request.

I'll also help to label your pull request and assign reviewers!
If you need help or see there might be an issue with me, open an issue here

@alalazo
Copy link
Member

alalazo commented Oct 15, 2022

Not sure why spackbot cannot be retriggered. The corresponding develop pipeline failed, but finished. This pipeline is still waiting though.

@haampie
Copy link
Member Author

haampie commented Oct 16, 2022

@spackbot run pipeline

@spackbot-app
Copy link

spackbot-app bot commented Oct 16, 2022

I've started that pipeline for you!

@alalazo
Copy link
Member

alalazo commented Oct 17, 2022

Python 2.7 failed on the infamous test_foreground_background. Restarted.

@haampie
Copy link
Member Author

haampie commented Oct 17, 2022 •
edited

btw @alalazo, what do you think about changing fetch/build/total into a dynamic list of actual phases followed by total? E.g. for cmake packages cmake build install etc... For binary installs it would be search fetch install.

(Let's first get this in though ;p)

@alalazo
Copy link
Member

alalazo commented Oct 17, 2022

That sounds good to me. I'd get this in first and refine later.

@haampie haampie mentioned this pull request Oct 17, 2022
Merged
@alalazo alalazo merged commit 2c802c1 into spack:develop Oct 17, 2022
luke-dt added a commit to dantaslab/spack that referenced this pull request Oct 17, 2022
@luke-dt @haampie
@jedwards4b @iarspider @scottwittenburg @adamjstewart @bonachea @aurianer @eugeneswalker @balay @jlnav @eschnett @gsjaardema @michaelkuhn @blue42u @mkstoyanov @fangohr @giordano @dialvarezs @wdconinc
merge with upstream (#7)
e552c78 
* py-execnet: 1.9.0 (spack#33282)

* py-execnet: 1.9.0

* bounds

* add version 1.12.3 of parallel-netcdf (spack#33286)

* Add checksum for py-psutil 5.9.2 (spack#33139)

* gitlab ci: Print better information about broken specs (spack#33124)

When a pipeline generation job is automatically failed because it
generated jobs for specs known to be broken on develop, print better
information about the broken specs that were encountered.  Include
at a minimum the hash and the url of the job whose failure caused it
to be put on the broken specs list in the first place.

* meson: remove slash in path (spack#33292)

* Add checksum for py-gitpython 3.1.27 (spack#33285)

* Add checksum for py-gitpython 3.1.27

* Update package.py

* Update var/spack/repos/builtin/packages/py-gitpython/package.py

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>

* UPC++/GASNet-EX 2022.9.0 update (spack#33277)

* gasnet: Add new release hash
* upcxx: Add new release hash
* gasnet: misc updates
* upcxx: misc updates

* Fix pika@0.9.0 sha (spack#33307)

* hip@5.2.0 onwards: set prefix properly (spack#33257)

* hip-set-prefix-rocm5.2.0-onwards

* Update var/spack/repos/builtin/packages/hip/package.py

Update description

Co-authored-by: Satish Balay <balay@mcs.anl.gov>

* petsc,py-petsc4py,slepc,py-slepc4py: add version 3.18.0 (spack#32938)

* petsc,py-petsc4py,slepc,py-slepc4py: add version 3.18.0

* workaround for dealii build failure [with petsc version check]

* pism: add compatibility fix to for petsc@3.18

* add in hipsolver dependency

* py-libensemble: updating package for v0.9.3 (spack#33298)

* commit updating py-libensemble package for 0.9.3

* removed commented-out lines

* Add checksum for py-oauthlib 3.2.1 (spack#33201)

* Add checksum for py-oauthlib 3.2.1

* Update package.py

* [@spackbot] updating style on behalf of iarspider

* Update package.py

* Update package.py

Co-authored-by: iarspider <iarspider@users.noreply.github.com>

* ninja: New version 1.11.1 (spack#33215)

* seacas: update to latest release (spack#33330)

Add checksum for latest tag/release

* gptl: new version 8.1.1; use the correct mpi fortran compiler (spack#33235)

* gptl: new version 8.1.1; use the correct `mpifc`
* add `F90` and `$F77`

* glib: add 2.74.0 and 2.72.4 (spack#33332)

* depfile: update docs (spack#33279)

* rocksdb: add 7.7.3 (spack#33341)

* Add checksum for py-seaborn 0.12.0 (spack#33145)

* Add checksum for py-seaborn 0.12.0

* Update package.py

* Update package.py

* [@spackbot] updating style on behalf of iarspider

* Update package.py

Co-authored-by: iarspider <iarspider@users.noreply.github.com>

* CI: allow multiple matches to combine tags (spack#32290)

Currently "spack ci generate" chooses the first matching entry in
gitlab-ci:mappings to fill attributes for a generated build-job,
requiring that the entire configuration matrix is listed out
explicitly. This unfortunately causes significant problems in
environments with large configuration spaces, for example the
environment in spack#31598 (spack.yaml) supports 5 operating systems,
3 architectures and 130 packages with explicit size requirements,
resulting in 1300 lines of configuration YAML.

This patch adds a configuraiton option to the gitlab-ci schema called
"match_behavior"; when it is set to "merge", all matching entries
are applied in order to the final build-job, allowing a few entries
to cover an entire matrix of configurations.

The default for "match_behavior" is "first", which behaves as before
this commit (only the runner attributes of the first match are used).

In addition, match entries may now include a "remove-attributes"
configuration, which allows matches to remove tags that have been
aggregated by prior matches. This only makes sense to use with
"match_behavior:merge". You can combine "runner-attributes" with
"remove-attributes" to effectively override prior tags.

* meson: update OneAPI compiler support patch (spack#33293)

* py-tensorflow: fix zlib (spack#33349)

* py-tensorflow: fix zlib

* [@spackbot] updating style on behalf of haampie

Co-authored-by: haampie <haampie@users.noreply.github.com>

* py-meson-python: add new versions (spack#33294)

* tasmanian: disable openmp by default (spack#33345)

* octopus: upgrade to 12.1 (spack#33343)

* py-sphinx: add v5.3 and v5.2 (spack#33356)

* py-setuptools: add v65.5.0 (spack#33353)

* libblastrampoline: Add versions 5.1.1, 5.2.0 (spack#33352)

* nextflow: add v20.10.0 (spack#33354)

* sdl2: add v2.0.22 and v2.24.1 (spack#33351)

* mariadb-c-client: add 3.3.2, 3.2.7, 3.1.18, 3.0.10 (spack#33335)

* py-tensorflow-hub: zlib, again. (spack#33359)

* Add checksum for py-sniffio 1.3.0 (spack#32975)

* py-numpy: add v1.23.4 (spack#33260)

* py-jupyterlab-pygments: install from wheel to avoid cyclic dependency (spack#33278)

* py-jupyterlab-pygments: avoid cyclic dependency

* Fix style

* Update package.py

* Update package.py

* [@spackbot] updating style on behalf of iarspider

* Update package.py

* Flake-8

* fix

Co-authored-by: iarspider <iarspider@users.noreply.github.com>

* installer.py: show timers for binary install (spack#33305)

Print a message of the form
```
Fetch mm:ss.  Build: mm:ss.  Total: mm:ss
```
when installing from buildcache. 

Previously this only happened for source builds.

* Add checksum for py-astroid 2.12.7, py-astroid 2.12.10, py-setuptools 62.6.0, py-wrapt 1.14.1, py-pylint 2.15.0 (spack#32976)

* Add checksum for py-astroid 2.12.7, py-setuptools 62.6.0

* Also add checksum for py-wrapt

* Update package.py

* Update package.py (spack#57)

* Update package.py

* Update package.py

* Update package.py

* gitlab ci: Do not force protected build jobs to run on aws runners (spack#33314)

* installer.py: fix/test get_deptypes (spack#33363)

Fixing an oversight in spack#32537

`get_deptypes` should depend on new `package/dependencies_cache_only`
props.

* Add checksum for py-grpcio-tools 1.48.1 (spack#33358)

* Add checksum for py-prompt-toolkit 3.0.31 (spack#33362)

Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
Co-authored-by: Jim Edwards <jedwards@ucar.edu>
Co-authored-by: iarspider <iarspider@gmail.com>
Co-authored-by: Scott Wittenburg <scott.wittenburg@kitware.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Dan Bonachea <dobonachea@lbl.gov>
Co-authored-by: Auriane R <48684432+aurianer@users.noreply.github.com>
Co-authored-by: eugeneswalker <38933153+eugeneswalker@users.noreply.github.com>
Co-authored-by: Satish Balay <balay@mcs.anl.gov>
Co-authored-by: John-Luke Navarro <33131245+jlnav@users.noreply.github.com>
Co-authored-by: iarspider <iarspider@users.noreply.github.com>
Co-authored-by: Erik Schnetter <schnetter@gmail.com>
Co-authored-by: Greg Sjaardema <gsjaardema@gmail.com>
Co-authored-by: WuK <i@wu-kan.cn>
Co-authored-by: Michael Kuhn <michael.kuhn@ovgu.de>
Co-authored-by: Jonathon Anderson <17242663+blue42u@users.noreply.github.com>
Co-authored-by: haampie <haampie@users.noreply.github.com>
Co-authored-by: Miroslav Stoyanov <30537612+mkstoyanov@users.noreply.github.com>
Co-authored-by: Hans Fangohr <fangohr@users.noreply.github.com>
Co-authored-by: Mosè Giordano <giordano@users.noreply.github.com>
Co-authored-by: Diego Alvarez <dialvarezs@gmail.com>
Co-authored-by: Wouter Deconinck <wdconinc@gmail.com>
@haampie haampie deleted the fix/buildcache-show-timers branch October 17, 2022 15:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@alalazo alalazo alalazo approved these changes

Assignees
No one assigned
Labels
core PR affects Spack core functionality
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@haampie @alalazo