Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When spack install checks for buildcaches only add urls for current arch #14467

Merged

Conversation

gartung
Copy link
Member

@gartung gartung commented Jan 11, 2020

spack install command calls spack.package.try_install_from_binary_cache() which calls spack.binary_distribution.get_specs().

spack.binary_distribution.get_specs() was modified in
#13002
to allow listing and installing buildcaches for macOS on a linux host.

In the case where a mirror contains buildcaches for multiple OSes the spack install command can spend more time downloading all of the specs from the build_cache directory than it takes to install the buildcache. The is especially true when using pipelines and the buildcache download directory is cleared between steps.

I modified the get_specs() command to accept a filter option and pass the current OS as the filter option when get_specs() is called from spack.package.try_install_from_binary_cache()

@gartung
Copy link
Member Author

gartung commented Jan 11, 2020

closes #14466

@gartung gartung merged commit d2098d3 into spack:develop Jan 26, 2020
Spack v0.14.0 release automation moved this from Todo to Done Jan 26, 2020
@scottwittenburg
Copy link
Contributor

@gartung It seems like this commit has broken automated pipelines for me, at the point when they try to install dependencies from the mirror. Here's an excerpt from a failed job log:

==> [2020-01-27-18:01:24.017545] Installing libiconv
==> [2020-01-27-18:01:24.017599] Searching for binary cache of libiconv
==> [2020-01-27-18:01:24.018127] Finding buildcaches at s3://spack-public/mirror/build_cache
Traceback (most recent call last):
  File "/builds/scott.wittenburg/spack-pipeline-demo/spack/bin/spack", line 64, in <module>
    sys.exit(spack.main.main())
  File "/builds/scott.wittenburg/spack-pipeline-demo/spack/lib/spack/spack/main.py", line 722, in main
    return _invoke_command(command, parser, args, unknown)
  File "/builds/scott.wittenburg/spack-pipeline-demo/spack/lib/spack/spack/main.py", line 456, in _invoke_command
    return_val = command(parser, args)
  File "/builds/scott.wittenburg/spack-pipeline-demo/spack/lib/spack/spack/cmd/install.py", line 368, in install
    install_spec(args, kwargs, abstract, concrete)
  File "/builds/scott.wittenburg/spack-pipeline-demo/spack/lib/spack/spack/cmd/install.py", line 226, in install_spec
    env.install(abstract_spec, spec, **kwargs)
  File "/builds/scott.wittenburg/spack-pipeline-demo/spack/lib/spack/spack/environment.py", line 1019, in install
    self._install(concrete, **install_args)
  File "/builds/scott.wittenburg/spack-pipeline-demo/spack/lib/spack/spack/environment.py", line 1022, in _install
    spec.package.do_install(**install_args)
  File "/builds/scott.wittenburg/spack-pipeline-demo/spack/lib/spack/spack/package.py", line 1652, in do_install
    dep.package.do_install(**dep_kwargs)
  File "/builds/scott.wittenburg/spack-pipeline-demo/spack/lib/spack/spack/package.py", line 1665, in do_install
    if self.try_install_from_binary_cache(explicit):
  File "/builds/scott.wittenburg/spack-pipeline-demo/spack/lib/spack/spack/package.py", line 1510, in try_install_from_binary_cache
    specs = binary_distribution.get_specs(use_arch=True)
  File "/builds/scott.wittenburg/spack-pipeline-demo/spack/lib/spack/spack/binary_distribution.py", line 704, in get_specs
    if use_arch and re.search(spack.architecture(), link):
TypeError: 'module' object is not callable

Does that ring any bells for you?

@gartung
Copy link
Member Author

gartung commented Jan 27, 2020

@scottwittenburg Sorry about that. I am reverting now. I should have done more thorough testing on it. @eugeneswalker asked for it to be merged so I assumed it worked for him.

@scottwittenburg
Copy link
Contributor

@gartung No worries, thanks for reverting in such a timely fashion 😄

@gartung
Copy link
Member Author

gartung commented Jan 27, 2020

I should really just make a pull request with the real fix. How can I reproduce your error?

@scottwittenburg
Copy link
Contributor

I'll have to try to reproduce outside of a pipeline, but it seems like it would happen just attempting to install any spec when you have a mirror configured and the mirror has a build_cache with some spec.yaml files in it.

@gartung
Copy link
Member Author

gartung commented Jan 27, 2020

OK. I will try to reproduce it locally.

@scottwittenburg
Copy link
Contributor

I just reproduced it by the following steps:

  1. install zlib
  2. create a buildcache entry for installed zlib (unsigned is fine)
  3. uninstall zlib
  4. try to install zlib using --cache-only

@gartung
Copy link
Member Author

gartung commented Jan 27, 2020

Can you try out #14639

@gartung gartung deleted the gartung-buildcache-install-arch-spec-filter branch January 27, 2020 19:05
@scottwittenburg
Copy link
Contributor

Thanks @gartung, I tried #14639 and it seems to have fixed the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
No open projects
Development

Successfully merging this pull request may close these issues.

None yet

2 participants