Allow packages without URLs #3161

adamjstewart · 2017-02-16T19:32:06Z

Fixes #98
Fixes #3042
Fixes #7971
Fixes #8046
Fixes #8173
Fixes #8733
Fixes #8745

Some packages do not have a url and are instead downloaded via git, hg, or svn. Some packages like spectrum-mpi cannot be downloaded at all, and are placeholder packages for system installations. Previously, __init__() in PackageBase crashed if a package did not have a url attribute defined.

I hacked this section of code out, but I have no idea what the repercussions of that are. Personally, I don't know why we would want to call url_for_version during package creation at all. This change probably breaks something, so let me know if you can think of anything that required this.

@tgamblin @citibeth

citibeth · 2017-02-16T19:51:14Z

I have no idea... Can you try it out for a while and see what happens? If it breaks, the break should be pretty obvious (missing attribute).

citibeth · 2017-02-16T19:52:25Z

Another alternative would be to initialize Package.url=""

SteVwonder · 2018-07-18T21:32:11Z

Chiming in to say this fixed the issue I ran into with #8745. I'll keep this commit cherry-picked to see if it causes any issues. What else needs to be done to get this PR merged?

adamjstewart · 2018-07-18T21:36:26Z

What else needs to be done to get this PR merged?

Mostly testing by people like you so that you can let us know if it breaks anything unexpected down the line. And a review from someone who knows more about Spack's internals than I do.

tgamblin · 2018-07-19T01:12:33Z

I’ve got some additions and tests I’ll push soon. I think it’ll be ready then.

tgamblin · 2018-07-19T03:41:45Z

@adamjstewart: I added some more machinery and more tests, and I think you are now in the odd position of needing to review your own PR. The details are in the commit messages but here they are for folks on this PR:

core: fixes and tests for handling of fetcher attributes in packages

Packages can remove the top-level url attribute and still work

These are now legal:

Packages with only version-specific URLs (even with gaps)

Packages with a top-level git/hg/svn attribute and version
directives for that.

If a package has both a top-level hg/git/svn attribute AND a top-level
url attribute, the url attribute takes precedence.

This probably fixes a few more issues than are listed in the main PR, as it should make top-level git, hg, and svn attributes work, too.

Also, while this keeps the policy for extrapolation we decided on in #8565 (trust the package-level URL first), it also brings back the nearest_url logic that was removed there, so that packages with only version-specific URLs will work, even if there are gaps in the list.

adamjstewart · 2018-07-19T03:52:17Z

This probably fixes a few more issues than are listed in the main PR, as it should make top-level git, hg, and svn attributes work, too.

I've been wanting this for the longest time! One step closer to the interface I envisioned in #2718.

I'll try to reproduce all of the bugs listed above and see if this does indeed fix them. If I don't get to this tomorrow, I'll definitely get to it by the weekend.

I would be interested in removing fake URLs and adding top-level git, hg, and svn attributes to dozens of packages in Spack. Should I add that to this PR or wait until after this is merged?

tgamblin · 2018-07-19T04:05:10Z

@adamjstewart: I think the interface you proposed there will work now. If it doesn't, add a test package for what it should look like and I don't think it would be too much more effort to make it work.

tgamblin · 2018-07-19T04:07:42Z

@HadrienG2: FYI

tgamblin · 2018-07-19T04:22:56Z

I would be interested in removing fake URLs and adding top-level git, hg, and svn attributes to dozens of packages in Spack. Should I add that to this PR or wait until after this is merged?

I think this is up to you. It sounds good either way! Also if you don't mind adding docs for it, that would be awesome.

citibeth · 2018-07-19T10:53:38Z

I would lean toward another PR. My two cents...

…

On Thu, Jul 19, 2018 at 12:23 AM Todd Gamblin ***@***.***> wrote: I would be interested in removing fake URLs and adding top-level git, hg, and svn attributes to dozens of packages in Spack. Should I add that to this PR or wait until after this is merged? I think this is up to you. It sounds good either way! Also if you don't mind adding docs for it, that would be awesome. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3161 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB1cd6EaHg5LCNeXHQB3qaBCCCZIOd_Eks5uIAmigaJpZM4MDcd2> .

adamjstewart · 2018-07-19T12:41:28Z

I can definitely add docs. If I make the package changes in this PR, it becomes easier to test that this PR actually works and none of the behavior will change.

citibeth · 2018-07-25T04:04:53Z

P.S. Still dreaming of the day when #1983 <#1983> is merged and we no longer have to tell people to rename their master branch to develop:

I would love to have something more functional too, but I think that #1983 has some serious shortcomings. I would love a process by which all parties involved can hammer out a plan that we can all support.

…

-- Elizbaeth

adamjstewart

Okay, docs have now been updated. I think this PR is ready for a final review as far as I'm concerned!

P.S. I'm moving to a new aprt this weekend, so I'm going to be somewhat busy (un)packing. I'll be able to respond to comments and make small changes, but if you need to make any big changes, feel free to push directly onto this PR. I'm hoping to get this merged quickly as it affects so many packages.

adamjstewart · 2018-07-26T01:37:28Z

lib/spack/docs/packaging_guide.rst


-^^^^^^^^^^^^^^^^^^^^^
-PyPI and version URLs


This section has been moved to http://spack.readthedocs.io/en/latest/build_systems/pythonpackage.html#url.

adamjstewart · 2018-07-26T01:38:07Z

lib/spack/docs/packaging_guide.rst

-levels of subdirectories, as the ``mpich`` website is structured much
+``list_url``.  ``list_depth = 1`` tells it to follow up to 1 level of
+links from the top-level page.  Note that here, this implies 1
+level of subdirectories, as the ``mpich`` website is structured much


Pretty sure this was a bug in list_depth that @tgamblin fixed a long time ago but the docs were never updated.

adamjstewart · 2018-07-26T01:39:18Z

lib/spack/docs/packaging_guide.rst

+  there is no way to verify that the repository has not been compromised, and
+  the commit you get when you install the package likely won't be the same
+  commit that was used when the package was first written. Additionally, the
+  default branch may change. It is best to at least specify a branch name.


I tried to expand on each of these "This download method is untrusted, and is not recommended" statements to explain the rationale and what alternatives are available. I think this is sufficient to close #6785.

adamjstewart · 2018-07-26T01:40:44Z

lib/spack/docs/packaging_guide.rst

+.. _go-fetch:
+
+^^
+Go


Apparently Spack supports fetching with Go? Well, it's documented now! Tbh, I don't know the first thing about Go, so if this section feels... wishy-washy, that's why. Someone should definitely take a close look at this section and correct me wherever I'm wrong.

Some packages do not have a `url` and are instead downloaded via `git`, `hg`, or `svn`. Some packages like `spectrum-mpi` cannot be downloaded at all, and are placeholder packages for system installations. Previously, `__init__()` in `PackageBase` crashed if a package did not have a `url` attribute defined. I hacked this section of code out, but I have no idea what the repercussions of that are.

- Packages can remove the top-level `url` attribute and still work - These are now legal: - Packages with *only* version-specific URLs (even with gaps) - Packages with a top-level git/hg/svn attribute and `version` directives for that. - If a package has both a top-level hg/git/svn attribute AND a top-level url attribute, the url attribute takes precedence.

- ensure that packages can't have more than one of git, hg, svn, or url

- packagers can specify two top-level fetch URLs if one is `url` - e.g., `url` and `git` or `url` and `svn` - allow only one VCS fetcher so we can differentiate between URL and VCS. - also clean up fetcher logic and class structure

- this test is checking the package *repository*, not the database.

- Previously, Spack didn't check the arguments you put in version() directives. - So, you could do something like this, where there are arguments for a URL fetcher AND for a git fetcher: version('1.0', md5='abc123', git='https://foo.bar', commit='feda2343') - Now, we check the arguments before constructing a fetcher, to ensure that each package has *only* arguments for a single type of fetcher. - Also added `test_package_version_consistency()` to the `package_sanity` test, so that all builtin packages are required to have valid `version()` directives.

tgamblin

@adamjstewart: I read through the docs. I think they're good enough for now. There are some clarifications we should make to the wording ("trusted" vs. verifiable, etc.), and a few things we should add to the fetch strategy to satisfy some of @HadrienG2's requests, but I think those belong in another PR. The issues were there before.

To me, this looks great -- thanks for fixing and documenting everything!

adamjstewart · 2020-02-28T02:16:13Z

@tgamblin there seems to be a regression of this feature causing #2788 to reappear.

ax3l · 2020-03-05T18:52:35Z

looks like this was just a cli usage error: #2788 (comment)

adamjstewart added the WIP label Feb 16, 2017

adamjstewart mentioned this pull request Feb 16, 2017

Overhaul Spack's URL parsing #2972

Merged

adamjstewart mentioned this pull request Feb 18, 2017

GISS ModelE: New Package #1544

Closed

adamjstewart mentioned this pull request Nov 23, 2017

Package: "url" member always required #2788

Closed

adamjstewart mentioned this pull request May 8, 2018

spack won't use graphviz@2.30.1 from RHEL7 installation #8046

Closed

adamjstewart mentioned this pull request May 17, 2018

Cannot uninstall package without url #8173

Closed

adamjstewart mentioned this pull request Jul 17, 2018

Fetch mechanics broken in develop for rare but present cases #8733

Closed

tgamblin self-requested a review July 18, 2018 00:46

adamjstewart added the fetching label Jul 18, 2018

adamjstewart mentioned this pull request Jul 18, 2018

Concretization failure when using system-installed CUDA #8745

Closed

adamjstewart mentioned this pull request Jul 18, 2018

External Packages Not Available in Spack #7971

Closed

tgamblin force-pushed the fixes/no-url branch 2 times, most recently from aaeb4e6 to 4df4193 Compare July 19, 2018 03:36

tgamblin requested review from tgamblin and removed request for tgamblin July 19, 2018 03:36

adamjstewart changed the title ~~[WIP] Allow packages without URLs~~ Allow packages without URLs Jul 19, 2018

adamjstewart removed the WIP label Jul 19, 2018

tgamblin force-pushed the fixes/no-url branch from 4df4193 to d2c084d Compare July 19, 2018 04:20

adamjstewart commented Jul 26, 2018

View reviewed changes

adamjstewart and others added 17 commits July 25, 2018 22:28

style: clean up exception definitions in package.py

64b1195

core: add check for conflicting top-level fetch attributes in packages

b15a95f

- ensure that packages can't have more than one of git, hg, svn, or url

tests: correct name of check_db to check_repo

d4885d3

- this test is checking the package *repository*, not the database.

tests: convert mirror test to pytest with free functions.

67fa808

Add top-level attributes for hg, svn, and git A-L packages

64b753e

Add top-level attributes for git M-P packages

d6752c7

Add top-level attributes for git R (Bioconductor) packages

2f1e4e6

Add top-level attributes for git R-Z packages

b80c356

Replace git=git with explicit branch names

90c239b

All git URLs end in .git

a0e9007

Further improvements to version parsing

ca06249

Unit test fixes

fd51b44

Add docs on new top-level VCS attributes

9d2cfe4

tgamblin force-pushed the fixes/no-url branch from 336c2f7 to 9d2cfe4 Compare July 26, 2018 05:30

tgamblin approved these changes Jul 26, 2018

View reviewed changes

tgamblin merged commit 7902f3e into spack:develop Jul 26, 2018

adamjstewart deleted the fixes/no-url branch July 26, 2018 13:10

This was referenced Jul 26, 2018

updated Tasmanian spack with latest options #8804

Merged

sz: new version 2.0.2.0 #8786

Merged

This was referenced Aug 9, 2018

Computing full_hash() of external package fails #8929

Open

Fix spack versions behavior when no URL #8967

Merged

adamjstewart mentioned this pull request Jul 31, 2020

Add fenicsx packages #17743

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow packages without URLs #3161

Allow packages without URLs #3161

adamjstewart commented Feb 16, 2017 •

edited

citibeth commented Feb 16, 2017

citibeth commented Feb 16, 2017

SteVwonder commented Jul 18, 2018

adamjstewart commented Jul 18, 2018

tgamblin commented Jul 19, 2018

tgamblin commented Jul 19, 2018

adamjstewart commented Jul 19, 2018

tgamblin commented Jul 19, 2018 •

edited

tgamblin commented Jul 19, 2018

tgamblin commented Jul 19, 2018

citibeth commented Jul 19, 2018 via email

adamjstewart commented Jul 19, 2018

citibeth commented Jul 25, 2018 via email

adamjstewart left a comment

adamjstewart Jul 26, 2018

adamjstewart Jul 26, 2018

adamjstewart Jul 26, 2018

adamjstewart Jul 26, 2018

tgamblin left a comment

adamjstewart commented Feb 28, 2020

ax3l commented Mar 5, 2020 •

edited

Allow packages without URLs #3161

Allow packages without URLs #3161

Conversation

adamjstewart commented Feb 16, 2017 • edited

citibeth commented Feb 16, 2017

citibeth commented Feb 16, 2017

SteVwonder commented Jul 18, 2018

adamjstewart commented Jul 18, 2018

tgamblin commented Jul 19, 2018

tgamblin commented Jul 19, 2018

adamjstewart commented Jul 19, 2018

tgamblin commented Jul 19, 2018 • edited

tgamblin commented Jul 19, 2018

tgamblin commented Jul 19, 2018

citibeth commented Jul 19, 2018 via email

adamjstewart commented Jul 19, 2018

citibeth commented Jul 25, 2018 via email

adamjstewart left a comment

Choose a reason for hiding this comment

adamjstewart Jul 26, 2018

Choose a reason for hiding this comment

adamjstewart Jul 26, 2018

Choose a reason for hiding this comment

adamjstewart Jul 26, 2018

Choose a reason for hiding this comment

adamjstewart Jul 26, 2018

Choose a reason for hiding this comment

tgamblin left a comment

Choose a reason for hiding this comment

adamjstewart commented Feb 28, 2020

ax3l commented Mar 5, 2020 • edited

adamjstewart commented Feb 16, 2017 •

edited

tgamblin commented Jul 19, 2018 •

edited

ax3l commented Mar 5, 2020 •

edited