Fixing "problems in spack" as identified by repology #25491

vsoch · 2021-08-18T22:05:45Z

I've added spack to repology (went live in the middle of today) and as part of that, I stumbled on this entire three page of "spack problems"

https://repology.org/repository/spack/problems

And of course could not simply leave them there, so I went through each one and fixed the package. There were a few that should be further discussed in #25490 but for the most part these are small changes to update urls to use https, or update the perl package manager, or replace urls that are now 404. For a few sites that triggered errors (that didn't have https) the certificate setup might be strange but it's not broken, so I didn't change those. It looks like from the file count I was able to fix 379/397, which is pretty good!

Signed-off-by: vsoch vsoch@users.noreply.github.com

alalazo

Reviewed the first 63 packages. Just a couple of questions so far. I'll also ping @adamjstewart to have another pair of eyes on this one.

var/spack/repos/builtin/packages/connect-proxy/package.py

var/spack/repos/builtin/packages/keepalived/package.py

alalazo

Got to half of the perl packages, just another minor comment so far. 165/379

alalazo · 2021-08-19T09:12:16Z

var/spack/repos/builtin/packages/parmgridgen/package.py

@@ -15,7 +15,7 @@ class Parmgridgen(Package):
    ParMGridGen is the parallel version of MGridGen.
    """

-    homepage = "http://www-users.cs.umn.edu/~moulitsa/software.html"
+    homepage = "https://github.com/mrklein/ParMGridGen"


This one is weird. The homepage points to a Github repository that has v0.0.2 as the latest version dating 2015. The url refers to v1.0 so 🤷‍♂️

I wasn't sure what to do about this one - the original home page is 404 https://www-users.cse.umn.edu/~moulitsa/software.html. I chose the repo because it seems to be linked to those same docs: https://github.com/mrklein/ParMGridGen

adamjstewart

I'm not particularly worried about homepage changes since Spack doesn't use them for anything. As long as you've checked any url changes to make sure that things can still be downloaded. If you ever want to go crazy on urls, Spack has a spack url stats command to see how many URLs still use http, and a spack url list command if you want to print them.

Personally I would leave the R and Perl changes out for now and do a separate PR that replaces everything with a cran = '...' or cpan = '...' similar to pypi = '...'. If @glennpj is busy I may be interested in working on this in the near future.

var/spack/repos/builtin/packages/boost/package.py

var/spack/repos/builtin/packages/grib-api/package.py

var/spack/repos/builtin/packages/gconf/package.py

adamjstewart · 2021-08-19T15:17:47Z

var/spack/repos/builtin/packages/perl-html-parser/package.py

+    homepage = "https://metacpan.org/pod/HTML::Parser"
    url      = "http://search.cpan.org/CPAN/authors/id/G/GA/GAAS/HTML-Parser-3.72.tar.gz"


I assume that if search.cpan.org is discouraged for homepages, it's also discouraged for URLs (even if repology doesn't test it). We should add fetching support like we did for pypi at some point. This can happen in a separate PR though.

yeah that's a good idea!

var/spack/repos/builtin/packages/py-nose-cov/package.py

var/spack/repos/builtin/packages/py-pager/package.py

var/spack/repos/builtin/packages/py-pytest-pep8/package.py

var/spack/repos/builtin/packages/py-wget/package.py

var/spack/repos/builtin/packages/py-zc-buildout/package.py

adamjstewart · 2021-08-19T15:25:41Z

var/spack/repos/builtin/packages/r-amap/package.py

+    homepage = "https://cran.r-project.org/web/packages/amap/index.html"
    url      = "https://cloud.r-project.org/src/contrib/amap_0.8-16.tar.gz"
    list_url = "https://cloud.r-project.org/src/contrib/Archive/amap/"


For R packages we really need a cran attribute similar to pypi that autopopulates all of these attributes. @glennpj are you interested in working on this?

Sure, I actually planned on doing that when I do my next wave of CRAN package updates. I have not been pushing for that in PRs though.

Just to make sure that I am clear about the goal, are we looking to have the cran attribute in all CRAN packages? The attribute already exists but its usage has not been standardized on as of yet.

Maybe y'all want to open an issue on this instead of the thread getting lost in this PR? There are already so many comments I can't find ones that I'm looking for with that accordian thing GitHub is doing!

Yes, the goal is to get a cran attribute for all packages. I didn't realize we already had support for it. I don't think an issue is necessary if you're already planning to do this in the future.

adamjstewart

Python packages look good to me. If I get bored I'll add cran and cpan attributes for fetching someday.

vsoch · 2021-08-23T22:26:06Z

okay @adamjstewart I fixed that one boost issue, let me know if there is anything else you'd like before this is ready for merge!

adamjstewart · 2021-08-23T22:27:43Z

Did you add deprecated=True to the known deprecated packages?

vsoch · 2021-08-23T22:28:52Z

@adamjstewart I asked questions about those requests, they might be hidden now in the compressed comments. I asked specifically about versions I think.

adamjstewart · 2021-08-23T22:31:07Z

For at least one of those I said to just deprecate all versions if the entire package is deprecated, sorry for the confusion.

Most of these are perl packages that need to point to the meta docs site, and then a fair amount of http addresses that need to be https, and then the rest are usually documentation sites that no longer exist or were otherwise changes Signed-off-by: vsoch <vsoch@users.noreply.github.com>

vsoch · 2021-08-23T23:11:48Z

No problem! All set.

vsoch · 2021-08-26T20:10:44Z

YES!!! 🥳

alalazo · 2021-08-27T06:17:13Z

Given the commit message with which this landed in develop, I think @adamjstewart was hungry 😆 For reference this is "speck"

vsoch · 2021-08-27T06:22:40Z

lol! So that was totally my mistake - I fixed the title from "speck" to "spack" but didn't update the commit message. Oups.

And just looked at the link, ewsies! @adamjstewart you can have all the speck, do not save any for me 😆

vsoch changed the title ~~Fixing "problems in speck" as identified by repology~~ Fixing "problems in spack" as identified by repology Aug 18, 2021

vsoch mentioned this pull request Aug 18, 2021

Publish package list in repology #25288

Closed

2 tasks

vsoch force-pushed the fix/repology-problems branch 3 times, most recently from f04c91b to 6d4be85 Compare August 18, 2021 23:47

vsoch requested a review from alalazo August 19, 2021 00:50

alalazo reviewed Aug 19, 2021

View reviewed changes

var/spack/repos/builtin/packages/connect-proxy/package.py Outdated Show resolved Hide resolved

var/spack/repos/builtin/packages/keepalived/package.py Show resolved Hide resolved

alalazo requested a review from adamjstewart August 19, 2021 07:14

alalazo reviewed Aug 19, 2021

View reviewed changes

vsoch force-pushed the fix/repology-problems branch from 6d4be85 to 3dc6637 Compare August 19, 2021 15:28

adamjstewart reviewed Aug 19, 2021

View reviewed changes

vsoch force-pushed the fix/repology-problems branch from 3dc6637 to 23fda60 Compare August 19, 2021 15:39

vsoch requested review from alalazo and adamjstewart August 23, 2021 20:49

adamjstewart previously approved these changes Aug 23, 2021

View reviewed changes

vsoch dismissed adamjstewart’s stale review via 6905557 August 23, 2021 22:25

vsoch force-pushed the fix/repology-problems branch from 23fda60 to 6905557 Compare August 23, 2021 22:25

vsoch force-pushed the fix/repology-problems branch from 6905557 to 7a1dd37 Compare August 23, 2021 23:11

vsoch requested a review from adamjstewart August 24, 2021 04:45

adamjstewart approved these changes Aug 24, 2021

View reviewed changes

vsoch mentioned this pull request Aug 26, 2021

ack: Switch to https #25639

Closed

adamjstewart merged commit 6a26322 into spack:develop Aug 26, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixing "problems in spack" as identified by repology #25491

Fixing "problems in spack" as identified by repology #25491

vsoch commented Aug 18, 2021

alalazo left a comment

alalazo left a comment

alalazo Aug 19, 2021

vsoch Aug 19, 2021

adamjstewart left a comment

adamjstewart Aug 19, 2021

vsoch Aug 19, 2021

adamjstewart Aug 19, 2021

glennpj Aug 24, 2021

glennpj Aug 24, 2021

vsoch Aug 24, 2021

adamjstewart Aug 26, 2021

adamjstewart left a comment

vsoch commented Aug 23, 2021

adamjstewart commented Aug 23, 2021

vsoch commented Aug 23, 2021

adamjstewart commented Aug 23, 2021

vsoch commented Aug 23, 2021

vsoch commented Aug 26, 2021

alalazo commented Aug 27, 2021

vsoch commented Aug 27, 2021

		homepage = "https://metacpan.org/pod/HTML::Parser"
		url = "http://search.cpan.org/CPAN/authors/id/G/GA/GAAS/HTML-Parser-3.72.tar.gz"

Fixing "problems in spack" as identified by repology #25491

Fixing "problems in spack" as identified by repology #25491

Conversation

vsoch commented Aug 18, 2021

alalazo left a comment

Choose a reason for hiding this comment

alalazo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adamjstewart left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adamjstewart left a comment

Choose a reason for hiding this comment

vsoch commented Aug 23, 2021

adamjstewart commented Aug 23, 2021

vsoch commented Aug 23, 2021

adamjstewart commented Aug 23, 2021

vsoch commented Aug 23, 2021

vsoch commented Aug 26, 2021

alalazo commented Aug 27, 2021

vsoch commented Aug 27, 2021