Cases when Hackage information is not updated

[Anton-Latukha]

I noticed that the project reports wrong Hackage versions, for example for the most famous text formats converter pandoc:

Repology reports to us that newest pandoc on Hackage is 2.2.1, but how can that be when all Haskell ecosystem today releases everything solely through Hackage.

Really, on Hackage the latest pandoc released is 2.9.2.1

2.2.1 version is 2018-05-11, it has been two years of constant development from that point, 2.2.1 -> 2.9.2.1 means that by Haskell version policy terms there were 7 major breakage releases in these two years, Haskell packages mostly curated by their xx.xx.**.** first and second digits since those mean the API stable releases/branches.

---

I know there is packdeps.haskellers.com and pacdeps utility that more canonically determine real last versions. Webservice currently does not respect deprecated versions (like when developers released it as API breakage but then due to no API breackage in real - rolled down the release number into a minor release and deprecated that major release increase). I submitted a report on that, for web service to respect the deprecations. pacdeps has a --preferred flag to properly respect the deprecations, deprecations of the major release increases by developers should be respected due to otherwise the service would indicate the old placebo major release, when the development goes and latest releases are in the previous numbers.

I do not use pacdeps at the moment, so I do not know if it would be useful to your project, but I think most probably.

[AMDmi3]

That's what Repology uses:

% wget -qO- https://hackage.haskell.org/packages/index.tar.gz | tar -tf- | awk -F/ '$1=="pandoc"{print $2}' | sort -g | tail -10
2.0.3
2.0.4
2.0.5
2.0.6
2.1
2.1.1
2.1.2
2.1.3
2.2
2.2.1

Given than overall hackage freshness is still >99%, it could be a problem with that index file generation which affects some individual packages.

currently does not respect deprecated versions

We don't either, see #838

[Anton-Latukha]

Ok, I would know about this and would look what I can contribute here.

Most probably there is a definite canonical Haskell tooling way or tool to directly resolve this information.

Maybe I even ask the most knowledgeable person and his club about this on the livestream in a couple of hours.

I would report here.

[Anton-Latukha]

Also this summer there is work happening on extending Hackage API, particularly for "badges", which in fact is API returning versioning info.

[Anton-Latukha]

So. We sorted it out.

---

As testing example here we would use binary package, it deprecated last two versions, 0.9, 0.10 and so 0.8 is the real latest branch.

New index link is http://hackage.haskell.org/01-index.tar.gz
Previous is http://hackage.haskell.org/00-index.tar.gz

So looks like they also can further increase version someday.

It (01-index.tar.gz) has a new pandoc version and the whole shebang.

But that does not respect the deprecated versions, we can observe there is binary 0.9, 0.10.

---

Also, sort has --version-sort sort option.

[Anton-Latukha]

Also peti has the hackage-db library that allows on top of it run cabal -v0 v2-run show-package-versions binary and it reports a clean list and respects the deprecations also, but this needs presetup and download/compilation of Haskell libraries, and better this tool be implemented separately by itself.

[Anton-Latukha]

Better to do it over official Haskell package management util - the cabal.

1. On your hosts to update the local Hackage DB with Cabal (cabal v2-update).

And you may even share its DB over the node instances in your setup if you want.

And so you can update Cabal Hackage DB once in a period you want and use one DB fetch everywhere in your infrastructure.

2. Then just do

cabal info binary
...
    Versions available: 0.6.4.0, 0.7.5.0, 0.7.6.0, 0.7.6.1, 0.8.6.0, 0.8.7.0,
                        0.8.8.0, (0.9.0.0), (0.10.0.0) (and 37 others)
...

Deprecated versions are put inside (..)

So you can parse this output and receive needed real latest version, this is 0.8.8.0 here.

---

Also as I mentioned soon there should be the propper support from Hackage API of badge requests, so you would be able to just negotiate a huge quota for repology, and use the API.

[Anton-Latukha]

So here we have.

If you would need a full list of all versions provided. Then - technically deprecated versions are legitimate, cabal just really avoids them to the end, and picks them only if there are no other options. For all list - you can parse http://hackage.haskell.org/01-index.tar.gz.

Or if you want the list of only really used versions. You can find a show-package-versions in haskell-db project and run cabal -v0 v2-run show-package-versions <package> on that file - it would print all versions but not deprecations.

If you want to get the last release version, or last versions on last several release branches - cabal info <package>.

[AMDmi3]

Switched to 01-index.tar.gz. Pandoc is fixed, hackage freshness rose to 99.9%.

Had to rewrite the parser to allow incremental tarfile in multiple passes. preferred-versions are collected, but not parsed and used yet. It shouldn't be too hard to implement parser for these and support deprecated versions, but that's not top priority right now.