livecheck: refactor url preprocessing

[vladimyr]

• Have you followed the guidelines in our Contributing document?
• Have you checked to ensure there aren't other open Pull Requests for the same change?
• Have you added an explanation of what your changes do and why you'd like us to include them?
• Have you written new tests for your changes? Here's an example.
• Have you successfully run brew style with your changes locally?
• Have you successfully run brew tests with your changes locally?
• Have you successfully run brew man locally and committed any changes?

---

Working with URI instances instead of Strings makes code much shorter and more readable. Special cases handling isn't needed anymore because matching is done on a host(name) level, paths ending with /releases/latest do not get altered and all those listed formulas already have formula specific livecheck blocks.
Added goodies are introducing support for tildegit.org (which follows Github's routing patterns) and sourcehut 🎉

[MikeMcQuaid]

Makes sense to me but would like @samford or @nandahkrishna to chip in. Thanks @vladimyr!

[samford]

I'm in the process of doing [verbose JSON] livecheck runs across homebrew/core with and without these changes, to identify any regressions/improvements. I'll review this once that's finished and I've done some manual testing to check specific cases as well.

Off hand, there's one type of GitHub URL that the previous code handled but this PR doesn't, so I'm going to label this "do not merge" for the moment. I'll address this in the forthcoming review but I at least wanted to give a brief update while I'm testing this.

[vladimyr]

I'm in the process of doing [verbose JSON] livecheck runs across homebrew/core with and without these changes, to identify any regressions/improvements. I'll review this once that's finished and I've done some manual testing to check specific cases as well.

I made a quick test comparing the output of the brew livecheck --debug command for special cases before and after proposed changes.

Source:

#!/bin/bash

formulae=(
  mednafen
  camlp5
  kotlin
  osrm-backend
  prometheus
  pyenv-virtualenv
  sysdig
  shairport-sync
  yuicompressor
)

export HOMEBREW_NO_AUTO_UPDATE=1
for formula in "${formulae[@]}"; do
  brew livecheck --debug "$formula" 2>/dev/null
done

Results:

results.before.txt
results.after.txt

FWIW is worth those are identical but sweep through the whole formula set is very much needed 👍

---

Off hand, there's one type of GitHub URL that the previous code handled but this PR doesn't, so I'm going to label this "do not merge" for the moment. I'll address this in the forthcoming review but I at least wanted to give a brief update while I'm testing this.

Let me know what did I miss and I'll try to fix it so you don't have to do redundant tests. 😉

[samford]

I've finished testing this and identifying regressions/improvements.

I like the idea of simplifying this code but this PR currently omits handling of certain types of URLs that the previous code supports. Namely:

• Older https://github.com/downloads/owner/repository/... URLs were handled properly before but aren't handled in this PR. These URLs should be processed into https://github.com/owner/repository.git but this PR produces a processed URL like https://github.com/downloads/owner.git.
• GitLab URLs that aren't on gitlab.com were handled properly before (this is the purpose of the url.include?("/-/archive/") block) but this PR only supports gitlab.com. Removing the broader GitLab support in favor of a domain-specific approach breaks checks for a number of formulae.

That said, it's easy to fix these issues and I've added suggestions that do so. I also did a livecheck run across homebrew/core with my suggestions below and confirmed they resolve these two regressions that I saw in the original run for this PR's changes.

There are some modifications that need to be made to specific formulae with this PR in mind but I've taken care of some of them and I'm in the process of working through the others. Most of these can be done before this is merged but at least one needs to be handled afterward.

Formulae with a https://github.com/downloads/... stable URL

The following formulae are affected by the removal of the code that handles https://github.com/downloads/owner/repository/... URLs. Some of the checks continue to work even though the stable URL can't be checked, as either the head URL is tried first or livecheck falls back to the homepage (since livecheck checks in order of head, stable, homepage when there isn't a livecheck block in the formula).

• btpd: check only works because it falls back to the homepage URL
• cssembed: broken, as no formula URLs are usable by default
• growly: check only works because it falls back to the homepage URL
• henplus: broken, as no formula URLs are usable by default
• midgard2: only works because the head URL is tried before the stable URL
• rpg: only works because the head URL is tried before the stable URL
• tidyp: check only works because it falls back to the homepage URL

Formulae with a Self-Hosted GitLab stable URL

The following formulae are affected by the removal of the /-/archive/ URL handling code, which was replaced with code that only handles gitlab.com URLs. As above, some of these checks are broken by this change and others continue to work due to the existence of other URLs in the formula (e.g., head, homepage).

• dav1d: https://code.videolan.org/videolan/dav1d/-/archive/0.7.1/dav1d-0.7.1.tar.bz2
• libcerf: https://jugit.fz-juelich.de/mlz/libcerf/-/archive/v1.14/libcerf-v1.14.tar.gz
• libhandy: https://gitlab.gnome.org/GNOME/libhandy/-/archive/1.0.1/libhandy-1.0.1.tar.gz
• libolm: https://gitlab.matrix.org/matrix-org/olm/-/archive/3.2.1/olm-3.2.1.tar.gz
• libxp: https://gitlab.freedesktop.org/xorg/lib/libxp/-/archive/libXp-1.0.3/libxp-libXp-1.0.3.tar.bz2
• lmdb: https://git.openldap.org/openldap/openldap/-/archive/LMDB_0.9.27/openldap-LMDB_0.9.27.tar.bz2
• mat2: https://0xacab.org/jvoisin/mat2/-/archive/0.11.0/mat2-0.11.0.tar.gz
• menhir: https://gitlab.inria.fr/fpottier/menhir/-/archive/20200624/menhir-20200624.tar.bz2
• ortp: https://gitlab.linphone.org/BC/public/ortp/-/archive/4.3.2/ortp-4.3.2.tar.bz2
• posh: https://salsa.debian.org/clint/posh/-/archive/debian/0.14.1/posh-debian-0.14.1.tar.bz2

Other Issues

URI.parse will give a URI::InvalidURIError error for some head URLs. The examples I encountered were:

• mandoc: URI::InvalidURIError (bad URI(is not URI?): "anoncvs@mandoc.bsd.lv:/cvs") (from head URL)
• ncp: URI::InvalidURIError (bad URI(is not URI?): ":pserver:cvs:@cvs.fefe.de:/cvs") (from head URL)

I'll defer to @MikeMcQuaid about what the best way of handling this would be. The basic idea is that these particular URLs shouldn't be processed. In general, livecheck won't do anything with these URLs (since no strategies will apply to them), so we only have to deal with this from the standpoint of #preprocess_url.

Improvements

• bombadillo: Unable to get versions before and now working properly using the Git strategy with the tildegit.org repository.
• scdoc: Unable to get versions before and now working properly using the Git strategy with the git.sr.ht repository.

[vladimyr]

@samford Before I start reading/addressing comments I just want to take a moment to appreciate both the effort and the outcome, tips 🎩

[vladimyr]

@samford I took another shot based on helpful comments and my own ideas and here is what I got:

Formulae with a https://github.com/downloads/... stable URL

$ brew livecheck --debug btpd cssembed growly henplus midgard2 rpg tidyp 2>&1

Show output

/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/btpd.rb
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/cssembed.rb
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/growly.rb
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/henplus.rb
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/midgard2.rb
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/rpg.rb
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/tidyp.rb

Formula:          btpd
Livecheckable?:   No

URL:              https://github.com/downloads/btpd/btpd/btpd-0.16.tar.gz
URL (processed):  https://github.com/btpd/btpd.git
Strategy:         Git

Matched Versions:
0.13, 0.14, 0.15, 0.16
btpd : 0.16 ==> 0.16

----------


Formula:          cssembed
Livecheckable?:   No

URL:              https://github.com/downloads/nzakas/cssembed/cssembed-0.4.5.jar
URL (processed):  https://github.com/nzakas/cssembed.git
Strategy:         Git

Matched Versions:
0.4.0, 0.4.5
cssembed : 0.4.5 ==> 0.4.5

----------


Formula:          growly
Livecheckable?:   No

URL:              https://github.com/ryankee/growly.git
Strategy:         Git

URL:              https://github.com/downloads/ryankee/growly/growly-v0.2.0.tar.gz
URL (processed):  https://github.com/ryankee/growly.git
Strategy:         Git

URL:              https://github.com/ryankee/growly
URL (processed):  https://github.com/ryankee/growly.git
Strategy:         Git
Error: growly: Unable to get versions

----------


Formula:          henplus
Livecheckable?:   No

URL:              https://github.com/downloads/neurolabs/henplus/henplus-0.9.8.tar.gz
URL (processed):  https://github.com/neurolabs/henplus.git
Strategy:         Git

URL:              https://github.com/neurolabs/henplus
URL (processed):  https://github.com/neurolabs/henplus.git
Strategy:         Git
Error: henplus: Unable to get versions

----------


Formula:          midgard2
Livecheckable?:   No

URL:              https://github.com/midgardproject/midgard-core.git
Strategy:         Git

Matched Versions:
10.05.0, 10.05.1, 10.05.2, 10.05.3, 10.05.4, 10.05.5, 10.05.5.1, 10.05.6, 10.05.7, 10.05.7.1, 10.12, 12.09.1, 9.09.0, 9.09.1, 9.09.2
midgard2 : 12.09 ==> 12.09.1

----------


Formula:          rpg
Livecheckable?:   No

URL:              https://github.com/rtomayko/rpg.git
Strategy:         Git

Matched Versions:
0.1.0, 0.2.0, 0.3.0
rpg : 0.3.0 ==> 0.3.0

----------


Formula:          tidyp
Livecheckable?:   No

URL:              https://github.com/downloads/petdance/tidyp/tidyp-1.04.tar.gz
URL (processed):  https://github.com/petdance/tidyp.git
Strategy:         Git

Matched Versions:
0.90, 0.99, 1.00, 1.02, 1.04
tidyp : 1.04 ==> 1.04

Formulae with a Self-Hosted GitLab stable URL

$ brew livecheck --debug dav1d libcerf libhandy libolm libxp lmdb mat2 menhir ortp posh 2>&1

Show output

/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/dav1d.rb
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/libcerf.rb
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/libhandy.rb
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/libolm.rb
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/libxp.rb
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/lmdb.rb
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/mat2.rb
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/menhir.rb
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/ortp.rb
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/posh.rb

Formula:          dav1d
Livecheckable?:   No

URL:              https://code.videolan.org/videolan/dav1d/-/archive/0.7.1/dav1d-0.7.1.tar.bz2
URL (processed):  https://code.videolan.org/videolan/dav1d.git
Strategy:         Git

Matched Versions:
0.1.0, 0.2.0, 0.2.1, 0.2.2, 0.3.0, 0.3.1, 0.4.0, 0.5.0, 0.5.1, 0.5.2, 0.6.0, 0.7.0, 0.7.1
dav1d : 0.7.1 ==> 0.7.1

----------


Formula:          libcerf
Livecheckable?:   Yes

URL:              https://jugit.fz-juelich.de/api/v4/projects/269/releases
Strategy:         PageMatch
Regex:            /libcerf[._-]v?(\d+(?:\.\d+)+)/i

Matched Versions:
1.14, 2.0, 1.13
libcerf : 1.14 ==> 2.0

----------


Formula:          libhandy
Livecheckable?:   No

URL:              https://gitlab.gnome.org/GNOME/libhandy/-/archive/1.0.1/libhandy-1.0.1.tar.gz
URL (processed):  https://gitlab.gnome.org/GNOME/libhandy.git
Strategy:         Git

Matched Versions:
0.81.0, 0.82.0, 0.83.0, 0.84.0, 0.85.0, 0.90.0, 0.91.0, 1.0.0, 1.0.1, 0.0.1, 0.0.10, 0.0.11, 0.0.12, 0.0.13, 0.0.2, 0.0.3, 0.0.4, 0.0.5, 0.0.6, 0.0.7, 0.0.8, 0.0.9, 0.80.0
libhandy : 1.0.1 ==> 1.0.1

----------


Formula:          libolm
Livecheckable?:   No

URL:              https://gitlab.matrix.org/matrix-org/olm/-/archive/3.2.1/olm-3.2.1.tar.gz
URL (processed):  https://gitlab.matrix.org/matrix-org/olm.git
Strategy:         Git

Matched Versions:
0.1.0, 1.0.0, 1.1.0, 1.2.0, 1.3.0, 2.0.0, 2.1.0, 2.2.0, 2.2.1, 2.2.2, 2.3.0, 3.0.0, 3.1.0, 3.1.0-pre1, 3.1.1, 3.1.2, 3.1.3, 3.1.4, 3.1.5, 3.2.0, 3.2.1, 0.1.0, 0.2.0, 2.0.0
libolm : 3.2.1 ==> 3.2.1

----------


Formula:          libxp
Livecheckable?:   No

URL:              https://gitlab.freedesktop.org/xorg/lib/libxp/-/archive/libXp-1.0.3/libxp-libXp-1.0.3.tar.bz2
URL (processed):  https://gitlab.freedesktop.org/xorg/lib/libxp.git
Strategy:         Git

Matched Versions:
6_8_0-MERGE, 6_8_1-MERGE, 6_8_2-MERGE, 6_8_0-MERGE, 1-MERGE, 6-REVIEW-BASE, 86-4_3_0_1, 6_7_0, 6_7_99_1, 6_7_99_2, 6_7_99_901, 6_7_99_902, 6_7_99_903, 6_7_99_904, 6_8_0, 6_8_1, 6_8_1_901, 6_8_1_902, 6_8_1_903, 6_8_1_904, 6_8_2, 6_8_99_1, 6_8_99_10, 6_8_99_11, 6_8_99_12, 6_8_99_13, 6_8_99_14, 6_8_99_15, 6_8_99_16, 6_8_99_2, 6_8_99_3, 6_8_99_4, 6_8_99_5, 6_8_99_6, 6_8_99_7, 6_8_99_8, 6_8_99_9, 6_8_99_900, 6_8_99_901, 6_8_99_902, 6_8_99_903, 6_99_99_900, 6_99_99_901, 6_99_99_902, 6_99_99_903, 6_99_99_904, 7_0, 7_0_99_901, 7_1, 1, 1-BASE, 1-MERGE, 1-STSF-FORK, 1-TM-BASE, 1-TM-CLOSED, 1-TM-MERGE, 20040421_xprint_branch_landing, 3d-base, 3d-rel-0-6-2, 3d-rel-0-7-0, 1.0.0, 1.0.1, 1.0.2, 1.0.3, 0-6-1, 86-012804-2330, 86-4_3_0_1, 86-4_3_99_16, 86-4_3_99_901, 86-4_3_99_902, 86-4_3_99_903, 86-4_3_99_903_special, 86-4_4_0, 86-4_4_99_1, 6_7_0
libxp : 1.0.3 ==> 20040421_xprint_branch_landing

----------


Formula:          lmdb
Livecheckable?:   Yes

URL:              https://git.openldap.org/openldap/openldap.git
Strategy:         Git
Regex:            /^LMDB[._-]v?(\d+(?:\.\d+)+)$/i

Matched Versions:
0.9.15, 0.9.16, 0.9.17, 0.9.18, 0.9.19, 0.9.20, 0.9.21, 0.9.22, 0.9.23, 0.9.24, 0.9.25, 0.9.26, 0.9.27
lmdb : 0.9.27 ==> 0.9.27

----------


Formula:          mat2
Livecheckable?:   No

URL:              https://0xacab.org/jvoisin/mat2/-/archive/0.11.0/mat2-0.11.0.tar.gz
URL (processed):  https://0xacab.org/jvoisin/mat2.git
Strategy:         Git

Matched Versions:
0.1.0, 0.1.1, 0.1.2, 0.1.3, 0.10.0, 0.10.1, 0.11.0, 0.2.0, 0.3.0, 0.3.1, 0.4.0, 0.5.0, 0.6.0, 0.7.0, 0.8.0, 0.9.0
mat2 : 0.11.0 ==> 0.11.0

----------


Formula:          menhir
Livecheckable?:   No

URL:              https://gitlab.inria.fr/fpottier/menhir/-/archive/20200624/menhir-20200624.tar.bz2
URL (processed):  https://gitlab.inria.fr/fpottier/menhir.git
Strategy:         Git

Matched Versions:
20141215, 20170418, 20170509, 20170607, 20170712, 20171013, 20171206, 20171222, 20180523, 20180528, 20180530, 20180703, 20180905, 20181005, 20181006, 20181025, 20181026, 20181112, 20181113, 20190613, 20190620, 20190626, 20190924, 20200121, 20200123, 20200211, 20200525, 20200612, 20200619, 20200624, 309
menhir : 20200624 ==> 20200624

----------


Formula:          ortp
Livecheckable?:   No

URL:              https://gitlab.linphone.org/BC/public/ortp.git
Strategy:         Git

Matched Versions:
0.16.5, 0.18.0, 0.19.0, 0.20.0, 0.21, 0.21.1, 0.22.0, 0.24.0, 0.24.1, 0.24.2, 0.25.0, 0.26.0, 0.27.0, 1.0.0, 1.0.1, 1.0.2, 4.3.0, 4.3.1, 4.3.2, 4.3.3, 4.4.0, 4.4.4, 4.4.5, 4.4.6
ortp : 4.3.2 ==> 4.4.6

----------


Formula:          posh
Livecheckable?:   Yes

URL:              https://salsa.debian.org/clint/posh/-/archive/debian/0.14.1/posh-debian-0.14.1.tar.bz2
URL (processed):  https://salsa.debian.org/clint/posh.git
Strategy:         Git
Regex:            /^debian\/v?(\d+(?:\.\d+)+)$/i

Matched Versions:
0.10, 0.10.1, 0.10.2, 0.11, 0.11.1, 0.12, 0.12.1, 0.12.2, 0.12.2.1, 0.12.3, 0.12.4, 0.12.5, 0.12.6, 0.13, 0.13.1, 0.13.2, 0.14, 0.14.1, 0.6.1, 0.6.10, 0.6.11, 0.6.12, 0.6.13, 0.6.14, 0.6.15, 0.6.16, 0.6.17, 0.6.18, 0.6.2, 0.6.3, 0.6.4, 0.6.5, 0.6.6, 0.6.7, 0.6.8, 0.6.9, 0.7, 0.7.1, 0.7.2, 0.8, 0.8.1, 0.8.2, 0.8.3, 0.8.4, 0.8.5, 0.8.6, 0.9, 0.9.1, 0.9.2
posh : 0.14.1 ==> 0.14.1

BTW, took #9105 and applied it locally resulting in green light too.

[vladimyr]

For reference here are router implementations from:

• Gitea → https://github.com/go-gitea/gitea/blob/v1.12.5/routers/routes/routes.go#L243
• Gogs → https://github.com/gogs/gogs/blob/v0.12.3/internal/route/api/v1/api.go#L170

Gogs seems to lack the latest release (%r{/releases/latest/?$}) route.

[samford]

I'm calling it a night over here but I'll take another pass at the latest revision when I get a chance.

[samford]

This pass addresses the strictness of the host comparisons in the if/else conditions, as described in the comments below.

---

If #9105 is merged before this PR, I'll push a commit to your branch to expand the #preprocess_url test cases to include the additional URLs supported here. I already have them worked out and this PR passed when I tested it locally.

Otherwise, if this is merged before #9105, I'll simply update that PR to include the additional test cases.

[MikeMcQuaid]

Running CI now #9105 is merged.

[MikeMcQuaid]

@vladimyr Any news here?

[samford]

To help move this forward, I rebased this branch and added a commit that expands the #preprocess_url test cases to also include the newly-supported URLs in this PR.

I also added a suggestion here to handle the remaining issue that hadn't yet been addressed (i.e., URI.parse will give a URI::InvalidURIError error for some head URLs).

At this point, we just need to incorporate the suggestions, give this a final review, and I believe it should be good to go. @vladimyr, let me know if you won't have time to finish this, as I can simply incorporate the requested changes and help shepherd this towards merging.

[MikeMcQuaid]

At this point, we just need to incorporate the suggestions, give this a final review, and I believe it should be good to go. @vladimyr, let me know if you won't have time to finish this, as I can simply incorporate the requested changes and help shepherd this towards merging.

@samford if you could do that when you get a chance it'd be good 🙇🏻

[samford]

if you could do that when you get a chance it'd be good

I'm already incorporating the suggestions here on my local branch as part of testing, so I'll push a commit once I'm done 👍

[samford]

I did another before/after livecheck run across homebrew/core and the only changes I observed were as follows:

• aerc: Unable to get versions before but now works by checking the sourcehut Git tags.
• asuka: Previously worked by checking the sourcehut "refs" page but Unable to get versions with this PR. This is because the URL in the livecheck block is being processed into the Git repo address but the existing regex no longer applies due to the change in context (HTML page vs. Git tags).
• bombadillo:Unable to get versions before but now works by checking the tildegit Git tags.
• scdoc: Unable to get versions before but now works by checking the sourcehut Git tags.

I'll create a follow-up PR in homebrew-core to address asuka after this is merged; reducing the livecheck block to url :stable will sort it out.

At this point, this just needs a final review to address any areas that may need to be improved/modified.

Requested changes have been incorporated.

[samford]

Looks good to me at this point. I'm going to go ahead and merge this, so we can revisit another open livecheck PR (cask support) and try to move that one toward merging.

Thanks, @vladimyr!