livecheck: crates.io strategy

[vladimyr]

• Have you followed the guidelines in our Contributing document?
• Have you checked to ensure there aren't other open Pull Requests for the same change?
• Have you added an explanation of what your changes do and why you'd like us to include them?
• Have you written new tests for your changes? Here's an example.
• Have you successfully run brew style with your changes locally?
• Have you successfully run brew tests with your changes locally?
• Have you successfully run brew man locally and committed any changes?

---

This adds a new livecheck strategy for software obtained from crates.io. Note that crates.io doesn't let bots access their data (https://crates.io/policies#crawlers) so docs.rs is used instead.

[samford]

From what I'm seeing, we don't currently have any formulae in homebrew/core that use a crate as the stable archive, homepage, etc. We prefer to check for a new version in the same place as the stable URL, whenever possible, so that usually ends up being GitHub for Rust-related formulae.

Could you explain a bit about what category of formulae would benefit from this strategy (with examples, if possible)? We usually add a strategy in response to a demonstrated need (i.e., multiple formulae doing something similar in their livecheck block, multiple formulae using the same stable source, etc.).

[vladimyr]

we don't currently have any formulae in homebrew/core that use a crate as the stable archive, homepage, etc.

Emphasis on the currently Homebrew/homebrew-core#66697 😁

I'm guessing that asking upstream to retroactively tag commits is no go (#9505 (comment)). And something tells me this isn't an isolated case and we might bump into it again so I decided to take care of it.
I already addressed it with a formula-specific livecheck block but then I saw PyPI, npm & Hackage specific strategies and one thing led to another...

Could you explain a bit about what category of formulae would benefit from this strategy (with examples, if possible)?

So to answer your question, the only category of formulae that would benefit from this strategy are weird ones without tagged releases on Github 🤷

I guess this falls into:

Your scientists were so preoccupied with whether or not they could, they didn’t stop to think if they should.

FWIW even though we agree that this was a stupid challenge and it shouldn't land in it was very much worth it. I figured out that all livecheck strategies lack (fake) user agent making them susceptible to data access restrictions like in crates.io case (https://crates.io/policies#crawlers).

This brings me to the next question. @samford how do you feel about me doing some actually useful PR ensuring Homebrew's fake Safari user agent is used for all livecheck requests?

[vladimyr]

Meanwhile, I aligned it with existing strategies by going back to matching URLs using regex and applying stuff from #9510

[samford]

Emphasis on the currently Homebrew/homebrew-core#66697 😁

There was one problematic formula in the past where I floated the idea of switching stable to a crate file (as there weren't any other good options at the time) but there was pushback against the idea. If that idea got anywhere, I was going to either add .crate to unpack_strategy/tar.rb or create a new strategy (like #9505), so it's interesting to see this.

Let's see how Homebrew/homebrew-core#66697 and #9505 play out before we move forward with this livecheck strategy. If the oakc formula ends up being merged while using the crate, I'll come back and review this and move it toward merging. I'm going to label this do not merge for now.

how do you feel about me doing some actually useful PR ensuring Homebrew's fake Safari user agent is used for all livecheck requests?

I'm glad you asked, as I have some local changes hanging around that address this as part of something larger. I've encountered issues with a small number of other websites in the past, which is why I made sure to add support for user agents as part of working on something else. I haven't created a PR for this yet because I promised not to work on notable livecheck PRs until my open documentation PR is pretty much done (or at least closer to merging).

I've been working on the docs PR a lot this week and I'm getting close to pushing a revision, so I should be working on livecheck's internals again very soon. At that time, I'll be creating a bunch of PRs for changes I've been sitting on, so it would be best to hold off for now. Thanks for the offer, though!

[vladimyr]

There was one problematic formula in the past where I floated the idea of switching stable to a crate file (as there weren't any other good options at the time)

It is kinda comforting to know I'm not the only one that hit that bummer. And you can definitely expect me to fix exotic and things not worth touching 😄

I've been working on the docs PR a lot this week and I'm getting close to pushing a revision, so I should be working on livecheck's internals again very soon. At that time, I'll be creating a bunch of PRs for changes I've been sitting on, so it would be best to hold off for now. Thanks for the offer, though!

It is equally satisfying to fix things myself or just see them being fixed by others so go ahead and make it better 🚀

[samford]

Let's see how Homebrew/homebrew-core#66697 and #9505 play out before we move forward with this livecheck strategy.

In light of #9505 being closed without merging, I think we should follow suit here. oakc is the only formula that uses a crate as the stable archive at the moment and I don't see that number notably increasing anytime soon. It seems likely that oakc will remain the only formula using a crate for some time, as it's not something that we do under normal circumstances. Even then, the long-term goal with oakc is to migrate its stable URL away from the crate if upstream ever starts tagging release versions on GitHub.

As mentioned before, we create new strategies based on a demonstrated need in enough formulae/casks and that's not the case here yet. With that in mind, I'm going to close this but feel free to save a patch or stash these changes in case the situation changes in the future.

Thanks for your understanding, @vladimyr!

[vladimyr]

It was the logical thing to do 👍
If we ever see an increase in cases where crate URL is used as stable this could be used as the basis for a new livecheck strategy.