test resources is not found by brew

[fxcoudert]

brew doctor output

Happens on all CI runners

Verification

• My "brew doctor output" above says Your system is ready to brew. and am still able to reproduce my issue.
• I ran brew update twice and am still able to reproduce my issue.
• This issue's title and/or description do not reference a single formula e.g. brew install wget. If they do, open an issue at https://github.com/Homebrew/homebrew-core/issues/new/choose instead.

brew config output

Happens on all CI runners

What were you trying to do (and why)?

brew test --verbose openai-whisper as part of CI

What happened (include all command output)?

  ==> Testing openai-whisper
  Error: openai-whisper: failed
  An exception occurred within a child process:
    ResourceMissingError: openai-whisper does not define resource "homebrew-test-audio"
  /opt/homebrew/Library/Homebrew/software_spec.rb:136:in `block in resource'
  /opt/homebrew/Library/Homebrew/software_spec.rb:136:in `fetch'
  /opt/homebrew/Library/Homebrew/software_spec.rb:136:in `resource'
  /opt/homebrew/Library/Homebrew/formula.rb:557:in `resource'
  /opt/homebrew/Library/Taps/homebrew/homebrew-core/Formula/o/openai-whisper.rb:95:in `block in <class:OpenaiWhisper>'

What did you expect to happen?

No error.

Step-by-step reproduction instructions (by running brew commands)

`brew test --verbose openai-whisper`

[MikeMcQuaid]

@fxcoudert Can you (or someone) try to git bisect to figure out when this happened?

I'm not sure if the issue is that the test resource cannot be found or that brew fetch is trying to fetch something found in a test do block.

[MikeMcQuaid]

regardless (and not a blocker for fixing this 🐛) we should probably have something like this in future instead:

resource "homebrew-test-audio" => :test do
  url "https://raw.githubusercontent.com/openai/whisper/7858aa9c08d98f75575035ecd6481f462d66ca27/tests/jfk.flac"
  sha256 "63a4b1e4c1dc655ac70961ffbf518acd249df237e5a0152faae9a4a836949715"
end

[fxcoudert]

I'm traveling so I can't debug properly for now, or bisect. But this happens only in the test step, all those steps before succeeded:

==> brew fetch --retry openai-whisper --build-bottle --force
==> brew install --verbose --build-bottle openai-whisper
==> brew style --formula openai-whisper
==> brew audit --formula openai-whisper --online --git --skip-style
==> brew bottle --verbose --json openai-whisper --only-json-tab
==> brew bottle --merge --write --no-commit --no-all-checks ./openai-whisper--20231117_3.arm64_sonoma.bottle.json
==> brew uninstall --force openai-whisper
==> brew uninstall --force asciidoc asciidoctor autoconf automake berkeley-db@5 bison boost cargo-c cmake curl docbook docbook-xsl doxygen flatbuffers gdbm gnu-getopt gobject-introspection googletest libgit2 libssh2 libtool lit llvm@17 lua m4 meson ncurses ninja openldap perl pkg-config protobuf@21 python-setuptools python@3.11 rtmpdump ruby rust scons source-highlight sphinx-doc swig util-macros vim xcb-proto xmlto xtrans
==> brew install --only-dependencies ./openai-whisper--20231117_3.arm64_sonoma.bottle.tar.gz
==> brew install ./openai-whisper--20231117_3.arm64_sonoma.bottle.tar.gz
==> brew linkage --test openai-whisper
==> brew linkage --cached --test --strict openai-whisper
==> brew linkage --cached openai-whisper
==> brew install --only-dependencies --include-test openai-whisper

and only this one fails:

==> brew test --verbose openai-whisper

https://github.com/Homebrew/homebrew-core/actions/runs/8899611195/job/24497962540?pr=170392

I've been able to test locally that it happens with any formula. It's also not a download failure, it doesn't even initiate the download.

In case it helps, the full backtrace (where I added the same in my local gmp formula for easier testing) is:

An exception occurred within a child process:
  ResourceMissingError: gmp does not define resource "homebrew-test-audio"
/opt/homebrew/Library/Homebrew/software_spec.rb:136:in `block in resource'
/opt/homebrew/Library/Homebrew/software_spec.rb:136:in `fetch'
/opt/homebrew/Library/Homebrew/software_spec.rb:136:in `resource'
/opt/homebrew/Library/Homebrew/formula.rb:557:in `resource'
/opt/homebrew/Library/Taps/homebrew/homebrew-core/Formula/g/gmp.rb:64:in `block in <class:Gmp>'
/opt/homebrew/Library/Homebrew/formula.rb:2793:in `block (3 levels) in run_test'
/opt/homebrew/Library/Homebrew/extend/kernel.rb:520:in `with_env'
/opt/homebrew/Library/Homebrew/formula.rb:2792:in `block (2 levels) in run_test'
/opt/homebrew/Library/Homebrew/formula.rb:1198:in `with_logging'
/opt/homebrew/Library/Homebrew/formula.rb:2791:in `block in run_test'
/opt/homebrew/Library/Homebrew/mktemp.rb:75:in `block in run'
/opt/homebrew/Library/Homebrew/mktemp.rb:75:in `chdir'
/opt/homebrew/Library/Homebrew/mktemp.rb:75:in `run'
/opt/homebrew/Library/Homebrew/formula.rb:3102:in `mktemp'
/opt/homebrew/Library/Homebrew/formula.rb:2785:in `run_test'
/opt/homebrew/Library/Homebrew/test.rb:46:in `block in <main>'
/opt/homebrew/Library/Homebrew/vendor/portable-ruby/3.1.4/lib/ruby/3.1.0/timeout.rb:107:in `block in timeout'
/opt/homebrew/Library/Homebrew/vendor/portable-ruby/3.1.4/lib/ruby/3.1.0/timeout.rb:36:in `block in catch'
/opt/homebrew/Library/Homebrew/vendor/portable-ruby/3.1.4/lib/ruby/3.1.0/timeout.rb:36:in `catch'
/opt/homebrew/Library/Homebrew/vendor/portable-ruby/3.1.4/lib/ruby/3.1.0/timeout.rb:36:in `catch'
/opt/homebrew/Library/Homebrew/vendor/portable-ruby/3.1.4/lib/ruby/3.1.0/timeout.rb:123:in `timeout'
/opt/homebrew/Library/Homebrew/test.rb:50:in `<main>'

[fxcoudert]

Done bisecting:

caf87c0336dea14bafe35441ea90dfd32be27d85 is the first bad commit
commit caf87c0336dea14bafe35441ea90dfd32be27d85
Author: Markus Reiter <me@reitermark.us>
Date:   Fri Apr 26 20:55:51 2024 +0200

    Warn about undocumented non-private APIs.

@reitermarkus

[fxcoudert]

Maybe we should revert temporarily. This is causing several big PR testing to fail on homebrew-core.

[reitermarkus]

The problem is Formula.resource works on class level, and I thought that Formula#resource at the instance level is only used for accessing the class level resource definitions, hence only the name argument is needed.

In openai-whisper we are using Formula#resource in test to define a new resource with a block, so the signature is wrong.

[Bo98]

Test resources working is purely accidental tbh so am not surprised it breaks.

That's why the signature is "wrong" - it was never supposed to be used this way, someone figured out it worked and it slowly became widespread from there.

Long-term ambition is full support via :test tag or similar but this hasn't been picked up yet.

[MikeMcQuaid]

resource "homebrew-test-audio" => :test do

this is what is needed