cache parallel tests log on ci

[hyuraku]

• Have you followed the guidelines in our Contributing document?
• Have you checked to ensure there aren't other open Pull Requests for the same change?
• Have you added an explanation of what your changes do and why you'd like us to include them?
• Have you written new tests for your changes? Here's an example.
• Have you successfully run brew style with your changes locally?
• Have you successfully run brew typecheck with your changes locally?
• Have you successfully run brew tests with your changes locally?

---

cache brew tests(parallel tests) log on ci.
Fixes #10507

[MikeMcQuaid]

Looks good @hyuraku! If you rebase on master the macOS test failures will be gone.

What do we need to look for in the GitHub Actions logs to see if this is working as expected?

[hyuraku]

@MikeMcQuaid

What do we need to look for in the GitHub Actions logs to see if this is working as expected?

I guess that We have to check the two expect test results.

1. Cache test log in Cache parallel tests log phase.
   On change order of caching test log · Homebrew/brew@a5c4a3d · GitHub,
   it is not good because cache error happensCache not found for input keys.
2. Test log shows Using recorded test runtime like a parallel_tests spec

[Bo98]

actions/cache@v1 does not support single file caches. You could create a directory and cache that.

[MikeMcQuaid]

Just noticed: probably want to include the test-flags in the key somehow (because otherwise the three matrix workers will overwrite each other's cache)

[MikeMcQuaid]

2. Test log shows Using recorded test runtime like a parallel_tests spec

Not seeing any of these yet unfortunately 😭

[Bo98]

The previous cache is probably in a broken state now.

The cache is never overwritten - you have something unique in the key. Use something unique to each test run (like perhaps ${{ github.sha }}), but have restore-keys be more relaxed to allow restoring from previous runs.

[MikeMcQuaid]

Use something unique to each test run (like perhaps ${{ github.sha }})

That would avoid having any speedup here. It's fine to overwrite the existing values here (as this is effectively what is done locally) it's just not fine to do so across test suites. Arguably, though, that's something that could be handled in brew tests itself to avoid brew tests && brew tests --generic on my local machine having incorrect parallel run logs.

[MikeMcQuaid]

Suggested change

	parallel_rspec_log_path = if ENV["CI"]
	"tests/parallel_runtime_rspec.log"
	else
	"#{HOMEBREW_CACHE}/tests/parallel_runtime_rspec.log"
	end
	parallel_rspec_log_name = "parallel_runtime_rspec"
	parallel_rspec_log_name = "#{parallel_rspec_log_name}.no_compat" if args.no_compat?
	parallel_rspec_log_name = "#{parallel_rspec_log_name}.generic" if args.generic?
	parallel_rspec_log_name = "#{parallel_rspec_log_name}.online" if args.online?
	parallel_rspec_log_name = "#{parallel_rspec_log_name}.log"
	parallel_rspec_log_path = if ENV["CI"]
	"tests/#{parallel_rspec_log_name}"
	else
	"#{HOMEBREW_CACHE}/tests/#{parallel_rspec_log_name}"
	end

[MikeMcQuaid]

Suggested change

	key: ${{ runner.os }}-parallel_runtime_rspec.log
	restore-keys: ${{ runner.os }}-parallel_runtime_rspec.log
	key: ${{ runner.os }}-${{ matrix.test-flags }}-parallel_runtime_rspec

[MikeMcQuaid]

Suggested change

restore-keys: ${{ runner.os }}-parallel_runtime_rspec.log

[Bo98]

That would avoid having any speedup here.

That's the point of restore keys. So anything is restored via a partial key match but you always save the updated log with a unique key.

Right now, as the key is, the cache will never be updated because the key never changes. So you are storing one log and using it permanently without changes.

(I'm not talking about the platform/flags - I agree they should be separated).

[MikeMcQuaid]

Right now, as the key is, the cache will never be updated because the key never changes. So you are storing one log and using it permanently without changes.

@Bo98 My understanding was it'd be overwritten each time: https://github.com/Homebrew/brew/pull/10989/checks?check_run_id=2274127135#step:17:3

Do you have a source for the above? Not disagreeing, just want to make sure I fully understand 😁

[Bo98]

That worked because it's the first run under the new key, but look at the previous commit run:

https://github.com/Homebrew/brew/runs/2270091160?check_suite_focus=true#step:17:2

[MikeMcQuaid]

That worked because it's the first run under the new key, but look at the previous commit run:

https://github.com/Homebrew/brew/runs/2270091160?check_suite_focus=true#step:17:2

Gotcha!

[MikeMcQuaid]

@Bo98 make sense to you?

[Bo98]

Yep that should work. Will need to run it a couple times to see if the restore works.

[MikeMcQuaid]

Great work here, thanks again @hyuraku!