GitHub Actions: Revdep error for 'furrr' on 'Snapshot of code has changed'

[HenrikBengtsson]

I've been getting the following reverse-dependency check errors for furrr on GitHub Actions for a long time now. It happens also over on the future repository:

* checking tests ...
  Running ‘testthat.R’ [40s/118s]
 [40s/118s] ERROR
Running the tests in ‘tests/testthat.R’ failed.
Last 13 lines of output:
  ── Failure ('test-deprecation.R:2'): `future_options()` is hard deprecated ─────
  Snapshot of code has changed:
  old[3:6] vs new[3:6]
    "Condition"
    "  Error:"
    "  ! `future_options()` was deprecated in furrr 0.3.0 and is now defunct."

  - "  Please use `furrr_options()` instead."
  + "  i Please use `furrr_options()` instead."
  
  * Run `testthat::snapshot_accept('deprecation')` to accept the change.
  * Run `testthat::snapshot_review('deprecation')` to interactively review the change.
  
  [ FAIL 1 | WARN 0 | SKIP 0 | PASS 855 ]
  Error: Test failures
  Execution halted

I've fully cleared the GitHub Action Caches (https://github.com/HenrikBengtsson/globals/actions/caches), but the above still shows up.

I do not get this when I run R CMD check --as-cran furrr_0.3.1.tar.gz locally.

@DavisVaughan, do you happen to know how I fix this?

[DavisVaughan]

You might need an updated version of {lifecycle} locally.

I'd need to rerun the snapshot tests locally and push updated ones to github to clear this up. Does this check of yours use the dev version of furrr? Or would i need to send in a patch release to CRAN for you?

[HenrikBengtsson]

You might need an updated version of {lifecycle} locally.

Locally, I have everything up-to-date per CRAN and Bioconductor. Are you thinking this is only happens on GitHub Actions right now, because there's a devel version of lifecycle might be used?

Does this check of yours use the dev version of furrr?

No, I test towards the CRAN source tarball.

Or would i need to send in a patch release to CRAN for you?

I'm not sure if I follow, probably because I don't understand how testthat snapshots work, but wouldn't I have a problem when I submit to CRAN, if you don't update there?

I've just add a job to test against furrr devel (v0.3.1-9000), but it also gives this error, cf. https://github.com/HenrikBengtsson/globals/actions/runs/3414650319/jobs/5682882329.

[DavisVaughan]

wouldn't I have a problem when I submit to CRAN, if you don't update there?

No, snapshot tests aren't ever run by CRAN. They are run in two main circumstances:

• Locally, when you run devtools::test() or devtools::check()
• By our CI, i.e. GitHub Actions

Snapshot tests are useful, but are by their very nature a bit fragile, which is why they aren't run on CRAN. We mainly use them to check against error messages that we don't own in that package (like how the lifecycle message is being thrown from furrr), so if the error message gets tweaked a bit upstream, we don't want to have to immediately fix all of the downstream packages. So generally we just update the snapshot tests when we get back around to working on that package to account for any changes that might have occurred in upstream packages since the last release.

Are you running the tests locally with devtools::check() or devtools::test()? That could be why you don't see the snapshot failures. You might be emulating CRAN closely enough that the test system thinks you are CRAN, preventing the snapshot tests from running.

[HenrikBengtsson]

No, snapshot tests aren't ever run by CRAN. They are run in two main circumstances:

• Locally, when you run devtools::test() or devtools::check()
• By our CI, i.e. GitHub Actions

I see. So, it sounds like I need to find a way to disable the snapshot checks on GitHub Actions? But does it also mean you need to regenerate the snapshots on furrr* devel, since that fails for me, and I guess anyone else who tries the same.

Are you running the tests locally with devtools::check() or devtools::test()? That could be why you don't see the snapshot failures.

I'm using R CMD check --as-cran furrr_0.3.1.tar.gz, and I've also tried with rcmdcheck::rcmdcheck("furrr_0.3.1.tar.gz"). I don't know how to check a tarball using devtools::check(); it seems to be targeting the developer of the package tests, i.e. assuming you've got a package folder.

[DavisVaughan]

it sounds like I need to find a way to disable the snapshot checks on GitHub Actions?

Yea, but I think my recommendation for you is that you should try and make your reverse dependency GitHub Actions workers "look like CRAN", and that will automatically disable snapshot test checking.

The way to do that is to set NOT_CRAN: false as an environment variable here

globals/.github/workflows/future-revdepcheck-top.yaml

Line 35 in d3a0b70

## R CMD check

NOT_CRAN is typically set to true by the setup-r action here, but if you explicitly set it to false ahead of time in the env: section, then that should be respected

globals/.github/workflows/future-revdepcheck-top.yaml

Line 50 in d3a0b70

- uses: r-lib/actions/setup-r@v2

https://github.com/r-lib/actions/blob/50d1eae9b8da0bb3f8582c59a5b82225fa2fe7f2/setup-r/src/installer.ts#L678

Give that a try and see if it "just works"

As mentioned before, we typically want our CI to look like our local workflow, i.e. "not CRAN", so that is why NOT_CRAN: true is the default.

---

does it also mean you need to regenerate the snapshots on furrr* devel

Yes, eventually, when I come back around to work on furrr again that will be the first thing I do. But the idea behind all this is to prevent us from having to rush back to a package and perform some kind of update as soon as a fragile snapshot test starts failing.

[HenrikBengtsson]

Yea, but I think my recommendation for you is that you should try and make your reverse dependency GitHub Actions workers "look like CRAN", and that will automatically disable snapshot test checking.

The way to do that is to set NOT_CRAN: false as an environment variable here

Thank you. This solved the problem (cf. https://github.com/HenrikBengtsson/globals/actions/runs/3457263520/jobs/5770573078)