`expect_snapshot_file()` should have some convention for varying file names

[hadley]

So that (e.g.) you can generate snap-windows.png, snap-osx.png, and snap-linux.png on separate machines, and not have the others automatically deleted.

[krlmlr]

Similarly for UTF8 vs. non-UTF8 output. Currently I'm working around in pillar by specifying the test name in testthat depending on the system setup.

[arnaud-feldmann]

I agree with that, and it's the same even for R versions. The point is that :

• making snapshot is cool because it catches even small modifcations.
• but catching small modifications makes also tests fragile hence it would be very useful to have some easy way to accept those modifications

With context I can use a simple trick by example context(if (R.version$major>=4 &&R.version$minor>=1.0) "plots-R-4-1" else "plots-R-4-0") at the begining of a file for vdiffr.
context being deprecated, that won't be possible anymore. Hence I'd love testthat 3.0 to provide a fine way to make the "I'm okay with that" easy without having to skip.

[beevabeeva]

This would be a most welcome feature.

A way to have multiple acceptable outcomes of an expect_snapshot() would be nice.

I use snapshots in testthat files that are initially tested using machine A.
Since the tests are DBI related, things like a db host name will change when running the same tests on a Gitlab runner. So I have to create and run a little script to accept the new snapshots, within the gitlab-ci.yaml config file 😆 . I still like to run the same tests on machine A.

Having expect_snapshot() take an optional path parameter seems to be an easy solution (easy to someone who has not authored this package and does not understand its nuances 😉 ).

Edit: The above scenario might not be what Hadley is describing in this issue.

[hadley]

Maybe a suffix argument would be sufficient?

[cderv]

I think I have another use case for snapshot with different Pandoc version. Only one is used at a time and currently I would need to use 2 snapshot call with a if clause on Pandoc version. The suffix argument could be way to have one call to expect_snapshot but several files created depending on the suffix value.

[arnaud-feldmann]

Maybe a suffix argument would be sufficient?

I think it would be fine

[hadley]

(I now think variant is probably a better name than suffix)

Let's just flesh out what might happen in a little more detail, using expect_snapshot() as an example. Say you have this test:

test_that("can access nickname", {
  expect_snapshot(
    version$nickname,
    variant = as.character(getRversion()[, 1:2])
  )
})

And when you run it on your local machine it generates:

# can access nickname (4.0)

    Code:
      version$nickname
    Output:
      [1] "Shake and Throw"

If run on CI that uses 4.0, then the test works as usual. What happens if the test is run on 4.1? Currently:

• The snapshot can access nickname (4.0) would be deleted
• You'd get a warning about creating a new snapshot

I think both of those things need to change slightly:

• Any snapshots where the variant doesn't match need to be left alone (so maybe the variant should become an h2?)
• Instead of a warning, you should get a skip (since those are reported more aggressively, and seem like a better fit here)

It feels like we're adding some hierarchy to the snapshots — you'd only delete a snapshot if no children were created. This suggests that maybe for expect_snapshot_file() the overall snapshot should use a directory? And maybe we could do it without an extra argument, so you'd activate this behaviour by using a subdirectory in the path? OTOH that would make it less consistent with expect_snapshot() which will need a separate argument.

cc @lionel- in case he has any thoughts from his vdiffr experience.

[lionel-]

That was my thought as well, instead of file suffixes this could be a folder prefix. This way we can reuse the existing snapshot mechanism without much change. UI-wise this could be implemented using a global option like testthat.snapshot_prefix that could be set globally from a helper- file or locally with local_options().

[hadley]

@lionel- Sounds like folder prefix is the way to go. So the code below would create _snaps/bar/foo.md and _snaps/bar/foo/baz.png.

# test-foo.R
test_that("a test", {
  expect_snapshot(variant = "bar")
  expect_snapshot_file("baz.png", variant = "bar")
})

And my previous example:

r_version <- function() as.character(getRversion()[, 1:2])

test_that("can access nickname", {
  expect_snapshot(version$nickname, variant = r_version())
})

Would yield _snaps/4.0/foo.md:

# can access nickname

    Code:
      version$nickname
    Output:
      [1] "Shake and Throw"

While one global variant name would make the implementation much simpler, I think it'll be a little too restrictive in practice. You might want some tests that are split out out by (e.g.) OS and others by R version, and think you'll want to keep the majority of tests variant free, so you'd then still need some way to manually opt in/out for each snapshot.

[hadley]

When you use a variant, there's no way that testthat can automatically delete outdated snaps, because there's no way for one session to know the full set of possible variants. That means only variants that correspond to test files that no longer exist can be deleted.

And it's not that we should change from a warn to a skip, it's that a test that only generates warnings should automatically generate an empty test skip (i.e. #1416).

[hadley]

Maybe it's worth supplying a vector of all variants?