Continuous Integration

[nashysolutions]

What do you recommend for using this with continuous integration? If the images need to be stored and then subsequently re-run so a comparison can be made, then how would you typically configure your CI setup to keep these images around? I appreciate that there are many CI tools out there and I'm not suggesting specifics. Just some general guidance in the README would be great. Thanks.

[zablaetna]

You would generate the reference screenshots in your branch work and store them. The CI doesn't generate them. It only compares them.

[nashysolutions]

You would generate the reference screenshots in your branch work and store them. The CI doesn't generate them. It only compares them.

🤯😟

[nashysolutions]

@zablaetna could you please elaborate from this starting point? I am genuinely confused and I would appreciate your help.

import SnapshotTesting
import XCTest

class MyViewControllerTests: XCTestCase {
  func testMyViewController() {
    let vc = MyViewController()

    assertSnapshot(matching: vc, as: .image)
  }
}

[zablaetna]

You shouldn't be generating the snapshots on CI. Those should be getting generated and committed. Run your UI tests and have them generate naturally. Then commit those once you do a visual inspection. After that, CI runs and does the comparisons.

Maybe I'm not following exactly what you're asking?

[nashysolutions]

Hi @zablaetna

Ok so I have taken the liberty of reading the code. There is some annotation in place that says, as follows:

Third party snapshot assert helpers can be built on top of this function. Simply invoke verifySnapshot with your own arguments

I can see that there is a directory parameter that I can pass in, which will allow me to specify where I want the generated images to be saved. Yes?

If I specify a location within the scope of my project, then the images will be indexed by my git repository and I can upload the images to CI.

At which point the CI will compare them, yes?

[zablaetna]

Correct.

[nashysolutions]

Now do you see the problem? @zablaetna

[zablaetna]

Nope.

[nashysolutions]

The problem: I had to ask the community questions and read the code itself, to learn about this API. I think the readme should have one or two sentences that highlights this particular API and maybe some simple guidance on where to stash the images.

It's the little things that matter. Developers love shortcuts that can save them brain energy right? Especially when we work 9 to 5 doing code. We love shortcuts, from the small ones like a keyboard hotkey combination or the bigger ones like a plugin for your IDE that re-formats your code.

Developers love saving brain power. And if we can save a couple of kilojoules by having a sentence or two in the readme, rather than having discussions in a lengthily thread like this, then well...hey...we have done our part for the next dev that comes along right?

[lukeredpath]

@nashysolutions I would suggest reading the README again. I think you've missed something here as I'm also not sure what issue you are having here.

Out of the box, assertSnapshot will generate images for you when you first run the test. These files will be written to a directory alongside your tests. You need to commit these to your repository - not just so CI can run the tests but also so the rest of your team can run the tests.

You didn't need to dig in to the code to find this out. It's right there in the README. You don't need to use the verifySnapshot function at all.

[nashysolutions]

"I think the readme should have one or two sentences that highlights this particular API and maybe some simple guidance on where to stash the images."

I can't explain it in terms simpler than that.

[lukeredpath]

Taken from the README

When an assertion first runs, a snapshot is automatically recorded to disk and the test will fail, printing out the file path of any newly-recorded reference.

No special configuration is needed. The reference snapshots will appear as uncommitted files when you run a git status command.

Repeat test runs will load this reference and compare it with the runtime value. If they don't match, the test will fail and describe the difference. Failures can be inspected from Xcode's Report Navigator or by inspecting the file URLs of the failure.

Here it explains how the snapshot library works. It seems obvious that you’d need to include these references in source control otherwise how else would you expect these tests to run on any other machine?

What particular API are you referring to? Is there a specific reason you want to store them somewhere other than the default location? What’s your use case here?

[nashysolutions]

It may seem obvious about what should happen.

But the README does not explain what does happen.

This is typical cognitive bias and I would invite you to be careful with words like obvious, because they are unhelpful ways of expressing yourself. And you will be at risk of offending people like myself who genuinely thought otherwise.

The README does not explain that the images are stored within the bounds of the project directory. And that running these tests will result in a change to source control. That's the reason I wanted to know if there was API's to control the location of the default directory, for generated images.

[intonarumori]

I completely agree that this is not documented properly.

2 years from when this issue is opened there's literally zero mention of CI workflow and best practices in the readme.
I'm assuming it's somehow intentional, but definitely a missing piece.