Golden Testing on iOS and Android

[MelbourneDeveloper]

Is there an existing feature request for this?

• I have searched the existing issues.

Command

Golden Testing on iOS and Android

Description

As a developer, I would like to do golden testing on iOS and Android

Reasoning

These are the platforms we target

Additional context and comments

We currently run golden testing on our pipelines but we cannot do golden testing on integration tests so we cannot test goldens on actual phones.

We currently produce goldens with lots of different sizes with a Linux desktop. That's a problem because the pipeline goldens don't match our local goldens (Mac / windows). We have to do eleborate hacks just to incorporate goldens in our flow.

We've done plenty of research on this and the only other way we can so this is to take literal screenshot and then do image compares ourself.

If this library would allow us to do golden testing on integration tests, especially with Android and iOS, it would be a quantum leap forward for us and basically every flutter developer on the planet that does golden testing.

Thanks

[definitelyokay]

Hi! Can you be a little more specific?

If this library would allow us to do golden testing on integration tests, especially with Android and iOS, it would be a quantum leap forward for us and basically every flutter developer on the planet that does golden testing.

I haven't tried it in the context of a flutter integration test, but I believe you can use the matcher matchesGoldenFile during an integration test. Under the hood, that's all Alchemist is doing to generate and compare rendered widgets with golden files. Alchemist is designed to make doing golden tests (which are ultimately just widget tests) easier and more reliable. Without knowing more about your situation, it's hard to understand what you'd like to use Alchemist for.

[MelbourneDeveloper]

I haven't tried it in the context of a flutter integration test, but I believe you can use the matcher matchesGoldenFile during an integration test.

We have tried many times and we do not believe it is possible. We have tried on Windows, Linux and Android.

If it's possible, we would love to hear about how you did it.

@definitelyokay

[definitelyokay]

We have tried many times and we do not believe it is possible. We have tried on Windows, Linux and Android.

AFAIK, Alchemist wasn't intended to be run within an integration test context (@jeroen-meijer, correct me if I'm wrong), and I don't think we want to move away from using Flutter's vanilla golden test infrastructure centered around matchesGoldenFile.

Here's an article I've stumbled across before that might be helpful to your situation, although I presume you've already seen it: https://blog.codemagic.io/flutter-automated-screenshot-testing/

Essentially, they recommend using binding.takeScreenshot to snap a picture in an android integration environment. I don't have much flutter driver experience, so I apologize if this a path you've already been down.

[MelbourneDeveloper]

The problem is with matchesGoldenFile. If Alchemist uses matchesGoldenFile, it's going to have the same limitations. If there is no intention to move away from using the vanilla matchesGoldenFile function, then I will close the issue.

I just want to make a point though that not being able to run matchesGoldenFile() on an actual device like iOS or Android is a massive limitation. Goldens DO vary from platform to platform and we need to compare goldens on actual phones, not desktops. This isn't Alchemist's issue, but I was hoping Alchemist would be using a different process to capture and compare the golden.

I just cannot understand why matchesGoldenFile() only works with widget tests. But, I will try to hunt down the original repo and raise the issue there.

[MelbourneDeveloper]

Essentially, they recommend using binding.takeScreenshot to snap a picture in an android integration environment. I don't have much flutter driver experience, so I apologize if this a path you've already been down.

@definitelyokay this is probably the best we can do

[definitelyokay]

That makes sense @MelbourneDeveloper.

If there is no intention to move away from using the vanilla matchesGoldenFile function, then I will close the issue.

I can't say it won't ever happen, but I don't think it's on the radar right now.

I don't think any of us working on Alchemist have explored generating screenshots on devices just yet — if we (or you, or anyone) ends up creating an engine that can act as a back-end to replace matchesGoldenFile which can gather screenshots while running on-device, that would likely be the start of a new open source project related to integration testing. Just guessing, though.