Improve support for running on cloud devices

[Android-s14]

Something I faced when trying to configure Shot for a real project.

The example setup looks something along the lines: the project is built on Circle CI (or any other provider), connected tests are run on Firebase Test Lab. The Test Lab takes an app apk + a test apk and runs them on a real device or an emulator, as an output producing contents of the device's sdcard. The further flow then is to download the sdcard contents, find the screenshots folder, and move it to a location Shot expects it in.

Then we can compare the screenshots against the baseline with the executeScreenshotTests task. The problem is the task is hardcoded to execute the connected tests on a device + do all the downloading / cleaning up stuff. So in the environment without any access to a device, the user of the library has to execute something like ./gradlew executeScreenshotTests -x downloadScreenshots -x removeScreenshots -x connectedAndroidTest, which is obviously a hack.

While I'm not sure what the best solution would be, maybe it makes sense to extract out a screenshot comparing task, so that it can be run separately if needed?

[pedrovgs]

Hi @Android-s14 unfortunately, this project is not compatible with Firebase Test Lab. The original facebook library we use under the hood is not compatible either. As the main image comparison logic is implemented using a Gradle plugin and this is not uploaded to the Firebase services we will never be able to use this approach in Firebase test lab without implementing the core library again. We created this project as a Gradle Plugin improvement and for now, we won't change the core library because we know Facebook is working on it. Sorry for the inconveniences.

[andrey-bolduzev]

I mean, as I mentioned above, there's a way to make it work and it does work, so it's not really incompatible. It's just the structure of the gradle tasks makes the setup a bit less obvious.

[pedrovgs]

Are you sure we can pull through adb all the screenshots saved on the device running on Firebase Test Lab?

[Android-s14]

They don't provide an adb access to the devices/emulators they run tests on. But they do dump the complete sdcard contents to a bucket in the GCP storage.

So assuming the team already has some flow in place to run connected tests remotely and pull some results back from the cloud storage (e.g., test execution and code coverage data), it's only one step away to find a folder named screenshots-default among this data and copy it over to some <project>/app/screenshots/ location.

At which point, it would be highly convenient to have a gradle task that is equivalent to this ./gradlew executeScreenshotTests -x downloadScreenshots -x removeScreenshots -x connectedAndroidTest, i.e., does the comparison magic without worrying about running the tests and communicating with the device.

[pedrovgs]

Interesting 🤔 Thanks for the information, I had no idea you can get the SD card data from a GCP storage. Instead of adding the task you mention I'd make the plugin support fetching the images from the GCP storage. I've tagged this issue as an enhancement in case anyone would like to collaborate.

[Serchinastico]

I'll join if you don't mind. I see two approaches here:

1. Create a separate task in Shot to just compare the screenshots just as @Android-s14 proposed
2. Create a way to integrate Shot into Firebase directly, I don't know how many other services are there available for running emulators in remote environments.

I personally like the first one as it is more versatile and cover (and will cover) other services as well.

[pedrovgs]

@Serchinastico ok, then I think we can split the tasks as a first approach and we'll see if we will implement that later.

[Laimiux]

I've added a simple initial implementation of this in a PR above. Let me know if you have any feedback or particular use cases that we should address before it is good to go.

[pedrovgs]

Hi @Laimiux sorry for the delay, I'm going to take a look right now. Thanks for the contribution 😃

[pedrovgs]

PR merged, I'm closing this issue and releasing a version. Thank you all for your help 👏