Add firefox-ios SyncIntegrationTests to our smoketest suite

[data-sync-user]

The firefox-ios repository contains some basic "live" integration tests for sync-related functionality, which attempt to connect a firefox-ios and a Desktop device to the same account and confirm that a bookmark/password/whatever can successfully sync between them:

https://github.com/mozilla-mobile/firefox-ios/tree/master/SyncIntegrationTests

I think these would be a great addition to the smoke-test-firefox-ios.py script, but unfortunately I have not yet been able to get them running reliably on my machine. I'm out of time to debug this further in the current testing-related epic, but wanted to file this issue to revisit another time.

(I managed to get the tests running with a couple of tweaks captured in mozilla-mobile/firefox-ios#6389 but they don't pass for me).

[data-sync-user]

➤ Ryan Kelly commented:

To add a bit more status, I was able to get them running and attempting to sync a bookmark between the two devices, but the desktop-side test would fail saying it was unable to find the resulting bookmark. These are the logs generated by running pipenv run pytext -xs:

• logs-firefox-ios-syncintegration-tests-20200409.zip ( https://github.com/mozilla/application-services/files/4453592/logs-firefox-ios-syncintegration-tests-20200409.zip )

But I'm not sure how to debug further.

[data-sync-user]

➤ isabelrios commented:

I was able to get them running and attempting to sync a bookmark between the two devices, but the desktop-side test would fail saying it was unable to find the resulting bookmark. These are the logs generated by running pipenv run pytext -xs:

This may be due to this error mozilla-mobile/firefox-ios#6311
Once the bookmark is created on Mobile we force the sync to be sure it is synced, but due to that issue, the test finishes on Mobile side before the sync process has finished.
Could you try to run this the test that starts on Desktop and check on Mobile?
Comment all the tests except for this one to try: https://github.com/mozilla-mobile/firefox-ios/blob/master/SyncIntegrationTests/test_integration.py#L6
Hopefully that will work.. 🤞

[data-sync-user]

➤ isabelrios commented:

FYi, There is an "external" error affecting these tests: mozilla-mobile/firefox-ios#6400