CI is broken and it seems unlikely that Browserstack will be able to resolve this on their own.

[romainmenke]

see : https://github.com/Financial-Times/polyfill-library/actions/runs/4501232331/jobs/8176234150

{
  error: javascript error: Error: Permission denied to access property "global_test_results"
  Build info: version: '3.2.0', revision: '8c03df6', time: '2017-03-02 09:34:51 -0800'
  System info: host: 'mac-207-254-53-192.browserstack.com', ip: 'fe80:0:0:0:108f:a82:d0fa:f85e%en0', os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.15.6', java.version: '1.8.0_51'
  Driver info: org.openqa.selenium.firefox.FirefoxDriver

I know that Browserstack is rolling out breaking changes to their platform.
This is a gradual roll out, older browser versions seem to update first.
I've already reached out to them and asked them to provide more info on this.
This is a slow and frustrating process because everything goes through non-technical support. It is impossible to contact an engineer and just get a direct answer on what is going on.

The most clear info I have gotten is that they are removing a lot of heuristics around the capabilities which need to be set. In the past it was fine to just request a session with a specific browser name and browser version. Now a lot more needs to be set, all driver versions, machine type, ... Without these the service is either flaky or plain broken. I don't know why these heuristics are being removed. It is very difficult to manually configure everything correctly for the large number of browsers and browser versions we test here.

One way I worked around this in other projects was by setting a flag to make sure the w3c standard protocol wasn't used. This seemed to trigger the old heuristics to kick in again. This no longer seems to work.

It is possible that they have moved on to a next step in these changes and that the current breakage is not caused by these capabilities, but by something else.

I don't currently have time to really debug this all over again and find out what the issues are this time and I also don't have time to hop on a call with Browserstack support and show that the issues are real.

---

Is there anyone with a Browserstack account and a setup of the CI for the polyfill-library that is willing to spend time on this?

[mhassan1]

I tried replicating this from my machine against my own BrowserStack, but I wasn't able to. With command node ./test/polyfills/remotetest.js test-modified-only targeted director browser=firefox, tests seem to be passing:

 • Browser: firefox/38.0 Test config: targeted /          ✓ 1088 tests  65 seconds to complete
 • Browser: firefox/39.0 Test config: targeted /          ✓ 1082 tests  63 seconds to complete
 • Browser: firefox/40.0 Test config: targeted /          91/113
 • Browser: firefox/41.0 Test config: targeted /          ✓ 1068 tests  60 seconds to complete
 • Browser: firefox/42.0 Test config: targeted /          ✓ 947 tests  56 seconds to complete
 • Browser: firefox/43.0 Test config: targeted /          49/97
 • Browser: firefox/44.0 Test config: targeted /          75/95
 • Browser: firefox/45.0 Test config: targeted /          71/90
 • Browser: firefox/46.0 Test config: targeted /          ✓ 909 tests  51 seconds to complete
 • Browser: firefox/47.0 Test config: targeted /          51/88

Is there something else I should do to replicate?

[romainmenke]

Thank you for checking @mhassan1 🙇

This at least gives some indication that CI can still pass.
Let me try a few things and I'll get back to you!

[romainmenke]

Re-tested :

• the failing job here in CI with FT Browserstack credentials
• main branch locally with my credentials
• the feature branch locally with my credentials

All are failing on exactly Firefox 45 with the same error.

---

The Firefox 45 session, was it running on Windows XP or a different OS, OS version?

[mhassan1]

Windows XP. Here is a video. It times out near the end, but I think that's because I interrupted the test. Also, here is a video of 45.0 on Sierra.

Are you able to see the job run in the Automate tab?

[romainmenke]

Interesting, that is not what I am seeing on my end :
https://automate.browserstack.com/builds/0979012ebb8baebed2654098ca55093912a27f7a/sessions/e45ac236a7cff15c0c115da569fc89e7943404e3?auth_token=40174601c50fea922f017b49bee24dbdc8374f926eff8243adeefa31052fafad

The browser never even starts as you can see in the video.

[mhassan1]

Hm. Maybe there is some difference between our plans; here is mine:

[romainmenke]

Personal plan :

I don't know the plan details of FT

[JakeChampion]

@notlee and/or @wheresrhys - could you let us know the FT BrowserStack plans please?

[wheresrhys]

You mean this?

It says our percy is stopped (some allowance limits have been exceeded recently, so maybe we are being penalised) - could that be the issue?

[romainmenke]

Maybe something specific about sponsored accounts?

[romainmenke]

I've been reminding Browserstack of these issues regularly and got some good news today :

We have made some changes at our end. Please try running a few sessions and let us know if the issue persists. If found any issue, please share the sessionID here please.

I've rerun some tests here and CI seems fine again 🎉
Thank you everyone for checking things and for providing more info.