Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Triage Safari differences between Azure Pipelines and Buildbot setup #646

Open
foolip opened this issue Jan 31, 2019 · 5 comments

Comments

@foolip
Copy link
Collaborator

commented Jan 31, 2019

With web-platform-tests/wpt#14836 we've begun running Safari Technology Preview on Azure Pipelines, currently every 12 hours.

The first aligned run between Azure Pipelines and Buildbot has now occured:
https://wpt.fyi/results/?diff&run_id=5992310068215808&run_id=6012154763280384

The latest aligned run at any time will be:
https://wpt.fyi/results/?label=master&label=experimental&product=safari%5Bbuildbot%5D&product=safari%5Bazure%5D&aligned&diff

And all of the runs that could be compared are:
https://wpt.fyi/runs?label=master&label=experimental&max-count=100&product=safari%5Bbuildbot%5D&product=safari%5Bazure%5D&aligned

There are some differences in results, and the differences in webdriver/ is the largest.

We should understand the differences before committing to relying on only Azure Pipelines.

@mariestaver we'll to discuss when to do this work, vs. the cost of keeping Buildbot running. Waiting has its benefits because it gives us more data to compare, which could help filter out flakiness.

@foolip foolip changed the title Triage differences between Azure Pipelines and Buildbot setup Triage Safari differences between Azure Pipelines and Buildbot setup Jan 31, 2019

@foolip

This comment has been minimized.

Copy link
Collaborator Author

commented Jan 31, 2019

One important difference is that Azure Pipelines runs 4 shards with --chunk-type hash. That combined with --no-restart-on-unexpected will probably result in some differences that are tricky to understand.

@jugglinmike

This comment has been minimized.

Copy link
Collaborator

commented Jan 31, 2019

Thanks for calling that out. Past experience has demonstrated that WPT isn't nearly hygienic enough to rule out test interaction. The discrepancies between collections for this revision seem tractable, but I think any investigation will be much more efficient if it starts from datasets which control for this variation.

@foolip I can replicate that configuration in the Buildbot environment. Sound good to you?

@jgraham

This comment has been minimized.

Copy link
Collaborator

commented Jan 31, 2019

The webdriver differences look like we're getting the wrong version of SafariDriver.

@foolip

This comment has been minimized.

Copy link
Collaborator Author

commented Feb 1, 2019

@jugglinmike aligning the two setups makes sense, but if you'd rather change the Azure Pipelines setup we could do that. I sort expect the hash chunking to have some unwanted side effects, which would show up in this triaging.

@gsnedders

This comment has been minimized.

Copy link
Collaborator

commented Feb 2, 2019

The webdriver differences look like we're getting the wrong version of SafariDriver.

That's weird, given it's very much tied to a single version of Safari (and bundled with it). Given safaridriver starts a constant build, that's very weird.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.