Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Triage Safari differences between Azure Pipelines and Buildbot setup #646
With web-platform-tests/wpt#14836 we've begun running Safari Technology Preview on Azure Pipelines, currently every 12 hours.
The first aligned run between Azure Pipelines and Buildbot has now occured:
The latest aligned run at any time will be:
And all of the runs that could be compared are:
There are some differences in results, and the differences in webdriver/ is the largest.
We should understand the differences before committing to relying on only Azure Pipelines.
@mariestaver we'll to discuss when to do this work, vs. the cost of keeping Buildbot running. Waiting has its benefits because it gives us more data to compare, which could help filter out flakiness.
Thanks for calling that out. Past experience has demonstrated that WPT isn't nearly hygienic enough to rule out test interaction. The discrepancies between collections for this revision seem tractable, but I think any investigation will be much more efficient if it starts from datasets which control for this variation.
@foolip I can replicate that configuration in the Buildbot environment. Sound good to you?