Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Safari Technology Preview is stuck at version 125 #31147

Closed
foolip opened this issue Oct 7, 2021 · 5 comments
Closed

Safari Technology Preview is stuck at version 125 #31147

foolip opened this issue Oct 7, 2021 · 5 comments
Labels
Azure Pipelines infra

Comments

@foolip
Copy link
Member

foolip commented Oct 7, 2021

The latest update of Safari Technology Preview version used in WPT was in #29133 back in May.

After that, STP began requiring macOS 11, which for a long time wasn't available on Azure Pipelines. When it finally became available, my attempt to upgrade in #30256 failed because installing certificates is no longer possible due to SIP. (WPT uses a "web-platform.test" domain and many subdomains.)

At this point there's seemingly nothing that can be done about the situation without some change to macOS or Safari itself.

I'm filing this issue to give it greater visibility/linkability, since I've gotten the question multiple times recently, and Jen asked on Twitter today.

cc @gsnedders @jensimmons
@foolip foolip mentioned this issue Oct 7, 2021
Open
@foolip
Copy link
Member Author

foolip commented Oct 7, 2021 •
edited
Loading

Since there have been frequent issues with running web-platform-tests against Safari in a CI system over the years, I'll take this chance to summarize what WPT needs and what has tended to break so far:

  • sudo safaridriver --enable has to work and make it possible for the user (not root) to start Safari via safaridriver. Different workarounds have been needed over time as this has only rarely worked without issue.
  • The Safari instance launched by safaridriver has to recognize a custom certificate root and certificate, used for the web-platform.test domain. This was possible using sudo security add-trusted-cert until macOS 11. But if there was a way to provide the certificate to safaridriver and it was only valid for automated testing, that would solve the problem too.
  • The WebDriver implementation has to be pretty solid. There hasn't been a problem recently, but infrastructure/ tests don't work with Safari Technology Preview 68 #13800 is an example from a few years ago.

The way that WPT has to use Safari is different from a web developer running tests on their own machine. It seems to me like this can only be fixed by testing Safari in public CI systems in the macOS and Safari release process, because once a problem is in a stable release, we end up blocked for a very long time.

Finally, Azure Pipelines and other public CI systems take time to support a new version of macOS, and we as the web-platform-tests project have no influence over this. However, as long as Safari supports the two latest versions of macOS, this isn't a big problem as long as upgrading isn't blocked like it is now.

@voxpelli voxpelli mentioned this issue Nov 2, 2021
Closed
@foolip
Copy link
Member Author

foolip commented Nov 11, 2021 •
edited
Loading

In web-platform-tests/wpt.fyi#2668, @stephenmcgruer made https://staging.wpt.fyi/compat2021?feature=summary&use_webkitgtk as a stop-gap to allow us to see the trends of Compat 2021 with WebKitGTK standing in for Safari.

gsnedders added a commit that referenced this issue Nov 15, 2021
@gsnedders
Try using acceptInsecureCerts with Safari using firstMatch
9db687c 
This removes the blocker to running STP on Azure Pipelines documented
in #31147, whereby trusting certificates in Big Sur or later requires
either user interaction or SIP being disabled, which is infeasible in
public automation.
@Schweinepriester
Copy link

Schweinepriester commented Nov 15, 2021 •
edited
Loading

https://webkit.org/blog/12040/release-notes-for-safari-technology-preview-135/

Added support for the acceptInsecureCerts capability (r285164)

Might that be something? Immediately thought of this issue :)

EDIT: Ah, @gsnedders is already trying it out, apologies 0:-)

@gsnedders
Copy link
Member

EDIT: Ah, @gsnedders is already trying it out, apologies 0:-)

It's almost like I was twiddling my thumbs waiting for STP 135 to go live 🙃 (and it turns out I'd forgotten about a bunch of bash/zsh differences, but hey)

But yeah, now have a full manually triggered run underway using STP 135, and I don't suspect that's going to go badly wrong, but we'll just have to wait and see.

gsnedders added a commit that referenced this issue Nov 16, 2021
@gsnedders
Try using acceptInsecureCerts with Safari using firstMatch
4a5966a 
This removes the blocker to running STP on Azure Pipelines documented
in #31147, whereby trusting certificates in Big Sur or later requires
either user interaction or SIP being disabled, which is infeasible in
public automation.
gsnedders added a commit that referenced this issue Nov 16, 2021
@gsnedders
Use acceptInsecureCerts with a new enough release of Safari
9a73151 
This removes the blocker to running STP on Azure Pipelines documented
in #31147, whereby trusting certificates in Big Sur or later requires
either user interaction or SIP being disabled, which is infeasible in
public automation.
gsnedders pushed a commit that referenced this issue Nov 16, 2021
@foolip
[Azure Pipelines] Upgrade to macOS 11 (Big Sur) (#30256)
5c2d714 
This moves to using acceptInsecureCerts with a new enough release of
Safari, which removes the blocker to running STP on Azure Pipelines
documented in #31147, whereby trusting certificates in Big Sur or
later requires either user interaction or SIP being disabled, which is
infeasible on Azure Pipelines.  See
https://developer.apple.com/documentation/macos-release-notes/macos-big-sur-11_0_1-release-notes#Security
for details about the behaviour change in Big Sur.

For now, this keeps Safari (stable) on Catalina, as it doesn't support
acceptInsecureCerts.

Additionally, this moves all usage of Python on Big Sur to 3.7, as 3.6
is not available Azure Pipelines' macOS 11 VMs.
@gsnedders
Copy link
Member

And we have STP135 on wpt.fyi. Not on the homepage because there's no aligned Edge run.

Gabisampaio pushed a commit to Gabisampaio/wpt that referenced this issue Nov 18, 2021
@foolip @Gabisampaio
[Azure Pipelines] Upgrade to macOS 11 (Big Sur) (web-platform-tests#3…
d01e8ed 
…0256)

This moves to using acceptInsecureCerts with a new enough release of
Safari, which removes the blocker to running STP on Azure Pipelines
documented in web-platform-tests#31147, whereby trusting certificates in Big Sur or
later requires either user interaction or SIP being disabled, which is
infeasible on Azure Pipelines.  See
https://developer.apple.com/documentation/macos-release-notes/macos-big-sur-11_0_1-release-notes#Security
for details about the behaviour change in Big Sur.

For now, this keeps Safari (stable) on Catalina, as it doesn't support
acceptInsecureCerts.

Additionally, this moves all usage of Python on Big Sur to 3.7, as 3.6
is not available Azure Pipelines' macOS 11 VMs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Azure Pipelines infra
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@gsnedders @foolip @Schweinepriester