Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Azure Pipelines] Enable `safaridriver --diagnose` logs #18444

Closed
wants to merge 7 commits into from

Conversation

@foolip
Copy link
Member

foolip commented Aug 15, 2019

For #16229.

@wpt-pr-bot wpt-pr-bot added the infra label Aug 15, 2019
@wpt-pr-bot wpt-pr-bot requested a review from jgraham Aug 15, 2019
displayName: 'Run tests (Safari Technology Preview)'
- task: PublishBuildArtifacts@1
displayName: 'Publish results'

This comment has been minimized.

Copy link
@foolip

foolip Aug 15, 2019

Author Member

I'm replacing this task rather than adding a new one because it hasn't been updating any artifacts since 7dd31d2, and could have been removed then. @Hexcles FYI.

@foolip
Copy link
Member Author

foolip commented Aug 15, 2019

I confirmed locally that no_proxy='*' ./wpt run --yes --no-manifest-update --manifest MANIFEST.json --metadata infrastructure/metadata/ --log-tbpl - --log-tbpl-level info --webdriver-arg=--diagnose --channel preview safari infrastructure/ produces a lot of logs in ~/Library/Logs/com.apple.WebDriver/.

foolip added 3 commits Aug 15, 2019
@foolip
Copy link
Member Author

foolip commented Aug 15, 2019

https://dev.azure.com/web-platform-tests/wpt/_build/results?buildId=27221 showed this to work for the infrastructure/ tests. There were 10 log files produced:

safaridriver.1705.oakdiu.txt
safaridriver.1720.fmgfke.txt
safaridriver.1724.edimyy.txt
safaridriver.1729.vnyfkp.txt
safaridriver.1731.mtllhj.txt
safaridriver.1741.hquxlw.txt
safaridriver.1745.zaxjwp.txt
safaridriver.1749.dasyrk.txt
safaridriver.1753.mlkuzq.txt
safaridriver.1759.cxlygb.txt

The first was the largest of them, and I've pasted it into https://gist.github.com/foolip/5bca5e1504bc9206e0763b114d8e798f.

@smfr @youennf @burg can any of you review this and say if it's adding valuable information?

Once #18397 has been resolved I can trigger full runs and see if there's something interesting in the logs.

@foolip
Copy link
Member Author

foolip commented Aug 15, 2019

Full run of the two Safaris in progress to test this: https://dev.azure.com/web-platform-tests/wpt/_build/results?buildId=27258

@foolip
Copy link
Member Author

foolip commented Aug 15, 2019

https://dev.azure.com/web-platform-tests/wpt/_build/results?buildId=27258 finished and has the logs as artifacts. safaridriver-logs.zip is over 450 MB, so always collecting these logs might not be a great idea.

There are 155 log files, and correlating them to a failed job isn't going to be easy since they're named by process ID and that isn't included in the main logs, right @gsnedders?

@smfr
Copy link
Contributor

smfr commented Aug 15, 2019

@smfr @youennf @burg can any of you review this and say if it's adding valuable information?

At the end of the log I see:
2019/08/15 07:33:21:339 AutomationProtocol: RECV {"error":{"code":-32000,"message":"JavaScriptTimeout;","data":[{"code":-32000,"message":"JavaScriptTimeout;"}]},"id":425} 2019/08/15 07:33:21:340 HTTPServer: --> HTTP 408 2019/08/15 07:33:21:340 HTTPServer: --> {"value":{"error":"timeout","message":"","stacktrace":""}}

@burg
Copy link
Contributor

burg commented Aug 15, 2019

@foolip, these logs are pretty much required to investigate any safaridriver failures. That said, I think they are too big to always save, and hard to sort out when you have a handful of them.

The --diagnose option is designed to handle situations where safaridriver crashes immediately or in response to New Session command failing. The 'safari:diagnose" capability is more fine-grained and creates log files with the session UUID in it. This may be better for your purposes if you plan to re-run failing tests with additional diagnostics enabled.

@foolip
Copy link
Member Author

foolip commented Aug 16, 2019

@burg since this will be used for debugging problems, we could set it up so that logs are only collected when system.debug is set in Azure Pipelines, or something like that. Let's see if we can use the logs to get to the bottom of #16229 before trying to land this, since we won't want to enable it by default we'll need to start debug runs and then we can just as well start them from a branch.

@youennf I've invited you and @burg to the Azure Pipelines project in the hope that you'll be able to trigger runs yourselves. I'd also be happy to start runs if you can't make it work.

@foolip
Copy link
Member Author

foolip commented Nov 19, 2019

I'm going to close this as it hasn't been necessary for debugging. @burg if you think having this for all runs would be valuable, or a way to enable it for select runs, please let me know.

@foolip foolip closed this Nov 19, 2019
Reliable Safari results automation moved this from Maybe do to Done Nov 19, 2019
@foolip foolip deleted the foolip/azure-safari-diagnose branch Nov 19, 2019
@burg
Copy link
Contributor

burg commented Nov 19, 2019

@foolip
Copy link
Member Author

foolip commented Nov 27, 2019

@burg yeah it would be easy enough to keep this configuration in the repo and ready to be triggered. Have you been using the triggers/safari_preview branch before, or triggering runs from the Azure Pipelines console? Depending on how you do it I'd need to put the control knob in a different place.

If you'd be willing to make changes to a branch to trigger this, then I could just add a variable that defaults to false that could could set to true in one of the YAML files.

@burg
Copy link
Contributor

burg commented Nov 27, 2019

@foolip foolip restored the foolip/azure-safari-diagnose branch Jan 23, 2020
@foolip foolip deleted the foolip/azure-safari-diagnose branch Jan 23, 2020
@foolip
Copy link
Member Author

foolip commented Jan 23, 2020

I've done this in #21367 now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Linked issues

Successfully merging this pull request may close these issues.

None yet

5 participants
You can’t perform that action at this time.