Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Specify WebDriver command for testing #89

Merged
merged 2 commits into from Jun 5, 2018
Merged

Conversation

paulmeyer90
Copy link
Contributor

@paulmeyer90 paulmeyer90 commented Jun 4, 2018

These changes add an "Automation" section, which defines a new WebDriver extension command, Generate Test Report. This command will allow generic reports to be generated for testing purposes, e.g. in web platform tests.


Preview | Diff

@paulmeyer90 paulmeyer90 requested a review from dcreager June 4, 2018 19:34
Copy link

@kereliuk kereliuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excited to see this PR! Looks good, but need a way to handle errors on the penultimate steps which.

8. Let |settings| be the <a>environment settings object</a> of the
<a>current browsing context</a>'s <a>active document</a>.

9. Execute [[#queue-report]] with |body|, "test", |group|, and |settings|.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if this fails/throws an error?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we're okay here. The #queue-report algorithm always succeeds — it just adds the report to a queue and immediately returns.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Though that does mean we might need another extension command that lets you trigger the #send-reports algorithm, and that one does have interesting failure modes]

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if there is some unforeseen memory error? Or it's implemented in some more complicated way in a different browser?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd argue that there's not much that you can do in that case. Report delivery is already best-effort, not guaranteed (see §1.1). A report can be dropped at any point down the line (by being evicted when a later report is queued, for instance, or during a GC run), and there's no way to signal that back to whatever originally generated the report.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahh I see what you are saying now. In the scope of just the reporting API that makes sense to me, but I think I would still prefer to have some sort of catch for an error. Maybe WebDriver is doing this by invoking a call to another system and that system changes the name of the method WebDriver usually calls. In that case we never reach the queue but we still error out. I think Unknown Error in the WebDriver spec could be used for this case.

WYT?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you give an example of a specific error condition that you'd want to write a test for?

Copy link

@kereliuk kereliuk Jun 5, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If ChromeDriver implements this it will probably use the DevTools Protocol to dispatch the queue-report. Would want to have an error catch error in case there is an error thrown in the protocol code.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After a conversation offline with Paul I've changed my opinion and I think this is good to go as is. For further reference this can be compared to step 6 in the maximize window command in the WebDriver (spec)[https://w3c.github.io/webdriver/#maximize-window] which is handled in a similar way to this.

Copy link
Member

@dcreager dcreager left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

8. Let |settings| be the <a>environment settings object</a> of the
<a>current browsing context</a>'s <a>active document</a>.

9. Execute [[#queue-report]] with |body|, "test", |group|, and |settings|.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we're okay here. The #queue-report algorithm always succeeds — it just adds the report to a queue and immediately returns.

index.src.html Outdated
<tr>
<td>POST</td>
<td>/session/{session id}/reporting</td>
<td>generate_test_report</td>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Super nit: can you wrap the prefix and name in backticks / tt to make them monospace?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants