Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement non-XR Gamepad discovery and input #31200

Merged
merged 35 commits into from Feb 17, 2024
Merged

Implement non-XR Gamepad discovery and input #31200

merged 35 commits into from Feb 17, 2024

Conversation

msub2
Copy link
Contributor

@msub2 msub2 commented Jan 27, 2024

Implement discovery and input updating for non-XR controllers for use with the Gamepad API.


  • There are tests for these changes OR
  • These changes do not require tests because ___

@msub2 msub2 changed the title Fully implement Gamepad API Implement non-XR Gamepad discovery and input Feb 4, 2024
@msub2 msub2 marked this pull request as ready for review February 4, 2024 08:43
@msub2
Copy link
Contributor Author

msub2 commented Feb 4, 2024

Okay, it's not 100% perfect yet but I think ready for a more complete review now. Some caveats of the current implementation:

  • Anything related to haptic actuators has not been implemented yet. The spec updated pretty recently on this and it'll require some WebIDL work so I'm saving it for a future PR.
  • Anything related to gamepad permissions is also TODO, I will also tackle that in a future PR.
  • Right now gamepad input seems to not get updated until a window event happens i.e. mouse moving. Mentioned this on Zulip, might just be where I'm handling events in servoshell right now.
  • A gamepad that is already plugged in upon starting servoshell will not be detected, the Embedder connection event seems to be getting dropped in constellation from not having a browsing context to reach

@msub2
Copy link
Contributor Author

msub2 commented Feb 8, 2024

I've added spec comments where applicable. Here's the full flow for reference:

Embedder

In ports/servoshell/app.rs, we call handle_gamepad_events in the handle_events function. handle_gamepad_events is a function in ports/servoshell/webview.rs that handles pumping messages from GilRs, sending controller connect/disconnect events, and updating axis/button values (after mapping to standard gamepad in the spec) via EmbedderEvents,

Constellation

Constellation receives these EmbedderEvents and handles them in handle_gamepad_msg in components/constellation/constellation.rs.

Script thread

Constellation sends the gamepad events to the script thread, where it then gets dispatched to the document via document.dispatch_gamepad_event. Here we finally handle the specific gamepad event (connect, disconnect, update) via queueing a task on the newly created gamepad task source.

Copy link
Member

@gterzian gterzian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great start, thanks! In general, instead of adding comments for steps, I prefer to try to break things down into method mapping to a link in the spec(because the step numbers usually change). Which steps are taken inside the method is usually clear enough that no comment is necessary. Comments for TODO's are useful on the other hand.

This is more like a first pass, with some suggested structural changes.

components/script/dom/gamepad.rs Outdated Show resolved Hide resolved
components/script/dom/document.rs Outdated Show resolved Hide resolved
components/script/dom/document.rs Outdated Show resolved Hide resolved
components/script/dom/document.rs Outdated Show resolved Hide resolved
@@ -437,6 +437,7 @@ impl App {
}

// Catch some keyboard events, and push the rest onto the WebViewManager event queue.
webviews.handle_gamepad_events();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it makes sense to put this below handle_window_events, and make it conditional on webview_id.is_some(). cc @wusyong @delan ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've gone ahead and made this change

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey sorry for the late reply. Yeah I think it's better to match all event variant into same method.

Btw I have some thoughts about some input events like keyboard and mouse should be more direct.
Right now it has to be passed from embedder to constellation and the to script thread.
In general, it should be fine. But for gaming usage, it usually requires more precision checks. Even a few nanoseconds lost may cause the trigger to miss the previous tick frame.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok thanks for the info! I think performance can be investigated in a follow-up. I agree there appears to be some redundant message passing.

components/script/dom/document.rs Outdated Show resolved Hide resolved
components/script/dom/document.rs Outdated Show resolved Hide resolved
components/script/dom/gamepad.rs Outdated Show resolved Hide resolved
components/script/dom/gamepad.rs Outdated Show resolved Hide resolved
components/script/dom/gamepad.rs Outdated Show resolved Hide resolved
if let Some(gamepad) = gamepad_list.IndexedGetter(index.0 as u32) {
// 1. Let now be the current high resolution time.
let current_time = time::get_time();
let now = (current_time.sec * 1000 + current_time.nsec as i64 / 1000000) as f64;
Copy link
Contributor

@Taym95 Taym95 Feb 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we are trying to remove time create, can you use std::time?

cc @mrobinson

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, done.

Copy link
Member

@gterzian gterzian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the changes, it is much easier to follow-along now.

I have first a few more "structural" comments, and after they have been addressed I will review the details of the spec algorithms.

components/constellation/constellation.rs Outdated Show resolved Hide resolved
components/script/dom/globalscope.rs Outdated Show resolved Hide resolved
components/script/dom/globalscope.rs Show resolved Hide resolved
components/script/dom/globalscope.rs Outdated Show resolved Hide resolved
components/script/dom/globalscope.rs Outdated Show resolved Hide resolved
components/shared/script/lib.rs Show resolved Hide resolved
components/shared/script/lib.rs Show resolved Hide resolved
components/script/task_source/mod.rs Show resolved Hide resolved
components/script/script_thread.rs Outdated Show resolved Hide resolved
components/script/dom/globalscope.rs Show resolved Hide resolved
Copy link
Member

@gterzian gterzian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the work, looks like very good progress. Couple of changes on the detailed algorithms.

@wusyong Could you please take a look at the embedding layer?

We also need to start looking at WPT updates, see https://github.com/servo/servo/tree/main/tests/wpt#updating-test-expectations

components/script/dom/gamepad.rs Outdated Show resolved Hide resolved
components/script/dom/globalscope.rs Show resolved Hide resolved
components/script/dom/gamepadlist.rs Show resolved Hide resolved
components/script/dom/globalscope.rs Show resolved Hide resolved
components/script/dom/globalscope.rs Outdated Show resolved Hide resolved
components/script/dom/gamepad.rs Outdated Show resolved Hide resolved
components/script/dom/gamepad.rs Outdated Show resolved Hide resolved
components/script/dom/gamepad.rs Outdated Show resolved Hide resolved
components/script/dom/gamepad.rs Outdated Show resolved Hide resolved
components/script/dom/gamepad.rs Outdated Show resolved Hide resolved
@msub2
Copy link
Contributor Author

msub2 commented Feb 14, 2024

In addition to the WPT changes there's one notable remaining issue that still needs fixing, which is that button inputs don't seem to register on their very first input, but respond without issue afterwards. Going to try and track that down later today

@gterzian
Copy link
Member

In addition to the WPT changes there's one notable remaining issue that still needs fixing, which is that button inputs don't seem to register on their very first input, but respond without issue afterwards. Going to try and track that down later today

Interesting. Is that after the connected event has been fired? I can imagine some input coming-in before that and being ignored.

@msub2
Copy link
Contributor Author

msub2 commented Feb 14, 2024

Interesting. Is that after the connected event has been fired? I can imagine some input coming-in before that and being ignored.

It is, that's what confuses me about it. Axes values seem to update immediately so I'll need to track down where it seems to be getting lost

@msub2
Copy link
Contributor Author

msub2 commented Feb 14, 2024

I've figured out the button issue! Turns out it stems from how I'm handling events from GilRs. There are ButtonPressed, ButtonReleased, and ButtonChanged events. I've been listening for ButtonChanged, but the very first time you press a button it only emits the ButtonPressed event, I assume because there's no prior state to identify a change with. Fixing this shouldn't be too difficult now, just need to add some checks on the embedder side

Copy link
Member

@gterzian gterzian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok to script all LGTM, would be good to:

  1. Update test expectations so that it can pass the merge queue.
  2. File an issue re TODO'(one issue can just give a sense of what needs to be addressed later).

The frozen array stuff can be done in a follow-up I would say(unless it makes new tests fails, but I don't think it does).

@wusyong All ok on the embedding side?

@sagudev I will be away next week, could you help to get this to merge?

@msub2
Copy link
Contributor Author

msub2 commented Feb 16, 2024

@sagudev Wu Yu Wei mentioned on Zulip already that it looks good

@sagudev
Copy link
Member

sagudev commented Feb 17, 2024

@sagudev Wu Yu Wei mentioned on Zulip already that it looks good

I see.

@sagudev sagudev added this pull request to the merge queue Feb 17, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Feb 17, 2024
@sagudev
Copy link
Member

sagudev commented Feb 17, 2024

Some more passing test on legacy-layout:

   OK /gamepad/idlharness.https.window.html
        PASS [expected FAIL] subtest: HTMLBodyElement interface: attribute ongamepadconnected
        PASS [expected FAIL] subtest: HTMLBodyElement interface: attribute ongamepaddisconnected
        PASS [expected FAIL] subtest: Window interface: attribute ongamepadconnected
        PASS [expected FAIL] subtest: Window interface: attribute ongamepaddisconnected
        PASS [expected FAIL] subtest: HTMLFrameSetElement interface: attribute ongamepadconnected
        PASS [expected FAIL] subtest: HTMLFrameSetElement interface: attribute ongamepaddisconnected

@sagudev
Copy link
Member

sagudev commented Feb 17, 2024

I think this is wrong:

prefs: ["dom.gamepad.enabled:true"]

so we should get even more PASS on legacy layout.

@sagudev sagudev added the T-linux-wpt-2013 Do a try run of the WPT (legacy layout) label Feb 17, 2024
@github-actions github-actions bot removed the T-linux-wpt-2013 Do a try run of the WPT (legacy layout) label Feb 17, 2024
Copy link

🔨 Triggering try run (#7940297235) for Linux WPT legacy-layout

Copy link

Test results for linux-wpt-layout-2013 from try job (#7940297235):

Flaky unexpected result (17)
  • TIMEOUT [expected OK] /_webgl/conformance/glsl/misc/shader-with-non-reserved-words.html (#16216)
    • NOTRUN [expected PASS] subtest: Overall test
  • TIMEOUT [expected OK] /_webgl/conformance/uniforms/out-of-bounds-uniform-array-access.html (#26225)
    • NOTRUN [expected PASS] subtest: Overall test
  • OK /css/cssom-view/MediaQueryList-addListener-removeListener.html (#24569)
    • PASS [expected FAIL] subtest: listeners are called correct number of times
  • OK /html/browsers/browsing-the-web/navigating-across-documents/empty-iframe-load-event.html (#29066)
    • PASS [expected FAIL] subtest: Check execution order on load handler
    • PASS [expected FAIL] subtest: Check execution order from nested timeout
  • OK /html/browsers/history/the-history-interface/traverse_the_history_5.html (#21383)
    • PASS [expected FAIL] subtest: Multiple history traversals, last would be aborted
  • OK /html/browsers/the-window-object/open-close/creating_browsing_context_test_01.html (#29046)
    • PASS [expected FAIL] subtest: first argument: absolute url
  • TIMEOUT [expected OK] /html/interaction/focus/the-autofocus-attribute/update-the-rendering.html (#24145)
    • TIMEOUT [expected FAIL] subtest: "Flush autofocus candidates" should be happen before a scroll event and animation frame callbacks Test timed out
  • CRASH [expected OK] /html/semantics/embedded-content/the-iframe-element/iframe_sandbox_popups_nonescaping-2.html (#22154)
  • CRASH [expected PASS] /html/semantics/embedded-content/the-img-element/image-loading-lazy-subframe-detached-crash.html
  • OK /html/semantics/links/links-created-by-a-and-area-elements/htmlanchorelement_noopener.html (#23205)
    • PASS [expected FAIL] subtest: Check that rel=noopener with target=_self does a normal load
  • OK /html/syntax/parsing/DOMContentLoaded-defer.html (#21550)
    • PASS [expected FAIL] subtest: The end: DOMContentLoaded and defer scripts
  • OK /html/webappapis/dynamic-markup-insertion/document-write/module-tla-delayed.html (#29137)
    • FAIL [expected PASS] subtest: document.write in an imported module assert_true: onload must be called expected true got false
  • TIMEOUT [expected OK] /html/webappapis/scripting/processing-model-2/integration-with-the-javascript-job-queue/promise-job-entry-different-function-realm.html (#25805)
    • TIMEOUT [expected FAIL] subtest: Fulfillment handler on pending-then-fulfilled promise Test timed out
    • TIMEOUT [expected FAIL] subtest: Rejection handler on pending-then-rejected promise Test timed out
  • TIMEOUT [expected OK] /html/webappapis/scripting/processing-model-2/integration-with-the-javascript-job-queue/promise-job-entry.html (#25805)
    • TIMEOUT [expected FAIL] subtest: Fulfillment handler on pending-then-fulfilled promise Test timed out
    • TIMEOUT [expected FAIL] subtest: Rejection handler on pending-then-rejected promise Test timed out
  • TIMEOUT /resource-timing/test_resource_timing.https.html (#25216)
    • PASS [expected FAIL] subtest: PerformanceEntry has correct name, initiatorType, startTime, and duration (img)
  • CRASH [expected PASS] /streams/readable-streams/crashtests/strategy-worker-terminate.html
  • TIMEOUT [expected OK] /webmessaging/with-ports/018.html (#24485)
    • TIMEOUT [expected PASS] subtest: origin of the script that invoked the method, javascript: Test timed out
Stable unexpected results that are known to be intermittent (16)
  • OK [expected CRASH] /_mozilla/mozilla/service-workers/service-worker-registration.https.html (#31036)
  • TIMEOUT [expected OK] /_webgl/conformance/glsl/misc/shader-uniform-packing-restrictions.html (#28103)
    • NOTRUN [expected PASS] subtest: Overall test
  • OK [expected TIMEOUT] /dom/events/webkit-transition-end-event.html (#26497)
    • FAIL [expected TIMEOUT] subtest: webkitTransitionEnd event listener should trigger for an animation assert_true: received webkitTransitionEnd event expected true got false
    • PASS [expected NOTRUN] subtest: webkitTransitionEnd event listener should not trigger if an unprefixed listener also exists
    • PASS [expected NOTRUN] subtest: webkitTransitionEnd event listener should not trigger if an unprefixed event handler also exists
    • FAIL [expected NOTRUN] subtest: event types for prefixed and unprefixed transitionend event listeners should be named appropriately assert_equals: expected (string) "webkitTransitionEnd" but got (undefined) undefined
    • PASS [expected NOTRUN] subtest: webkitTransitionEnd event listener is case sensitive
  • TIMEOUT /fetch/metadata/generated/css-images.sub.tentative.html (#29047)
    • PASS [expected FAIL] subtest: border-image sec-fetch-dest - Not sent to non-trustworthy same-origin destination
  • OK [expected TIMEOUT] /html/browsers/browsing-the-web/navigating-across-documents/about-srcdoc-navigation-blocked.window.html (#31025)
  • OK /html/browsers/browsing-the-web/navigating-across-documents/navigate-to-unparseable-url.html (#29050)
    • FAIL [expected PASS] subtest: <a> tag navigate fails for unparseable URLs promise_test: Unhandled rejection with value: object "Error: Navigation was attempted to unparseable URL"
  • OK /html/browsers/history/the-history-interface/traverse_the_history_2.html (#21383)
    • FAIL [expected PASS] subtest: Multiple history traversals, last would be aborted assert_array_equals: Pages opened during history navigation lengths differ, expected array [6, 3] length 2, got [6, 1, 1] length 3
  • CRASH [expected FAIL] /html/canvas/element/manual/text/canvas.2d.disconnected.html (#30063)
  • CRASH [expected OK] /html/semantics/embedded-content/the-iframe-element/iframe_sandbox_popups_escaping-1.html (#22647)
  • CRASH [expected TIMEOUT] /html/semantics/embedded-content/the-iframe-element/iframe_sandbox_popups_escaping-2.html (#22667)
  • OK /html/semantics/forms/form-submission-0/text-plain.window.html (#28687)
    • FAIL [expected PASS] subtest: text/plain: 0x00 in value (formdata event) assert_equals: expected "a=b\0c\r\n" but got ""
    • FAIL [expected PASS] subtest: text/plain: \n in value (normal form) assert_equals: expected "a=b\r\nc\r\n" but got ""
    • FAIL [expected PASS] subtest: text/plain: \r\n in filename (formdata event) assert_equals: expected "a=b\r\nc\r\n" but got ""
  • ERROR [expected OK] /html/semantics/scripting-1/the-script-element/defer-script/async-script.html?reload (#29054)
  • TIMEOUT /html/webappapis/scripting/events/compile-event-handler-settings-objects.html (#24246)
    • PASS [expected FAIL] subtest: The entry settings object while executing the compiled callback via Web IDL's invoke must be that of the node document
  • OK [expected TIMEOUT] /webmessaging/without-ports/017.html (#24486)
    • PASS [expected TIMEOUT] subtest: origin of the script that invoked the method, about:blank
  • ERROR [expected OK] /workers/constructors/Worker/Worker-constructor.html (#22991)
  • OK /xhr/xhr-timeout-longtask.any.worker.html (#27663)
    • PASS [expected FAIL] subtest: Long tasks should not trigger load timeout
Stable unexpected results (1)
  • OK /_webgl/conformance2/renderbuffers/framebuffer-object-attachment.html
    • FAIL [expected PASS] subtest: WebGL test Fix out-of-tree builds #2: gl.getParameter(gl.RED_BITS) + gl.getParameter(gl.GREEN_BITS) + gl.getParameter(gl.BLUE_BITS) + gl.getParameter(gl.ALPHA_BITS) >= 16 should be true. Was false. assert_true: gl.getParameter(gl.RED_BITS) + gl.getParameter(gl.GREEN_BITS) + gl.getParameter(gl.BLUE_BITS) + gl.getParameter(gl.ALPHA_BITS) >= 16 should be true. Was false. expected true got false
    • FAIL [expected PASS] subtest: WebGL test Updated a cast in rust-mozjs #6: gl.getParameter(gl.RED_BITS) + gl.getParameter(gl.GREEN_BITS) + gl.getParameter(gl.BLUE_BITS) + gl.getParameter(gl.ALPHA_BITS) >= 16 should be true. Was false. assert_true: gl.getParameter(gl.RED_BITS) + gl.getParameter(gl.GREEN_BITS) + gl.getParameter(gl.BLUE_BITS) + gl.getParameter(gl.ALPHA_BITS) >= 16 should be true. Was false. expected true got false
    • FAIL [expected PASS] subtest: WebGL test Allow cairo to be placed anywhere or build it in-tree #7: gl.getParameter(gl.DEPTH_BITS) >= 16 should be true. Was false. assert_true: gl.getParameter(gl.DEPTH_BITS) >= 16 should be true. Was false. expected true got false
    • FAIL [expected PASS] subtest: WebGL test Fixed an html-parsing crash caused by refactor #10: gl.getParameter(gl.RED_BITS) + gl.getParameter(gl.GREEN_BITS) + gl.getParameter(gl.BLUE_BITS) + gl.getParameter(gl.ALPHA_BITS) >= 16 should be true. Was false. assert_true: gl.getParameter(gl.RED_BITS) + gl.getParameter(gl.GREEN_BITS) + gl.getParameter(gl.BLUE_BITS) + gl.getParameter(gl.ALPHA_BITS) >= 16 should be true. Was false. expected true got false
    • FAIL [expected PASS] subtest: WebGL test Implemented a parser for a subset of css. #11: gl.getParameter(gl.DEPTH_BITS) >= 16 should be true. Was false. assert_true: gl.getParameter(gl.DEPTH_BITS) >= 16 should be true. Was false. expected true got false
    • FAIL [expected PASS] subtest: WebGL test Added explicit casts from ints to machine types #12: gl.getParameter(gl.STENCIL_BITS) >= 8 should be true. Was false. assert_true: gl.getParameter(gl.STENCIL_BITS) >= 8 should be true. Was false. expected true got false
    • FAIL [expected PASS] subtest: WebGL test Added css selector matching and now render specified background colors #16: gl.getParameter(gl.RED_BITS) + gl.getParameter(gl.GREEN_BITS) + gl.getParameter(gl.BLUE_BITS) + gl.getParameter(gl.ALPHA_BITS) >= 16 should be true. Was false. assert_true: gl.getParameter(gl.RED_BITS) + gl.getParameter(gl.GREEN_BITS) + gl.getParameter(gl.BLUE_BITS) + gl.getParameter(gl.ALPHA_BITS) >= 16 should be true. Was false. expected true got false
    • FAIL [expected PASS] subtest: WebGL test Fix borrowck errors and erros in layout positioning #17: gl.getParameter(gl.DEPTH_BITS) >= 16 should be true. Was false. assert_true: gl.getParameter(gl.DEPTH_BITS) >= 16 should be true. Was false. expected true got false
    • FAIL [expected PASS] subtest: WebGL test build fails on ubuntu 12.04 "I need autoconf 2.13" #21: gl.getParameter(gl.RED_BITS) + gl.getParameter(gl.GREEN_BITS) + gl.getParameter(gl.BLUE_BITS) + gl.getParameter(gl.ALPHA_BITS) >= 16 should be true. Was false. assert_true: gl.getParameter(gl.RED_BITS) + gl.getParameter(gl.GREEN_BITS) + gl.getParameter(gl.BLUE_BITS) + gl.getParameter(gl.ALPHA_BITS) >= 16 should be true. Was false. expected true got false
    • FAIL [expected PASS] subtest: WebGL test Fail to build on Mac OS X 10.7 #22: gl.getParameter(gl.DEPTH_BITS) >= 16 should be true. Was false. assert_true: gl.getParameter(gl.DEPTH_BITS) >= 16 should be true. Was false. expected true got false
    • And 9 more unexpected results...

Copy link

⚠️ Try run (#7940297235) failed.

@sagudev sagudev added this pull request to the merge queue Feb 17, 2024
Merged via the queue into servo:main with commit c999d45 Feb 17, 2024
36 of 37 checks passed
@msub2 msub2 deleted the gamepad branch February 17, 2024 19:59
@gterzian
Copy link
Member

@msub2 thanks for the contrib!

@sagudev @wusyong thanks for your help!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement non-XR gamepad discovery and input
5 participants