Hubs rooms don't load

[jdm]

Trying to load http://hubs.mozilla.com/mB8Wn7L/talkative-simplistic-sphere

Recommended setup:

• add debug=true to the query string of the URL to get useful debug info from various vendored libraries
• run with ./mach run [url] --pref dom.gamepad.enabled --pref dom.svg.enabled --pref dom.webrtc.enabled --pref dom.canvas_capture.enabled --pref dom.webrtc.transceiver.enabled --userscripts=resources/user-agent-js
• apply #26097 (comment) to resources/user-agent-js/00.example.js

To modify hubs for debugging:

• run with --unminify-js once, then use --local-script-source=unminified-js/ and modify the appropriate scripts as needed, OR
• clone https://github.com/mozilla/hubs/, run npm run dev, connect to https://localhost:8080/hub.html?hub_id=SjXspyZ (the room link on the default index page won't do what you want)
• TODO: figure out how to modify npm dependencies when running the server locally

Known issues:

• JS error getting navigator.gamepads
  • workaround: run with --pref dom.gamepad.enabled
• JS error getting window.SVGElement
  • workaround: run with --pref dom.svg.enabled
• JS error getting window.RTCPeerConnection
  • workaround: run with --pref dom.webrtc.enabled
• Big banner that says "your browser does not support webrtc data channels, which is required to use hubs"
  • workaround: user script in #26097 (comment) or https://github.com/servo/hubs/tree/checkpoint
• default self-signed cert for hubs dev server needs a keySize of 2048 (https://github.com/MozillaReality/ieeevr-hubs/blob/12a52b10a5f72b7c32b9b7ce10cde05453cd24c0/webpack.config.js#L32-L34)
  • mozilla/hubs#2343
• servo won't validate self-signed cert from dev server
  • workaround: append hubs/certs/cert.pem to servo/resources/certs
• websocket connection errors out repeatedly
  • #26622
• webaudio panic
  • #26611
• we have an incomplete createImageBitmap implementation which isn't pref-gated, which interferes with the texture loader
  • #26623
• JS error trying to call this.audioContext.createMediaStreamDestination
  • comment out all uses of media stream destination nodes in hub.js, components/camera-tools.js, systems/audio-system.js (see https://github.com/servo/hubs/tree/checkpoint)
• JS error: TypeError: conn.createDataChannel is not a function
  • workaround: user script in #26097 (comment)
• JS errors: console.group is not a function (also a use of console.groupEnd)
  • workaround: user script in #26097 (comment)
• JS error: [2020-06-10T20:41:16Z ERROR script::dom::bindings::error] Error at :0:0 DataCloneError: The object can not be cloned.
  • #26874, coming from https://github.com/InfiniteLee/three-ammo/blob/26717e12e05d2e936e8349e9dc165b95f5103e92/index.js#L100-L106 via https://github.com/servo/hubs/blob/7e9a9c0d9343704d6e37aafd0311cc501e6775e9/src/systems/physics-system.js#L94-L121
• websocket handshake error: X-WebSocket-Reject-Reason: invalid/missing Sec-WebSocket-Protocol
  • workaround: use latest revision of checkpoint branch of servo/hubs
• WebRTC creation failed: Backend("webrtcbin element not found") (thread ScriptThread PipelineId { namespace_id: PipelineNamespaceId(1), index: PipelineIndex(1) }, at /Users/jdm/.cargo/git/checkouts/media-c23a3cd5aa97076a/62cd58a/backends/gstreamer/lib.rs:314)
  • workaround: update to the latest gstreamer version
• JS error: TypeError: canvas.captureStream is not a function
  • this comes from the mediasoup-client library
  • workaround: run with --pref dom.canvas_capture.enabled
• JS output: %cnaf-dialog-adapter:error %c_joinRoom() failed:%o%c +311ms
  • https://github.com/servo/hubs/blob/7e9a9c0d9343704d6e37aafd0311cc501e6775e9/src/naf-dialog-adapter.js#L460
  • run with --pref dom.webrtc.transceiver.enabled
• JS output: SecurityError: The operation is insecure.
• JS error: TypeError: mediaDevices.enumerateDevices is undefined
• movement is mostly broken
  • #26880
• any sound from another connected client triggers a panic
  • #24826
• hubs.mozilla.org doesn't show the lobby dialog
  • #27199
• avatar selection is broken
  • #27210

[Manishearth]

JS error getting navigator.gamepads

This might be something that needs to be fixed upstream in Hubs. However, I'm curious as to how much the Hubs folks are interested in supporting devices that have no gamepads, since you can't do as much.

[jdm]

They are interested in supporting the HoloLens.

[Manishearth]

This will probably require UI changes on their side so that hololens' interaction patterns can work.

Once immersive-web/webxr#997 is resolved that will also probably trigger some change. We should wait for that to resolve and then open a dialogue with them.

[jdm]

With some judicious commenting in the Hubs source code, the following userscript, and a couple bug fixes, we can successfully render the 3d observer view!

window.MediaDevices.prototype.enumerateDevices = function() {
    return Promise.resolve([]);
};

[jdm]

Hubs patches pushed to https://github.com/servo/hubs/tree/checkpoint.

[Manishearth]

Filed servo/media#342 , servo/media#343 for the media element stuff

[jdm]

With the latest user script revision from the servo/hubs checkpoint branch, and the latest userscript from #26097 (comment), I am able to complete the UI flow to enter the room. I can't move around with the keyboard; it's not yet clear if that's because the data channels are totally fake or the correct key event listeners aren't being triggered.

[jdm]

With #26899 and #26880 fixed, movement works as expected.

[jdm]

With the latest Servo revision we can see the room contents for https://hubs.mozilla.com/mB8Wn7L/talkative-simplistic-sphere, but the lobby dialog never appears. I've rebased https://github.com/servo/hubs/tree/checkpoint against upstream Hubs and the lobby dialog appears, so it seems like servo/hubs@1e5f21c may still be commenting out some code that interrupts execution.

[jdm]

With the latest Servo revision, the only user-agent script stub that's still required is MediaDevices.enumerateDevices. Additionally, with my latest hubs-investigation servo branch, I am able to enter rooms on hubs.mozilla.org, rather than requiring a hacked-up local server!

[asajeffrey]

So I can now get servo to enter a hubs room, with the magic incantation:L

./mach run -r "https://hubs.mozilla.com/$ROOM_ID?no_force_webvr&vr_entry_type=2d_now --pref dom.gamepad.enabled --pref dom.svg.enabled --pref dom.webrtc.enabled --pref dom.canvas_capture.enabled --pref dom.webrtc.transceiver.enabled"

The URL parameters are the key, they are:

• no_force_webvr which gets hubs to use WebXR rather than WebVR
• vr_entry_type=2d_now goes straight into the room, bypassing the avatar and audio dialogs.

There is also vr_entry_type=vr_now but at the moment that will cause a security error, since there was no user intent. We could add a pref to indicate user intent, in which case we could enter hubs directly.

[asajeffrey]

#27429 adds a pref to indicate user intent, which allows entering a hubs room in webxr.

[asajeffrey]

#27456 is the changes needed to enter hubs rooms in the gstreamer plugin.