Added AudioTrack, AudioTrackList, VideoTrack, and VideoTrackList inte… #22622

sreeise · 2019-01-05T06:07:24Z

…rfaces

Added AudioTrack, AudioTrackList, VideoTrack, VideoTrackList, and TrackEvent interfaces to support multiple media tracks.

./mach build -d does not report any errors
./mach test-tidy does not report any errors
These changes fix #22301 (GitHub issue number if applicable)

There are tests for these changes OR
These changes do not require tests because ___

This change is

components/script/dom/htmlmediaelement.rs

jdm

Since the code for the audio and video tracks is so similar, please consider whether each comment made about the video track list implementation could be applied to the audio track list implementation. My one concern about this PR is that we're adding a bunch of code that does not get executed, since there are no callers for AudioTrackList::add and VideoTrackList::add. You can get the information about audio and video tracks from the Metadata value that's in PlayerEvent::MetadataUpdated.

components/script/dom/videotracklist.rs

jdm · 2019-01-07T20:25:24Z

components/script/dom/videotracklist.rs

+    }
+
+    pub fn is_empty(&self) -> bool {
+        self.tracks.borrow().is_empty().clone()


The clone should not be necessary here.

jdm · 2019-01-07T20:25:24Z

components/script/dom/videotracklist.rs

+    }
+
+    pub fn set_selected(&self, idx: usize, value: bool) {
+        self.item(idx).unwrap().set_selected(value);


We should probably silently ignore this instead of unwrapping.

jdm · 2019-01-07T20:25:24Z

components/script/dom/videotracklist.rs

+                self.tracks.borrow_mut().push(Dom::from_ref(track));
+            }
+        } else {
+            self.tracks.borrow_mut().push(Dom::from_ref(track));


This looks like we can rewrite this as:

if track.selected() { if let Some(idx) = self.selected_index() { self.set_selected(idx, false); } } self.tracks.borrow_mut().push(Dom::from_ref(track));

jdm · 2019-01-07T20:25:24Z

components/script/dom/videotracklist.rs

+        } else {
+            self.tracks.borrow_mut().push(Dom::from_ref(track));
+        }
+        self.upcast::<EventTarget>().fire_event(atom!("addtrack"));


According to the steps in https://html.spec.whatwg.org/multipage/media.html#media-data-processing-steps-list that create these tracks, the events are supposed to be TrackEvent objects, not just simple Event objects.

jdm · 2019-01-07T20:25:24Z

components/script/dom/videotracklist.rs

+        self.tracks
+            .borrow()
+            .iter()
+            .filter(|track| track.id() == id)


This can use find instead of filter & next.

components/script/dom/htmlmediaelement.rs

jdm · 2019-01-07T20:26:31Z

Welp, I totally forgot to read the long useful description in the pull request that you provided, but I think we came to similar conclusions.

jdm · 2019-01-07T20:29:46Z

As for the contents of those vectors that you mentioned, the string values are the names of the codecs associated with those media tracks. We can add a TODO comment about using the codec information to choose the optimal track to enable.

sreeise · 2019-01-07T23:38:22Z

Welp, I totally forgot to read the long useful description in the pull request that you provided, but I think we came to similar conclusions.

No worries! You actually answered all of questions I had and was actually going to ask now so it works for me. Thanks for the helpful information and ill get back with a much cleaner pr soon :).

sreeise · 2019-01-14T07:39:37Z

@jdm
Hey Josh, I am trying to figure out how to update the tracks correctly but unsure on this. When looking at the metadata from here:

servo/components/script/dom/htmlmediaelement.rs

Line 1134 in 6197f1a

PlayerEvent::MetadataUpdated(ref metadata) => {

It seems that only the metadata.format would be relevant (besides the codecs you mentioned) from this part? I was trying to use GDB to figure out what exactly the metadata format can look like to determine the best route but could not debug it successfully. Is the format showing the type such as "audio" or "video" or is there something more specific here?

Also, is the correct way to retrieve the rest of the track values by getting the attribute with the local_name! macro? Something like:

element.get_string_attribute(&local_name!("kind"));

Thanks for the help!

jdm · 2019-01-16T19:20:21Z

The format contains a string that corresponds with https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-bad-libs/html/GstPlayerMediaInfo.html#gst-player-media-info-get-container-format, which doesn't give any meaningful information about what it might contain. I'm not sure that it will be useful here.

If the media tracks have a kind content attribute, then that would be the appropriate way to retrieve if it element is an HTMLTrackElement (or an Element that is actually an HTMLTrackElement under the hood).

bors-servo · 2019-01-17T18:46:13Z

☔ The latest upstream changes (presumably #22692) made this pull request unmergeable. Please resolve the merge conflicts.

components/script/dom/htmlmediaelement.rs

sreeise · 2019-01-30T16:58:51Z

Hello, I am sorry for getting back to this so late. I added a part of the media processing steps but I am currently stuck on the kind attribute. I added an inline comment discussing the video and audio formats and how they are determined for this.

Thanks for the help.

jdm · 2019-01-31T21:08:27Z

error: unused import: `crate::dom::htmlaudioelement::HTMLAudioElement`
  --> components/script/dom/htmlmediaelement.rs:32:5
   |
32 | use crate::dom::htmlaudioelement::HTMLAudioElement;
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: `-D unused-imports` implied by `-D warnings`

jdm

This is very close!

jdm · 2019-01-31T21:31:37Z

components/script/dom/audiotracklist.rs

+    }
+
+    pub fn is_empty(&self) -> bool {
+        self.tracks.borrow().is_empty().clone()


No need to clone this value.

jdm · 2019-01-31T21:31:37Z

components/script/dom/audiotracklist.rs

+    event_handler!(onaddtrack, GetOnaddtrack, SetOnaddtrack);
+
+    // https://html.spec.whatwg.org/multipage/#handler-tracklist-onremovetrack
+    event_handler!(onremovetrack, GetOnremovetrack, SetOnremovetrack);


Remove the on prefix from the first argument for event handlers.

jdm · 2019-01-31T21:31:37Z

components/script/dom/htmlmediaelement.rs

+                        }
+
+                        // Steps 7
+                        // Fire an event named addtrack at this AudioTrackList object, using TrackEvent


Can we implement this step in this PR?

jdm · 2019-01-31T21:31:37Z

components/script/dom/htmlmediaelement.rs

+                        }
+
+                        // Steps 7
+                        // Fire an event named addtrack at this VideoTrackList object, using TrackEvent


Can we implement this step in this PR?

jdm · 2019-01-31T21:31:37Z

components/script/dom/videotracklist.rs

+                self.set_selected(idx, false);
+            }
+        }
+        self.tracks.borrow_mut().push(Dom::from_ref(track));


We should push this before calling set_selected, otherwise the event handler for the change event could see an unexpected intermediate state.

jdm · 2019-01-31T21:31:37Z

components/script/dom/videotracklist.rs

+    event_handler!(onaddtrack, GetOnaddtrack, SetOnaddtrack);
+
+    // https://html.spec.whatwg.org/multipage/#handler-tracklist-onremovetrack
+    event_handler!(onremovetrack, GetOnremovetrack, SetOnremovetrack);


Remove the on prefix from the first argument to these event handlers.

components/script/dom/audiotracklist.rs

jdm · 2019-01-31T21:31:37Z

components/script/dom/videotracklist.rs

+        if let Some(track) = self.item(idx) {
+            track.set_selected(value);
+        }
+        self.upcast::<EventTarget>().fire_event(atom!("change"));


Per https://html.spec.whatwg.org/multipage/media.html#toggle-audio-track, this needs to be a queued task like

servo/components/script/dom/htmlmediaelement.rs

Lines 328 to 334 in f1dd31f

task_source

.queue(

task!(resolve_pending_play_promises: move || {

let this = this.root();

if generation_id != this.generation_id.get() {

return;

}

. Also, we should only queue the task if an actual change occurred.

sreeise · 2019-03-03T07:14:06Z

Hopefully the changes are correct. I am having an issue testing these locally. When I try run tests/wpt/web-platform-tests/html/dom/interfaces.https.html it crashes with this error:

pid:7447 VMware, Inc.
 0:03.95 pid:7447 softpipe
 0:03.95 pid:7447 3.3 (Core Profile) Mesa 18.3.0-devel
 0:03.96 pid:7447 assertion failed: `(left == right)`
 0:03.96 pid:7447   left: `1282`,
 0:03.96 pid:7447  right: `0`

which begins at .cargo/registry/src/github.com-1ecc6299db9ec823/gleam-0.6.8/src/gl.rs:93

The other thing I noticed on development builds, which happened before I changed the test expectations, is this warning:

 --> components/script/dom/bindings/conversions.rs:74:5
   |
74 |     rustc_on_unimplemented = "The IDL interface `{Self}` is not derived from `{T}`."
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: #[warn(unused_attributes)] on by default

I am not sure what the cause of these are. They could just be happening for me locally.

jdm · 2019-03-03T11:14:09Z

@bors-servo r+

bors-servo · 2019-03-03T11:14:10Z

📌 Commit 094777c has been approved by jdm

bors-servo · 2019-03-03T12:04:29Z

⌛ Testing commit 094777c with merge 543bf19...


        Auto merge of #22622 - sreeise:audio_video_tracks, r=jdm

Added AudioTrack, AudioTrackList, VideoTrack, and VideoTrackList inte… …rfaces  Added AudioTrack, AudioTrackList, VideoTrack, VideoTrackList, and TrackEvent interfaces to support multiple media tracks. ---  - [x] `./mach build -d` does not report any errors - [x] `./mach test-tidy` does not report any errors - [x] These changes fix #22301 (GitHub issue number if applicable)  - [x] There are tests for these changes OR - [ ] These changes do not require tests because ___    --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/22622)

bors-servo · 2019-03-03T12:38:47Z

💔 Test failed - linux-rel-wpt


        Added AudioTrack, AudioTrackList, VideoTrack, VideoTrackList, and Tra…

…ckEvent interfaces

sreeise · 2019-03-03T14:10:07Z

Yep I missed one, sorry!. Hopefully this one will finally pass.

jdm · 2019-03-03T15:31:41Z

@bors-servo r+

bors-servo · 2019-03-03T15:31:42Z

📌 Commit cac4aa5 has been approved by jdm

bors-servo · 2019-03-03T15:31:46Z

⌛ Testing commit cac4aa5 with merge 4ccbd1e...


        Auto merge of #22622 - sreeise:audio_video_tracks, r=jdm

Added AudioTrack, AudioTrackList, VideoTrack, and VideoTrackList inte… …rfaces  Added AudioTrack, AudioTrackList, VideoTrack, VideoTrackList, and TrackEvent interfaces to support multiple media tracks. ---  - [x] `./mach build -d` does not report any errors - [x] `./mach test-tidy` does not report any errors - [x] These changes fix #22301 (GitHub issue number if applicable)  - [x] There are tests for these changes OR - [ ] These changes do not require tests because ___    --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/22622)

bors-servo · 2019-03-03T16:44:05Z

☀️ Test successful - android-mac, arm32, arm64, linux-rel-css, linux-rel-wpt, mac-rel-css1, mac-rel-css2, mac-rel-wpt1, mac-rel-wpt2, mac-rel-wpt3, mac-rel-wpt4, magicleap, status-taskcluster
Approved by: jdm
Pushing 4ccbd1e to master...

jdm · 2019-03-03T17:23:44Z

Woohoo! Thanks for sticking with this!

jdm added the S-awaiting-review label Jan 5, 2019

jdm self-assigned this Jan 5, 2019

sreeise reviewed Jan 6, 2019

View changes

components/script/dom/htmlmediaelement.rs Show resolved Hide resolved

jdm requested changes Jan 7, 2019

View changes

jdm added S-needs-code-changes and removed S-awaiting-review labels Jan 7, 2019

ferjm added this to In progress in Media playback Jan 10, 2019

highfive added the S-needs-rebase label Jan 17, 2019

sreeise force-pushed the sreeise:audio_video_tracks branch from 6707d45 to 5507362 Jan 30, 2019

highfive added S-awaiting-review and removed S-needs-code-changes labels Jan 30, 2019

sreeise force-pushed the sreeise:audio_video_tracks branch from 5507362 to 652f949 Jan 30, 2019

sreeise reviewed Jan 30, 2019

View changes

components/script/dom/htmlmediaelement.rs Show resolved Hide resolved

sreeise force-pushed the sreeise:audio_video_tracks branch from 652f949 to b4c8b40 Jan 30, 2019

jdm mentioned this pull request Jan 31, 2019

Integrate DOM audio and video track selection with media player #22799

Closed

jdm requested changes Jan 31, 2019

View changes

jdm added S-needs-code-changes and removed S-awaiting-review S-needs-rebase labels Jan 31, 2019

sreeise force-pushed the sreeise:audio_video_tracks branch from b4c8b40 to fa722a7 Feb 4, 2019

highfive added S-awaiting-review and removed S-needs-code-changes labels Feb 4, 2019

sreeise force-pushed the sreeise:audio_video_tracks branch from 4c79042 to 094777c Mar 3, 2019

highfive added S-awaiting-review and removed S-tests-failed labels Mar 3, 2019

highfive added S-awaiting-merge and removed S-awaiting-review labels Mar 3, 2019

highfive added S-tests-failed and removed S-awaiting-merge labels Mar 3, 2019

Added AudioTrack, AudioTrackList, VideoTrack, VideoTrackList, and Tra…

Loading status checks…

cac4aa5

…ckEvent interfaces

sreeise force-pushed the sreeise:audio_video_tracks branch from 094777c to cac4aa5 Mar 3, 2019

highfive added S-awaiting-review and removed S-tests-failed labels Mar 3, 2019

highfive added S-awaiting-merge and removed S-awaiting-review labels Mar 3, 2019

Media playback automation moved this from In progress to Done Mar 3, 2019

highfive removed the S-awaiting-merge label Mar 3, 2019

sreeise deleted the sreeise:audio_video_tracks branch Mar 3, 2019

servo / servo

Added AudioTrack, AudioTrackList, VideoTrack, and VideoTrackList inte… #22622

Added AudioTrack, AudioTrackList, VideoTrack, and VideoTrackList inte… #22622

sreeise commented Jan 5, 2019 •

edited

jdm left a comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

jdm commented Jan 7, 2019

jdm commented Jan 7, 2019

sreeise commented Jan 7, 2019

sreeise commented Jan 14, 2019

jdm commented Jan 16, 2019

bors-servo commented Jan 17, 2019

sreeise commented Jan 30, 2019

jdm commented Jan 31, 2019

jdm left a comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

sreeise commented Mar 3, 2019

jdm commented Mar 3, 2019

bors-servo commented Mar 3, 2019

bors-servo commented Mar 3, 2019

bors-servo commented Mar 3, 2019

sreeise commented Mar 3, 2019

jdm commented Mar 3, 2019

bors-servo commented Mar 3, 2019

bors-servo commented Mar 3, 2019

bors-servo commented Mar 3, 2019

jdm commented Mar 3, 2019

	task_source
	.queue(
	task!(resolve_pending_play_promises: move \|\| {
	let this = this.root();
	if generation_id != this.generation_id.get() {
	return;
	}

servo / servo

Join GitHub today

Added AudioTrack, AudioTrackList, VideoTrack, and VideoTrackList inte… #22622

Added AudioTrack, AudioTrackList, VideoTrack, and VideoTrackList inte… #22622

Conversation

sreeise commented Jan 5, 2019 • edited

jdm left a comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

jdm commented Jan 7, 2019

jdm commented Jan 7, 2019

sreeise commented Jan 7, 2019

sreeise commented Jan 14, 2019

jdm commented Jan 16, 2019

bors-servo commented Jan 17, 2019

sreeise commented Jan 30, 2019

jdm commented Jan 31, 2019

jdm left a comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

sreeise commented Mar 3, 2019

jdm commented Mar 3, 2019

bors-servo commented Mar 3, 2019

bors-servo commented Mar 3, 2019

bors-servo commented Mar 3, 2019

sreeise commented Mar 3, 2019

jdm commented Mar 3, 2019

bors-servo commented Mar 3, 2019

bors-servo commented Mar 3, 2019

bors-servo commented Mar 3, 2019

jdm commented Mar 3, 2019

sreeise commented Jan 5, 2019 •

edited