Added AudioTrack, AudioTrackList, VideoTrack, and VideoTrackList inte… #22622

sreeise · 2019-01-05T06:07:24Z

…rfaces

Added AudioTrack, AudioTrackList, VideoTrack, VideoTrackList, and TrackEvent interfaces to support multiple media tracks.

./mach build -d does not report any errors
./mach test-tidy does not report any errors
These changes fix Support media resources with multiple media tracks #22301 (GitHub issue number if applicable)

There are tests for these changes OR
These changes do not require tests because ___

This change is

components/script/dom/htmlmediaelement.rs

jdm

Since the code for the audio and video tracks is so similar, please consider whether each comment made about the video track list implementation could be applied to the audio track list implementation. My one concern about this PR is that we're adding a bunch of code that does not get executed, since there are no callers for AudioTrackList::add and VideoTrackList::add. You can get the information about audio and video tracks from the Metadata value that's in PlayerEvent::MetadataUpdated.

components/script/dom/videotracklist.rs

jdm · 2019-01-07T20:08:53Z

components/script/dom/videotracklist.rs

+    }
+
+    pub fn is_empty(&self) -> bool {
+        self.tracks.borrow().is_empty().clone()


The clone should not be necessary here.

jdm · 2019-01-07T20:09:48Z

components/script/dom/videotracklist.rs

+    }
+
+    pub fn set_selected(&self, idx: usize, value: bool) {
+        self.item(idx).unwrap().set_selected(value);


We should probably silently ignore this instead of unwrapping.

jdm · 2019-01-07T20:12:08Z

components/script/dom/videotracklist.rs

+                self.tracks.borrow_mut().push(Dom::from_ref(track));
+            }
+        } else {
+            self.tracks.borrow_mut().push(Dom::from_ref(track));


This looks like we can rewrite this as:

if track.selected() { if let Some(idx) = self.selected_index() { self.set_selected(idx, false); } } self.tracks.borrow_mut().push(Dom::from_ref(track));

jdm · 2019-01-07T20:15:44Z

components/script/dom/videotracklist.rs

+        } else {
+            self.tracks.borrow_mut().push(Dom::from_ref(track));
+        }
+        self.upcast::<EventTarget>().fire_event(atom!("addtrack"));


According to the steps in https://html.spec.whatwg.org/multipage/media.html#media-data-processing-steps-list that create these tracks, the events are supposed to be TrackEvent objects, not just simple Event objects.

jdm · 2019-01-07T20:16:46Z

components/script/dom/videotracklist.rs

+        self.tracks
+            .borrow()
+            .iter()
+            .filter(|track| track.id() == id)


This can use find instead of filter & next.

components/script/dom/htmlmediaelement.rs

jdm · 2019-01-07T20:26:31Z

Welp, I totally forgot to read the long useful description in the pull request that you provided, but I think we came to similar conclusions.

jdm · 2019-01-07T20:29:46Z

As for the contents of those vectors that you mentioned, the string values are the names of the codecs associated with those media tracks. We can add a TODO comment about using the codec information to choose the optimal track to enable.

sreeise · 2019-01-07T23:38:22Z

Welp, I totally forgot to read the long useful description in the pull request that you provided, but I think we came to similar conclusions.

No worries! You actually answered all of questions I had and was actually going to ask now so it works for me. Thanks for the helpful information and ill get back with a much cleaner pr soon :).

sreeise · 2019-01-14T07:39:37Z

@jdm
Hey Josh, I am trying to figure out how to update the tracks correctly but unsure on this. When looking at the metadata from here:

servo/components/script/dom/htmlmediaelement.rs

Line 1134 in 6197f1a

PlayerEvent::MetadataUpdated(ref metadata) => {

It seems that only the metadata.format would be relevant (besides the codecs you mentioned) from this part? I was trying to use GDB to figure out what exactly the metadata format can look like to determine the best route but could not debug it successfully. Is the format showing the type such as "audio" or "video" or is there something more specific here?

Also, is the correct way to retrieve the rest of the track values by getting the attribute with the local_name! macro? Something like:

element.get_string_attribute(&local_name!("kind"));

Thanks for the help!

jdm · 2019-01-16T19:20:21Z

The format contains a string that corresponds with https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-bad-libs/html/GstPlayerMediaInfo.html#gst-player-media-info-get-container-format, which doesn't give any meaningful information about what it might contain. I'm not sure that it will be useful here.

If the media tracks have a kind content attribute, then that would be the appropriate way to retrieve if it element is an HTMLTrackElement (or an Element that is actually an HTMLTrackElement under the hood).

bors-servo · 2019-01-17T18:46:13Z

☔ The latest upstream changes (presumably #22692) made this pull request unmergeable. Please resolve the merge conflicts.

components/script/dom/htmlmediaelement.rs

sreeise · 2019-01-30T16:58:51Z

Hello, I am sorry for getting back to this so late. I added a part of the media processing steps but I am currently stuck on the kind attribute. I added an inline comment discussing the video and audio formats and how they are determined for this.

Thanks for the help.

jdm · 2019-01-31T21:08:27Z

error: unused import: `crate::dom::htmlaudioelement::HTMLAudioElement`
  --> components/script/dom/htmlmediaelement.rs:32:5
   |
32 | use crate::dom::htmlaudioelement::HTMLAudioElement;
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: `-D unused-imports` implied by `-D warnings`

jdm

This is very close!

jdm · 2019-01-31T21:09:13Z

components/script/dom/audiotracklist.rs

+    }
+
+    pub fn is_empty(&self) -> bool {
+        self.tracks.borrow().is_empty().clone()


No need to clone this value.

jdm · 2019-01-31T21:13:23Z

components/script/dom/audiotracklist.rs

+    event_handler!(onaddtrack, GetOnaddtrack, SetOnaddtrack);
+
+    // https://html.spec.whatwg.org/multipage/#handler-tracklist-onremovetrack
+    event_handler!(onremovetrack, GetOnremovetrack, SetOnremovetrack);


Remove the on prefix from the first argument for event handlers.

jdm · 2019-01-31T21:17:06Z

components/script/dom/htmlmediaelement.rs

+                        }
+
+                        // Steps 7
+                        // Fire an event named addtrack at this AudioTrackList object, using TrackEvent


Can we implement this step in this PR?

jdm · 2019-01-31T21:19:49Z

components/script/dom/htmlmediaelement.rs

+                        }
+
+                        // Steps 7
+                        // Fire an event named addtrack at this VideoTrackList object, using TrackEvent


Can we implement this step in this PR?

jdm · 2019-01-31T21:22:06Z

components/script/dom/videotracklist.rs

+                self.set_selected(idx, false);
+            }
+        }
+        self.tracks.borrow_mut().push(Dom::from_ref(track));


We should push this before calling set_selected, otherwise the event handler for the change event could see an unexpected intermediate state.

jdm · 2019-01-31T21:22:30Z

components/script/dom/videotracklist.rs

+    event_handler!(onaddtrack, GetOnaddtrack, SetOnaddtrack);
+
+    // https://html.spec.whatwg.org/multipage/#handler-tracklist-onremovetrack
+    event_handler!(onremovetrack, GetOnremovetrack, SetOnremovetrack);


Remove the on prefix from the first argument to these event handlers.

components/script/dom/audiotracklist.rs

jdm · 2019-01-31T21:29:34Z

components/script/dom/videotracklist.rs

+        if let Some(track) = self.item(idx) {
+            track.set_selected(value);
+        }
+        self.upcast::<EventTarget>().fire_event(atom!("change"));


Per https://html.spec.whatwg.org/multipage/media.html#toggle-audio-track, this needs to be a queued task like

servo/components/script/dom/htmlmediaelement.rs

Lines 328 to 334 in f1dd31f

task_source

.queue(

task!(resolve_pending_play_promises: move || {

let this = this.root();

if generation_id != this.generation_id.get() {

return;

}

. Also, we should only queue the task if an actual change occurred.

sreeise · 2019-03-03T07:14:06Z

Hopefully the changes are correct. I am having an issue testing these locally. When I try run tests/wpt/web-platform-tests/html/dom/interfaces.https.html it crashes with this error:

pid:7447 VMware, Inc.
 0:03.95 pid:7447 softpipe
 0:03.95 pid:7447 3.3 (Core Profile) Mesa 18.3.0-devel
 0:03.96 pid:7447 assertion failed: `(left == right)`
 0:03.96 pid:7447   left: `1282`,
 0:03.96 pid:7447  right: `0`

which begins at .cargo/registry/src/github.com-1ecc6299db9ec823/gleam-0.6.8/src/gl.rs:93

The other thing I noticed on development builds, which happened before I changed the test expectations, is this warning:

 --> components/script/dom/bindings/conversions.rs:74:5
   |
74 |     rustc_on_unimplemented = "The IDL interface `{Self}` is not derived from `{T}`."
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: #[warn(unused_attributes)] on by default

I am not sure what the cause of these are. They could just be happening for me locally.

jdm · 2019-03-03T11:14:09Z

@bors-servo r+

bors-servo · 2019-03-03T11:14:10Z

📌 Commit 094777c has been approved by jdm

bors-servo · 2019-03-03T12:04:29Z

⌛ Testing commit 094777c with merge 543bf19...

Added AudioTrack, AudioTrackList, VideoTrack, and VideoTrackList inte… …rfaces  Added AudioTrack, AudioTrackList, VideoTrack, VideoTrackList, and TrackEvent interfaces to support multiple media tracks. ---  - [x] `./mach build -d` does not report any errors - [x] `./mach test-tidy` does not report any errors - [x] These changes fix #22301 (GitHub issue number if applicable)  - [x] There are tests for these changes OR - [ ] These changes do not require tests because ___    --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/22622)

bors-servo · 2019-03-03T12:38:47Z

💔 Test failed - linux-rel-wpt

…ckEvent interfaces

sreeise · 2019-03-03T14:10:07Z

Yep I missed one, sorry!. Hopefully this one will finally pass.

jdm · 2019-03-03T15:31:41Z

@bors-servo r+

bors-servo · 2019-03-03T15:31:42Z

📌 Commit cac4aa5 has been approved by jdm

bors-servo · 2019-03-03T15:31:46Z

⌛ Testing commit cac4aa5 with merge 4ccbd1e...

Added AudioTrack, AudioTrackList, VideoTrack, and VideoTrackList inte… …rfaces  Added AudioTrack, AudioTrackList, VideoTrack, VideoTrackList, and TrackEvent interfaces to support multiple media tracks. ---  - [x] `./mach build -d` does not report any errors - [x] `./mach test-tidy` does not report any errors - [x] These changes fix #22301 (GitHub issue number if applicable)  - [x] There are tests for these changes OR - [ ] These changes do not require tests because ___    --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/22622)

bors-servo · 2019-03-03T16:44:05Z

☀️ Test successful - android-mac, arm32, arm64, linux-rel-css, linux-rel-wpt, mac-rel-css1, mac-rel-css2, mac-rel-wpt1, mac-rel-wpt2, mac-rel-wpt3, mac-rel-wpt4, magicleap, status-taskcluster
Approved by: jdm
Pushing 4ccbd1e to master...

jdm · 2019-03-03T17:23:44Z

Woohoo! Thanks for sticking with this!

jdm added the S-awaiting-review There is new code that needs to be reviewed. label Jan 5, 2019

jdm self-assigned this Jan 5, 2019

sreeise commented Jan 6, 2019

View reviewed changes

components/script/dom/htmlmediaelement.rs Show resolved Hide resolved

jdm requested changes Jan 7, 2019

View reviewed changes

jdm added S-needs-code-changes Changes have not yet been made that were requested by a reviewer. and removed S-awaiting-review There is new code that needs to be reviewed. labels Jan 7, 2019

highfive added the S-needs-rebase There are merge conflict errors. label Jan 17, 2019

sreeise force-pushed the audio_video_tracks branch from 6707d45 to 5507362 Compare January 30, 2019 16:01

highfive added S-awaiting-review There is new code that needs to be reviewed. and removed S-needs-code-changes Changes have not yet been made that were requested by a reviewer. labels Jan 30, 2019

sreeise force-pushed the audio_video_tracks branch from 5507362 to 652f949 Compare January 30, 2019 16:07

sreeise commented Jan 30, 2019

View reviewed changes

components/script/dom/htmlmediaelement.rs Show resolved Hide resolved

sreeise force-pushed the audio_video_tracks branch from 652f949 to b4c8b40 Compare January 30, 2019 17:24

jdm mentioned this pull request Jan 31, 2019

Integrate DOM audio and video track selection with media player #22799

Closed

jdm requested changes Jan 31, 2019

View reviewed changes

jdm added S-needs-code-changes Changes have not yet been made that were requested by a reviewer. and removed S-awaiting-review There is new code that needs to be reviewed. S-needs-rebase There are merge conflict errors. labels Jan 31, 2019

sreeise force-pushed the audio_video_tracks branch from b4c8b40 to fa722a7 Compare February 4, 2019 09:39

highfive added S-awaiting-review There is new code that needs to be reviewed. and removed S-needs-code-changes Changes have not yet been made that were requested by a reviewer. labels Feb 4, 2019

sreeise force-pushed the audio_video_tracks branch from 4c79042 to 094777c Compare March 3, 2019 06:58

highfive added S-awaiting-review There is new code that needs to be reviewed. and removed S-tests-failed The changes caused existing tests to fail. labels Mar 3, 2019

highfive added S-awaiting-merge The PR is in the process of compiling and running tests on the automated CI. and removed S-awaiting-review There is new code that needs to be reviewed. labels Mar 3, 2019

highfive added S-tests-failed The changes caused existing tests to fail. and removed S-awaiting-merge The PR is in the process of compiling and running tests on the automated CI. labels Mar 3, 2019

Added AudioTrack, AudioTrackList, VideoTrack, VideoTrackList, and Tra…

cac4aa5

…ckEvent interfaces

sreeise force-pushed the audio_video_tracks branch from 094777c to cac4aa5 Compare March 3, 2019 14:06

highfive added S-awaiting-review There is new code that needs to be reviewed. and removed S-tests-failed The changes caused existing tests to fail. labels Mar 3, 2019

highfive added S-awaiting-merge The PR is in the process of compiling and running tests on the automated CI. and removed S-awaiting-review There is new code that needs to be reviewed. labels Mar 3, 2019

bors-servo merged commit cac4aa5 into servo:master Mar 3, 2019

highfive removed the S-awaiting-merge The PR is in the process of compiling and running tests on the automated CI. label Mar 3, 2019

sreeise deleted the audio_video_tracks branch March 3, 2019 18:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added AudioTrack, AudioTrackList, VideoTrack, and VideoTrackList inte… #22622

Added AudioTrack, AudioTrackList, VideoTrack, and VideoTrackList inte… #22622

sreeise commented Jan 5, 2019 •

edited

Loading

jdm left a comment

jdm Jan 7, 2019

jdm Jan 7, 2019

jdm Jan 7, 2019

jdm Jan 7, 2019

jdm Jan 7, 2019

jdm commented Jan 7, 2019

jdm commented Jan 7, 2019

sreeise commented Jan 7, 2019

sreeise commented Jan 14, 2019

jdm commented Jan 16, 2019

bors-servo commented Jan 17, 2019

sreeise commented Jan 30, 2019

jdm commented Jan 31, 2019

jdm left a comment

jdm Jan 31, 2019

jdm Jan 31, 2019

jdm Jan 31, 2019

jdm Jan 31, 2019

jdm Jan 31, 2019

jdm Jan 31, 2019

jdm Jan 31, 2019

sreeise commented Mar 3, 2019

jdm commented Mar 3, 2019

bors-servo commented Mar 3, 2019

bors-servo commented Mar 3, 2019

bors-servo commented Mar 3, 2019

sreeise commented Mar 3, 2019

jdm commented Mar 3, 2019

bors-servo commented Mar 3, 2019

bors-servo commented Mar 3, 2019

bors-servo commented Mar 3, 2019

jdm commented Mar 3, 2019

	task_source
	.queue(
	task!(resolve_pending_play_promises: move \|\| {
	let this = this.root();
	if generation_id != this.generation_id.get() {
	return;
	}

Added AudioTrack, AudioTrackList, VideoTrack, and VideoTrackList inte… #22622

Added AudioTrack, AudioTrackList, VideoTrack, and VideoTrackList inte… #22622

Conversation

sreeise commented Jan 5, 2019 • edited Loading

jdm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jdm commented Jan 7, 2019

jdm commented Jan 7, 2019

sreeise commented Jan 7, 2019

sreeise commented Jan 14, 2019

jdm commented Jan 16, 2019

bors-servo commented Jan 17, 2019

sreeise commented Jan 30, 2019

jdm commented Jan 31, 2019

jdm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sreeise commented Mar 3, 2019

jdm commented Mar 3, 2019

bors-servo commented Mar 3, 2019

bors-servo commented Mar 3, 2019

bors-servo commented Mar 3, 2019

sreeise commented Mar 3, 2019

jdm commented Mar 3, 2019

bors-servo commented Mar 3, 2019

bors-servo commented Mar 3, 2019

bors-servo commented Mar 3, 2019

jdm commented Mar 3, 2019

sreeise commented Jan 5, 2019 •

edited

Loading