Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Media Session API #24499

Merged
merged 37 commits into from Nov 20, 2019
Merged
Changes from all commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
4b5b4d1
MediaSession API DOM bindings
ferjm Oct 1, 2019
1ab6500
Set MediaMetadata MediaSession owner
ferjm Oct 2, 2019
f824680
MediaSession registration
ferjm Oct 4, 2019
7101a9d
Use BrowsingContextId for MediaSession registration
ferjm Oct 4, 2019
6233f78
MediaSessionAction message from embedder to script thread
ferjm Oct 4, 2019
fa61191
Make MediaSession handle embedder requested action
ferjm Oct 4, 2019
ec7a4bf
MediaSession: update action handler algorithm
ferjm Oct 7, 2019
31ce7a2
Handle media session action, no default handling yet
ferjm Oct 7, 2019
d7d775a
Extend libsimpleservo API with a method to request a media session ac…
ferjm Oct 7, 2019
fd040b0
Extend libsimpleservo API with a callback for media session activation
ferjm Oct 7, 2019
9c329a7
Add embedder message to (de)activate media session
ferjm Oct 7, 2019
4d147d2
Register media instance with session and prepare communication with e…
ferjm Oct 8, 2019
89d9e3a
Introduce embedder MediaSessionEvent and move active session to Servo
ferjm Oct 9, 2019
7ca74d5
MediaSession: refactor embedder API. Fix Android build
ferjm Oct 9, 2019
3f65694
MediaSession Android bits
ferjm Oct 10, 2019
761f21f
Send MediaSessionEvent::PlaybackStateChange when needed
ferjm Oct 16, 2019
dd63ba4
MediaSession show media controls on Android
ferjm Oct 18, 2019
0bbacdd
Move media session related code out of MainActivity
ferjm Oct 18, 2019
b494acb
Adapt MediaMetadata interface to new way of declaring constructors
ferjm Oct 18, 2019
08f9f17
Send MediaSessionAction from Android
ferjm Oct 18, 2019
85ec66b
Move active media session logic to constellation
ferjm Oct 28, 2019
68baabb
Format code and fix rebase errors
ferjm Nov 6, 2019
9da1dd3
Default media session actions
ferjm Nov 12, 2019
b048d7f
Fix media session action handling
ferjm Nov 12, 2019
b5b8c6c
Remove prev and next track action buttons for now
ferjm Nov 13, 2019
07483f1
Switch play and pause buttons according to playback state
ferjm Nov 13, 2019
7b5b46f
Fix play action
ferjm Nov 13, 2019
caedc28
Do not play notification sound when creating media session on Android
ferjm Nov 13, 2019
6ee21af
Update media session metadata according to media player
ferjm Nov 14, 2019
d33c96b
Fallback to current url as metadata title
ferjm Nov 15, 2019
f65c400
Update media session metadata and show content text with artist and a…
ferjm Nov 15, 2019
35c4c35
Fix metadata update
ferjm Nov 18, 2019
59f22ab
Remove stop button for now and show play/pause in compact mode
ferjm Nov 18, 2019
ba48e54
Enable MediaSession WPT
ferjm Nov 18, 2019
ca1a2e0
Update test expectations
ferjm Nov 19, 2019
6dcdfef
Fix simpleservo build
ferjm Nov 19, 2019
9f77ea1
Mach fmt, this time including C++ files
ferjm Nov 20, 2019
File filter...
Filter file types
Jump to…
Jump to file
Failed to load files.

Always

Just for now

@@ -10,7 +10,7 @@ use euclid::Scale;
use gleam::gl;
use keyboard_types::KeyboardEvent;
use msg::constellation_msg::{PipelineId, TopLevelBrowsingContextId, TraversalDirection};
use script_traits::{MouseButton, TouchEventType, TouchId, WheelDelta};
use script_traits::{MediaSessionActionType, MouseButton, TouchEventType, TouchId, WheelDelta};
use servo_geometry::DeviceIndependentPixel;
use servo_media::player::context::{GlApi, GlContext, NativeDisplay};
use servo_url::ServoUrl;
@@ -102,6 +102,9 @@ pub enum WindowEvent {
CaptureWebRender,
/// Toggle sampling profiler with the given sampling rate and max duration.
ToggleSamplingProfiler(Duration, Duration),
/// Sent when the user triggers a media action through the UA exposed media UI
/// (play, pause, seek, etc.).
MediaSessionAction(MediaSessionActionType),
}

impl Debug for WindowEvent {
@@ -132,6 +135,7 @@ impl Debug for WindowEvent {
WindowEvent::CaptureWebRender => write!(f, "CaptureWebRender"),
WindowEvent::ToggleSamplingProfiler(..) => write!(f, "ToggleSamplingProfiler"),
WindowEvent::ExitFullScreen(..) => write!(f, "ExitFullScreen"),
WindowEvent::MediaSessionAction(..) => write!(f, "MediaSessionAction"),
}
}
}
@@ -112,6 +112,7 @@ use compositing::SendableFrameTree;
use crossbeam_channel::{after, never, unbounded, Receiver, Sender};
use devtools_traits::{ChromeToDevtoolsControlMsg, DevtoolsControlMsg};
use embedder_traits::{Cursor, EmbedderMsg, EmbedderProxy, EventLoopWaker};
use embedder_traits::{MediaSessionEvent, MediaSessionPlaybackState};
use euclid::{default::Size2D as UntypedSize2D, Size2D};
use gfx::font_cache_thread::FontCacheThread;
use gfx_traits::Epoch;
@@ -139,7 +140,6 @@ use net_traits::{self, FetchResponseMsg, IpcSend, ResourceThreads};
use profile_traits::mem;
use profile_traits::time;
use script_traits::CompositorEvent::{MouseButtonEvent, MouseMoveEvent};
use script_traits::MouseEventType;
use script_traits::{webdriver_msg, LogEntry, ScriptToConstellationChan, ServiceWorkerMsg};
use script_traits::{
AnimationState, AnimationTickType, AuxiliaryBrowsingContextLoadInfo, CompositorEvent,
@@ -153,6 +153,7 @@ use script_traits::{
IFrameLoadInfo, IFrameLoadInfoWithData, IFrameSandboxState, TimerSchedulerMsg,
};
use script_traits::{LayoutMsg as FromLayoutMsg, ScriptMsg as FromScriptMsg, ScriptThreadFactory};
use script_traits::{MediaSessionActionType, MouseEventType};
use script_traits::{MessagePortMsg, PortMessageTask, StructuredSerializedData};
use script_traits::{SWManagerMsg, ScopeThings, UpdatePipelineIdReason, WebDriverCommandMsg};
use serde::{Deserialize, Serialize};
@@ -474,6 +475,9 @@ pub struct Constellation<Message, LTF, STF> {

/// Mechanism to force the compositor to process events.
event_loop_waker: Option<Box<dyn EventLoopWaker>>,

/// Pipeline ID of the active media session.
active_media_session: Option<PipelineId>,
}

/// State needed to construct a constellation.
@@ -843,6 +847,7 @@ where
glplayer_threads: state.glplayer_threads,
player_context: state.player_context,
event_loop_waker: state.event_loop_waker,
active_media_session: None,
};

constellation.run();
@@ -1541,6 +1546,9 @@ where
FromCompositorMsg::ExitFullScreen(top_level_browsing_context_id) => {
self.handle_exit_fullscreen_msg(top_level_browsing_context_id);
},
FromCompositorMsg::MediaSessionAction(action) => {
self.handle_media_session_action_msg(action);
},
}
}

@@ -1771,6 +1779,31 @@ where
new_value,
);
},
FromScriptMsg::MediaSessionEvent(pipeline_id, event) => {
// Unlikely at this point, but we may receive events coming from
// different media sessions, so we set the active media session based
// on Playing events.
// The last media session claiming to be in playing state is set to
// the active media session.
// Events coming from inactive media sessions are discarded.
if self.active_media_session.is_some() {
match event {
MediaSessionEvent::PlaybackStateChange(ref state) => {
match state {
MediaSessionPlaybackState::Playing |
MediaSessionPlaybackState::Paused => (),
_ => return,
};
},
_ => (),
};
}
self.active_media_session = Some(pipeline_id);
self.embedder_proxy.send((
Some(source_top_ctx_id),
EmbedderMsg::MediaSessionEvent(event),
));
},
}
}

@@ -5019,4 +5052,29 @@ where
.send(ToCompositorMsg::SetFrameTree(frame_tree));
}
}

fn handle_media_session_action_msg(&mut self, action: MediaSessionActionType) {
if let Some(media_session_pipeline_id) = self.active_media_session {
let result = match self.pipelines.get(&media_session_pipeline_id) {
None => {
return warn!(
"Pipeline {} got media session action request after closure.",
media_session_pipeline_id,
)
},
Some(pipeline) => {
let msg = ConstellationControlMsg::MediaSessionAction(
media_session_pipeline_id,
action,
);
pipeline.event_loop.send(msg)
},
};
if let Err(e) = result {
self.handle_send_error(media_session_pipeline_id, e);
}
} else {
error!("Got a media session action but no active media session is registered");
}
}
}
@@ -162,6 +162,9 @@ pub enum EmbedderMsg {
Shutdown,
/// Report a complete sampled profile
ReportProfile(Vec<u8>),
/// Notifies the embedder about media session events
/// (i.e. when there is metadata for the active media session, playback state changes...).
MediaSessionEvent(MediaSessionEvent),
}

impl Debug for EmbedderMsg {
@@ -194,6 +197,7 @@ impl Debug for EmbedderMsg {
EmbedderMsg::AllowOpeningBrowser(..) => write!(f, "AllowOpeningBrowser"),
EmbedderMsg::BrowserCreated(..) => write!(f, "BrowserCreated"),
EmbedderMsg::ReportProfile(..) => write!(f, "ReportProfile"),
EmbedderMsg::MediaSessionEvent(..) => write!(f, "MediaSessionEvent"),
}
}
}
@@ -202,3 +206,45 @@ impl Debug for EmbedderMsg {
/// the `String` content is expected to be extension (e.g, "doc", without the prefixing ".")
#[derive(Clone, Debug, Deserialize, Serialize)]
pub struct FilterPattern(pub String);

/// https://w3c.github.io/mediasession/#mediametadata
#[derive(Clone, Debug, Deserialize, Serialize)]
pub struct MediaMetadata {
/// Title
pub title: String,
/// Artist
pub artist: String,
/// Album
pub album: String,
}

impl MediaMetadata {
pub fn new(title: String) -> Self {
Self {
title,
artist: "".to_owned(),
album: "".to_owned(),
}
}
}

/// https://w3c.github.io/mediasession/#enumdef-mediasessionplaybackstate
#[repr(i32)]
#[derive(Clone, Debug, Deserialize, Serialize)]
pub enum MediaSessionPlaybackState {
/// The browsing context does not specify whether it’s playing or paused.
None_ = 1,
/// The browsing context is currently playing media and it can be paused.
Playing,
/// The browsing context has paused media and it can be resumed.
Paused,
}

/// Type of events sent from script to the embedder about the media session.
#[derive(Clone, Debug, Deserialize, Serialize)]
pub enum MediaSessionEvent {
/// Indicates that the media metadata is available.
SetMetadata(MediaMetadata),
/// Indicates that the playback state has changed.
PlaybackStateChange(MediaSessionPlaybackState),
}
@@ -57,7 +57,7 @@ use content_security_policy::CspList;
use crossbeam_channel::{Receiver, Sender};
use cssparser::RGBA;
use devtools_traits::{CSSError, TimelineMarkerType, WorkerId};
use embedder_traits::EventLoopWaker;
use embedder_traits::{EventLoopWaker, MediaMetadata};
use encoding_rs::{Decoder, Encoding};
use euclid::default::{Point2D, Rect, Rotation3D, Transform2D, Transform3D};
use euclid::Length as EuclidLength;
@@ -94,8 +94,8 @@ use profile_traits::time::ProfilerChan as TimeProfilerChan;
use script_layout_interface::rpc::LayoutRPC;
use script_layout_interface::OpaqueStyleAndLayoutData;
use script_traits::transferable::MessagePortImpl;
use script_traits::DrawAPaintImageResult;
use script_traits::{DocumentActivity, ScriptToConstellationChan, TimerEventId, TimerSource};
use script_traits::{DocumentActivity, DrawAPaintImageResult};
use script_traits::{MediaSessionActionType, ScriptToConstellationChan, TimerEventId, TimerSource};
use script_traits::{UntrustedNodeAddress, WindowSizeData, WindowSizeType};
use selectors::matching::ElementSelectorFlags;
use serde::{Deserialize, Serialize};
@@ -536,6 +536,8 @@ unsafe_no_jsmanaged_fields!(WindowGLContext);
unsafe_no_jsmanaged_fields!(VideoFrame);
unsafe_no_jsmanaged_fields!(WebGLContextId);
unsafe_no_jsmanaged_fields!(Arc<Mutex<dyn AudioRenderer>>);
unsafe_no_jsmanaged_fields!(MediaSessionActionType);
unsafe_no_jsmanaged_fields!(MediaMetadata);

unsafe impl<'a> JSTraceable for &'a str {
#[inline]
@@ -15,8 +15,10 @@ use crate::dom::bindings::codegen::Bindings::HTMLMediaElementBinding::HTMLMediaE
use crate::dom::bindings::codegen::Bindings::HTMLSourceElementBinding::HTMLSourceElementMethods;
use crate::dom::bindings::codegen::Bindings::MediaErrorBinding::MediaErrorConstants::*;
use crate::dom::bindings::codegen::Bindings::MediaErrorBinding::MediaErrorMethods;
use crate::dom::bindings::codegen::Bindings::NavigatorBinding::NavigatorBinding::NavigatorMethods;
use crate::dom::bindings::codegen::Bindings::NodeBinding::NodeBinding::NodeMethods;
use crate::dom::bindings::codegen::Bindings::TextTrackBinding::{TextTrackKind, TextTrackMode};
use crate::dom::bindings::codegen::Bindings::WindowBinding::WindowBinding::WindowMethods;
use crate::dom::bindings::codegen::InheritTypes::{ElementTypeId, HTMLElementTypeId};
use crate::dom::bindings::codegen::InheritTypes::{HTMLMediaElementTypeId, NodeTypeId};
use crate::dom::bindings::codegen::UnionTypes::{
@@ -65,6 +67,7 @@ use crate::script_thread::ScriptThread;
use crate::task_source::TaskSource;
use dom_struct::dom_struct;
use embedder_traits::resources::{self, Resource as EmbedderResource};
use embedder_traits::{MediaSessionEvent, MediaSessionPlaybackState};
use euclid::default::Size2D;
use headers::{ContentLength, ContentRange, HeaderMapExt};
use html5ever::{LocalName, Prefix};
@@ -592,7 +595,6 @@ impl HTMLMediaElement {
match (old_ready_state, ready_state) {
(ReadyState::HaveNothing, ReadyState::HaveMetadata) => {
task_source.queue_simple_event(self.upcast(), atom!("loadedmetadata"), &window);

// No other steps are applicable in this case.
return;
},
@@ -1725,6 +1727,17 @@ impl HTMLMediaElement {
if self.Controls() {
self.render_controls();
}

let global = self.global();
let window = global.as_window();

// Update the media session metadata title with the obtained metadata.
window.Navigator().MediaSession().update_title(
metadata
.title
.clone()
.unwrap_or(window.get_url().into_string()),
);
},
PlayerEvent::NeedData => {
// The player needs more data.
@@ -1782,13 +1795,33 @@ impl HTMLMediaElement {
};
ScriptThread::await_stable_state(Microtask::MediaElement(task));
},
PlayerEvent::StateChanged(ref state) => match *state {
PlaybackState::Paused => {
if self.ready_state.get() == ReadyState::HaveMetadata {
self.change_ready_state(ReadyState::HaveEnoughData);
}
},
_ => {},
PlayerEvent::StateChanged(ref state) => {
let mut media_session_playback_state = MediaSessionPlaybackState::None_;
match *state {
PlaybackState::Paused => {
media_session_playback_state = MediaSessionPlaybackState::Paused;
if self.ready_state.get() == ReadyState::HaveMetadata {
self.change_ready_state(ReadyState::HaveEnoughData);
}
},
PlaybackState::Playing => {
media_session_playback_state = MediaSessionPlaybackState::Playing;
},
PlaybackState::Buffering => {
// Do not send the media session playback state change event
// in this case as a None_ state is expected to clean up the
// session.
return;
},
_ => {},
};
debug!(
"Sending media session event playback state changed to {:?}",
media_session_playback_state
);
self.send_media_session_event(MediaSessionEvent::PlaybackStateChange(
media_session_playback_state,
));
},
}
}
@@ -1883,6 +1916,15 @@ impl HTMLMediaElement {
self.media_element_load_algorithm();
}
}

fn send_media_session_event(&self, event: MediaSessionEvent) {
let global = self.global();
let media_session = global.as_window().Navigator().MediaSession();

media_session.register_media_instance(&self);

media_session.send_event(event);
}
}

// XXX Placeholder for [https://github.com/servo/servo/issues/22293]
ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.