Skip to content
Permalink
Browse files
[GStreamer] Track handling fixes 
https://bugs.webkit.org/show_bug.cgi?id=239702

Patch by Philippe Normand <pnormand@igalia.com> on 2022-04-27
Reviewed by Xabier Rodriguez-Calvar.

When the player is using playbin3 the audio/video/text tracks are associated to the
corresponding GstStream, itself part of a single GstStreamCollection. There is no need to
use a GRefPtr for GstStream in this case because those streams are not meant to be modified
and they remain valid as long as the parent collection is alive. So the player now keeps
track of the current stream collection and the private tracks handle GstStream pointers.

The stream collection handling was refactored, removing redundant logging, making use of
ScopeExit and removing the special case for text tracks creation.

This patch also changes the internal storage of tracks from HashMap<AtomString,RePtr<T>> to
HashMap<AtomString,Ref<T>>, bringing us a bit closer to the AVF implementation.

* platform/graphics/gstreamer/AudioTrackPrivateGStreamer.cpp:
(WebCore::AudioTrackPrivateGStreamer::AudioTrackPrivateGStreamer):
(WebCore::AudioTrackPrivateGStreamer::updateConfigurationFromTags):
(WebCore::AudioTrackPrivateGStreamer::updateConfigurationFromCaps):
(WebCore::AudioTrackPrivateGStreamer::kind const):
(WebCore::AudioTrackPrivateGStreamer::disconnect):
* platform/graphics/gstreamer/AudioTrackPrivateGStreamer.h:
* platform/graphics/gstreamer/InbandTextTrackPrivateGStreamer.cpp:
(WebCore::InbandTextTrackPrivateGStreamer::InbandTextTrackPrivateGStreamer):
* platform/graphics/gstreamer/InbandTextTrackPrivateGStreamer.h:
(WebCore::InbandTextTrackPrivateGStreamer::create):
* platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.cpp:
(WebCore::MediaPlayerPrivateGStreamer::notifyPlayerOfTrack):
(WebCore::MediaPlayerPrivateGStreamer::updateEnabledVideoTrack):
(WebCore::MediaPlayerPrivateGStreamer::updateEnabledAudioTrack):
(WebCore::MediaPlayerPrivateGStreamer::playbin3SendSelectStreamsIfAppropriate):
(WebCore::MediaPlayerPrivateGStreamer::updateTracks):
(WebCore::MediaPlayerPrivateGStreamer::handleStreamCollectionMessage):
(WebCore::MediaPlayerPrivateGStreamer::handleMessage):
* platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.h:
* platform/graphics/gstreamer/TrackPrivateBaseGStreamer.cpp:
(WebCore::TrackPrivateBaseGStreamer::TrackPrivateBaseGStreamer):
(WebCore::TrackPrivateBaseGStreamer::disconnect):
(WebCore::TrackPrivateBaseGStreamer::tagsChanged):
* platform/graphics/gstreamer/TrackPrivateBaseGStreamer.h:
(WebCore::TrackPrivateBaseGStreamer::stream const):
(WebCore::TrackPrivateBaseGStreamer::stream): Deleted.
* platform/graphics/gstreamer/VideoTrackPrivateGStreamer.cpp:
(WebCore::VideoTrackPrivateGStreamer::VideoTrackPrivateGStreamer):
(WebCore::VideoTrackPrivateGStreamer::updateConfigurationFromTags):
(WebCore::VideoTrackPrivateGStreamer::updateConfigurationFromCaps):
(WebCore::VideoTrackPrivateGStreamer::kind const):
(WebCore::VideoTrackPrivateGStreamer::disconnect):
* platform/graphics/gstreamer/VideoTrackPrivateGStreamer.h:

Canonical link: https://commits.webkit.org/250060@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@293531 268f45cc-cd09-0410-ab3c-d52691b4dbfc
  • Loading branch information
@philn @webkit-commit-queue
philn authored and webkit-commit-queue committed Apr 27, 2022
1 parent 26e0840 commit 62a72631a0fe144c4eb12b3936b69e7bacaab8dc
Show file tree
Hide file tree
Showing 11 changed files with 170 additions and 113 deletions.
54 Source/WebCore/ChangeLog
View file
@@ -1,3 +1,57 @@
2022-04-24 Philippe Normand <philn@igalia.com>

[GStreamer] Track handling fixes
https://bugs.webkit.org/show_bug.cgi?id=239702

Reviewed by Xabier Rodriguez-Calvar.

When the player is using playbin3 the audio/video/text tracks are associated to the
corresponding GstStream, itself part of a single GstStreamCollection. There is no need to
use a GRefPtr for GstStream in this case because those streams are not meant to be modified
and they remain valid as long as the parent collection is alive. So the player now keeps
track of the current stream collection and the private tracks handle GstStream pointers.

The stream collection handling was refactored, removing redundant logging, making use of
ScopeExit and removing the special case for text tracks creation.

This patch also changes the internal storage of tracks from HashMap<AtomString,RePtr<T>> to
HashMap<AtomString,Ref<T>>, bringing us a bit closer to the AVF implementation.

* platform/graphics/gstreamer/AudioTrackPrivateGStreamer.cpp:
(WebCore::AudioTrackPrivateGStreamer::AudioTrackPrivateGStreamer):
(WebCore::AudioTrackPrivateGStreamer::updateConfigurationFromTags):
(WebCore::AudioTrackPrivateGStreamer::updateConfigurationFromCaps):
(WebCore::AudioTrackPrivateGStreamer::kind const):
(WebCore::AudioTrackPrivateGStreamer::disconnect):
* platform/graphics/gstreamer/AudioTrackPrivateGStreamer.h:
* platform/graphics/gstreamer/InbandTextTrackPrivateGStreamer.cpp:
(WebCore::InbandTextTrackPrivateGStreamer::InbandTextTrackPrivateGStreamer):
* platform/graphics/gstreamer/InbandTextTrackPrivateGStreamer.h:
(WebCore::InbandTextTrackPrivateGStreamer::create):
* platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.cpp:
(WebCore::MediaPlayerPrivateGStreamer::notifyPlayerOfTrack):
(WebCore::MediaPlayerPrivateGStreamer::updateEnabledVideoTrack):
(WebCore::MediaPlayerPrivateGStreamer::updateEnabledAudioTrack):
(WebCore::MediaPlayerPrivateGStreamer::playbin3SendSelectStreamsIfAppropriate):
(WebCore::MediaPlayerPrivateGStreamer::updateTracks):
(WebCore::MediaPlayerPrivateGStreamer::handleStreamCollectionMessage):
(WebCore::MediaPlayerPrivateGStreamer::handleMessage):
* platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.h:
* platform/graphics/gstreamer/TrackPrivateBaseGStreamer.cpp:
(WebCore::TrackPrivateBaseGStreamer::TrackPrivateBaseGStreamer):
(WebCore::TrackPrivateBaseGStreamer::disconnect):
(WebCore::TrackPrivateBaseGStreamer::tagsChanged):
* platform/graphics/gstreamer/TrackPrivateBaseGStreamer.h:
(WebCore::TrackPrivateBaseGStreamer::stream const):
(WebCore::TrackPrivateBaseGStreamer::stream): Deleted.
* platform/graphics/gstreamer/VideoTrackPrivateGStreamer.cpp:
(WebCore::VideoTrackPrivateGStreamer::VideoTrackPrivateGStreamer):
(WebCore::VideoTrackPrivateGStreamer::updateConfigurationFromTags):
(WebCore::VideoTrackPrivateGStreamer::updateConfigurationFromCaps):
(WebCore::VideoTrackPrivateGStreamer::kind const):
(WebCore::VideoTrackPrivateGStreamer::disconnect):
* platform/graphics/gstreamer/VideoTrackPrivateGStreamer.h:

2022-04-27 Eric Carlson <eric.carlson@apple.com>

[iOS] unable to start playing audio when device is locked
22 Source/WebCore/platform/graphics/gstreamer/AudioTrackPrivateGStreamer.cpp
View file
@@ -40,22 +40,22 @@ AudioTrackPrivateGStreamer::AudioTrackPrivateGStreamer(WeakPtr<MediaPlayerPrivat
{
}

AudioTrackPrivateGStreamer::AudioTrackPrivateGStreamer(WeakPtr<MediaPlayerPrivateGStreamer> player, unsigned index, GRefPtr<GstStream>&& stream)
: TrackPrivateBaseGStreamer(TrackPrivateBaseGStreamer::TrackType::Audio, this, index, WTFMove(stream))
AudioTrackPrivateGStreamer::AudioTrackPrivateGStreamer(WeakPtr<MediaPlayerPrivateGStreamer> player, unsigned index, GstStream* stream)
: TrackPrivateBaseGStreamer(TrackPrivateBaseGStreamer::TrackType::Audio, this, index, stream)
, m_player(player)
{
int kind;
auto tags = adoptGRef(gst_stream_get_tags(m_stream.get()));
auto tags = adoptGRef(gst_stream_get_tags(m_stream));

if (tags && gst_tag_list_get_int(tags.get(), "webkit-media-stream-kind", &kind) && kind == static_cast<int>(AudioTrackPrivate::Kind::Main)) {
auto streamFlags = gst_stream_get_stream_flags(m_stream.get());
gst_stream_set_stream_flags(m_stream.get(), static_cast<GstStreamFlags>(streamFlags | GST_STREAM_FLAG_SELECT));
auto streamFlags = gst_stream_get_stream_flags(m_stream);
gst_stream_set_stream_flags(m_stream, static_cast<GstStreamFlags>(streamFlags | GST_STREAM_FLAG_SELECT));
}

g_signal_connect_swapped(m_stream.get(), "notify::caps", G_CALLBACK(+[](AudioTrackPrivateGStreamer* track) {
g_signal_connect_swapped(m_stream, "notify::caps", G_CALLBACK(+[](AudioTrackPrivateGStreamer* track) {
track->updateConfigurationFromCaps();
}), this);
g_signal_connect_swapped(m_stream.get(), "notify::tags", G_CALLBACK(+[](AudioTrackPrivateGStreamer* track) {
g_signal_connect_swapped(m_stream, "notify::tags", G_CALLBACK(+[](AudioTrackPrivateGStreamer* track) {
track->updateConfigurationFromTags();
}), this);

@@ -65,7 +65,7 @@ AudioTrackPrivateGStreamer::AudioTrackPrivateGStreamer(WeakPtr<MediaPlayerPrivat

void AudioTrackPrivateGStreamer::updateConfigurationFromTags()
{
auto tags = adoptGRef(gst_stream_get_tags(m_stream.get()));
auto tags = adoptGRef(gst_stream_get_tags(m_stream));
unsigned bitrate;
if (!tags || !gst_tag_list_get_uint(tags.get(), GST_TAG_BITRATE, &bitrate))
return;
@@ -79,7 +79,7 @@ void AudioTrackPrivateGStreamer::updateConfigurationFromTags()

void AudioTrackPrivateGStreamer::updateConfigurationFromCaps()
{
auto caps = adoptGRef(gst_stream_get_caps(m_stream.get()));
auto caps = adoptGRef(gst_stream_get_caps(m_stream));
if (!caps || !gst_caps_is_fixed(caps.get()))
return;

@@ -102,7 +102,7 @@ void AudioTrackPrivateGStreamer::updateConfigurationFromCaps()

AudioTrackPrivate::Kind AudioTrackPrivateGStreamer::kind() const
{
if (m_stream.get() && gst_stream_get_stream_flags(m_stream.get()) & GST_STREAM_FLAG_SELECT)
if (m_stream && gst_stream_get_stream_flags(m_stream) & GST_STREAM_FLAG_SELECT)
return AudioTrackPrivate::Kind::Main;

return AudioTrackPrivate::kind();
@@ -111,7 +111,7 @@ AudioTrackPrivate::Kind AudioTrackPrivateGStreamer::kind() const
void AudioTrackPrivateGStreamer::disconnect()
{
if (m_stream)
g_signal_handlers_disconnect_matched(m_stream.get(), G_SIGNAL_MATCH_DATA, 0, 0, nullptr, nullptr, this);
g_signal_handlers_disconnect_matched(m_stream, G_SIGNAL_MATCH_DATA, 0, 0, nullptr, nullptr, this);

m_player = nullptr;
TrackPrivateBaseGStreamer::disconnect();
6 Source/WebCore/platform/graphics/gstreamer/AudioTrackPrivateGStreamer.h
View file
@@ -42,9 +42,9 @@ class AudioTrackPrivateGStreamer final : public AudioTrackPrivate, public TrackP
return adoptRef(*new AudioTrackPrivateGStreamer(player, index, WTFMove(pad), shouldHandleStreamStartEvent));
}

static Ref<AudioTrackPrivateGStreamer> create(WeakPtr<MediaPlayerPrivateGStreamer> player, unsigned index, GRefPtr<GstStream>&& stream)
static Ref<AudioTrackPrivateGStreamer> create(WeakPtr<MediaPlayerPrivateGStreamer> player, unsigned index, GstStream* stream)
{
return adoptRef(*new AudioTrackPrivateGStreamer(player, index, WTFMove(stream)));
return adoptRef(*new AudioTrackPrivateGStreamer(player, index, stream));
}

Kind kind() const final;
@@ -66,7 +66,7 @@ class AudioTrackPrivateGStreamer final : public AudioTrackPrivate, public TrackP

private:
AudioTrackPrivateGStreamer(WeakPtr<MediaPlayerPrivateGStreamer>, unsigned index, GRefPtr<GstPad>&&, bool shouldHandleStreamStartEvent);
AudioTrackPrivateGStreamer(WeakPtr<MediaPlayerPrivateGStreamer>, unsigned index, GRefPtr<GstStream>&&);
AudioTrackPrivateGStreamer(WeakPtr<MediaPlayerPrivateGStreamer>, unsigned index, GstStream*);

WeakPtr<MediaPlayerPrivateGStreamer> m_player;
};
10 Source/WebCore/platform/graphics/gstreamer/InbandTextTrackPrivateGStreamer.cpp
View file
@@ -43,15 +43,15 @@ InbandTextTrackPrivateGStreamer::InbandTextTrackPrivateGStreamer(unsigned index,
{
}

InbandTextTrackPrivateGStreamer::InbandTextTrackPrivateGStreamer(unsigned index, GRefPtr<GstStream>&& stream)
InbandTextTrackPrivateGStreamer::InbandTextTrackPrivateGStreamer(unsigned index, GstStream* stream)
: InbandTextTrackPrivate(CueFormat::WebVTT)
, TrackPrivateBaseGStreamer(TrackPrivateBaseGStreamer::TrackType::Text, this, index, WTFMove(stream))
, TrackPrivateBaseGStreamer(TrackPrivateBaseGStreamer::TrackType::Text, this, index, stream)
{
m_id = AtomString::fromLatin1(gst_stream_get_stream_id(m_stream.get()));
m_id = AtomString::fromLatin1(gst_stream_get_stream_id(m_stream));
GST_INFO("Track %d got stream start for stream %s.", m_index, m_id.string().utf8().data());

GST_DEBUG("Stream %" GST_PTR_FORMAT, m_stream.get());
auto caps = adoptGRef(gst_stream_get_caps(m_stream.get()));
GST_DEBUG("Stream %" GST_PTR_FORMAT, m_stream);
auto caps = adoptGRef(gst_stream_get_caps(m_stream));
const char* mediaType = capsMediaType(caps.get());
m_kind = g_str_has_prefix(mediaType, "closedcaption/") ? Kind::Captions : Kind::Subtitles;
}
6 Source/WebCore/platform/graphics/gstreamer/InbandTextTrackPrivateGStreamer.h
View file
@@ -47,9 +47,9 @@ class InbandTextTrackPrivateGStreamer : public InbandTextTrackPrivate, public Tr
return create(index, WTFMove(pad));
}

static Ref<InbandTextTrackPrivateGStreamer> create(unsigned index, GRefPtr<GstStream>&& stream)
static Ref<InbandTextTrackPrivateGStreamer> create(WeakPtr<MediaPlayerPrivateGStreamer>, unsigned index, GstStream* stream)
{
return adoptRef(*new InbandTextTrackPrivateGStreamer(index, WTFMove(stream)));
return adoptRef(*new InbandTextTrackPrivateGStreamer(index, stream));
}

Kind kind() const final { return m_kind; }
@@ -62,7 +62,7 @@ class InbandTextTrackPrivateGStreamer : public InbandTextTrackPrivate, public Tr

private:
InbandTextTrackPrivateGStreamer(unsigned index, GRefPtr<GstPad>&&, bool shouldHandleStreamStartEvent);
InbandTextTrackPrivateGStreamer(unsigned index, GRefPtr<GstStream>&&);
InbandTextTrackPrivateGStreamer(unsigned index, GstStream*);

void notifyTrackOfSample();

0 comments on commit 62a7263

Please sign in to comment.