From 296d132241e6d3edbbcabdb591abea5ac2004ae4 Mon Sep 17 00:00:00 2001 From: Philippe Normand Date: Wed, 25 May 2022 08:00:26 +0000 Subject: [PATCH] [MSE][GStreamer] Support Google Dynamic Ad Insertion (DAI) https://bugs.webkit.org/show_bug.cgi?id=216039 Patch by Philippe Normand on 2022-05-25 Reviewed by Alicia Boya Garcia. The "simple DAI" demo was not working as expected because the new source pad of the demuxer in the append pipeline was sinking buffers to a "black hole". That was happening because the second time the no-more-pads signal was emitted, the previous source pad was still linked to a track, and there was no code to unlink/relink demuxer pads. This is now handled... The specific "simple DAI" demo generates 2 AAC streams, with different tkhd track-id fields, we had no layout test reproducing this use-case, so this patch provides one. Test: media-source/media-source-audio-track-id-switch.html * LayoutTests/media/media-source/content/test-48kHz-new-track-id.m4a: Added. * LayoutTests/media/media-source/content/test-48khz-new-track-id-manifest.json: Added. * LayoutTests/media/media-source/media-source-audio-track-id-switch-expected.txt: Added. * LayoutTests/media/media-source/media-source-audio-track-id-switch.html: Added. * Source/WebCore/platform/graphics/gstreamer/mse/AppendPipeline.cpp: (WebCore::AppendPipeline::handleErrorSyncMessage): (WebCore::AppendPipeline::didReceiveInitializationSegment): (WebCore::createOptionalParserForFormat): (WebCore::AppendPipeline::recycleTrackForPad): (WebCore::AppendPipeline::linkPadWithTrack): (WebCore::AppendPipeline::tryMatchPadToExistingTrack): Deleted. * Source/WebCore/platform/graphics/gstreamer/mse/AppendPipeline.h: (WebCore::AppendPipeline::Track::isLinked const): Canonical link: https://commits.webkit.org/250950@main git-svn-id: https://svn.webkit.org/repository/webkit/trunk@294791 268f45cc-cd09-0410-ab3c-d52691b4dbfc --- .../content/test-48kHz-new-track-id.m4a | Bin 0 -> 18361 bytes .../test-48khz-new-track-id-manifest.json | 19 ++++ ...-source-audio-track-id-switch-expected.txt | 12 +++ .../media-source-audio-track-id-switch.html | 55 ++++++++++ .../graphics/gstreamer/mse/AppendPipeline.cpp | 97 ++++++++++++++---- .../graphics/gstreamer/mse/AppendPipeline.h | 4 +- 6 files changed, 165 insertions(+), 22 deletions(-) create mode 100644 LayoutTests/media/media-source/content/test-48kHz-new-track-id.m4a create mode 100644 LayoutTests/media/media-source/content/test-48khz-new-track-id-manifest.json create mode 100644 LayoutTests/media/media-source/media-source-audio-track-id-switch-expected.txt create mode 100644 LayoutTests/media/media-source/media-source-audio-track-id-switch.html diff --git a/LayoutTests/media/media-source/content/test-48kHz-new-track-id.m4a b/LayoutTests/media/media-source/content/test-48kHz-new-track-id.m4a new file mode 100644 index 0000000000000000000000000000000000000000..eb6074397990fc1b0f2457f6b3c3a0510949cc0e GIT binary patch literal 18361 zcmeHPby!r}-X6L^8WDkkp%m#-K|s1hT3Wgl>5>qTuAxCdk!}$sB?KJ0LkW?VlJ1yq z55hTm?>YDS+~==k8gjL2`BJaBl(L#qNb8`DY~JTSI4Iqf@bWAVTgIDWz2=BJB8jK8;2 z8QYm4jw2G{v2h*ktZC26+4c^G)<9X?-ogHSz!81qfQdmoFd`x9-xD(E0;2H!kwHu* z_Qv*zI?6?0he{|y3Zek!@X_fH4Vf?!9SJdhB*3NJ5Z!)A7l1aSpX&~%mpW?$jvGDv z;kbZ}lZgvZzhGl>{)(q_LhE36+K%Xoc+5W^7107h>H)r|)1k33JwFf82dM&>^V4<* zMEmzyMx0CHU~25}^F@$C9qbVEfP!dd2hRNnyes13=x2HLS)O~A|KsuVe!ugd$L&8D z4-|PDV*>{;EsBUa6TSZ&_`S<7ddd7R+NybB2<9o z9<=bXmSqA;y{{ZE%b7ZDa zDDRhRn~X9m1#>6`vnu(sL!H!fVYd@lo`kX(hB8mQKT+FPgw@_$`#4LAd$03VJ5{kPZRin&`fL$OkSLKTdG33D%8U?# zg(?A6pz_&uJ~fy*ico^Imy=y^b3k=xYZolkZB=3Kdd^!Y$=L#)F+<%<}O=W-T|1*%GHMBKFDV$5DXj`U8* zsCLv8)HQ~ImXZ;j?C*xSaKB5 zcsErc0w!Q&*d*KA}w|b96+^vK;a{Im#n1~0N-0EY_Xw&_KsCr4lR#puQ*892{ zuYC9P(bx3JW>R9eCT=h3mG0=x&3<;8*SoUyn4LefcR}#5yXHtqf{Z&+sjb0aCs0H1 zqu#5XH?yvf^50roW#mvXW`66yb52>A zmzEDR{QP3Es_fDme0-kVwj`pSnD?TZ$EU$)*tpTdjOm5I!|%^5jdr4ypAVEk86&tLJup? zUwW}VU98{7D<3%sfLkK!GHg^~Kz{hAwyN-FNqR`H>bvJ9S!5;5^lQDDxe(dkSeB|# zaibSfhjGb7ip)ukOiPAL3ll0X3cLx{_a9KWf{Tj_ycMul%}EnxcyPIbDJp{Fe!f8| z3Xq7D^^GIj(BEDc*vhsedc}Xc{i0rG)PWKTcW^M(uMpQX#g&H52hNlW{Ac&N~1mz$H~r zqhkw2*b&ulNdFs1NBLz)M?x^{e+%h{XAs_UmM8s&^#3o>gktj}q}NFV3^%<7=6=dhjsZp5<4tLTQ`s-0XC!T@KA?Waypt4@Ya*#BlfTNX2Xo)t^=ERQ&aYy zep?;O-Y(4Z5>@((kc)4x8^y@W6BDBhKX$L*V@LVo4e|!j9D41*S3pRwU%xH|G`q<) zj}&tXRN;K&K!6GZZ}S9r8w`M00!WXb%N>y#2b(8p5jVPGTj6Q*yky+21-E&UwH12^qPhJedKbz?9sz-l*Z3$VW}wq=8NO zEB3y;7Xo1yQ)^JaHhzcD^LyI<22TL^TK>XT0Q_YmmHd*e0GS5?o_;^r3fO>p&(Gk9 zbnAN^Dd?>I$=00ZH=qo_?$A6R*P_U$kA@m@qwJJ3rLMm1&nlo9@fju zh(5f1st}pBXv_Qe??*-^tC!rrZzwi;v^xC^wz=zWfB*qqbCU0f`<-3?!41#IAjBI4 zqJe`j0HV$Pfa`+O=2}^TYb?)9vx_k;jgQ+fAqBQ-J*7OygJ*c`gQsuTw6HxSy&tqh$&MVRA|h zYjWPrt4XRBN*Q5ScoH)uvb4k@5XP6BtnY3yzDk-(*yNm3EBS=v;}Z$d+F=uH8W3l~ zH+`X=3SGZ+RYs*`nB-!%eOPI0?z47HRST|m`8P5#QGd{LfT*Jng2A-)9ezQ?HzQ+% z>fVK4#!G1`W1W%nJ=WttsCl5^wjH0N#>0r6j&+^$dfxJ*B(?rdeKkeiA!&4kVYN2HT zR_lzm@uDiL%!M~8&svj&SNCqg0V)f8)F2eZ=a-EPN9Z63f^JTN`G-yd)Y-ra!iu8> zEu?v25*Xbum3@byU3lQ;P=2AF!5;(|-7nX05tqA%@`}Jt9a}mLGmcAJ$I2YTN8|#< z5F-u^k$AcmQ4xa@X>3TZ*o;#u8R?YX#e0#w8g7p_bVr=+Ulz?CwLhq00T;+o*F9%H zl@Nn832-fybvV-hw7~=c<9{I<(17+|Gb1A}19SXuc{Sj6K=4SU+h;k$nP`ytEg}%( z@=MVFi~UdeHzihlzoE?Bxmt0RB~DR5Emot3UG-LuM~J)(a=nZhS|*oL zoXTavNi8+OPwF>ZCpGQ1B^;acXmm8mLZPDGNnLcRtxBv}vaavTN(=H_7G_mo8bx{) zuS15(*K02i3rYht9zpEihboznoo+}HG5TwDv7RdHtKmm{*CoWrxUKDA%n1pMjHt5G z7)1xWK5Kdy4G2#O2nGJ|WIPfCQo`ugLPsJ%2ur}2>J~{0kaVO)k#r$xa#$g95C*`~ zkHxf~2433<#5r_>dlj1>!S%c*imMW*ssEa&SA8eF96Fn!yU0U!{F|MladV&f`MI8_ ztw}?U4E?gB-BLCQbewrIZ_KG?QWfHXDlLrKC1~=)94}?z#i6p5diK^l{l>6rP!%03 zY2=94b4Q#D+<=?$h^a(Rf{*xx0_rly3;eAFaA`Elr92k;=k?nZ7cbRbn;={5yX_gB zCo+pt?X)tQDH~Kp?C{hKB8E!!^tQ?Pg;IP6CL&FNjn+H@UE%Tv}gcRYBb|h zn-!W^lo!I_{4a2OON}`URAt6;p>#Mho}LyOO$veUd?7N3VC#!0Vw{G3L_CClvW)A1 z6};pCBoDl>U$%3wkh!zXaO(oj{8k;ExH;Vu7dGXRp7Fhx_7zDTF|mQWj|$b_+E9gg z%3x4=;1ECp)-fu|fji`~ta<-WGxom$WPmgOA|RvW03Y_h1>~(WAO`%Br@;L3HxU1~ zf%yD-Xp}zyvcmBVjN$d&rPU(V?WI|yAs-R3raB+}%@y#8h5iO3;R)r$6J>qNT3W&r z^oh7B{bye9#JzkK=RU?S4zFWM?Oi#s3i5pIMbzZw>pUkOzudomM{3JwcOZTaXNi;M zOT^w+hC{0(M#5vg<9dbzrUR(dPW-NF{1W%#E1EvxZm51rM69tF`vhsLr_>1{;gQW@ zVEmNTfVdZSsx!mhrHvOfwW(wbJHrjeo}Jr<`Z}(zG&x7c@qObmG?$tfruw4RF@yFT zPz9N2)4XKx1h5vc#YF5}2QG9nr<(Y2g)%)A&%t|oqz3|}(_pmVmtiAQBOHQqW4UO7 zyeBvIrtc!G0cVQIsde@T#Y9Meyw5(cx#8&gHMtF}K(lOB#d6!Ygdp=w<3H@vp-rV* zTVkHt#8W=F{%?{?EA*q8XRaNs?37Cmz4VB%-bzWSD!3d#>G9z0!qs@*X0E}j)p-+% zQ!B?Y4_!8=oSYw;jo9=i-<=(J7Z72p{`t;Q!HV{d-gr$~wmN|WGm6?*U#NLZHlioTx<))-nBW&KbSyj5&Pei}LaBX3iKTax`(}Ku- zHUUm8g)j}Blj<-iW-_#8uIKrtyZ{Bah zm+Lx?RoLJsvtI18jrZV(59hlFJ;dPOZe?ptN0@rNJHP1jyWo=_OzYcu!ki~%+0!R0 z$`{v1WSRK76?B8Smu~DDcDd*{Q+HW&4lZ2$I9Hh8$1ZQC&1<6<1vMHEr`X;#(svRt zVi4Fh(&Gj6JLTo&%}T6$$^IEfHU5vM2}IGOZe~A7c_FyjW#7#^&1$%05S_-9R0O*? z`?jbs=&_pM@FlG;1+VgAlAuw>&G{yb85!EZ09`7^wQZzgajjZ|;aWL4*_)Y}5j)tL znek7*U_j>^$JY&5${YM>@Py)L9VJhUu!}_nox~ON>Ph)myoQJ_({Q?jw7^tIcDGJrnUFa`qAF|$iazd?+jFITwYIn; zQz=5f*A3x_@Sp)9Ash%lIv@V1EA2;YXA0Bo{5!DqB;4Rx z4n*FZBlGVZ7{LFcH4ZK7M@e_cA27TWzcq1md@ywt*4>72MzQG_4r`A)2v5k;>RZBh zy`j@b1Mx{m!D4d(C4r6Ua_;+SOw+r2@$rS(DP! zmP8Rc5wO*ZM0+nMk%e(?TCjVMt435^UuQSByUlxrC`}~v3H0)9o=9oizylNyxlV1m zTUvcY8>ZqB;jd6@v~5EMWLW4LWO^CYpp?>U;`eosSH_04HO-8!F1=laI$*epJMdJZ z>w8ozqzs^#^T`K4?@CWpGAIp7JlPB=6+sHUMG$$ThvU>gOb*D=n-e4nFHqrcJ>hk4 zb5p5LeFCU}hlJqNr%(&nR-}c`I67fGB(|!_M4Z7+iK3GMm=`kB*UBX+w|mi;Zm@}5 z6d1@hB{mr~j;icO~#!USO;9scTz4~jz0_?-tB;yM)Vb^ zDv8F+4Gyrlwt3UAbokDDfb{u9K-;CvXk#>=kJN=8kGYe?Ml!1Qh;4i>jZ?WwkngHh zB3R3Yl=T%wc4}TBOQfN`c^}`3md?c11B3H;zOOnvxu8$|_P&1jPO8&PivOHoW{G{= z-e-6i^q|ps9%rKOX1A$<7t#kedX|-H^-;r38m_N`e7=S1R$BNvhYA+?gQR)}3*e2Q zLvhdBp2&4!Ha8XMs%>CKj|P(lS?!lh*t?5+@{@|IdZ9<6yArv2Z`>spNh6Dl_@A0^ zH@<&Ip)xwR0#ogi-<0lCPyv4QSMX4fLvEz|H0tzNk9+d#jH6vrs(IwBT9MCSoE>}? zq4PJh1h`GsMYVGR8TQ2Ss%8Pr>7;mld&T9XZd7<;btMlbEWE^DH+L{pba={}zTBlD zI8{qYda3rd0( zEdoz0Qe=?u3@q50i@(KqSt)kpxT-m|bbZ3`u4@pPOtmB3ed?MmjO4Xca>M*`GLEwA zL@AFVD{-LE)~E0<&@9=ljaJs^c6z#Rn|ISjVhY4K3cV4d>JfIUiZDyE`X*-~P|Noc?!G@c(u zKl^z^?)}(u4AnmQl=TXq&FE>zb)cWz4tQ*W@yG(^WzusDfTVBTG4zWiMiuWw;Ds#$TwA@>64|HzOREVR#)Z zZB0iCMdr@qF4@2;p^Fk&HX45cA(VeX|3Z^>Bydhv?h2KARPL(@&qAqP_L9B$mg6da zjB0m+5bIA)qyd!q;) zcCl=#aDdUSYWRLQzocD* zc+ja}p0vT(PHoT>hpt_S9s(eHka&zjoSUTVJ9-8>3M8MIrEjomDyyt2D+2?A4M)H= zCj>-dg_-AIg6iuVz`WPAP6OF)>y8C=iRH*bu8MDIcVG50pG&QT=`;tvzRc@eF|Zut*;-+fIuiNMn=&+ z5IY6%zpxFQyVd}iDeH}b&~5iX^u@}?_}h}Sg&Gi1y=)823wuYdJ5!^GXh3SWVbc5w z`0H-iWqqXy_}GCWOeJV`$E}C?!XLgUgobMDj0$WHBIPS08gefj| zbn96kWDzK2*ZL~7B|mikyrFSRH-O3)j`2Mkl85_i~%d6#{!Z5C%RVt zkK!J8@(gm5O|u@_tJIiTwms5QYjfE}x$!CAiZ@PH(#bX7w6IiNb8$H)M7>k{qFZWF zt8^97?yMNkXGMo+wZ(>z~dC2TC2u#uB(?RZpaBBTMthF@IMbF{KKjIMruxYDCWou_FJj_8}W+u>*96nSH!FLnOJQ&%K?Mi zZ)Fx3xc{Oudp@3M#XpMIRY43O>WKYFcw(tr!e!+0&p};u`WH{GMeSQ`u!g##=6s~5 z1Ye16Y4(;aP*yRH?7WvOgOa2O@ok*U@PEN%#w>et9c~6w_wuUAsl%9?x<$8aT$N2r zPfD?uVT}Ln7Rr@UQEHne8QT;kqL96&yEqgAIEzMvE6-ZuMJ6*`@mu1tx}!v8Q2EV; zArgkL#@Awd>6|DL@usO+gw*ZLtq$SHQi)crcDVb5p^-?(X$^$T9ALQFuDA{>mJ62k zRMgaUE*%0QDSMB&T3E+ce96+P%*um?kAuv4_dS{V0v8sDdgSSoLgiR+!-ZyES$(Xg z6s>*LA(v-XNbjmpylOC0WHYy-k{9?HeASTtKDnb6Uy6d{Rdk0|g?I-t)|G%8_p&H0 z4Sc0J797er&6TwpC(*0|7tvs06v{otBxS{nsU*8|G+u}EGjP>|x>lj0jZnIxM><_H zA0D+4vb|XI2>9747vhajfk!hfL&+eb!V$@E#d$Tg$?uVY=g!N2jGJ5kd*yd51vh~s zBaBg@x#J~+|0MKz(B$>)1UeUX@kmh6_?}8M?wz`}4)1jP+Mv-;CgR1dy(zQ9%j|o* z&a$5c$G7=I-%z7XF3-u;Bg2fVT5U^pGPd@wSwh_MR!FMS4H zvE^d<{>f(Yk;Q!IQv6S@MZ}^TU6De~NZ%)~nJ^e%-zhqP#4>I%FTR-<4;Qq+dFdVb z5gP}taCPR>{Ib?AlGhWTM!|z+{3Hz1_XDZU%VES_oi^pF&Ncyf{D<07!UnAchHO&ou$^P!@w7W)) zeME{gdGqz9+7M066vKCIqitUKk6fDXr6qKI)WPEd)(xzAoG!S+Wiw~qLK{@tx6CZw<|%#kR|jt5iLyrm5tjcz5(O#Hc;D5HsZ zfvZDbug83~vwm^s^_t8j@&dh*%8!03{KC-2_-`2;1koXPyQcpH9o}){8_8-PN ziye1@S`DlQKV6|%!sFKbs;{=JlO17uQvd?RbISIxojkJ?x@Sbw{5XX z=rNyzWiO;po|ToAIWUhCzcCC#1-?0UhgYzPty-&9dZl7{vHixUh4B91l+;EI<5Xj9 zUgNFiXtKhDB;)3&XPk|c3m>YivdR$)ikzH*JtMtGfO!0^kHS9RTsmtTj6&O%2^UEd zSiS3;V&cU8z@6z(G%i;gYdA;Ts)TsB_q+x)8TkZlwzK)5W3VDFe%Z$AMv5tPT7hyz zX66yS+8r3ev*~8|Ov+sHt+GYFX;wy3so_c=Gp}_95HP96s3l>km1q%VJg7RTkQp1T zCK*eitmTWQpe83@)-GSxrkW^27KI%L%8g4jXF>U6EmQq3uVoU~SB>0qqv>UkSLleg#6@T z4iw2QM+OU3M81Ii+fn@cU2=-q`*7#Q3HnH%ZT)f&f|t5b-R9DJ6sMY(bzo4 zxRPV3i6JzCBmD?+Po2RKBS|j8ytP35TKNI03Mh1;G;n=Cu)uCg9SkqCCE6KXiS!9$j z%H&wC1dypqEw+%byhGVtpIAA$8%R5Z8ZByZWr)n&(eulk^|u;7t+898u?jjJsPkFuanBw%hM>u%^U8eO4+z4 zD{kGwa2r4DZFU0R`1>phXRXBod$8rDHeRm>qa#9(2^+f^+(aqYf|MO!3rXA`=x8Pl zuzcod+@U6KgXg8p`P#os(lSWqIzps0Dy4kvilwkQvPd{y0()#= z&nbpvf+R+QM!x(w{>{pv-iS9&6a>U;IS9mxfg8WN*=ZQ{mK*5udX5q zhjA{91UOcq#naSS7W+8+*v4eNKy4=UBZ=u(O?v>}l*)cayE#E{5G!NR}} zw$#tx9rU&D;IdaS)p8{7`qLZ@uHt8UyjIvTO+Nh4eKZH#425?mrp58pT{G1;p7r=7 zvu}3ub4=RmEp|x*FYnEFuJ#9}@J7ZH!4r4i$YUeB1zG;ViU*~cu37riYTIFKy&9{} z9$qwBqoTfDSE9WX#m-zSKD8Mz5wo|L&o^mvWy!3)i0g}fYm|NN!q5z-Biji5wN?V}8r7$7 zS!f7$(gm$}?kFJJGe>nBv9k!=!r5jG@33PjC>g+5Ayp^;7O-KqxC%a z1!3})6~!LsXIH=@P!wO8X87#xx4dy--QxoH%SJSx{eb^jfzgdv5t1S;B%|nu|9|zC BRVDxc literal 0 HcmV?d00001 diff --git a/LayoutTests/media/media-source/content/test-48khz-new-track-id-manifest.json b/LayoutTests/media/media-source/content/test-48khz-new-track-id-manifest.json new file mode 100644 index 000000000000..449699efd654 --- /dev/null +++ b/LayoutTests/media/media-source/content/test-48khz-new-track-id-manifest.json @@ -0,0 +1,19 @@ +{ + "url": "content/test-48kHz-new-track-id.m4a", + "type": "audio/mp4; codecs=\"mp4a.40.2\"", + "init": { "offset": 0, "size": 624 }, + "duration": 10.0906, + "media": [ + { "offset": 624, "size": 1308, "timestamp": 0, "duration": 1.0027}, + { "offset": 1932, "size": 1491, "timestamp": 1.0027, "duration": 1.0027}, + { "offset": 3423, "size": 1341, "timestamp": 2.0053, "duration": 1.0027}, + { "offset": 4764, "size": 1563, "timestamp": 3.008, "duration": 0.9813}, + { "offset": 6327, "size": 1425, "timestamp": 3.9893, "duration": 1.0027}, + { "offset": 7752, "size": 1770, "timestamp": 4.992, "duration": 1.0027}, + { "offset": 9522, "size": 1910, "timestamp": 5.9947, "duration": 1.0027}, + { "offset": 11432, "size": 1887, "timestamp": 6.9973, "duration": 1.0027}, + { "offset": 13319, "size": 1919, "timestamp": 8, "duration": 1.0027}, + { "offset": 15238, "size": 1892, "timestamp": 9.0027, "duration": 1.0027}, + { "offset": 17130, "size": 1231, "timestamp": 10.0053, "duration": 0.0853} + ] +} diff --git a/LayoutTests/media/media-source/media-source-audio-track-id-switch-expected.txt b/LayoutTests/media/media-source/media-source-audio-track-id-switch-expected.txt new file mode 100644 index 000000000000..66441ee6d8cd --- /dev/null +++ b/LayoutTests/media/media-source/media-source-audio-track-id-switch-expected.txt @@ -0,0 +1,12 @@ + +RUN(audio.src = URL.createObjectURL(source)) +EVENT(sourceopen) +RUN(source.duration = loader.duration()) +RUN(sourceBuffer = source.addSourceBuffer(loader.type())) +RUN(sourceBuffer.appendBuffer(loader.initSegment())) +EVENT(updateend) +Switching to a similar audio stream but with different mp4 track ID. +RUN(sourceBuffer.appendBuffer(loader.initSegment())) +EVENT(updateend) +END OF TEST + diff --git a/LayoutTests/media/media-source/media-source-audio-track-id-switch.html b/LayoutTests/media/media-source/media-source-audio-track-id-switch.html new file mode 100644 index 000000000000..f421e8627603 --- /dev/null +++ b/LayoutTests/media/media-source/media-source-audio-track-id-switch.html @@ -0,0 +1,55 @@ + + + + media-source-audio-track-id-switch + + + + + +