Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support new RTE Player #18417

Open
outhud opened this issue Dec 5, 2018 · 18 comments
Open

Support new RTE Player #18417

outhud opened this issue Dec 5, 2018 · 18 comments

Comments

@outhud
Copy link

@outhud outhud commented Dec 5, 2018

Please follow the guide below

  • You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
  • Put an x into all the boxes [ ] relevant to your issue (like this: [x])
  • Use the Preview tab to see what your issue will actually look like

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2018.12.03. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

  • I've verified and I assure that I'm running youtube-dl 2018.12.03

Before submitting an issue make sure you have:

  • At least skimmed through the README, most notably the FAQ and BUGS sections
  • Searched the bugtracker for similar issues including closed ones
  • Checked that provided video/audio/playlist URLs (if any) are alive and playable in a browser

What is the purpose of your issue?

  • Bug report (encountered problems with youtube-dl)
  • Site support request (request for adding support for a new site)
  • Feature request (request for a new functionality)
  • Question
  • Other

youtube-dl -v https://beta.rte.ie/player/series/whistleblower--the-maurice-mccabe-story/SI0000004409?epguid=IH000374544
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'https://beta.rte.ie/player/series/whistleblower--the-maurice-mccabe-story/SI0000004409?epguid=IH000374544']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.12.03
[debug] Python version 2.7.15rc1 (CPython) - Linux-4.15.0-42-generic-x86_64-with-LinuxMint-19-tara
[debug] exe versions: ffmpeg 3.4.4-0ubuntu0.18.04.1, ffprobe 3.4.4-0ubuntu0.18.04.1, rtmpdump 2.4
[debug] Proxy map: {}
[generic] SI0000004409?epguid=IH000374544: Requesting header
WARNING: Falling back on generic information extractor.
[generic] SI0000004409?epguid=IH000374544: Downloading webpage
[generic] SI0000004409?epguid=IH000374544: Extracting information
ERROR: Unsupported URL: https://beta.rte.ie/player/series/whistleblower--the-maurice-mccabe-story/SI0000004409?epguid=IH000374544
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/generic.py", line 2339, in _real_extract
    doc = compat_etree_fromstring(webpage.encode('utf-8'))
  File "/usr/local/bin/youtube-dl/youtube_dl/compat.py", line 2542, in compat_etree_fromstring
    doc = _XML(text, parser=etree.XMLParser(target=_TreeBuilder(element_factory=_element_factory)))
  File "/usr/local/bin/youtube-dl/youtube_dl/compat.py", line 2531, in _XML
    parser.feed(text)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1659, in feed
    self._raiseerror(v)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1523, in _raiseerror
    raise err
ParseError: syntax error: line 1, column 0
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 792, in extract_info
    ie_result = ie.extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 508, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/generic.py", line 3318, in _real_extract
    raise UnsupportedError(url)
UnsupportedError: Unsupported URL: https://beta.rte.ie/player/series/whistleblower--the-maurice-mccabe-story/SI0000004409?epguid=IH000374544


...
<end of log>

Sample url: https://beta.rte.ie/player/series/whistleblower--the-maurice-mccabe-story/SI0000004409?epguid=IH000374544

@Vangelis66
Copy link

@Vangelis66 Vangelis66 commented Dec 6, 2018

While I was able to start watching provided sample programme from a southern European location (i.e. outside of Ireland) and without the need to sign-in, things look pretty grim:

  1. On a desktop browser, I was being served MPEG-DASH streams encrypted with cenc DRM, which required my browser to have enabled EME and a recent version of the WidevineCDM (>= 1.4.8.984, with support for Verified Media Path (VMP)).
  2. Manipulating the MPD URI so as to change it to a HLS .m3u8 master playlist URI, I was successful in getting HLS streams but, sadly, they were encrypted with Apple Fairplay DRM:
#EXTM3U
#EXT-X-VERSION:4
## Created with Unified Streaming Platform(version=1.8.4)
#EXT-X-SESSION-KEY:METHOD=SAMPLE-AES,URI="skd://fairplay.entitlement.theplatform.eu/fpls/web/FairPlay?expiry=1544740740",KEYFORMAT="com.apple.streamingkeydelivery",KEYFORMATVERSIONS="1"
  1. Pretending to be either an Android or iOS mobile device, I was instructed to head to the respective app store to download and install the "new" RTE app...

TL;DR: If this "beta" RTE player goes into production, you can probably forget about downloading RTE shows (with yt-dl or otherwise...); it's UKTVplay all over again...

@ealgase
Copy link
Contributor

@ealgase ealgase commented Dec 6, 2018

Hopefully, someone will make an open-source implementation of Widevine (although the odds of that happening seem low).

@parshy
Copy link

@parshy parshy commented Feb 25, 2019

This doesn't sound like good news, at all. No!!

@stephen147
Copy link

@stephen147 stephen147 commented Mar 30, 2019

Obviously the beta stage is over now with RTE Player. My bug report below.

I doubt anything has changed to make it easier to DL anything from RTE.

youtube-dl "https://www.rte.ie/player/movie/shortscreen-digs-for-pennies/94356520117/" --match-title REGEX ".*video.*mp4" -v
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['https://www.rte.ie/player/movie/shortscreen-digs-for-pennies/94356520117/', '--match-title', 'REGEX', '.*video.*mp4', '-v']
[debug] Encodings: locale cp1252, fs mbcs, out cp850, pref cp1252
[debug] youtube-dl version 2019.03.18
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.17763
[debug] exe versions: ffmpeg 4.1.1, ffprobe N-86809-g012620a
[debug] Proxy map: {}
[generic] 94356520117: Requesting header
WARNING: Falling back on generic information extractor.
[generic] 94356520117: Downloading webpage
[generic] 94356520117: Extracting information
ERROR: Unsupported URL: https://www.rte.ie/player/movie/shortscreen-digs-for-pennies/94356520117/
Traceback (most recent call last):
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpcs52imf5\build\youtube_dl\YoutubeDL.py", line 794, in extract_info
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpcs52imf5\build\youtube_dl\extractor\common.py", line 529, in extract
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpcs52imf5\build\youtube_dl\extractor\generic.py", line 3320, in _real_extract
youtube_dl.utils.UnsupportedError: Unsupported URL: https://www.rte.ie/player/movie/shortscreen-digs-for-pennies/94356520117/
@macplaxton
Copy link

@macplaxton macplaxton commented Apr 23, 2019

I don't know if this is of any help, but here goes...someone might be able to fill in the missing pieces between the player page URL and the actual video URL.

OS: openSUSE Leap 15.0
Browser: Version 73.0.3683.75 (openSUSE Build) (64-bit)
youtube-dl --version 2019.04.17

Starting form here
https://www.rte.ie/player/series/the-late-late-show/SI0000001694?epguid=IH000368289

I was able to watch the adverts, but then ran up against a DRM error of some sort. To solve that error, I installed a further package chromium-plugin-widevinecdm 1.49.1076-lp150.1.1 from the Packman repository. This enable me to watch the main programme in the browser.

Whilst playing the content in the browser, right-clicking and opening inspect element shows the browser console. From this I was able to pluck the correct video link:

Cross-Origin Read Blocking (CORB) blocked cross-origin response https://vod.rte.ie/rte/vod/RTE_Prod_-_Prod/406/504/IH000368289/2cdd5b50-6ef8-4e46-858a-a01e1baadd6b_f73e1a09-a085-40e8-ad42-1f0333e215e0-1555752767765.ism/.mpd?available=1555713840&expiry=1558305840&ip=xxx.xxx.xxx.xxx&token1=206c47ca9750935254bb44a00700fcd60ae86c0fc2d56741bad97a976a0a4eca&filter=systemBitrate%3C%3D7000000 with MIME type application/dash+xml. See https://www.chromestatus.com/feature/5629709824032768 for more details.

Note: ip=xxx.xxx.xxx.xxx Irish IP address redacted.

Copied link and trimmed down to the end of the IH number.
https://vod.rte.ie/rte/vod/RTE_Prod_-_Prod/406/504/IH000368289/

Browsed this which has a selection of stream links. There is a massive .ismv stream link, but there is also a mp4 link with a [VID] tag. Copy this link and right-click and save. This link will download with youtube-dl, the page of links won't, nor will the link of the programme main page.

Despite the size being marked 6.1M it was ~930MB, video plays fine in VLC.

@ealgase
Copy link
Contributor

@ealgase ealgase commented Apr 23, 2019

Seems like a hacky workaround but that MP4 appears to not have any DRM.

@stephen147
Copy link

@stephen147 stephen147 commented Apr 23, 2019

Thanks @macplaxton .

Just one thing to add for others. The URL must end with /.

In my case RTE link:
https://www.rte.ie/player/movie/shortscreen--digs-for-pennies/94356520117

translated to DIR:
https://vod.rte.ie/rte/vod/RTE_Prod_-_Prod/619/728/AI000003079/

This short film has only 2 days left so it'll be interesting if that FTP link is still active after that time.

Edit: Are the ISMV files protected in any way?

@macplaxton
Copy link

@macplaxton macplaxton commented Apr 23, 2019

It is hacky, but a finer mind than mine might be able to suss out an automated way of retrieving the links from the site.

I did download a massive (c.12GB) ismv and it played on VLC sluggishly and out of sync on VLC, but it did play. My machine isn't all at new (Athlon 64X2 c. 2007) and so I daren't try out a file conversion with FFmpeg.

@stephen147
Copy link

@stephen147 stephen147 commented Apr 23, 2019

Working here for me with playing the ISMV file on MPC-BE x64. Will FFmpeg that file tomorrow. But this is truly fantastic. Just in the nick of time. That short-film had only 2 days remaining to view.

Subtitles are also available in different formats as far as I can tell. They're found under the [TXT] tag.

Here are the specs of that FULL HD file (which was the best available for this particular link):

Complete name                  : C:\Users\username_redacted\Desktop\M3664947-15531847403816328_12a4a4968e634be1951de66a7e42eaae.ismv
Format                         : MPEG-4
Format profile                 : ISML
Codec ID                       : isml (iso2/piff)
Codec ID/Hint                  : IIS Smooth Streaming file format
File size                      : 686 MiB
Duration                       : 15 min 49 s
Overall bit rate               : 6 064 kb/s
Encoded date                   : UTC 2019-03-21 16:25:42
Tagged date                    : UTC 2019-03-21 16:25:42

Video
ID                             : 7
Format                         : AVC
Format/Info                    : Advanced Video Codec
Format profile                 : High@L4
Format settings                : CABAC / 4 Ref Frames
Format settings, CABAC         : Yes
Format settings, ReFrames      : 4 frames
Codec ID                       : avc1
Codec ID/Info                  : Advanced Video Coding
Duration                       : 15 min 49 s
Bit rate                       : 6 200 kb/s
Width                          : 1 920 pixels
Height                         : 1 080 pixels
Display aspect ratio           : 16:9
Frame rate mode                : Constant
Frame rate                     : 25.000 FPS
Color space                    : YUV
Chroma subsampling             : 4:2:0
Bit depth                      : 8 bits
Scan type                      : Progressive
Bits/(Pixel*Frame)             : 0.120
Stream size                    : 678 MiB (99%)
Title                          : Video
Writing library                : x264 core 148
Encoding settings              : cabac=1 / ref=1 / deblock=1:0:0 / analyse=0x3:0x113 / me=hex / subme=7 / psy=0 / mixed_ref=0 / me_range=16 / chroma_me=1 / trellis=1 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=0 / threads=7 / lookahead_threads=1 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / paff=0 / bluray_compat=0 / constrained_intra=0 / bframes=2 / b_pyramid=2 / b_adapt=1 / b_bias=0 / direct=1 / weightb=1 / open_gop=0 / weightp=2 / keyint=50 / keyint_min=5 / scenecut=40 / intra_refresh=0 / rc_lookahead=40 / rc=cbr / mbtree=1 / bitrate=6200 / ratetol=1.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / vbv_maxrate=6200 / vbv_bufsize=6200 / nal_hrd=none / filler=0 / ip_ratio=1.40 / aq=1:0.00
Language                       : English
Encoded date                   : UTC 2019-03-21 16:25:42
Tagged date                    : UTC 2019-03-21 16:25:42
Color range                    : Limited
Color primaries                : BT.709
Transfer characteristics       : BT.709
Matrix coefficients            : BT.709
Codec configuration box        : avcC

Audio
ID                             : 10
Format                         : AAC LC SBR PS
Format/Info                    : Advanced Audio Codec Low Complexity with Spectral Band Replication and Parametric Stereo
Commercial name                : HE-AACv2
Format settings                : Explicit
Codec ID                       : mp4a-67-2
Duration                       : 15 min 49 s
Bit rate mode                  : Constant
Bit rate                       : 64.0 kb/s
Nominal bit rate               : 128 kb/s
Channel(s)                     : 2 channels / 1 channel / 1 channel
Channel layout                 : C
Sampling rate                  : 48.0 kHz / 48.0 kHz / 24.0 kHz
Frame rate                     : 23.438 FPS (1024 SPF)
Compression mode               : Lossy
Stream size                    : 7.24 MiB (1%)
Title                          : Sound
Encoded date                   : UTC 2019-03-21 16:25:42
Tagged date                    : UTC 2019-03-21 16:25:42
@Vangelis66
Copy link

@Vangelis66 Vangelis66 commented Apr 24, 2019

It appears that if you possess a European Union (EU) IP address, you can still access streams 😉 .
For https://www.rte.ie/player/series/the-late-late-show/SI0000001694 I didn't even have to start watching (RTE Player was nagging me to disable uBlock Origin...).

Via Web Console (Basilisk web browser, a FirefoxESR 52 fork) I sniffed a lengthy "iframe" URI:

https://www.rte.ie/player/iframe/1.1.64/iframeEmbedSource-vjs.html?pid=https://link.eu.theplatform.com/s/1uC-gC/media/9dRvdSjq6SNM&signlanguage=off&mediatype=&playerprofile=391515:RTE_HTML5_Live&networkid=391515&serverurl=https://5f95b.v.fwmrm.net/&sitesectionid=RTE%20Player_Desktop_Web_NotRegistered&customid=fwVideoId&keyvaluesfield=fwKeyValues&mux_video_id=IH000368289&mux_video_title=Fri%2019%20April%202019&mux_video_series=The%20Late%20Late%20Show&mux_env_key=7rro99i9482rqqokaa3kuvgs4&mux_video_content_type=episode&mux_video_duration=6520.1&mux_video_stream_type=on-demand&mux_video_producer=No%20Provider&mux_player_version=1.1.64

They are in fact using the theplatform.com API to generate MPDs (mpeg-dash manifests, DRM'ed with common encryption, ergo WidevineCDM):
https://link.eu.theplatform.com/s/1uC-gC/media/9dRvdSjq6SNM?formats=mpeg-dash&format=SMIL =>

<smil xmlns="http://www.w3.org/2005/SMIL21/Language">
<head>
	<meta name="title" content="The Late Late Show E15"/>
	<meta name="startingBitrate" content="2500000"/>
</head>
<body>
<seq>
	<ref src="https://ads.freewheel.tv" no-skip="true" tags="preroll">
	</ref>
<switch>
	<video src="https://vod.rte.ie/rte/vod/RTE_Prod_-_Prod/406/504/IH000368289/2cdd5b50-6ef8-4e46-858a-a01e1baadd6b_f73e1a09-a085-40e8-ad42-1f0333e215e0-1555752767765.ism/.mpd?available=1555713840&amp;expiry=1558305840&amp;ip=<snipped>&amp;token1=91d637deeeb62b0074d387a7cf8744eb17c372aa40c8417209df8fa10947bc0c&amp;filter=systemBitrate%3C%3D7000000" system-bitrate="0" height="1080" width="1920"/>
	<ref src="https://vod.rte.ie/rte/vod/RTE_Prod_-_Prod/406/504/IH000368289/2cdd5b50-6ef8-4e46-858a-a01e1baadd6b_f73e1a09-a085-40e8-ad42-1f0333e215e0-1555752767765.ism/.mpd?available=1555713840&amp;expiry=1558305840&amp;ip=<snipped>&amp;token1=91d637deeeb62b0074d387a7cf8744eb17c372aa40c8417209df8fa10947bc0c&amp;filter=systemBitrate%3C%3D7000000" title="1-1" dur="6524887ms" guid="IH000368289" type="application/dash+xml" height="1080" width="1920" clipBegin="0:00" clipEnd="25:54.985" security="commonEncryption">
		<param name="fwVideoId" value="IH000368289"/>
	</ref>
</switch>
	<ref src="https://ads.freewheel.tv" no-skip="true" tags="midroll">
	</ref>
<switch>
	<video src="https://vod.rte.ie/rte/vod/RTE_Prod_-_Prod/406/504/IH000368289/2cdd5b50-6ef8-4e46-858a-a01e1baadd6b_f73e1a09-a085-40e8-ad42-1f0333e215e0-1555752767765.ism/.mpd?available=1555713840&amp;expiry=1558305840&amp;ip=<snipped>&amp;token1=91d637deeeb62b0074d387a7cf8744eb17c372aa40c8417209df8fa10947bc0c&amp;filter=systemBitrate%3C%3D7000000" system-bitrate="0" height="1080" width="1920"/>
	<ref src="https://vod.rte.ie/rte/vod/RTE_Prod_-_Prod/406/504/IH000368289/2cdd5b50-6ef8-4e46-858a-a01e1baadd6b_f73e1a09-a085-40e8-ad42-1f0333e215e0-1555752767765.ism/.mpd?available=1555713840&amp;expiry=1558305840&amp;ip=<snipped>&amp;token1=91d637deeeb62b0074d387a7cf8744eb17c372aa40c8417209df8fa10947bc0c&amp;filter=systemBitrate%3C%3D7000000" title="The Late Late Show E15" dur="6524887ms" guid="IH000368289" type="application/dash+xml" height="1080" width="1920" clipBegin="25:54.985" security="commonEncryption">
		<param name="fwVideoId" value="IH000368289"/>
	</ref>
</switch>
	<ref src="https://ads.freewheel.tv" no-skip="true" tags="postroll">
	</ref>
</seq>
</body>
</smil>

From that SMIL file you can take the .mpd URI up until the .ism part and substitute the .ism extension with .mp4:

-https://vod.rte.ie/rte/vod/RTE_Prod_-_Prod/406/504/IH000368289/2cdd5b50-6ef8-4e46-858a-a01e1baadd6b_f73e1a09-a085-40e8-ad42-1f0333e215e0-1555752767765.ism
+https://vod.rte.ie/rte/vod/RTE_Prod_-_Prod/406/504/IH000368289/2cdd5b50-6ef8-4e46-858a-a01e1baadd6b_f73e1a09-a085-40e8-ad42-1f0333e215e0-1555752767765.mp4

... and there you have the direct HTTPS link to the video file... What @macplaxton has discovered is the unencrypted SOURCE video file at the root of the directory, which is being streamed via encrypted (cenc) MPEG-DASH format 🥇 ...

Sadly, now that this is out in the open, I expect this loophole to be swiftly patched... Enjoy while you can 😜

@outhud
Copy link
Author

@outhud outhud commented Apr 24, 2019

Thanks, very useful.

The .ismv files are not DRM protected and can be remuxed to mp4 or mkv using ffmpeg.

Browsing around https://vod.rte.ie/rte/vod/RTE_Prod_-_Prod/XXX/ (replacing XXX with 3 digits, some combinations don't work) reveals a lot of unprotected content (both inside and outside Ireland). I guess "IH" at the start of some directory names might stand for "in-house", as it seems to be mostly news and weather bulletins in these folders. (Other directory names start with "AQ" or "IP") You can also find .vtt subtitle files inside many of the directories.

@remitamine remitamine added the DRM label Apr 24, 2019
@Vangelis66
Copy link

@Vangelis66 Vangelis66 commented Apr 24, 2019

Unfortunately, things aren't always that rosy 😭 , especially for lengthy programs like The Late Late Show that was mentioned previously by @macplaxton; both downloaded MP4 video files,

Fri 19 April 2019
https://www.rte.ie/player/series/the-late-late-show/SI0000001694?epguid=IH000368289 => 
https://vod.rte.ie/rte/vod/RTE_Prod_-_Prod/406/504/IH000368289/2cdd5b50-6ef8-4e46-858a-a01e1baadd6b_f73e1a09-a085-40e8-ad42-1f0333e215e0_1aa7becb687e4b0ca94448376373fb17.mp4

Fri 12 April 2019
https://www.rte.ie/player/series/the-late-late-show/SI0000001694?epguid=IH000368288 => 
https://vod.rte.ie/rte/vod/RTE_Prod_-_Prod/857/908/IH000368288/0e908db3-b34f-4067-bea6-0ab0588f72a9_c48f7199-73fd-4750-ad1d-76bfc890bc55-1555159154222.mp4

are in fact CORRUPTED 👎 ; in the first one, audio sync is lost pretty quickly after the first 10mins and it continues to play up to a point (ca. 24min), then NOTHING! In the second file, audio remains in sync with video but content stops at around 40min, then again NOTHING (both files have an advertised duration of ca. 1h48min).

The "huge" .ismv files are a totally different beast!

ffmpeg -i "https://vod.rte.ie/rte/vod/RTE_Prod_-_Prod/406/504/IH000368289/2cdd5b50-6ef8-4e46-858a-a01e1baadd6b_f73e1a09-a085-40e8-ad42-1f0333e215e0_1aa7becb687e4b0ca94448376373fb17.ismv"

waiting for 2mins, followed by CTRL+C yields: 

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'https://vod.rte.ie/rte/vod/RTE_Prod_-_P
rod/406/504/IH000368289/2cdd5b50-6ef8-4e46-858a-a01e1baadd6b_f73e1a09-a085-40e8-
ad42-1f0333e215e0_1aa7becb687e4b0ca94448376373fb17.ismv':
  Metadata:
    major_brand     : iso6
    minor_version   : 0
    compatible_brands: iso6
  Duration: 00:00:06.08, bitrate: N/A
    Stream #0:0(eng): Data: none (dfxp / 0x70786664), 137 kb/s (default)
    Metadata:
      handler_name    : USP Text Handler
    Stream #0:1(gle): Audio: aac (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 93
 kb/s (default)
    Metadata:
      handler_name    : USP Sound Handler
    Stream #0:2(eng): Audio: aac (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 12
5 kb/s (default)
    Metadata:
      handler_name    : USP Sound Handler
    Stream #0:3(eng): Audio: aac (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 63
 kb/s (default)
    Metadata:
      handler_name    : USP Sound Handler
    Stream #0:4(eng): Audio: aac (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 94
 kb/s (default)
    Metadata:
      handler_name    : USP Sound Handler
    Stream #0:5(eng): Audio: aac (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 93
 kb/s (default)
    Metadata:
      handler_name    : USP Sound Handler
    Stream #0:6(und): Video: h264 (avc1 / 0x31637661), none, 416x234, 147 kb/s,
25 fps, 10000k tbr, 10000k tbn, 20000k tbc (default)
    Metadata:
      handler_name    : USP Video Handler
      encoder         : AVC Coding
    Stream #0:7(und): Video: h264 (avc1 / 0x31637661), none, 480x270, 380 kb/s,
25 fps, 10000k tbr, 10000k tbn, 20000k tbc (default)
    Metadata:
      handler_name    : USP Video Handler
      encoder         : AVC Coding
    Stream #0:8(und): Video: h264 (avc1 / 0x31637661), none, 640x360, 782 kb/s,
25 fps, 10000k tbr, 10000k tbn, 20000k tbc (default)
    Metadata:
      handler_name    : USP Video Handler
      encoder         : AVC Coding
    Stream #0:9(und): Video: h264 (avc1 / 0x31637661), none, 768x432, 1169 kb/s,
 25 fps, 10000k tbr, 10000k tbn, 20000k tbc (default)
    Metadata:
      handler_name    : USP Video Handler
      encoder         : AVC Coding
    Stream #0:10(und): Video: h264 (avc1 / 0x31637661), none, 960x540, 2027 kb/s
, 25 fps, 10000k tbr, 10000k tbn, 20000k tbc (default)
    Metadata:
      handler_name    : USP Video Handler
      encoder         : AVC Coding
    Stream #0:11(und): Video: h264 (avc1 / 0x31637661), none, 1280x720, 4490 kb/
s, 25 fps, 10000k tbr, 10000k tbn, 20000k tbc (default)
    Metadata:
      handler_name    : USP Video Handler
      encoder         : AVC Coding
    Stream #0:12(und): Video: h264 (avc1 / 0x31637661), none, 1920x1080, 5570 kb
/s, 25 fps, 10000k tbr, 10000k tbn, 20000k tbc (default)
    Metadata:
      handler_name    : USP Video Handler
      encoder         : AVC Coding

This suggests it's somewhat of a merge of 5 audio + 7 video streams, but I was unsuccessful in getting anything to download as separate streams via, e.g. -map 0:11 -map 0:2 switches... Did not bother fetching to disk the whole 11.7GiB of it...

TL;DR : for files under 45min (to, maybe, 1h), the "hack" should work; for anything longer, YMMV - but my experience thus far has shown you'll just waste bandwidth and end up with partially playable video files... 😉

@macplaxton
Copy link

@macplaxton macplaxton commented Apr 24, 2019

Your One Nikita, might be more suitable for testing, as they are short animations.
https://www.rte.ie/player/series/your-one-nikita/SI0000004462?epguid=PL000001100

Might yield some correlation to the amount of audio drift?

@outhud
Copy link
Author

@outhud outhud commented Apr 25, 2019

@Vangelis66 yep I see the same issue with the mp4 going out of sync. It seems it is not a gradual audio drift throughout the file, but rather something that happens suddenly at one point in the video.

I tried with the 19th April episode, it happens at 10:46, you'll see they go to show a brief clip of a movie, which they have removed from the stream for licensing reasons I suppose. It goes out of sync after this cut. The RTE Player itself has some issues at this point in the video too (it pauses, and I need to skip ahead in the video to see more, but in the RTE Player there is no loss of av sync after that point).

I downloaded the full 12GB ismv. In many of the directories, it seems there is a separate ISMV for each quality (1080p, 720p, etc.) but in this one, all those video streams are in the one file. I used ffmpeg to copy the 1080p video stream and one of the audio streams to an mkv file. The mkv is complete, and it seems to play OK all the way through, apart from the audio going out of sync after 10:46, but I think this is because it was cut incorrectly, rather than it being an issue that only affects large / long shows.

@Vangelis66
Copy link

@Vangelis66 Vangelis66 commented Apr 25, 2019

@outhud wrote:

I downloaded the full 12GB ismv.
(snipped)
but in this one, all those video streams are in the one file.
I used ffmpeg to copy the 1080p video stream and one of the audio streams to an mkv file.
The mkv is complete, and it seems to play OK all the way through,
apart from the audio going out of sync after 10:46,
but I think this is because it was cut incorrectly

... OK, I caved in and downloaded all 11.67 GiB of the (full) .ISMV file that corresponds to the Fri Apr 12th 2019 episode of The Late Late Show (the one in which Una Healy 💘 is guest...); then, like you, used FFmpeg to create a 720p25 h264 + 128kbps AAC LC matroska file:

ffmpeg -v 16 -stats -i "TheLateLateShow_20190412.ismv" -map 0:11 -map 0:2 -c copy -f matroska "TheLateLateShow_20190412_720p.mkv"

That got me a 3.49 GiB matroska file with video+audio throughout its full duration 👍 . BUT:

Audio/Video sync is retained until the end of Part Two (ca. 01:07:54); in Part Three, the video precedes audio by at least 500ms; obviously, during editing the commercials out, they left 0.5sec worth of audio stream in, relative to edited-out video stream 😞

Things get even worse during Part Four (starting at ca. 01:30:25); from that point on, audio lags even more compared to video by at least 800ms, so, again, a cock-up during editing-out the commercial break between Parts Three & Four 😠

The easiest solution I found for rectifying the loss of sync in Parts 3+4 was to use VLC, which has a feature - not present in MPC-BE - which allows one to manually set an audio offset compared to the video stream:

Tools => Track Synchronization => Audio Track Synchronization

and set value -0,500 s during Part Three, increased to -0,800 s during (final) Part Four...

Returning on topic (well, sort of...), I still think it's impractical to download (via youtube-dl or other app) an obscenely sized file to then extract the resolution you want; especially when more tinkering is needed for a proper playback...

The case of standalone resolution .ismv files (offered for other programs), that could be looked at, I suppose... 😜

rather than it being an issue that only affects large / long shows.

I was referring to the .MP4 links; please, be my guest: Choose a long program (e.g. Friday Night Soccer), download the MP4 file under the [VID] tag and check whether it plays in full (i.e. till the end); my bet is it only plays up to a point (usually < 1h) 😉 ...

@philroche
Copy link

@philroche philroche commented May 10, 2019

I have a nasty hack way to download the mp4 using a combination of a javascript bookmarklet and python

Javascript bookmarket: https://gist.github.com/philroche/dc9fb6d148d2bdc667aa063b00a86438

Add this bookmarklet as a bookmark on your bookmark bar (see https://help.crossbrowsertesting.com/live-testing/getting-started/installing-the-bookmarklet/).

Once you start playing an episode on rte player click on your new bookmarklet and it will print a url to the javascript console with starts with "https://link.eu.theplatform.com".

Copy this to your clipboard.

You then need the second part of my hack which is the python script - https://gist.github.com/philroche/52a907d8d7b15e87f8c1d29599910b92

Install the python requirements requests, BeautifulSoup and youtube_dl then run

python3 download-rte-player-mp4.py "URL YOU COPIED ABOVE"

This will result in a correctly named mp4 in your current directory.

It's a hack but it has worked for me and have had no issues with the mp4 - but I have not tried it with programmes with running length greater than 1 hour.

@philroche
Copy link

@philroche philroche commented May 13, 2019

I have now updated the downloader script to a proper python package

pip install rteplayer-dl

rteplayer_dl --debug --video-directory="DIRECTORY ON YOUR LOCAL FILESYSTEM" --video-xml="VIDEO EMBED URL" 

I've also updated the bookmarklet https://gist.github.com/philroche/dc9fb6d148d2bdc667aa063b00a86438 to use a javascript prompt instead of just writing to the javascript console.

See https://rteplayer-dl.readthedocs.io/en/latest/usage.html for more info

@philroche
Copy link

@philroche philroche commented Nov 24, 2019

The ismvs that we discussed above and that are used in https://github.com/philroche/rteplayer_dl are now all returning HTTP 415 Unsupported Media Type. I have opened philroche/rteplayer_dl#6 to try and investigate but at first glance it does not look promising. If anyone knows why they have started returning HTTP 415 now please share.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
9 participants
You can’t perform that action at this time.