Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ARD] HD format as mp4 file (best) not found #17744

Open
andreygursky opened this issue Oct 2, 2018 · 9 comments · May be fixed by #17851
Open

[ARD] HD format as mp4 file (best) not found #17744

andreygursky opened this issue Oct 2, 2018 · 9 comments · May be fixed by #17851

Comments

@andreygursky
Copy link

@andreygursky andreygursky commented Oct 2, 2018

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2018.09.26. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

  • I've verified and I assure that I'm running youtube-dl 2018.09.26

Before submitting an issue make sure you have:

  • At least skimmed through the README, most notably the FAQ and BUGS sections
  • Searched the bugtracker for similar issues including closed ones
  • Checked that provided video/audio/playlist URLs (if any) are alive and playable in a browser

What is the purpose of your issue?

  • Bug report (encountered problems with youtube-dl)
  • Site support request (request for adding support for a new site)
  • Feature request (request for a new functionality)
  • Question
  • Other

If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

youtube-dl -v -F 'https://www.ardmediathek.de/tv/Babylon-Berlin/1929-Das-Jahr-Babylon/Das-Erste/Video?bcastId=54319834&documentId=56517232'
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'-F', u'https://www.ardmediathek.de/tv/Babylon-Berlin/1929-Das-Jahr-Babylon/Das-Erste/Video?bcastId=54319834&documentId=56517232']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.09.26
[debug] Python version 2.7.15 (CPython) - Linux-4.14.0-3-amd64-x86_64-with-debian-9.0                                                                                
[debug] exe versions: ffmpeg 3.4.2-1, ffprobe 3.4.2-1, rtmpdump 2.4
[debug] Proxy map: {}
[ARD:mediathek] 56517232: Downloading webpage
[ARD:mediathek] 56517232: Downloading media JSON
[ARD:mediathek] 56517232: Downloading f4m manifest
[ARD:mediathek] 56517232: Downloading m3u8 information
[ARD:mediathek] 56517232: Downloading m3u8 information
[info] Available formats for 56517232:
format code  extension  resolution note
a0-mp4-2     mp4        unknown
a1-mp4-0     mp4        unknown
a1-mp4-1-0   mp4        unknown
a1-mp4-1-1   mp4        unknown
a1-mp4-2-0   mp4        unknown
a1-mp4-2-1   mp4        unknown
a1-mp4-3     mp4        unknown
hls-188-0    mp4        320x180     188k , avc1.66.30, mp4a.40.2
hls-188-1    mp4        320x180     188k , avc1.66.30, mp4a.40.2
hls-316-0    mp4        480x270     316k , avc1.66.30, mp4a.40.2
hls-316-1    mp4        480x270     316k , avc1.66.30, mp4a.40.2
hds-604      flv        unknown     604k
hls-604-0    mp4        512x288     604k , avc1.77.30, mp4a.40.2
hls-604-1    mp4        512x288     604k , avc1.77.30, mp4a.40.2
hds-1212     flv        unknown    1212k
hls-1212-0   mp4        640x360    1212k , avc1.77.30, mp4a.40.2
hls-1212-1   mp4        640x360    1212k , avc1.77.30, mp4a.40.2
hds-1988     flv        unknown    1988k
hls-1988-0   mp4        960x540    1988k , avc1.77.30, mp4a.40.2
hls-1988-1   mp4        960x540    1988k , avc1.77.30, mp4a.40.2
hls-3771     mp4        1280x720   3771k , avc1.64001f, mp4a.40.2 (best)

Description of your issue, suggested solution and other information

1280x720 mp4 file (https://pdvideosdaserste-a.akamaihd.net/int/2018/09/18/196f8dd1-4491-4405-be65-fff78fcc8e0d/1280-1.mp4) has not been discovered. It should be selected as "best" instead of hls-3771 ts stream (which would need ffmpeg postprocessing).

P.S. above is only an example for the issue with ARD Mediathek.

@Hrxn
Copy link

@Hrxn Hrxn commented Oct 2, 2018

Well, seems like the same result, basically.. (720p)
Did you check the differences between format hls-3771 and the other format you mentioned?

@andreygursky
Copy link
Author

@andreygursky andreygursky commented Oct 2, 2018

diff ffprobe HLS stream <-> HTTP file download:

--- "1929 - Das Jahr Babylon-56517232.mp4.ffprobe"	2018-10-02 14:26:43.750085877 +0200
+++ 1280-1.mp4.ffprobe	2018-10-02 14:27:02.566086703 +0200
@@ -1,13 +1,16 @@
-Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '1929 - Das Jahr Babylon-56517232.mp4':
+Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '1280-1.mp4':                                               
   Metadata:
-    major_brand     : isom
-    minor_version   : 512
-    compatible_brands: isomiso2avc1mp41
-    encoder         : Lavf57.83.100
-  Duration: 00:43:50.08, start: 0.000000, bitrate: 3777 kb/s
-    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709), 1280x720 [SAR 1:1 DAR 16:9], 3583 kb/s, 25 fps, 25 tbr, 90k tbn, 50 tbc (default)  
+    major_brand     : mp42
+    minor_version   : 0
+    compatible_brands: isom
+    creation_time   : 2018-09-18T20:48:26.000000Z
+  Duration: 00:43:50.08, start: 0.000000, bitrate: 3775 kb/s
+    Stream #0:0(eng): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709), 1280x720 [SAR 1:1 DAR 16:9], 3582 kb/s, 25 fps, 25 tbr, 25k tbn, 50 tbc (default)  
     Metadata:
-      handler_name    : VideoHandler
-    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 189 kb/s (default)                                                                
+      creation_time   : 2018-09-18T20:48:26.000000Z
+      handler_name    : MP4 Video Media Handler
+      encoder         : AVC Coding
+    Stream #0:1(deu): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 189 kb/s (default)                                                                
     Metadata:
-      handler_name    : SoundHandler
+      creation_time   : 2018-09-18T20:48:26.000000Z
+      handler_name    : MP4 Sound Media Handler

diff mediainfo HLS stream <-> HTTP file download:

--- "1929 - Das Jahr Babylon-56517232.mp4.mediainfo"    2018-10-02 14:23:12.784419862 +0200
+++ 1280-1.mp4.mediainfo        2018-10-02 14:22:49.024420168 +0200
@@ -1,13 +1,14 @@
 General
-Complete name                            : 1929 - Das Jahr Babylon-56517232.mp4
+Complete name                            : 1280-1.mp4
 Format                                   : MPEG-4
-Format profile                           : Base Media
-Codec ID                                 : isom (isom/iso2/avc1/mp41)
+Format profile                           : Base Media / Version 2
+Codec ID                                 : mp42 (isom)
 File size                                : 1.16 GiB
 Duration                                 : 43 min 50 s
-Overall bit rate mode                    : Constant
-Overall bit rate                         : 3 777 kb/s
-Writing application                      : Lavf57.83.100
+Overall bit rate mode                    : Variable
+Overall bit rate                         : 3 775 kb/s
+Encoded date                             : UTC 2018-09-18 20:48:26
+Tagged date                              : UTC 2018-09-18 20:48:26

 Video
 ID                                       : 1
@@ -35,6 +36,9 @@ Bit depth
 Scan type                                : Progressive
 Bits/(Pixel*Frame)                       : 0.156
 Stream size                              : 1.10 GiB (95%)
+Language                                 : English
+Encoded date                             : UTC 2018-09-18 20:48:26
+Tagged date                              : UTC 2018-09-18 20:48:26
 Color range                              : Limited
 Color primaries                          : BT.709
 Transfer characteristics                 : BT.709
@@ -47,15 +51,17 @@ Format/Info
 Format profile                           : LC
 Codec ID                                 : mp4a-40-2
 Duration                                 : 43 min 50 s
-Bit rate mode                            : Constant
+Bit rate mode                            : Variable
 Bit rate                                 : 192 kb/s
+Maximum bit rate                         : 305 kb/s
 Channel(s)                               : 2 channels
 Channel positions                        : Front: L R
 Sampling rate                            : 48.0 kHz
 Frame rate                               : 46.875 FPS (1024 SPF)
 Compression mode                         : Lossy
 Stream size                              : 59.4 MiB (5%)
-Default                                  : Yes
-Alternate group                          : 1
+Language                                 : German
+Encoded date                             : UTC 2018-09-18 20:48:26
+Tagged date                              : UTC 2018-09-18 20:48:26

HLS stream:

  • need to call ffmpeg
  • download time: 05:17

HTTP file download:

  • faster download: 03:33
  • encoding timestamp preserved
  • audio language preserved
  • md5 checksum provided
@Hrxn
Copy link

@Hrxn Hrxn commented Oct 2, 2018

And bitrate 3583 kb/s vs. 3582 kb/s.. 😅

No really, thanks for this comparison. And I agree, the MP4 direct download is actually the better default and should be preferred, for the reasons you already mentioned.

BTW, what did you use to find the links? One of these?

https://mediathekdirekt.de/
https://mediathekviewweb.de/
@andreygursky
Copy link
Author

@andreygursky andreygursky commented Oct 2, 2018

BTW, what did you use to find the links? One of these?

Exactly. https://mediathekviewweb.de - is my favorite because of being fast due to using AJAX. In case I need a database in a local file, I go to https://mediathekdirekt.de/.

@ngdio ngdio linked a pull request that will close this issue Oct 10, 2018
5 of 9 tasks complete
@ngdio
Copy link

@ngdio ngdio commented Oct 10, 2018

Could you try out the version from my pull request? It solve this issue for all media from Das Erste.

@ngdio ngdio mentioned this issue Oct 11, 2018
5 of 9 tasks complete
@andreygursky
Copy link
Author

@andreygursky andreygursky commented Jul 1, 2020

Sorry for not replying before. For now the patch cannot be cleanly applied. But looking at https://github.com/ytdl-org/youtube-dl/pull/17851/files#diff-2ebe82369b35395b419db1575a615d9dR27

    # IMPORTANT: Not valid for media from WDR/BR/etc. servers

it seems it is not exactly what this issue is about, since the video from 'Das Erste' was only one example for an ARD URL (www.ardmediathek.de / classic.ardmediathek.de).

@andreygursky
Copy link
Author

@andreygursky andreygursky commented Jul 1, 2020

For ARD download URLs can be obtained generally as follows (in a bash shell):

URL_CLASSIC='https://classic.ardmediathek.de/tv/mareTV/Estlands-verwunschene-Inseln/NDR-Fernsehen/Video?bcastId=14049152&documentId=66593126'
ID=`echo "${URL_CLASSIC}" | sed -e 's|.*documentId=\([0-9]*\)|\1|'`
JSON=`wget -q -O - "https://www.ardmediathek.de/play/media/${ID}"`
echo $JSON | jq -r '._mediaArray[]._mediaStreamArray[]._stream'

Result:

https://ndrod-vh.akamaihd.net/i/ndr/2019/0909/TV-20190909-1144-2800.,hd,hq,ln,hi,mn,lo,ao,.mp4.csmil/master.m3u8
https://mediandr-a.akamaihd.net/progressive/2019/0909/TV-20190909-1144-2800.lo.mp4
[
  "https://mediandr-a.akamaihd.net/progressive/2019/0909/TV-20190909-1144-2800.ln.mp4",
  "https://mediandr-a.akamaihd.net/progressive/2019/0909/TV-20190909-1144-2800.hi.mp4",
  "https://mediandr-a.akamaihd.net/progressive/2019/0909/TV-20190909-1144-2800.mn.mp4"
]
https://mediandr-a.akamaihd.net/progressive/2019/0909/TV-20190909-1144-2800.hq.mp4
https://mediandr-a.akamaihd.net/progressive/2019/0909/TV-20190909-1144-2800.hd.mp4
@andreygursky
Copy link
Author

@andreygursky andreygursky commented Jul 1, 2020

It looks like for quality parameter applies:

  • 0 is "Low Quality"
  • 1 is "Medium Quality"
  • 2 is "High Quality" (actually "SD")
  • 3 is "HD"
  • 4 is "FullHD"

Thus one can select the desired download URL (e.g. "HD"):

echo "$JSON" | jq -r '._mediaArray[]._mediaStreamArray[] as $data | select($data._quality == 3) | $data._stream' 

Result:

https://mediandr-a.akamaihd.net/progressive/2019/0909/TV-20190909-1144-2800.hd.mp4
@ngdio
Copy link

@ngdio ngdio commented Aug 26, 2020

I might take a look at this again next week. Things have changed significantly since 2018, so it's likely all going to be different now. Shouldn't be difficult to find out, though.

it seems it is not exactly what this issue is about, since the video from 'Das Erste' was only one example for an ARD URL (www.ardmediathek.de / classic.ardmediathek.de).

At the time, there weren't any higher quality streams/files from WDR/BR/etc. that weren't found by the existing code. Might have changed now but only ARD-Mediathek-hosted videos had higher quality back in 2018.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

3 participants
You can’t perform that action at this time.