Skip to content

/ youtube-dl

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TF1/WAT.tv support broken on some videos #10281

Closed
sbouafif opened this issue Aug 10, 2016 · 7 comments
Closed

TF1/WAT.tv support broken on some videos #10281

sbouafif opened this issue Aug 10, 2016 · 7 comments

Comments

@sbouafif
Copy link

@sbouafif sbouafif commented Aug 10, 2016 •
edited

TF1 support is not working on some videos.

I'm getting different errors (403 Forbidden, Geoblocked content while I can watch the video in browser).

Yet it does work on some videos too.

s:~/ $ youtube-dl -v http://www.tf1.fr/tf1/ninja-warrior/videos/ninja-warrior-5-aout-2016.html             [12:16:07]
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-v', u'http://www.tf1.fr/tf1/ninja-warrior/videos/ninja-warrior-5-aout-2016.html']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.08.07
[debug] Python version 2.7.10 - Darwin-15.6.0-x86_64-i386-64bit
[debug] exe versions: avconv 11.4, avprobe 11.4, ffmpeg 3.1.1, ffprobe 3.1.1, rtmpdump 2.4
[debug] Proxy map: {}
[TF1] ninja-warrior-5-aout-2016: Downloading webpage
[wat.tv] 13234582: Downloading JSON metadata
[wat.tv] 13234582: Extracting m3u8 url
[wat.tv] 13234582: Extracting http url
[wat.tv] 13234582: Downloading m3u8 information
[wat.tv] 13234582: Downloading f4m manifest
[debug] Invoking downloader on u'http://dnl.adv.tf1.fr/2/USP-0x0/45/82/13234582/ssm/13234582-2500-64k.mp4'
ERROR: unable to download video data: HTTP Error 403: Forbidden
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1683, in process_info
    success = dl(filename, info_dict)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1625, in dl
    return fd.download(name, info)
  File "/usr/local/bin/youtube-dl/youtube_dl/downloader/common.py", line 350, in download
    return self.real_download(filename, info_dict)
  File "/usr/local/bin/youtube-dl/youtube_dl/downloader/http.py", line 58, in real_download
    data = self.ydl.urlopen(request)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1985, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 437, in open
    response = meth(req, response)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 550, in http_response
    'http', request, response, code, msg, hdrs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 475, in error
    return self._call_chain(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 558, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 403: Forbidden

s:~/ $ youtube-dl -v http://www.tf1.fr/tf1/flash/videos/flash-episode-10-saison-02-course-ralenti.html     [12:16:20]
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-v', u'http://www.tf1.fr/tf1/flash/videos/flash-episode-10-saison-02-course-ralenti.html']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.08.07
[debug] Python version 2.7.10 - Darwin-15.6.0-x86_64-i386-64bit
[debug] exe versions: avconv 11.4, avprobe 11.4, ffmpeg 3.1.1, ffprobe 3.1.1, rtmpdump 2.4
[debug] Proxy map: {}
[TF1] flash-episode-10-saison-02-course-ralenti: Downloading webpage
[wat.tv] 13235543: Downloading JSON metadata
[wat.tv] 13235543: Extracting m3u8 url
ERROR: wat.tv said: Sorry, this video is not available from your country.
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 682, in extract_info
    ie_result = ie.extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 347, in extract
    return self._real_extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/wat.py", line 86, in _real_extract
    m3u8_url = extract_url('ipad/%s.m3u8', 'm3u8')
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/wat.py", line 83, in extract_url
    expected=True)
ExtractorError: wat.tv said: Sorry, this video is not available from your country.

s:~/ $ youtube-dl -v http://www.tf1.fr/tf1/une-famille-formidable/videos/une-famille-formidable-l-enigme-papas-perdus-1.html
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-v', u'http://www.tf1.fr/tf1/une-famille-formidable/videos/une-famille-formidable-l-enigme-papas-perdus-1.html']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.08.07
[debug] Python version 2.7.10 - Darwin-15.6.0-x86_64-i386-64bit
[debug] exe versions: avconv 11.4, avprobe 11.4, ffmpeg 3.1.1, ffprobe 3.1.1, rtmpdump 2.4
[debug] Proxy map: {}
[TF1] une-famille-formidable-l-enigme-papas-perdus-1: Downloading webpage
[wat.tv] 13235312: Downloading JSON metadata
[wat.tv] 13235312: Extracting m3u8 url
[wat.tv] 13235312: Extracting http url
[wat.tv] 13235312: Downloading m3u8 information
[wat.tv] 13235312: Downloading f4m manifest
[debug] Invoking downloader on u'http://dnl.adv.tf1.fr/2/USP-0x0/53/12/13235312/ssm/13235312-2500-64k.mp4'
[download] Destination: Une famille formidable - L'énigme des papas perdus-13235312.mp4
[download]   0.1% of 1.73GiB at  2.02MiB/s ETA 14:38^C
ERROR: Interrupted by user
@sbouafif sbouafif changed the title TF1 support broken on some video TF1 support broken on some videos Aug 10, 2016
@sbouafif sbouafif mentioned this issue Aug 10, 2016
Closed
4 of 8 tasks complete
@sbouafif sbouafif changed the title TF1 support broken on some videos TF1/WAT support broken on some videos Aug 10, 2016
@sbouafif sbouafif changed the title TF1/WAT support broken on some videos TF1/WAT.tv support broken on some videos Aug 10, 2016
@sbouafif
Copy link
Author

@sbouafif sbouafif commented Aug 10, 2016

@yan12125 It seems that the 2 errors are completely different.

I found a fix for the 403 Forbidden.

youtube-dl -v -F http://www.tf1.fr/tf1/ninja-warrior/videos/ninja-warrior-5-aout-2016.html
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-v', u'-F', u'http://www.tf1.fr/tf1/ninja-warrior/videos/ninja-warrior-5-aout-2016.html']
[debug] Encodings: locale ANSI_X3.4-1968, fs ANSI_X3.4-1968, out ANSI_X3.4-1968, pref ANSI_X3.4-1968
[debug] youtube-dl version 2016.08.07
[debug] Python version 2.7.10 - Linux-4.2.0-38-generic-x86_64-with-Ubuntu-15.10-wily
[debug] exe versions: ffmpeg 2.7.6-0ubuntu0.15.10.1, ffprobe 2.7.6-0ubuntu0.15.10.1
[debug] Proxy map: {}
[TF1] ninja-warrior-5-aout-2016: Downloading webpage
[wat.tv] 13234582: Downloading JSON metadata
[wat.tv] 13234582: Extracting m3u8 url
[wat.tv] 13234582: Extracting http url
[wat.tv] 13234582: Downloading m3u8 information
[wat.tv] 13234582: Downloading f4m manifest
[info] Available formats for 13234582:
format code  extension  resolution note
hls-meta     mp4        multiple   Quality selection URL
hls-67       mp4        audio only   67k , mp4a.40.2@ 64k
hds-264      flv        416x234     264k
hls-279      mp4        416x234     279k , avc1.66.30@ 200k, mp4a.40.2@ 64k
http-279     mp4        416x234     279k , avc1.66.30@ 200k, mp4a.40.2@ 64k
hds-464      flv        480x270     464k
hls-491      mp4        480x270     491k , avc1.66.30@ 400k, mp4a.40.2@ 64k
http-491     mp4        480x270     491k , avc1.66.30@ 400k, mp4a.40.2@ 64k
hds-664      flv        640x360     664k
hls-703      mp4        640x360     703k , avc1.66.30@ 600k, mp4a.40.2@ 64k
http-703     mp4        640x360     703k , avc1.66.30@ 600k, mp4a.40.2@ 64k
hds-1264     flv        640x360    1264k
hls-1339     mp4        640x360    1339k , avc1.77.31@1200k, mp4a.40.2@ 64k
http-1339    mp4        640x360    1339k , avc1.77.31@1200k, mp4a.40.2@ 64k
hds-1864     flv        960x540    1864k
hls-1975     mp4        960x540    1975k , avc1.77.31@1800k, mp4a.40.2@ 64k
http-1975    mp4        960x540    1975k , avc1.77.31@1800k, mp4a.40.2@ 64k
hds-2564     flv        1280x720   2564k
hls-2717     mp4        1280x720   2717k , avc1.77.31@2500k, mp4a.40.2@ 64k
http-2717    mp4        1280x720   2717k , avc1.77.31@2500k, mp4a.40.2@ 64k (best)

When I use the format hls-2717 instead of the default one, http-2717, it works well.

I tried the same for the geo blocked content but couldn't get the manifest

[wat.tv] 13235543: Downloading JSON metadata
[wat.tv] 13235543: Extracting m3u8 url
ERROR: wat.tv said: Sorry, this video is not available from your country.
@sbouafif
Copy link
Author

@sbouafif sbouafif commented Aug 10, 2016 •
edited

For the geo blocked content. It seems that the manifest is hosted on a different domain for those videos.

For a "normal" video.

url: http://www.tf1.fr/tf1/une-famille-formidable/videos/une-famille-formidable-l-enigme-papas-perdus-1.html
SD manifest url: http://web.q1.tf1.fr/2/USP-0x0/53/12/13235312/ssm/13235312.ism/13235312.f4m?vk=MTMyMzUzMTIuZjRt&st=IywUQCMFFccnWjqCrrW45g&e=1470937847&t=1470808247&bu=WAT&login=hd1-une-famille-formidable&i=IP_ADDRESS&u=3de6e9347fdbf21a18de7efa0b558703&sum=1e6907bd578ecaa9050570aac8d0f437&min_bitrate=400000&max_bitrate=1000000&seek=wat&hdcore=2.11.3&g=OOMGLRBVPLMT

For a geo blocked one:

video url: http://www.tf1.fr/tf1/flash/videos/flash-episode-9-saison-02-plus-fort.html
SD manifest url: http://tf1vodhdscatchup-vh.akamaihd.net/z/H264-384x288/55/39/13235539.h264/manifest.f4m?hdnea=st=1470807619~exp=1470809419~acl=/*~hmac=742ae55f3687986514307440fea46e6eb723d21bfbeb62ae6e5f839c11929c31&bu=WAT&login=flash-tf1&i=IP_ADDRESS&u=3de6e9347fdbf21a18de7efa0b558703&sum=1e6907bd578ecaa9050570aac8d0f437&hdcore=2.11.3&g=GSWWCMROXFWJ
HD manifest url: http://tf1vodhdscatchup-vh.akamaihd.net/z/HD-1280x720/55/39/13235539.hd/manifest.f4m?hdnea=st=1470807626~exp=1470809426~acl=/*~hmac=13e3baf2dc91b0b297ef39956e107e8f844ac32b80984fb8929f78b3ad813ccb&bu=WAT&login=flash-tf1&i=IP_ADDRESS&u=3de6e9347fdbf21a18de7efa0b558703&sum=1e6907bd578ecaa9050570aac8d0f437&hdcore=2.11.3&g=FJHAMKAFXWXB

I'm not sure if that can help.
@remitamine
Copy link
Collaborator

@remitamine remitamine commented Aug 10, 2016 •
edited

in c5f5155, i have changed the extraction logic in order to extract more formats, but i didn't conside that there is a change of server between geo and non geo restricted videos.
i have a simple solution to be able to download any video in the website even when it's geo restricted or removed from public access, but it will depend on hardcoded values i don't know if this is acceptable as a fallback.
remitamine added a commit that referenced this issue Aug 10, 2016
@remitamine
[wat] improve extraction(#10281)
Loading status checks…
57ce8a6 
add alternative method to extract http formats
works even if the video is geo-restricted or removed
from public access(most of the cases)
@remitamine
Copy link
Collaborator

@remitamine remitamine commented Aug 10, 2016 •
edited

ok, i have pushed the method that i was talking about to master branch, if you can test it in the next version and confirm if it's working for you.
@sbouafif
Copy link
Author

@sbouafif sbouafif commented Aug 10, 2016

I just saw that. I will test this in the next 24 hours.
@sbouafif
Copy link
Author

@sbouafif sbouafif commented Aug 10, 2016

@remitamine Just compiled the new version. It works well. Thanks a lot

➜  youtube-dl git:(master) ✗ ./youtube-dl -v http://www.tf1.fr/tf1/ninja-warrior/videos/ninja-warrior-5-aout-2016.html
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-v', u'http://www.tf1.fr/tf1/ninja-warrior/videos/ninja-warrior-5-aout-2016.html']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.08.10
[debug] Python version 2.7.12 - Linux-4.4.0-21-generic-x86_64-with-Ubuntu-16.04-xenial
[debug] exe versions: ffmpeg 2.8.6-1ubuntu2, ffprobe 2.8.6-1ubuntu2
[debug] Proxy map: {}
[TF1] ninja-warrior-5-aout-2016: Downloading webpage
[wat.tv] 13234582: Downloading JSON metadata
[wat.tv] 13234582: Extracting http url
[wat.tv] 13234582: Extracting m3u8 url
[wat.tv] 13234582: Downloading m3u8 information
[wat.tv] 13234582: Downloading f4m manifest
[wat.tv] 13234582: Checking http-279 URL
[wat.tv] 13234582: Checking http-491 URL
[wat.tv] 13234582: Checking http-703 URL
[wat.tv] 13234582: Checking http-1339 URL
[wat.tv] 13234582: Checking http-1975 URL
[wat.tv] 13234582: Checking http-2717 URL
[wat.tv] 13234582: http-2717 URL is invalid, skipping
[debug] Invoking downloader on u'http://ios.q1.tf1.fr/2/USP-0x0/45/82/13234582/ssm/13234582.ism/13234582-audio%3D64000-video%3D2500125.m3u8?vk=MTMyMzQ1ODIubTN1OA==&st=3Ctyllz0k8qKIEjiyrxqBw&e=1470851229&t=1470840429&min_bitrate='
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 737
[download] Destination: Ninja warrior du 5 août 2016 (1_3)-13234582.mp4
[download]   0.2% of ~1.87GiB at  2.12MiB/s ETA 13:10^C
ERROR: Interrupted by user
➜  youtube-dl git:(master) ✗ youtube-dl -v http://www.tf1.fr/tf1/flash/videos/flash-episode-10-saison-02-course-ralenti.html
zsh: command not found: youtube-dl
➜  youtube-dl git:(master) ✗ ./youtube-dl -v http://www.tf1.fr/tf1/flash/videos/flash-episode-10-saison-02-course-ralenti.html
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-v', u'http://www.tf1.fr/tf1/flash/videos/flash-episode-10-saison-02-course-ralenti.html']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.08.10
[debug] Python version 2.7.12 - Linux-4.4.0-21-generic-x86_64-with-Ubuntu-16.04-xenial
[debug] exe versions: ffmpeg 2.8.6-1ubuntu2, ffprobe 2.8.6-1ubuntu2
[debug] Proxy map: {}
[TF1] flash-episode-10-saison-02-course-ralenti: Downloading webpage
[wat.tv] 13235543: Downloading JSON metadata
[wat.tv] 13235543: Extracting http url
[wat.tv] 13235543: Checking http-264 URL
[wat.tv] 13235543: Checking http-464 URL
[wat.tv] 13235543: Checking http-664 URL
[wat.tv] 13235543: Checking http-1264 URL
[wat.tv] 13235543: Checking http-1864 URL
[wat.tv] 13235543: Checking http-2564 URL
[debug] Invoking downloader on u'http://dnl.adv.tf1.fr/2/USP-0x0/55/43/13235543/ssm/13235543-2500-64k.mp4'
[download] Destination: Flash - Episode 10 Saison 02 - Course au ralenti-13235543.mp4
[download]   0.3% of 744.35MiB at  1.81MiB/s ETA 06:49^C
@remitamine
Copy link
Collaborator

@remitamine remitamine commented Aug 10, 2016

ok, closing.
@remitamine remitamine closed this Aug 10, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
@sbouafif @remitamine
You can’t perform that action at this time.