Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

VGTV.no protected content fails #5853

Open
kristofferR opened this issue May 30, 2015 · 9 comments
Open

VGTV.no protected content fails #5853

kristofferR opened this issue May 30, 2015 · 9 comments

Comments

@kristofferR
Copy link

@kristofferR kristofferR commented May 30, 2015

Youtube-DL fails to download certain content on VGTV.no, like this one:
http://www.vgtv.no/#!/video/110355/den-tapte-narkotikakrigen

╰─> youtube-dl --verbose http://www.vgtv.no/\#\!/video/110355/den-tapte-narkotikakrigen
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'--verbose', u'http://www.vgtv.no/#!/video/110355/den-tapte-narkotikakrigen']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2015.05.29
[debug] Python version 2.7.6 - Darwin-14.3.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 2.6.3, ffprobe 2.6.3, rtmpdump 2.4
[debug] Proxy map: {}
[VGTV] 110355: Downloading media JSON
[VGTV] 110355: Downloading m3u8 information
ERROR: Failed to download m3u8 information: HTTP Error 403: Forbidden (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 312, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1713, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 410, in open
    response = meth(req, response)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 523, in http_response
    'http', request, response, code, msg, hdrs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 448, in error
    return self._call_chain(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 531, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
@dstftw
Copy link
Collaborator

@dstftw dstftw commented May 30, 2015

It does not work for me in browser either. Can you watch it in browser?

@kristofferR
Copy link
Author

@kristofferR kristofferR commented May 31, 2015

Yeah, it works fine for me. It may be geo-protected though.

I compared the f4m manifest with a f4m manifest from a ripable video, and this element was the only major difference:

 <pv-2.0>;hdntl=exp=1433106405~acl=%2fz%2f2015%2f03%2f20150331_551a7c5145dd2_vg01%2f,240_160_64,320_180_250,480_270_500,640_360_800,960_540_1500,1280_720_3500,.mp4.csmil%2f*~data=hdntl~hmac=8ebacb238edddce2a4b5ce7e66e91ec61f2494ce1bafcda72a007d2594fc90d7</pv-2.0>

The token/protection probably prevents the current script from working on the video.

@dstftw
Copy link
Collaborator

@dstftw dstftw commented May 31, 2015

It tells me

En uventet feil oppstod. Vennligst prøv igjen senere.

so nothing about geo restriction at all.
How did you manage to get f4m if it throws you 403 according to the log?

@kristofferR
Copy link
Author

@kristofferR kristofferR commented May 31, 2015

The error means "An unknown error occurred. Please try again later", so they may just have been too lazy to implement a proper error message for geo blocking. Or there's a completely different error on your end.

Flashgot (for Firefox) was able to find the manifest and download it, + all the segments, without any issues.

I MITM'ed my iPad and watched both a ripable video and a protected one. The difference is clearly the HMAC/token protection. On the unprotected video the segments has URLs like this:

http://svpvod-vh.akamaihd.net/i/2015/05/20150526_5563c4fbc012c_vg01/,240_160_64,320_180_250,480_270_500,640_360_800,960_540_1500,1280_720_3500,.mp4.csmil/segment2_4_av.ts?null=

while the protected video has segment URLs like this:

http://svpsecurehdvod-vh.akamaihd.net/i/2015/03/20150331_551a7c5145dd2_vg01/,240_160_64,320_180_250,480_270_500,640_360_800,960_540_1500,1280_720_3500,.mp4.csmil/segment6_4_av.ts?null=&id=AgAlv5PQXZ5udWDmalWzdK34lRRrMbRNtTE2HYCwwPUGY5aO5twSSvtKt5FWLQB17DZALMCKZ3dR+A%3d%3d

The first URL is likely still accessible, but the protected one immediately returned 403 errors (likely because of time+something missing from the HTTP request)

Flashgot saved the manifest for the protected video from this URL:

http://svpsecurehdvod-vh.akamaihd.net/z/2015/03/20150331_551a7c5145dd2_vg01/,240_160_64,320_180_250,480_270_500,640_360_800,960_540_1500,1280_720_3500,.mp4.csmil/manifest.f4m?hdnea=st=1433070341~exp=1433070431~acl=/z/2015/03/20150331_551a7c5145dd2_vg01/,240_160_64,320_180_250,480_270_500,640_360_800,960_540_1500,1280_720_3500,.mp4.csmil/*~hmac=48faaea846f886715a08630c1aaebfd2c402d0570abbe718e640e893548efcf6&hdcore=2.10.3&g=KBIQZLNPNMUC

Here's the manifest for an unprotected video:

http://svpvod-vh.akamaihd.net/z/2015/05/20150526_5563c4fbc012c_vg01/,240_160_64,320_180_250,480_270_500,640_360_800,960_540_1500,1280_720_3500,.mp4.csmil/manifest.f4m?hdcore=2.10.3&g=BMLUHBEWHVQC
@kristofferR
Copy link
Author

@kristofferR kristofferR commented Jun 1, 2015

I managed to grab the video by MITM my iPad and really quickly copying the m3u8 manifest URL into livestreamer with the command:

livestreamer "hlsvariant://https://svpsecurehdvod-vh.akamaihd.net/i/2015/03/20150331_551a7c5145dd2_vg01/,240_160_64,320_180_250,480_270_500,640_360_800,960_540_1500,1280_720_3500,.mp4.csmil/master.m3u8?hdnea=st=1433073503~exp=1433073593~acl=/i/2015/03/20150331_551a7c5145dd2_vg01/,240_160_64,320_180_250,480_270_500,640_360_800,960_540_1500,1280_720_3500,.mp4.csmil/*~hmac=f89beaebda388cfe399703e231372aedf5b8202846295fc672a1c863359b50a4" best -o video.ts

The link expires really quickly, I had to attempt it a few times to make it work. At least it proves that ripping the video via a script like youtube-dl is definitely possible.

@remitamine
Copy link
Collaborator

@remitamine remitamine commented Jan 26, 2017

this patch will extract the token and add it to manifest url(need someone to test it):

diff --git a/youtube_dl/extractor/vgtv.py b/youtube_dl/extractor/vgtv.py
index 8a574bc26..c85e47d23 100644
--- a/youtube_dl/extractor/vgtv.py
+++ b/youtube_dl/extractor/vgtv.py
@@ -176,16 +176,28 @@ class VGTVIE(XstreamIE):
 
         formats = []
 
+        token = None
         hls_url = streams.get('hls')
+        hds_url = streams.get('hds')
+        manifest_url = hls_url or hds_url
+        if manifest_url and re.match(r'https?://[^/]*secure', manifest_url):
+            token = self._download_webpage(
+                'http://svp.vg.no/svp/api/v1/token?vendor=%s&acl=/*' % vendor,
+                video_id, fatal=False)
+
         if hls_url:
+            if token:
+                hls_url += '?hdnea=' + token
             formats.extend(self._extract_m3u8_formats(
                 hls_url, video_id, 'mp4', m3u8_id='hls', fatal=False))
 
-        hds_url = streams.get('hds')
         if hds_url:
             hdcore_sign = 'hdcore=3.7.0'
+            hds_url += '?' + hdcore_sign
+            if token:
+                hds_url += '&hdnea=' + token
             f4m_formats = self._extract_f4m_formats(
-                hds_url + '?%s' % hdcore_sign, video_id, f4m_id='hds', fatal=False)
+                hds_url, video_id, f4m_id='hds', fatal=False)
             if f4m_formats:
                 for entry in f4m_formats:
                     # URLs without the extra param induce an 404 error
@vidaritos
Copy link

@vidaritos vidaritos commented May 29, 2018

@remitamine Tried your patch, but no luck
This video video works with and without the patch:

python -m youtube_dl https://www.vgtv.no/video/159109/sa-hun-ble-utsatt-for-overgrep-saa-slapp-politiet-videoen -v

Output:

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'https://www.vgtv.no/video/159109/sa-hun-ble-utsatt-for-overgrep-saa-slapp-politiet-videoen', u'-v']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.05.26
[debug] Git HEAD: a07879d6b
[debug] Python version 2.7.10 (CPython) - Darwin-17.5.0-x86_64-i386-64bit
[debug] exe versions: avconv 12.2, avprobe 12.2
[debug] Proxy map: {}
[VGTV] 159109: Downloading media JSON
[VGTV] 159109: Downloading m3u8 information
[VGTV] 159109: Downloading f4m manifest
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on u'https://svpvodps-vh.akamaized.net/2018/05/ed948533282609e710d4b4adc7583c00/1280_720_3500.mp4'
[download] Destination: Sa hun ble utsatt for overgrep - så slapp politiet videoen-159109.mp4
[download] 100% of 25.77MiB in 00:05

But this fails:
Input:

python -m youtube_dl https://www.vgtv.no/video/158983/reporter-takles-av-loepsk-hest-paa-direkten -v

Output:

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'https://www.vgtv.no/video/158983/reporter-takles-av-loepsk-hest-paa-direkten', u'-v']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.05.26
[debug] Git HEAD: a07879d6b
[debug] Python version 2.7.10 (CPython) - Darwin-17.5.0-x86_64-i386-64bit
[debug] exe versions: avconv 12.2, avprobe 12.2
[debug] Proxy map: {}
[VGTV] 158983: Downloading media JSON
[VGTV] 158983: Downloading m3u8 information
[VGTV] 158983: Downloading f4m manifest
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on u'https://svpvod-vh.akamaized.net/i/2018/05/c1701634646613c085adf05f3b39bde7/,480_480_500,.mp4.csmil/index_0_av.m3u8'
[download] Destination: Reporter takles av løpsk hest på direkten-158983.mp4
[debug] avconv command line: avconv -y -loglevel verbose -headers 'Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip, deflate
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0 (Chrome)
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Cookie: _alid_=Zk/87GzIwRawKpdRzSBzpg==
' -i 'https://svpvod-vh.akamaized.net/i/2018/05/c1701634646613c085adf05f3b39bde7/,480_480_500,.mp4.csmil/index_0_av.m3u8' -c copy -f mp4 'file:Reporter takles av løpsk hest på direkten-158983.mp4.part'
avconv version 12.2, Copyright (c) 2000-2017 the Libav developers
  built on Oct  1 2017 01:28:28 with Apple LLVM version 9.0.0 (clang-900.0.37)
  configuration: --disable-debug --disable-shared --disable-indev=jack --prefix=/usr/local/Cellar/libav/12.2 --enable-gpl --enable-nonfree --enable-version3 --enable-vda --cc=clang --host-cflags= --host-ldflags= --enable-libfaac --enable-libfdk-aac --enable-libfreetype --enable-libmp3lame --enable-libopus --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libxvid
  libavutil     55. 20. 0 / 55. 20. 0
  libavcodec    57. 25. 0 / 57. 25. 0
  libavformat   57.  7. 2 / 57.  7. 2
  libavdevice   56.  1. 0 / 56.  1. 0
  libavfilter    6.  7. 0 /  6.  7. 0
  libavresample  3.  0. 0 /  3.  0. 0
  libswscale     4.  0. 0 /  4.  0. 0
https://svpvod-vh.akamaized.net/i/2018/05/c1701634646613c085adf05f3b39bde7/,480_480_500,.mp4.csmil/index_0_av.m3u8: Protocol not found


ERROR: ffmpeg exited with code 1
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Users/myuser/Documents/code/youtube-dl/youtube_dl/__main__.py", line 19, in <module>
    youtube_dl.main()
  File "youtube_dl/__init__.py", line 472, in main
    _real_main(argv)
  File "youtube_dl/__init__.py", line 462, in _real_main
    retcode = ydl.download(all_urls)
  File "youtube_dl/YoutubeDL.py", line 2001, in download
    url, force_generic_extractor=self.params.get('force_generic_extractor', False))
  File "youtube_dl/YoutubeDL.py", line 803, in extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "youtube_dl/YoutubeDL.py", line 857, in process_ie_result
    return self.process_video_result(ie_result, download=download)
  File "youtube_dl/YoutubeDL.py", line 1635, in process_video_result
    self.process_info(new_info)
  File "youtube_dl/YoutubeDL.py", line 1908, in process_info
    success = dl(filename, info_dict)
  File "youtube_dl/YoutubeDL.py", line 1847, in dl
    return fd.download(name, info)
  File "youtube_dl/downloader/common.py", line 364, in download
    return self.real_download(filename, info_dict)
  File "youtube_dl/downloader/external.py", line 64, in real_download
    self.get_basename(), retval))
  File "youtube_dl/downloader/common.py", line 165, in report_error
    self.ydl.report_error(*args, **kargs)
  File "youtube_dl/YoutubeDL.py", line 620, in report_error
    self.trouble(error_message, tb)
  File "youtube_dl/YoutubeDL.py", line 582, in trouble
    tb_data = traceback.format_list(traceback.extract_stack())
@remitamine
Copy link
Collaborator

@remitamine remitamine commented May 29, 2018

@vidaritos the version of avconv that you're using doesn't support HTTPS URLs, either use a version that supports it or download another format.

@vidaritos
Copy link

@vidaritos vidaritos commented May 29, 2018

@remitamine Gotcha, I added --prefer-ffmpeg and it worked fine!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.