403 on cinemassacre #5615

TheKorn2 · 2015-05-05T09:20:09Z

Using youtube-dl on any page on cinemassacre.com generates a 403 error. Pages play fine in any web browser from same IP.

Example page: http://cinemassacre.com/2015/03/30/probotectorcontra-nes-guest-doug-walker/

$ youtube-dl http://cinemassacre.com/2015/03/30/probotectorcontra-nes-guest-doug-walker/
[Cinemassacre] probotectorcontra-nes-guest-doug-walker: Downloading webpage
[ScreenwaveMedia] Cinemassacre-55185591835ec: Downloading player webpage
ERROR: unable to download video data: HTTP Error 403: Forbidden

dstftw · 2015-05-13T16:33:21Z

Post the full output you get when running with --verbose option.

marek-saji · 2015-05-14T08:26:35Z

$ ./youtube-dl --version
2015.05.10
$ ./youtube-dl --verbose http://cinemassacre.com/2015/03/30/probotectorcontra-nes-guest-doug-walker/
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'--verbose', u'http://cinemassacre.com/2015/03/30/probotectorcontra-nes-guest-doug-walker/']
[debug] Encodings: locale UTF-8, fs UTF-8, out None, pref UTF-8
[debug] youtube-dl version 2015.05.10
[debug] Python version 2.7.9 - Linux-3.19.0-15-generic-x86_64-with-Ubuntu-15.04-vivid
[debug] exe versions: avconv 11.2-6, avprobe 11.2-6, ffmpeg present, rtmpdump 2.4
[debug] Proxy map: {}
[Cinemassacre] probotectorcontra-nes-guest-doug-walker: Downloading webpage
[ScreenwaveMedia] Cinemassacre-55185591835ec: Downloading player webpage
[debug] Invoking downloader on u'http://video2.screenwavemedia.com/vod/Cinemassacre-55185591835ec_hd1.mp4'
ERROR: unable to download video data: HTTP Error 403: Forbidden
Traceback (most recent call last):
  File "./youtube-dl/youtube_dl/YoutubeDL.py", line 1417, in process_info
    success = dl(filename, info_dict)
  File "./youtube-dl/youtube_dl/YoutubeDL.py", line 1359, in dl
    return fd.download(name, info)
  File "./youtube-dl/youtube_dl/downloader/common.py", line 342, in download
    return self.real_download(filename, info_dict)
  File "./youtube-dl/youtube_dl/downloader/http.py", line 59, in real_download
    data = self.ydl.urlopen(request)
  File "./youtube-dl/youtube_dl/YoutubeDL.py", line 1710, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python2.7/urllib2.py", line 437, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 550, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 469, in error
    result = self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 656, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/usr/lib/python2.7/urllib2.py", line 437, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 550, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 475, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 558, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 403: Forbidden

dstftw · 2015-05-14T08:28:02Z

Can you watch it in browser?

marek-saji · 2015-05-16T17:47:36Z

Can you watch it in browser?

Video embedded on a site works without a problem.

URL resolved by youtube-dl also opens and plays in the browser (after being redirected to URL?nimblesessionid=NUMBER). I've used Chrome's „copy as cURL” feature and added --head --location:

$ youtube --get-url http://cinemassacre.com/2015/03/30/probotectorcontra-nes-guest-doug-walker/
http://video2.screenwavemedia.com/vod/Cinemassacre-55185591835ec_hd1.mp4
# Chrome's request + `--head --location`:
$ curl --head --location 'http://video2.screenwavemedia.com/vod/Cinemassacre-55185591835ec_hd1.mp4' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' -H 'Connection: keep-alive' -H 'DNT: 1' -H 'Accept-Encoding: gzip, deflate, sdch' -H 'Accept-Language: en-GB,en;q=0.8,en-US;q=0.6,pl;q=0.4' -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36' --compressed
HTTP/1.1 302 Found
Server: Nimble/2.6.0-9
Date: Sat, 16 May 2015 17:35:54 GMT
Location: http://video2.screenwavemedia.com/vod/Cinemassacre-55185591835ec_hd1.mp4?nimblesessionid=3482958
Content-Length: 0
Connection: close

HTTP/1.1 200 OK
Date: Sat, 16 May 2015 17:35:54 GMT
Server: Nimble/2.6.0-9
Content-Length: 200084801
Cache-Control: no-cache
Content-Type: video/mp4
Access-Control-Allow-Origin: *
Connection: Keep-Alive

Also, just realised that these work:

wget $( youtube-dl --get-url http://cinemassacre.com/2015/03/30/probotectorcontra-nes-guest-doug-walker/ )
curl --location -O $( youtube-dl --get-url http://cinemassacre.com/2015/03/30/probotectorcontra-nes-guest-doug-walker/ )

Maybe redirection is the problem? This line in the trackback would suggst so:

  File "/usr/lib/python3.4/urllib/request.py", line 684, in http_error_302
    return self.parent.open(new, timeout=req.timeout)

dstftw · 2015-05-16T18:02:33Z

This is probably due to HLS being used for streaming in browser. youtube-dl now downloads direct HD link, however for me this approach still works.

marek-saji · 2015-05-18T21:42:06Z

This is probably due to HLS being used for streaming in browser.

This does not explain why this works:

wget $( youtube-dl --get-url http://cinemassacre.com/2015/03/30/probotectorcontra-nes-guest-doug-walker/ )

Here are versions of things I am using:

$ youtube-dl --version
2015.05.15
$ lsb_release -d
Description:    Ubuntu 15.04
# (I've tried both python 2 and three with same result)
$ python2 --version
Python 2.7.9
$ python3 --version
Python 3.4.3
# traceback mentioned urllib2, so I'm showing that as well:
dpkg -S /usr/lib/python2.7/urllib2.py
libpython2.7-stdlib:amd64: /usr/lib/python2.7/urllib2.py

marek-saji · 2015-05-18T21:43:00Z

(Forgot about youtube-dl --get-url inside $(…) at the end of #5615 (comment), I've edited and added that.)

dstftw · 2015-05-19T13:57:16Z

Post the output of youtube-dl --print-traffic http://cinemassacre.com/2015/03/30/probotectorcontra-nes-guest-doug-walker/.

marek-saji · 2015-05-20T17:30:39Z

$ youtube-dl --version
2015.05.20
$ youtube-dl --print-traffic http://cinemassacre.com/2015/03/30/probotectorcontra-nes-guest-doug-walker/
[Cinemassacre] probotectorcontra-nes-guest-doug-walker: Downloading webpage
send: u'GET /2015/03/30/probotectorcontra-nes-guest-doug-walker/ HTTP/1.1\r\nHost: cinemassacre.com\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/20.0 (Chrome)\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nConnection: close\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Date: Wed, 20 May 2015 17:18:59 GMT
header: Content-Type: text/html; charset=UTF-8
header: Transfer-Encoding: chunked
header: Connection: close
header: Set-Cookie: __cfduid=d75f25d961e556d14eec8dafb7c38c2761432142339; expires=Thu, 19-May-16 17:18:59 GMT; path=/; domain=.cinemassacre.com; HttpOnly
header: Last-Modified: Wed, 20 May 2015 17:01:59 GMT
header: Cache-Control: max-age=3, must-revalidate
header: Expires: Wed, 20 May 2015 17:17:39 GMT
header: Vary: Accept-Encoding,Cookie
header: Server: cloudflare-nginx
header: CF-RAY: 1e99b43487bd0b7b-LHR
header: Content-Encoding: gzip
[ScreenwaveMedia] Cinemassacre-55185591835ec: Downloading player webpage
send: u'GET /play/player.php?id=Cinemassacre-55185591835ec HTTP/1.1\r\nHost: player.screenwavemedia.com\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/20.0 (Chrome)\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nConnection: close\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: Apache/2.4.12 (Ubuntu)
header: Vary: Accept-Encoding
header: Content-Encoding: gzip
header: Access-Control-Allow-Origin: *
header: Access-Control-Allow-Headers: origin, x-requested-with, content-type
header: Access-Control-Allow-Methods: PUT, GET, POST, DELETE, OPTIONS
header: Content-Type: text/html; charset=utf-8
header: x-url: /play/player.php?id=Cinemassacre-55185591835ec
header: x-host: player.screenwavemedia.com
header: Cache-Control: max-age=180, public
header: Content-Length: 2405
header: Accept-Ranges: bytes
header: Date: Wed, 20 May 2015 17:18:59 GMT
header: X-Varnish: 1403963510 1403963376
header: Age: 16
header: Via: 1.1 varnish
header: Connection: close
header: X-Cache: HIT
header: X-Cache-Hits: 1
send: u'GET /vod/Cinemassacre-55185591835ec_hd1.mp4 HTTP/1.1\r\nHost: video2.screenwavemedia.com\r\nAccept-Encoding: identity\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nConnection: close\r\nAccept-Language: en-us,en;q=0.5\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/20.0 (Chrome)\r\n\r\n'
reply: 'HTTP/1.1 302 Found\r\n'
header: Server: Nimble/2.6.0-9
header: Date: Wed, 20 May 2015 17:19:00 GMT
header: Location: http://video2.screenwavemedia.com/vod/Cinemassacre-55185591835ec_hd1.mp4?nimblesessionid=3922849
header: Content-Length: 0
header: Connection: close
send: u'GET /vod/Cinemassacre-55185591835ec_hd1.mp4?nimblesessionid=3922849 HTTP/1.1\r\nHost: video2.screenwavemedia.com\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/20.0 (Chrome)\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nConnection: close\r\n\r\n'
reply: 'HTTP/1.1 403 Forbidden\r\n'
header: Date: Wed, 20 May 2015 17:19:00 GMT
header: Server: Nimble/2.6.0-9
header: Content-Length: 72
header: Connection: close
header: Content-Type: text/html
ERROR: unable to download video data: HTTP Error 403: Forbidden

I’ve recreated second request (after redirection) using curl and it did fail:

$ curl --head \
  -H 'Host: video2.screenwavemedia.com' \
  -H 'Accept-Language: en-us,en;q=0.5' \
  -H 'Accept-Encoding: gzip, deflate' \
  -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' \
  -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/20.0 (Chrome)' \
  -H 'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7' \
  -H 'Connection: close' \
  'http://video2.screenwavemedia.com/vod/Cinemassacre-55185591835ec_hd1.mp4?nimblesessionid=3922849'
HTTP/1.1 403 Forbidden
Date: Wed, 20 May 2015 17:27:34 GMT
Server: Nimble/2.6.0-9
Content-Length: 72
Connection: close
Content-Type: text/html

… but removing Connection: close header fixes that:

$ curl --head \
  -H 'Host: video2.screenwavemedia.com' \
  -H 'Accept-Language: en-us,en;q=0.5' \
  -H 'Accept-Encoding: gzip, deflate' \
  -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' \
  -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/20.0 (Chrome)' \
  -H 'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7' \
  'http://video2.screenwavemedia.com/vod/Cinemassacre-55185591835ec_hd1.mp4?nimblesessionid=3922849'
HTTP/1.1 200 OK
Date: Wed, 20 May 2015 17:28:06 GMT
Server: Nimble/2.6.0-9
Content-Length: 200084801
Cache-Control: no-cache
Content-Type: video/mp4
Access-Control-Allow-Origin: *
Connection: close

etnmich · 2015-05-25T19:14:39Z

I spotted a curious thing while messing around with the same issue. It seems to not happen 100% of the time so removing the 'Connection: close' header doesn't actually fix anything; the server just decided to accept the connection that time. If you just try running youtube-dl again, it might suddenly start the dl without any issues. I did a few tries on a recent video (shortened a bit):

$ ./youtube-dl -v http://cinemassacre.com/2015/05/13/darkwing-duck-angry-video-game-nerd/
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-v', u'http://cinemassacre.com/2015/05/13/darkwing-duck-angry-video-game-nerd/']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2015.05.20
[debug] Python version 2.7.6 - Darwin-14.1.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 2.6.1, ffprobe 2.6.1
[debug] Proxy map: {}
[Cinemassacre] darkwing-duck-angry-video-game-nerd: Downloading webpage
[ScreenwaveMedia] Cinemassacre-55430f9321f38: Downloading player webpage
[debug] Invoking downloader on u'http://video2.screenwavemedia.com/vod/Cinemassacre-55430f9321f38_hd1.mp4'
[download] Destination: Darkwing Duck – Angry Video Game Nerd-Cinemassacre-55430f9321f38.mp4
[download]   0.2% of 150.59MiB at 108.32KiB/s ETA 23:41^C
ERROR: Interrupted by user

$ ./youtube-dl -v http://cinemassacre.com/2015/05/13/darkwing-duck-angry-video-game-nerd/
[debug] System config: []
(...)
[debug] Invoking downloader on u'http://video2.screenwavemedia.com/vod/Cinemassacre-55430f9321f38_hd1.mp4'
[download] Resuming download at byte 261120
ERROR: unable to download video data: HTTP Error 403: Forbidden
Traceback (most recent call last): 
  (...)
  HTTPError: HTTP Error 403: Forbidden

$ ./youtube-dl -v http://cinemassacre.com/2015/05/13/darkwing-duck-angry-video-game-nerd/
[debug] System config: []
(...)
[debug] Invoking downloader on u'http://video2.screenwavemedia.com/vod/Cinemassacre-55430f9321f38_hd1.mp4'
[download] Resuming download at byte 261120
[download] Destination: Darkwing Duck – Angry Video Game Nerd-Cinemassacre-55430f9321f38.mp4
[download]   0.5% of 150.59MiB at 77.93KiB/s ETA 32:49^C
ERROR: Interrupted by user

The last invocations were only seconds apart. I tried this with cURL too with similiar results so it seems like the issue is that ScreenwaveMedia's server sometimes sends out 403s for a perfectly fine request. I'd say the only way to fix this would be to make it retry after a few seconds on a 403 error, as the issue seems to be completely on their end.

PS. I had to edit youtube_dl/extractor/cinemassacre.py to include the new player2.screenwavemedia.com-urls as Cinemassacre uses those now. Just a heads up.

marek-saji · 2015-05-26T09:37:04Z

seems to not happen 100% of the time so removing the 'Connection: close' header doesn't actually fix anything; the server just decided to accept the connection that time

Interesting. I've never managed to download the video using youtube-dl before and never hit 403 using curl, besides when I've included Connection: close.

I’ve checked out master, run downloading in a loop:

while youtube-dl --verbose … ; sleep 10 ; done

and only after 13 minutes video started downloading. After I Ctrl-C it and retried, I started getting 403s again.

dstftw · 2016-05-14T00:27:48Z

Cinemassacre now uses JW Platform. This should work now.

yan12125 mentioned this issue May 26, 2015

Unable to download cinemassacre.com's videos again. #5821

Closed

dstftw closed this May 14, 2016

ytdl-org / youtube-dl

403 on cinemassacre #5615

403 on cinemassacre #5615

TheKorn2 commented May 5, 2015

dstftw commented May 13, 2015

marek-saji commented May 14, 2015

dstftw commented May 14, 2015

marek-saji commented May 16, 2015

dstftw commented May 16, 2015

marek-saji commented May 18, 2015

marek-saji commented May 18, 2015

dstftw commented May 19, 2015

marek-saji commented May 20, 2015

etnmich commented May 25, 2015

marek-saji commented May 26, 2015

dstftw commented May 14, 2016

ytdl-org / youtube-dl

Join GitHub today

GitHub is where the world builds software

403 on cinemassacre #5615

403 on cinemassacre #5615

Comments

TheKorn2 commented May 5, 2015

dstftw commented May 13, 2015

marek-saji commented May 14, 2015

dstftw commented May 14, 2015

marek-saji commented May 16, 2015

dstftw commented May 16, 2015

marek-saji commented May 18, 2015

marek-saji commented May 18, 2015

dstftw commented May 19, 2015

marek-saji commented May 20, 2015

etnmich commented May 25, 2015

marek-saji commented May 26, 2015

dstftw commented May 14, 2016

Essential cookies

Always active

Analytics cookies