Escapist not working

[terabyte]

See #5068 for details, even with the newest version as of this writing (I think?) I am getting a video but it is only a 45-second ad, not the real video

$ youtube-dl --verbose http://www.escapistmagazine.com/videos/view/zero-punctuation/10044-Evolve-Review
[debug] System config: []                                       
[debug] User config: []
[debug] Command-line args: ['--verbose', 'http://www.escapistmagazine.com/videos/view/zero-punctuation/10044-Evolve-Review']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2015.02.26.1
[debug] Python version 3.4.2 - Linux-3.16.0-4-amd64-x86_64-with-debian-8.0
[debug] exe versions: avconv 11.2-6, avprobe 11.2-6, rtmpdump 2.4
[debug] Proxy map: {}
[Escapist] 10044: Downloading webpage
[Escapist] 10044: Downloading normal configuration
[Escapist] 10044: Downloading hq configuration
[debug] Invoking downloader on 'http://video.escapistmagazine.com/mp4/escapist/sales-marketing/5387-4a7b988602b3d53a02653768b43bb191.mp4'
[download] Destination: Evolve - One vs Multiplayer-10044.mp4
[download] 100% of 2.38MiB in 00:02

[terabyte]

(also, mad props for this awesome tool, you guys rock, etc etc)

[phihag]

Thank you for the report. It seems like once one requests the configuration with a user agent The Escapist does not recognize, the IP address gets blocked. I've added detection for that as well as a new UA that should not be triggered in youtube-dl 2015.02.26.2. See our FAQ if you need help updating.

[terabyte]

All my IPs are getting blocked (even a fresh one I just tried for the first time - a linode). This is crazy (and obviously not your fault). Fuck escapist magazine, apparently I don't want to watch their shit anyways. I'll send Yatzee an email telling him he should find a new home that doesn't mistreat his fans so badly. (I am only using youtube-dl in this instance because my linux chrome UA also, apparently, got me banned).

[terabyte]

(but also, mad thanks for the super-fast update)

[achilleas-k]

Great. I'm blocked too!
Looks like we'll have to watch stuff a week late on their YouTube channel.

[Tailszefox]

I can confirm the same issue even with the latest version. I tried from a new IP, which I checked wasn't blocked by watching a video on the website, and the result was this:

[~/Bureau]$ youtube-dl --verbose http://www.escapistmagazine.com/videos/view/zero-punctuation/10051-The-Order-1886-Review\?utm_source\=rss\&utm_medium\=rss\&utm_campaign\=videos                                                                 
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['--verbose', 'http://www.escapistmagazine.com/videos/view/zero-punctuation/10051-The-Order-1886-Review?utm_source=rss&utm_medium=rss&utm_campaign=videos']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2015.03.09
[debug] Python version 2.7.5+ - Linux-3.11-2-amd64-x86_64-with-LinuxMint-1-debian
[debug] exe versions: ffmpeg 2.1.1, ffprobe 2.1.1, rtmpdump 2.4
[debug] Proxy map: {}
[Escapist] 10051: Downloading webpage
[Escapist] 10051: Downloading normal configuration
[Escapist] 10051: Downloading hq configuration
ERROR: This IP address has been blocked by The Escapist
Traceback (most recent call last):
  File "/home/tails/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 649, in extract_info
    ie_result = ie.extract(url)
  File "/home/tails/bin/youtube-dl/youtube_dl/extractor/common.py", line 275, in extract
    return self._real_extract(url)
  File "/home/tails/bin/youtube-dl/youtube_dl/extractor/escapist.py", line 101, in _real_extract
    raise ExtractorError('This IP address has been blocked by The Escapist', expected=True)
ExtractorError: This IP address has been blocked by The Escapist

Now I get the standard 45 seconds trailer.

I tried downloading a video directly with an unbanned IP using Firefox (going through the config file and launching the download of the .mp4 file through Firefox), and it worked correctly. So there's something youtube-dl does that gets it detected...

I'm not sure it's related to the UA. Maybe it's because of the order of operations, or because it's missing cookies when making the request, or...No real idea, unfortunately making tests is complicated considering any mistake gets you banned and prevents you from trying again.

@terabyte & @achilleas-k First time it happened, while I knew what was up, I sent an email to the support team playing dumb and telling them I just got a standard 45 seconds trailer on each video. If you email them and tell them you have no idea why it happens, they seem to unban you. After a few exchanges, I got this answer and I got unbanned:

Looks like someone was caught using an unauthorized service to download our videos from your IP. We've removed the block but please note continued action will get you re-added to the block.

I can't do it again because they'll probably realize I'm messing with them, but if enough people do it maybe they'll think there's a false positive and stop doing that.

Of course you can also tell them the truth, but that probably won't get you unbanned.

[achilleas-k]

@Tailszefox I wouldn't be surprised if they knew about ytdl and this bug. Not being paranoid, but it's not exactly secret.

As for emailing support to get unbanned, I got banned again some time ago (well, more like a couple of years) for using an XBMC addon. I'm not going to bother getting my IP unbanned, this time. ZP is the only show left on the site that I watch regularly and I'll just keep to watching the YouTube version a week late.

[Ghoughpteighbteau]

Maybe the plugin should be disabled for the time being until we can determine what the ban is switching on?

edit:

Message from the developers:

" flashvars="config=You do know the advertisement views on these videos pay for the rent/insurance/food for several very nice people? A few of them sadly feel they need to eat every day, but we're working on curbing that unfortunate instinct. :("

hahahahahahahaha. remember. If you arn't willing submitting yourself to tracking/aggregation/malware vectors, gratuitous sexualization and pandering, gigantic background links designed to be accidentally clicked on, and annoying flashing advertisements, you are a monster.

[Tailszefox]

Wow. Talk about passive-aggressive.

I think the easiest way to determine what makes this happen would be to step through the code line by line and check, each time, if we've been banned. At some point the offending line should be somewhat obvious. I'll give it a go when I can, since I can easily change my IP.

EDIT: After testing real quick, and while I'll have to go deeper (literally), this is where the ban was triggered:

> /home/tails/youtube-dl/youtube-dl/youtube_dl/extractor/escapist.py(96)_real_extract()
-> _add_format('normal', config_url, quality=0)
(Pdb) n
[Escapist] 10051: Downloading normal configuration
> /home/tails/youtube-dl/youtube-dl/youtube_dl/extractor/escapist.py(97)_real_extract()
-> hq_url = (config_url +

So it happens even before the video starts getting downloaded, it actually happens when it downloads the configuration file to the video.

I'm thinking cookies, but I'll have to test.

EDIT: Okay, my bad, it's definitely because of the user agent. I've changed it to the one I'm currently using in Firefox and suddenly, it started downloading the video just fine. Have they really just banned an user agent? Could someone else try changing their user agent in escapist.py to check?

EDIT 2: Yep, it's the user agent. I tried a bunch of others, it went all okay. Tried the original: banned. Seriously?

[Tailszefox]

Any news from anyone else? I switched the user agent and I had no issue for a while, so it's definitely because of it.

Of course, this can just turn into a game of cat and mouse: if it's changed, they'll just ban it again. They're probably not basing themselves just on the UA, though, otherwise there would be a lot of false positive. But since we can't seem to figure out what...

My best guess at this point would be to randomly generate a valid UA for each request. That way they couldn't just ban a specific one.

[achilleas-k]

I just assumed they use a whitelist, not a blacklist.

[Tailszefox]

I don't think they're really using either, at least not based on the UA. The current UA used by youtube-dl is the one used by IE 11 on Windows 7. There's no way they would blacklist this specific UA, otherwise they'd just end up blacklisting a big chunk of their visitors. I even tried setting that UA on my Firefox and didn't get banned despite doing the same requests.