HD content (but not SD content) from BBC is broken #467
Comments
|
One final note: Audio downloads are also unaffected and work normally. It is only HD video that is failing, and that is 100% reproducible. |
|
I am having the same problem too (reported in BBC Downloads Fail). Just tried the SD format download and it worked as you described, but no luck with HD. |
|
Confirmed the same issue. In my case, after rtmpdump retries, GIA uses lots of memory and becomes unresponsive. If I run get_iplayer.pl manually, rtmpdump stops at exactly the same point and retries continually. |
|
I agree this can not be a proxy or outside UK issue. Try downloading older versions of Silent Witness from the GIA using "Use this current Webpage" and you can still get the downloads in HD. I am using a proxy and otherwise have not changed a thing - so the Beeb appear to have done something which appears to have happened between the 26th and 27th Jan since Part 1 of Silent Witness on the 26th comes down HD, Part 2 on the 27th refuses to even start with HD. Just an observation. Hope the team (who to be fair are doing a brilliant job with all of this) can find the solution. |
|
Actually, after a little more debugging, this absolutely is a "non-UK" issue. From the UK, it work's just fine. So, I guess the BBC are protecting their newer content more intensely with their CDNs. Can't blame them really, I suppose. |
So, geo-blocking per se isn't the whole story, but it seems clear to me that non-UK access is a primary factor. It may be narrower in scope than that since I would expect more complaints if this were affecting users on all non-UK networks. But whatever the BBC has done, it doesn't affect UK users. And don't be surprised that it affects only HD or only programmes broadcast after a certain date. Different resources are used for different programming, and the iPlayer production process changes from time to time. GiA and get_iplayer are both working exactly as they should. No change to code in GiA or get_iplayer could possibly affect the configuration or performance of the CDN servers. That doesn't even make sense. Neither GiA nor get_iplayer even connect to the CDN servers. HD downloads are the job of rtmpdump. From all reports to date, and my own testing, rtmpdump has worked just fine. It reports whatever error the server generates and exits. If the CDN servers won't serve the media streams, there is absolutely nothing rtmpdump can do about it. There is no magic bullet. Now, if you know some changes to rtmpdump that will get around the BBC changes, don't be stingy - let me see your code. I don't maintain rtmpdump, but I can incorporate a custom build in GiA if warranted. Although this report is a duplicate of an existing closed issue (#464), I'll leave it open through the weekend to avoid more repetition and to see if any useful information crops up. |
|
@stickenhoffen: The problem of runaway/hanging rtmpdump processes has been around a long time, on all platforms. Sometimes when rtmpdump is fed garbage by the CDN server, it goes haywire and either hangs or, more typically, starts eating CPU. It might be related to this HD download issue in your case, but isn't unique to it. Either way, nobody has yet come up with a solution. |
|
@dinkypumpkin many thanks for the clear details and explanation. Understand totally what you have written and will have to live with it. Just hope they don't take it any lower to the other formats. What you have here is a massive life line for many expats and I thank you for that! |
|
In case there's anything useful, I'd tried downloading episode 4 of Crims in HD, as the OP did, earlier today. First, with a UK VPN, download was successful. https://gist.github.com/mostlyotter/0adfc32b15cfa59e1d16 Next, with the provided proxy, download failed. https://gist.github.com/mostlyotter/f73e5f488d71bc6ec4c6 Finally, with a custom proxy on a friend's UK VPS, download failed. https://gist.github.com/mostlyotter/c6fbfb63c087142dddf4 Using GIA 1.3.7.1 with get_iplayer.pl updated manually on 28 January (and a few other changes that may well make no sense). See http://i.imgur.com/9ciWRpr.png |
|
mostlyotter, could you please contact me directly? I've added a public email address to my profile to make it possible. I have a question about your UK VPN provider, but don't want to misuse this issue thread. |
|
@mostlyotter: Thanks for those logs. They square with my observations, and provide a timely reminder of the difference between using a VPN vs. just a proxy or DNS diddler. |
|
Based on my experience with the iPlayer plugin for Kodi (XBMC); If you are outside the UK the only CDN source that works properly with proxies is LimeLight. Other's that aunty uses (Level3, Akamai) have geo blocks on the CDN itself that require a full VPN service (and no proxy) for GiA to work. I've never looked at the get_iplayer code in detail but I've noticed in GiA logs that the first attempt to download flashhd1 (or whatever it's called) always fails then flashhd2 succeeds. I presume that flashhd1 is a reference to L3 or Akamai and flashhd2 is LimeLight because I see the same behaviour with Kodi. The iPlayer plugin has CDN source as a selectable option so you can always use LimeLight to improve user experience. At the moment i'm still able to stream things via the Kodi plugin and a private self-hosted proxy service (although stream performance from the UAE has been bad in recent weeks) while in GiA the download starts and then fails after 10-15 seconds before cycling through a resume failure (as others have described). |
|
@chewitt: I think you're still getting Akamai/Level3 streams for HD on-demand in Kodi even if you have Limelight set as your preference. If you're bored one day, make a Kodi debug log for playback of an on-demand HD programme and look for a Limelight stream. If such exists, I would definitely like to know about it. To my knowledge, it's never served up to get_player or Kodi iPlayer add-on, at least not in the UK. |
|
There are now at least some recent programmes whose Level3 HD streams also fail from within the UK. I suspect this is cock-up rather conspiracy since the Level3 streams I tested also broke playback on the iPlayer site. So, whatever afflicts Level3 would seem to be the ultimate villain of the piece. Again, there is nothing GiA can do about that. UK users should still get HD programmes via Akamai even if Level3 is playing up. |
|
@dinkypumpkin: I did some playing about and in the iplayer plugin the logic for stream size appears to be applied before selecting stream source. With auto for size and Limelight for stream it will still serve the 2800 stream for a current TopGear episode (available on AK and L3) from L3 ..which plays for about 15 seconds before it craps out (consistently at the same point). If I drop the stream size to 1500 (available on LL) it plays from LL and the entire show plays without stopping (the odd "buffering" but that's normal where I am). Original advice from the plugin author was to set LL when "abroad" ..I'll have to go ask if the logic is deliberate or got lost in a refactoring at some point. |
|
@chewitt: The plugin CDN setting is a preference rather than a requirement. It makes sense to me that bitrate would trump CDN preference since, technically, the CDN links could change at any time (they have to be discovered dynamically). It has presumably never been much of an issue before the recent L3 problems. AK has been geo-blocking for a while, but L3 would have taken up the slack for HD streams. |
|
@dinkypumpkin You mention that AK has been geo-blocking for a while. As far as I know, they all geo-block, so what is AK doing differently than LL and L3? Will it still work with a proxy? Presumably it will work with a VPN connection, right? Are AK (Akamai), LL (Limelight), and L3 (?) all of the CDNs, or is there a list somewhere to consult? If any of these would successfully download HD content under current circumstances, is there any chance of prioritizing an enhancement request to make it possible to specify the CDN in GiA's preferences (since get_iplayer apparently supports it)? Sorry for all the questions. Just trying to find a way to solve this. I'm now paying for a VPN account, which is mostly working, but it takes the machine offline for some other purposes, which is highly problematic. I miss a functional GiA, a lot. |
|
For HD, only AK and L3 are used AFAIK, and L3 is HD-only. As for what AK does differently, all I can tell you is that their Flash media servers are known to reject non-UK HD connections, as was demonstrated by the logs posted in #464. An HTTP proxy won't help. I don't know to what extent AK blocks VPNs, only that some have been blocked in the past. If your VPN doesn't support RTMP streaming, it won't matter anyway. VPNs that provide access to the iPlayer web site don't necessarily work with get_iplayer, HD in particular. Specifying the CDN won't help. As I've said, the ultimate problem appears to be that L3 is broken/unreliable. if Akamai blocks you, and L3 doesn't work properly, there isn't much you can do. |
|
Thanks. Experimenting with command-line get_iplayer today it appears that you can't actually specify the CDN anyway. You can use the --tvmode parameter and add a "1" or a "2" to flashhd, but which is which is not guaranteed to be stable. Also, through a proxy both were failing, though in different ways. One said "ERROR: Closing connection: NetStream.Play.StreamNotFound". The other said "ERROR: rtmp server sent error" and Through a VPN I still got the latter response from one server. The other connected and started to stream, but failed after about 1600 KB. It kept restarting and getting a couple of KB farther, but that's all. It was consistent about always adding a very few KB, but never much more. Just to sanity check I went back to GiA at various stages, and it was behaving the same, including the frequent restarts when it could connect, until, possibly coincidentally, I dropped then restarted the VPN, and after one of the frequent restarts the download continued to work. It's now 50% done and I'm vaguely hopeful. What a drag. I'd come to really rely on GiA for my BBC (and ITV) fix. But I realize this is happening well below GiA, and probably below get_iplayer. I see a "WARNING: Larger timestamp than 24-bit: 0xffffffea" immediately before all these restarts, which I'm guessing is coming from rtmpdump. Someone on a Linux & Unix troubleshooting forum thought that might be salient. He had a nice, verbose log if you want to stare at it: |
|
I already saw that SP post. There wasn't enough information there to draw any conclusions except that patching rtmpdump makes no difference, as you surmised. The "Larger timestamp" warning seems to be more symptom than cause since the server has already signalled stream stop at that point. The patch doesn't appear to take that into account. Few people will even see the (possibly spurious) warning since the patch only went into rtmpdump a month ago. Ignore "Invalid HLS playlist" warnings. get_iplayer hasn't yet caught up with all BBC changes, but that's irrelevant. GiA doesn't use the HLS streams. |
|
I posted that message, I didn't thought the "Larger timestamp" warning was relevant but I genuinely thought it was a bug in rtmpdump, hence the (futile) attempt with the latest version. |
|
I do think there's a bug in rtmpdump, but, yeah, that's not it. Or perhaps more a design flaw than a bug, proper. And it's not entirely clear if the fix should be in rtmpdump or get_iplayer, but even with the other problems the CDNs are currently presenting, a malformed buffer should not be written to disk. Currently it's this unresumable file problem that is making it impossible for repeated restarts to perhaps chunk along making progress. No guarantees that would be a useful workaround, but as long as these bad files are being written we'll never know. I started poking around a little bit and see how the CDN is sending what at least is being interpreted as a Play.Complete or Play.Stop, and it's presumably then that a final bad buffer is being written. Although I suppose it could be worse, and the buffer just prior to the termination could have been bad. But if the exact heuristic currently being used to identify the file as unresumable was applied to each block before it was written, perhaps at least this particular piece of the problem could be fixed. I doubt seriously that I'm going to have time to solve this, but will try to steal some time to poke around a little bit more. |
|
BBC updated their backend CDN handling a month ago. If you have a long latency VPN or Proxy then get_iplayer will quickly switch down from hd to sd in a matter of about one or two seconds after testing all available resolution types within a split second. To elaborate just a little further since I'm not a programmer, once get_iplayer has performed a lookup for the program ID there's a point where it just sits there and probably talks to the flash server through a reverse proxy to authenticate itself before RTMP dump starts. When RTMPdump begins I used to be able to kill the proxy/VPN and it would then quit its connection to flashhd1 and switch to flashhd2 ultimately allowing me to grab the 720p file without using an unreliable UK VPN. Now the switch between available resolutions happens in a split second and it quickly drops down to the lowest SD version. Not killing the proxy/VPN connection during the download typically doesn't cause issues. Of course, the latency still needs to be low. Probably no higher than 150ms. At 200ms the connection becomes unstable and get_iplayer will stop and start and ultimately quit if if happens more than once or twice. I've tried more than half a dozen so called "UK VPNs" and they all suck because they really weren't setup for streaming media even though they specifically advertise as such. VYPRvpn being one of the worst. Guess I'll have have to suck it up and pay for a UK VPS. I just wish the BBC would syndicate it's documentaries in the US on BBC America or sell it all on itunes then I'd pay for it and never worry about get_iplayer. Their loss.... |
|
I am no longer able to download anything from BBC - all downloads fail immediately. Changing the BBC formats in preferences has no effect. This started at around 4PM Central Time in the USA on 26 FEB. |
|
I'm still able to download formats other than HD. I just tested this by downloading the latest Weekly Wipe in Flash - Very High. I hope to find it somewhere else in HD to actually watch, but the download worked. I've also been able to download programs from ITV in Flash - High. But HD is well and truly dead, even when using a for-pay VPN service. Planning to investigate VPS services now, sigh. |
|
I've seen a report from a user of the iplayer plugin for xbmc (which, if I'm not mistaken, uses rtmpdump) that has the same problem even in the UK. |
|
Oh my gosh, they fixed it. I've just watched GiA download 5 HD programs, WITHOUT A SINGLE RESTART! I don't think I've ever seen it get through even a single episode without restarting a few times, and certainly not multiple episodes. So whatever they did fixed not only the hard failures, but the soft ones as well. They must be doing some buffering or timing differently or honoring acks or throttling requests or some darn thing, and it is a huge win, now that it works at all. Please let it stay this way. |
|
yes it does appear they updated it except now I am receiving a new error (most likely associated with a VPN): video:982020kB audio:39921kB subtitle:0 data:0 global headers:0kB muxing overhead 0.224642% INFO: Downloaded Thumbnail to 'C:\Users\my_computer\Desktop\iPlayer Recordings\Clima INFO: Command exit code 1 (raw code = 256) |
|
FWIW, just to test it, I downloaded the Climate Change - A Horizon Guide show, in HD, without a problem. I'm using a for-pay VPN rather than a proxy, in case that makes a difference (though you mention a VPN as well). I'm using BolehVPN, and highly recommend them. I only tried one other VPN service, but it was unstable and slow. BolehVPN is extremely stable and fast. And very important for me, besides the more traditional, whole-machine VPN service, they provide a local proxy service to access their VPN... It's unique amongst VPN providers, to my knowledge. What it lets you do is point GiA (or your browser or whatever), at a local proxy address (10.10.10.1:808) which serves as the VPN conduit to their server, in whatever location you choose. Meanwhile, the rest of the machine's network is completely unaffected, which is especially handy for me, given that the machine that runs GiA is also my main web server, and I need it online and accessible externally 24/7. And on the service front, when I first tested them they had this local-proxy service up and running in many countries, but not yet in the UK. They got right on it, at my request, and brought the service up in the UK in a couple of days. It works a treat. How's that for responsive service? (I have nothing to do with them besides being a happy customer.) |
|
I'll have to look into that larryy. thanks for the heads up and yes I have been paying for most VPN/Proxies I've tested, but not tried yours yet. I've been testing get_iplayer for the last 24hrs and it appears get_iplayer has no issues aside from thumbnail image tagging of MP4. However I notice the PVR recursive command doesn't work yet. If there are multiple shows in one series then using the web pvr's service and get_iplayer's recursive command doesn't appear to work. web pvr manager .\get_iplayer.cgi.cmd example result: ERROR: RTMP_Connect1, handshake failed. full result: |
|
I'm using a free vpn limited to 300MB daily just for the bbc network. Since it's only needed to obtain the metadata I need just a fraction of the daily allowance, the cdn is on a different network and can be accessed directly. |
|
Interesting, oliviuca. So what's your standard workflow? Do you manually bring up the VPN, relaunch GiA to get it to update is indexes, do your searching and adding to the queue, then manually bring down the VPN, and start GiA's download queue? |
|
I don't use GiA, just get_iplayer, but what I do should work even with GiA (though I see that it provides its own proxy, so this shouldn't be really necessary). |
|
Thanks, makes perfect sense. |
|
@larryy thanks for the BolehVPN recommendation. I signed up and used larryy as the referrer username, which the service appeared to recognize as yourself. I don't mean to hijack this issue but I'm curious about how you're using the local proxy address feature. I think my use case is the same as the issue described here, where a proxy isn't enough and a VPN is required but I also don't want to route all traffic to the VPN. My attempts to use any of the servers prefixed with "Proxied-" and route traffic to the local IP address produce the same results as a regular proxy server. |
|
@paulyoung You're welcome, but I'm afraid things have changed since I wrote that, gosh, 7 months ago. I used BolehVPN in the mode I described with great success for many months. All I did was bring up the "Proxied-UK" connection, tell GiA to use a proxy of 10.10.10.1:808, and everything worked great. That proxy let GiA talk to the BBC network for show info, and even though Akamai had been geo-blocking for a long time, Level3 wasn't, so I could download the shows I wanted to watch in HD. (The flash/RTMP stream carrying the video doesn't go through the proxy, but it was okay as long as Level3 was serving HD content without geo-blocking.) SD shows didn't work without a full-network, "xCloakRouted-UK" connection, but I mostly didn't care about those. Then a month or so back, Level3 started geo-blocking flash/RMTP streams, and everything broke. I could still enable whole-machine VPN and download, but I can't leave this machine like that. I was thinking about bringing Boleh and GiA up on another machine, that I could leave in full xCloakRouted-UK mode, when someone came up with a workaround... switch to HLS streams. That ruled out using GiA, so I started using get_iplayer directly, from a tiny script I threw together. I'd hand copy the PIDs into the script and kick it off to do downloads. This worked reasonably well until just about a week ago. Then about a week ago, Auntie got mean, in multiple ways at once... On the same day, I think, they geo-blocked all HLS streams, and, the worst bit of all, blacklisted various VPN providers, including BolehVPN. So even a full, xCloakRouted-UK, whole-machine VPN connection no longer works, for flash/RTMP or HLS streams. I was completely dead in the water. There's an article about this crackdown on VPN providers here: https://www.vpncompare.co.uk/bbc-iplayer-cracks-down-and-blocks-vpn-users/ I have since found a VPN provider that has not been blacklisted, and set things up on a machine that can stay fully VPN-routed and am back to using GiA. Someone also contacted me directly, because of something I posted elsewhere, to suggest another VPN provider that is known to work, and that is quite a bit cheaper than the one I found. I'm not going to mention these other providers in a public forum, because I really need them to continue to work, but I've enabled my email address to show on my profile temporarily, and you can email me directly if you want suggestions. There's also (not great, but mildly hopeful) news about Boleh I'll share privately. It's so depressing and such a hassle that they're working so hard to keep us from watching their shows. I would happily pay a license fee and do all this completely above board if they would just let me. But greed, regional copyrights, and too many lawyers mean it is highly unlikely in my lifetime, sigh. BBC America just isn't a useful alternative. Sure, they get Doctor Who. Yay! But they'll never bring over QI, Have I Got News for You, Mock the Week, Weekly Wipe, 10 O'Clock Live, and myriad others that are deemed to be either too risqué, too UK-specific, or both, but which are some of my favorite shows, period. |
|
just as @larryy I'm using a vpn, and just like him I won't say its name in a public forum. In my case it's a free vpn: slow as hell but I don't mind since I just use it for get_iplayer (thanks to linux network namespaces). |
|
The article in VPNCompare linked above (and also in the get_iplayer mailing list) seems to refer to streaming video from the BBC iPlayer site, not to using get_iplayer to download video. I use two different commercial VPN providers: one for PPTP VPN and one for openvpn. Both continue to work perfectly well for streaming from iPlayer. Both have also continued to allow GiA to download. For a few days, downloads were unusually slow and subject to far more frequent reinitializing. However, I assume VPN IP address blacklists or other geo-blocking would block access rather than reduce speed. My approach to GiA over the past few years has been been to wait and try failed downloads again in a day or so. This may not be technically advanced, but often works. All VPNs I've tried have been fastest at around 8 AM UK time and limped along during evening UK hours. I've no idea how much my east coast US location or local cable internet service may affect this. When using GiA with a proxy, speed was much faster and far more consistent. VPNCompare may well be one of many sites that essentially advertise VPNs. In any case, I tried their fastest-rated UK VPN, LiquidVPN, and found it to be the most useless VPN I've ever tried. YMMV but don't give LiquidVPN much money based on that review. I'd urge anyone to look for VPN reviews in an expat forum or other source that doesn't accept VPN advertisements. If not, start with VPNs that offer free trials with no payment. |
As of about five days ago all HD content from BBC has been broken. SD content (vhigh, high) continues to work normally. Typical failure mode is for the download stream to either fail immediately or after a short time, and attempted restarts always fail.
The problem has occurred with 10.9.5 both before and after the recent security update. The problem occurs with GiA 1.8.4, 1.8.3, and 1.7.2, with identical symptoms. The problem occurs whether one is using a commercial proxy or GiA's current default proxy.
It would appear that something about HD streams has changed and GiA can no longer download them.
After the failure, the error diagnosis reported is just "Problem Unknown. Please submit a bug report from the application menu." (Which I did, but saw no followup.) Here is the complete log for a typical failed attempt:
Get iPlayer Automator 1.8.4 Initialized.
Updating Program Index Feeds from Server...
Retrieving tv index feeds.
Retrieving itv index feeds.
Retrieving radio index feeds.
Retrieving podcast index feeds.
AppController: Index Updated.
INFO: Loading proxy settings...
INFO: Loading provided proxy (may take up to 20 seconds)...
INFO: Proxy load complete.
INFO: Using proxy: http://82.146.147.112:80
AppController: Starting Downloads
Downloading Show 1/1:
BBC Download (ID=372): Downloading Crims - Episode 4
INFO: Using Proxy http://82.146.147.112:80
372: Crims - Episode 4, BBC Three, Comedy,Sign Zone,Sitcoms, default,signed
INFO: 1 Matching Programmes
INFO: Checking existence of default version
INFO: flashhd2,flashhd1 modes will be tried for version default
INFO: Trying flashhd2 mode to record tv: Crims - 4. Episode 4
INFO: File name prefix = Crims - 4. Episode 4 ((flashhd))
RTMPDump v2.4-77-gdc76f0a
(c) 2010 Andrej Stepanchuk, Howard Chu, The Flvstreamer Team; license: GPL
Connecting ...
INFO: Connected...
INFO: Command exit code 1 (raw code = 256)
WARNING: Failed to stream file /Volumes/video/downloads/Get iPlayer Downloads/Crims/Crims - 4. Episode 4 ((flashhd)).partial.mp4.flv via RTMP
INFO: skipping flashhd2 mode
INFO: Trying flashhd1 mode to record tv: Crims - 4. Episode 4
INFO: File name prefix = Crims - 4. Episode 4 ((flashhd))
RTMPDump v2.4-77-gdc76f0a
(c) 2010 Andrej Stepanchuk, Howard Chu, The Flvstreamer Team; license: GPL
Connecting ...
INFO: Connected...
INFO: Command exit code 1 (raw code = 256)
WARNING: Failed to stream file /Volumes/video/downloads/Get iPlayer Downloads/Crims/Crims - 4. Episode 4 ((flashhd)).partial.mp4.flv via RTMP
INFO: skipping flashhd1 mode
ERROR: Failed to record 'Crims - 4. Episode 4 (b050qq66)'
BBC Download (ID=372): Crims - Episode 4 Failed
AppController: Downloads Finished
It only tries HD because I have removed the SD formats, on purpose. If SD formats are present GiA will fall back to vhigh or high and succeed, and thus think it has successfully completed the download. Accordingly, some people may fail to notice the problem until they look closely at their downloaded files.
If there's any chance of getting this fixed before current shows on iPlayer expire it would be much appreciated.
The text was updated successfully, but these errors were encountered: