livestream.com not working issue

[puncy]

Hi,

livestream.com seem to have a new way of parsing the URL for some of their videos.

could you possibly take a look at

http://livestream.com/nrl/stormdragons2016

the error im getting is:

C:\Users\admin\Desktop\New folder (2)>youtube-dl.exe http://livestream.com/nrl/stormdragons2016
[livestream] Downloading JSON metadata
[livestream] 4865403: Downloading page 1
[livestream] 4865403: Downloading page 2
[download] Downloading playlist: Storm vs. Dragons
[livestream] playlist Storm vs. Dragons: Collected 1 video ids (downloading 1 of them)
[download] Downloading video 1 of 1
[livestream] 114681772: Downloading JSON metadata
[livestream] 114681772: Downloading SMIL file
Traceback (most recent call last):
File "main.py", line 19, in
File "youtube_dl__init__.pyo", line 412, in main
File "youtube_dl__init__.pyo", line 402, in _real_main
File "youtube_dl\YoutubeDL.pyo", line 1718, in download
File "youtube_dl\YoutubeDL.pyo", line 679, in extract_info
File "youtube_dl\YoutubeDL.pyo", line 839, in process_ie_result
File "youtube_dl\YoutubeDL.pyo", line 731, in process_ie_result
File "youtube_dl\YoutubeDL.pyo", line 679, in extract_info
File "youtube_dl\YoutubeDL.pyo", line 724, in process_ie_result
File "youtube_dl\YoutubeDL.pyo", line 1364, in process_video_result
File "youtube_dl\YoutubeDL.pyo", line 1626, in process_info
File "youtube_dl\YoutubeDL.pyo", line 1568, in dl
File "youtube_dl\downloader\common.pyo", line 344, in download
File "youtube_dl\downloader\http.pyo", line 58, in real_download
File "youtube_dl\YoutubeDL.pyo", line 1931, in urlopen
File "urllib2.pyo", line 431, in open
File "urllib2.pyo", line 449, in _open
File "urllib2.pyo", line 409, in _call_chain
File "youtube_dl\utils.pyo", line 719, in http_open
File "urllib2.pyo", line 1194, in do_open
File "httplib.pyo", line 1053, in request
File "httplib.pyo", line 1093, in _send_request
File "httplib.pyo", line 1049, in endheaders
File "httplib.pyo", line 893, in _send_output
File "httplib.pyo", line 855, in send
File "httplib.pyo", line 832, in connect
File "socket.pyo", line 557, in create_connection
File "encodings\idna.pyo", line 164, in encode
File "encodings\idna.pyo", line 73, in ToASCII
UnicodeError: label empty or too long

thanks

[puncy]

using -v

C:\Users\admin\Desktop\nrldump>youtube-dl.exe -v http://livestream.com/nrl/raiderspanthers2016
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-v', u'http://livestream.com/nrl/raiderspanthers2016']
[debug] Encodings: locale cp1252, fs mbcs, out cp850, pref cp1252
[debug] youtube-dl version 2016.03.06
[debug] Python version 2.7.10 - Windows-8-6.2.9200
[debug] exe versions: rtmpdump 2.4
[debug] Proxy map: {}
[livestream] Downloading JSON metadata
[livestream] 4865342: Downloading page 1
[livestream] 4865342: Downloading page 2
[download] Downloading playlist: Raiders vs. Panthers
[livestream] playlist Raiders vs. Panthers: Collected 1 video ids (downloading 1 of them)
[download] Downloading video 1 of 1
[livestream] 114326130: Downloading JSON metadata
[livestream] 114326130: Downloading SMIL file
[debug] Invoking downloader on u'http://livestream-f.akamaihd.net3161248_4865342_01e81ad1cfe44d13a2e9052261e0d9e1_1_2320@103172?v=3.0.3&fp=WIN%2014,0,0,145&ssek=1131'
Traceback (most recent call last):
File "main.py", line 19, in
File "youtube_dl__init__.pyo", line 412, in main
File "youtube_dl__init__.pyo", line 402, in _real_main
File "youtube_dl\YoutubeDL.pyo", line 1718, in download
File "youtube_dl\YoutubeDL.pyo", line 679, in extract_info
File "youtube_dl\YoutubeDL.pyo", line 839, in process_ie_result
File "youtube_dl\YoutubeDL.pyo", line 731, in process_ie_result
File "youtube_dl\YoutubeDL.pyo", line 679, in extract_info
File "youtube_dl\YoutubeDL.pyo", line 724, in process_ie_result
File "youtube_dl\YoutubeDL.pyo", line 1364, in process_video_result
File "youtube_dl\YoutubeDL.pyo", line 1626, in process_info
File "youtube_dl\YoutubeDL.pyo", line 1568, in dl
File "youtube_dl\downloader\common.pyo", line 344, in download
File "youtube_dl\downloader\http.pyo", line 58, in real_download
File "youtube_dl\YoutubeDL.pyo", line 1931, in urlopen
File "urllib2.pyo", line 431, in open
File "urllib2.pyo", line 449, in _open
File "urllib2.pyo", line 409, in _call_chain
File "youtube_dl\utils.pyo", line 719, in http_open
File "urllib2.pyo", line 1194, in do_open
File "httplib.pyo", line 1053, in request
File "httplib.pyo", line 1093, in _send_request
File "httplib.pyo", line 1049, in endheaders
File "httplib.pyo", line 893, in _send_output
File "httplib.pyo", line 855, in send
File "httplib.pyo", line 832, in connect
File "socket.pyo", line 557, in create_connection
File "encodings\idna.pyo", line 164, in encode
File "encodings\idna.pyo", line 73, in ToASCII
UnicodeError: label empty or too long

[Ranutso]

I believe issue #8797 I've just opened is a clone of this one. It is indeed a problem with the URL, there is a forward slash missing after akamaihd.net. I've tested coping yt-dl's URL and fixing it manually, it works.

[puncy]

Smart boy ranutso

I didn't even notice that.

Did you just copy that URL with the additional / and put it in your download program and it worked?

[Ranutso]

It is not working as intended for NRL videos, I am not sure about other videos. If I use wget or curl the connection is closed (by the server) always at the same offset and neither wget nor curl seem to be able to continue from that point. So I always end up downloading the exact same 10 seconds video. Youtube-dl must do some magic behind the scenes, so I would appreciate if they fixed it.

[puncy]

"If I use wget or curl the connection is closed (by the server) always at the same offset and neither wget nor curl seem to be able to continue from that point."

yep exactly the same here, with any type of downloader, always at the same point in the stream it cuts out :(

[Ranutso]

So, I believe this is something you'll see in the future. I've applied the livestream.py patch myself on the current build and it now downloads. However, not even youtube-dl can work its magic on the FLV file, unfortunately. It is returning the same exact error as curl or wget: "content too short". :(

$ ./yt-dl -v "http://livestream.com/nrl/roostersrabbitohs2016"
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-v', u'http://livestream.com/nrl/roostersrabbitohs2016']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.03.06
[debug] Python version 2.7.11 - Darwin-15.3.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 3.0, ffprobe 3.0, rtmpdump 2.4
[debug] Proxy map: {}
[livestream] Downloading JSON metadata
[livestream] 4865369: Downloading page 1
[livestream] 4865369: Downloading page 2
[download] Downloading playlist: Roosters vs. Rabbitohs
[livestream] playlist Roosters vs. Rabbitohs: Collected 1 video ids (downloading 1 of them)
[download] Downloading video 1 of 1
[livestream] 114483875: Downloading JSON metadata
[livestream] 114483875: Downloading SMIL file
[debug] Invoking downloader on u'http://livestream-f.akamaihd.net/3161248_4865369_a3721ed24b4a4f7fa6e960bba2bb2218_1_2320@103172?fp=WIN%25+14%2C0%2C0%2C145&v=3.0.3&ssek=1598'
[download] Destination: Roosters vs Rabbitohs-114483875.flv
[download]   0.1% of 2.00GiB at 121.00KiB/s ETA 04:48:42ERROR: content too short (expected 2147483647 bytes and served 1234019)
Traceback (most recent call last):
  File "./yt-dl/youtube_dl/YoutubeDL.py", line 1626, in process_info
    success = dl(filename, info_dict)
  File "./yt-dl/youtube_dl/YoutubeDL.py", line 1568, in dl
    return fd.download(name, info)
  File "./yt-dl/youtube_dl/downloader/common.py", line 344, in download
    return self.real_download(filename, info_dict)
  File "./yt-dl/youtube_dl/downloader/http.py", line 236, in real_download
    raise ContentTooShortError(byte_counter, int(data_en))
ContentTooShortError

I think this might have something to do with Akamai's download protection. If it detects it is not a Flash player downloading it, it aborts the connection. So currently some of our NRL games in Livestream are unwatchable (since they won't play on the official iOS app either, that's great). What an amazing service Livestream. Amazing!

[puncy]

Damn that sucks ranutso.

If your looking at just viewing the game have you tried "livestreamer"

I'm actually trying to download them to keep offline

[Ranutso]

I really can't download those FLVs. I was having a look at the headers the official player send to the server and tried reproducing them with curl, to no avail. These are the file request headers (youtube-dl doesn't set any of them).

GET /3161248_4865369_a3721ed24b4a4f7fa6e960bba2bb2218_1_2320@103172?v=3.6.0&fp=MAC%2021,0,0,182&r=GXQMJ&g=XDFCSAUTIFPW&seek=1598 HTTP/1.1
Host: livestream-f.akamaihd.net
Connection: keep-alive
X-Requested-With: ShockwaveFlash/21.0.0.182
Cache-Control: no-cache
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36
Accept: */*
Referer: http://livestream.com/nrl/roostersrabbitohs2016
Accept-Encoding: gzip, deflate, such
Accept-Language: en-US,en;q=0.8,pt-BR;q=0.6,pt;q=0.4,es;q=0.2

I thought the 'Connection: keep-alive' would do it, but it doesn't. The server will always drop the connection at the exact same offset (or at the same byte).

[puncy]

Ranutso,

looks like this round of videos is does work by using the new version. its pissing me off that they keep changing it every couple of games.

give the knights and raiders round 3 match a try and see if you can get it to work, i can.

[Ranutso]

Yes, they are working. If the game has been published in HLS it works, if the game has been published in Flash only, it doesn't. I believe only some of the games in the first round weren't published in HLS. We'll see.

[puncy]

Ranutso,

do you know how to get them to download quicker through youtube-dl? im stuck downloading at 100k/sec when i can download from other places at over 1mb/sec

[Ranutso]

No, that's not youtube-dl's issue, this is their CDN. Same as the "Content too short" error, not youtube-dl's fault.

I believe their CDN limits the download speed to this bandwidth because I get the same speed as you. That is more than enough for streaming that video, so why would the CDN spend more money on faster downloads?

[puncy]

so they are working for now, but when i finish the download a get a message saying something about the audio and to use ffmpeg or avconv to fix

the files still play in VLC though so i guess ill leave them be

[puncy]

Ranutso,

after about 4 hours of searching the internet about these new ways of splitting the video for the stream (split it into over 1000 parts) i have found another project being worked on on github and i can now download the files at 1mb/sec using 8 parrallel download streams.

let me know if you want some more info on it.

it took me a while to set it up but once setup it is by far the easiest and quickest way to download them

[JaKXz]

$ youtube-dl http://livestream.com/UWFeds/GM2016/videos/116754333
[livestream] 116754333: Downloading JSON metadata
[livestream] 116754333: Downloading SMIL file
[livestream] 116754333: Downloading m3u8 information
[livestream] 116754333: Downloading f4m manifest
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 2185
[download] Destination: March General...-116754333.mp4
[download]  15.1% of ~3.36GiB at Unknown speed ETA Unknown ETAERROR: giving up after 10 retries

I have a similar issue, a completed video stream [with the video ID] is not working 😞

Anyone have any workarounds?

[Ranutso]

Thanks, puncy, I am good. This is called HLS, it splits the video into small segments. As far as the audio goes, no worries, youtube-dl fixes it for you.
I have no hurry downloading these files as I can leave them downloading all day long. So I put them to download in the morning and I can watch the show at night. It works for me. :)