Can't download recorded video from ustream

[calexicoz]

Hi,

I'm trying to download a video from the Github Universe conference, but getting an error:

$ youtube-dl -v "http://www.ustream.tv/recorded/91343263"
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-v', u'http://www.ustream.tv/recorded/91343263']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.09.18
[debug] Python version 2.7.12 - Darwin-16.0.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 3.1.3, ffprobe 3.1.3
[debug] Proxy map: {}
[ustream] 91343263: Downloading JSON metadata
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/local/bin/youtube-dl/__main__.py", line 19, in <module>
  File "/usr/local/bin/youtube-dl/youtube_dl/__init__.py", line 449, in main
  File "/usr/local/bin/youtube-dl/youtube_dl/__init__.py", line 439, in _real_main
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1791, in download
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 694, in extract_info
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 355, in extract
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/ustream.py", line 90, in _real_extract
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 994, in _sort_formats
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 949, in _formats_key
  File "/usr/local/bin/youtube-dl/youtube_dl/utils.py", line 2274, in determine_protocol
AttributeError: 'NoneType' object has no attribute 'startswith'

Thanks!

[yan12125]

DASH like streams. Fragment URLs are fetched via websocket.

[remitamine]

can be downloaded using HLS manifest: http://c742ef02-lp-omega.ums.ustream.tv/playlist/recorded/91343263/playlist.m3u8?token=1137814_1474223185739&appType=11&appVersion=2&ts=1474223185&cdn=uhs_akamai&sgn=1161eeb9687505a80fdc33995aa9fd2a565258eb

[yan12125]

403 for me. How did you get the HLS manifest URL?

[remitamine]

youtube-dl 'http://c742ef4c-lp-omega.ums.ustream.tv/playlist/recorded/91343263/playlist.m3u8?token=1157125_1474223224941&appType=11&appVersion=2&ts=1474223226&cdn=uhs_akamai&sgn=fbaa0761d51a4caff6c28ab0a204104addc37021'
[generic] playlist: Requesting header
[generic] playlist: Downloading m3u8 information
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 1092
[download] Destination: playlist-playlist.mp4
[download]   0.5% of ~192.20MiB at 150.98KiB/s ETA 52:21

from http://c742ef02-lp-omega.ums.ustream.tv/1/ustream?connectionId=vsjsp9df12je62ipg1639kokim-2

[yan12125]

Well, it says "Invalid connectionId" from my side. Which browser/user-agent are you using? I always get segmented mp4 streams in Firefox and Chrome.

[remitamine]

Firefox 48.0.2(flash disabled), i see this requests in the network traffic.

[yan12125]

I was trying on Firefox Nightly, but I still get segmented mp4 on Firefox 48.0.2. Maybe A/B testing.

[remitamine]

i also see the segmented mp4 requests after the request that contain the hls url, so the hls url is not used by default may be as fallback.

[yan12125]

Found no requests with "connectionId". They don't want to serve me HLS streams at all.

[remitamine]

the dash segments are also served for me in a similar request: http://c742ef4c-lp-omega.ums.ustream.tv/1/ustream?connectionId=gs41rbodprl44q2pbtpkr572e4-8.

[yan12125]

I don't know why they always use websockets here. If you are interested in fixing this, could you have a look? I can't write a patch without introducing websocket dependencies.

[remitamine]

i guess the requests with connectionId is a representation of websocket requests in the network tab, they happen after the switch to websocket protocol.

[yan12125]

Dunno. Chrome can dump websocket frames, and I didn't see requests with connectionId either.

[remitamine]

there is also a fallback to long polling when websocket is not supported.

[yan12125]

Finally get plain GET requests after blocking outgoing traffic via port 1935 in my firewall. Data are the same as those appeared in websocket frames. Thanks @remitamine.

[yan12125]

Thanks for the report. New ustream videos will be recognized in the next version.

[calexicoz]

Awesome, thanks!