Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[twitter] Support live streams #10806

Closed
ugukkylbklaom opened this issue Sep 30, 2016 · 19 comments
Closed

[twitter] Support live streams #10806

ugukkylbklaom opened this issue Sep 30, 2016 · 19 comments
Labels

Comments

@ugukkylbklaom
Copy link

@ugukkylbklaom ugukkylbklaom commented Sep 30, 2016

Please follow the guide below

  • You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
  • Put an x into all the boxes [ ] relevant to your issue (like that [x])
  • Use Preview tab to see how your issue will actually look like

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2016.09.27. If it's not read this FAQ entry and update. Issues with outdated version will be rejected.

  • I've verified and I assure that I'm running youtube-dl 2016.09.27

Before submitting an issue make sure you have:

  • At least skimmed through README and most notably FAQ and BUGS sections
  • Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

  • [X ] Bug report (encountered problems with youtube-dl)
  • Site support request (request for adding support for a new site)
  • Feature request (request for a new functionality)
  • Question
  • Other

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue


If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

Add -v flag to your command line you run youtube-dl with, copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):

proxychains4 youtube-dl -v https://hlslive-llc.med1.m.lv.twimg.com/ls01/twitter/2016/09/27/TWITTER_VIDEO_bloomberg_debate_1_20160927/3500K/3500_slide_twv1.m3u8
[proxychains] config file found: /usr/local/Cellar/proxychains-ng/4.7/etc/proxychains.conf
[proxychains] preloading /usr/local/Cellar/proxychains-ng/4.7/lib/libproxychains4.dylib
[proxychains] DLL init
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-v', u'https://hlslive-llc.med1.m.lv.twimg.com/ls01/twitter/2016/09/27/TWITTER_VIDEO_bloomberg_debate_1_20160927/3500K/3500_slide_twv1.m3u8']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.09.27
[proxychains] DLL init
[debug] Python version 2.7.9 - Darwin-14.5.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 3.0.1-tessus
[debug] Proxy map: {}
[generic] 3500_slide_twv1: Requesting header
[proxychains] Strict chain  ...  127.0.0.1:1080  ...  hlslive-llc.med1.m.lv.twimg.com:443  ...  OK
[generic] 3500_slide_twv1: Downloading m3u8 information
[proxychains] Strict chain  ...  127.0.0.1:1080  ...  hlslive-llc.med1.m.lv.twimg.com:443  ...  OK
[debug] Invoking downloader on u'https://hlslive-llc.med1.m.lv.twimg.com/ls01/twitter/2016/09/27/TWITTER_VIDEO_bloomberg_debate_1_20160927/3500K/3500_slide_twv1.m3u8'
[download] Destination: 3500_slide_twv1-3500_slide_twv1.mp4
[debug] ffmpeg command line: ffmpeg -y -headers 'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip, deflate
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/47.0 (Chrome)
' -i https://hlslive-llc.med1.m.lv.twimg.com/ls01/twitter/2016/09/27/TWITTER_VIDEO_bloomberg_debate_1_20160927/3500K/3500_slide_twv1.m3u8 -c copy -f mp4 -bsf:a aac_adtstoasc file:3500_slide_twv1-3500_slide_twv1.mp4.part
[proxychains] DLL init
ffmpeg version 3.0.1-tessus Copyright (c) 2000-2016 the FFmpeg developers
  built with Apple LLVM version 6.0 (clang-600.0.57) (based on LLVM 3.5svn)
  configuration: --cc=/usr/bin/clang --prefix=/opt/ffmpeg --as=yasm --extra-version=tessus --enable-avisynth --enable-fontconfig --enable-gpl --enable-libass --enable-libbluray --enable-libfreetype --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopus --enable-libschroedinger --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libvidstab --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzmq --enable-version3 --disable-ffplay --disable-indev=qtkit --disable-indev=x11grab_xcb
  libavutil      55. 17.103 / 55. 17.103
  libavcodec     57. 24.102 / 57. 24.102
  libavformat    57. 25.100 / 57. 25.100
  libavdevice    57.  0.101 / 57.  0.101
  libavfilter     6. 31.100 /  6. 31.100
  libswscale      4.  0.100 /  4.  0.100
  libswresample   2.  0.101 /  2.  0.101
  libpostproc    54.  0.100 / 54.  0.100
[proxychains] Strict chain  ...  127.0.0.1:1080  ...  hlslive-llc.med1.m.lv.twimg.com:443  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:1080  ...  media-api.m.lv.twimg.com:443  ...  OK
Unable to open key file https://media-api.m.lv.twimg.com/programs/38129399/keys/id/20186121
[proxychains] Strict chain  ...  127.0.0.1:1080  ...  hlslive-llc.med1.m.lv.twimg.com:443  ...  OK
[hls,applehttp @ 0x7fe651805600] Error when loading first segment 'https://hlslive-llc.med1.m.lv.twimg.com/ls01/twitter/2016/09/27/TWITTER_VIDEO_bloomberg_debate_1_20160927/3500K/03/21/54.ts'
https://hlslive-llc.med1.m.lv.twimg.com/ls01/twitter/2016/09/27/TWITTER_VIDEO_bloomberg_debate_1_20160927/3500K/3500_slide_twv1.m3u8: Invalid data found when processing input


ERROR: ffmpeg exited with code 1
  File "/Library/Frameworks/Python.framework/Versions/2.7/bin/youtube-dl", line 11, in <module>
    sys.exit(main())
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/youtube_dl/__init__.py", line 449, in main
    _real_main(argv)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/youtube_dl/__init__.py", line 439, in _real_main
    retcode = ydl.download(all_urls)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/youtube_dl/YoutubeDL.py", line 1791, in download
    url, force_generic_extractor=self.params.get('force_generic_extractor', False))
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/youtube_dl/YoutubeDL.py", line 705, in extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/youtube_dl/YoutubeDL.py", line 751, in process_ie_result
    return self.process_video_result(ie_result, download=download)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/youtube_dl/YoutubeDL.py", line 1435, in process_video_result
    self.process_info(new_info)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/youtube_dl/YoutubeDL.py", line 1699, in process_info
    success = dl(filename, info_dict)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/youtube_dl/YoutubeDL.py", line 1641, in dl
    return fd.download(name, info)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/youtube_dl/downloader/common.py", line 354, in download
    return self.real_download(filename, info_dict)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/youtube_dl/downloader/external.py", line 43, in real_download
    self.get_basename(), retval))
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/youtube_dl/downloader/common.py", line 162, in report_error
    self.ydl.report_error(*args, **kargs)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/youtube_dl/YoutubeDL.py", line 568, in report_error
    self.trouble(error_message, tb)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/youtube_dl/YoutubeDL.py", line 530, in trouble
    tb_data = traceback.format_list(traceback.extract_stack())

@yan12125
Copy link
Collaborator

@yan12125 yan12125 commented Sep 30, 2016

It's common that a standalone m3u8 file does not work. Could you share where you get this m3u8 link?

@yan12125
Copy link
Collaborator

@yan12125 yan12125 commented Sep 30, 2016

Seems Twitter uses exotic encryption to prevent downloading. Things are not clear until the next live show.

About the video on Bloomberg: apparently they have re-encoded the video.

@yan12125 yan12125 changed the title [error] Can't download m3u8 from twitter [twitter] Support live streams Sep 30, 2016
@yan12125
Copy link
Collaborator

@yan12125 yan12125 commented Sep 30, 2016

Is it on Twitter too? I didn't follow US presidential election.

@ugukkylbklaom
Copy link
Author

@ugukkylbklaom ugukkylbklaom commented Sep 30, 2016

@yan12125
Thank you for your replay.

I extracted m3u8 address from https://mobile.twitter.com/i/live/778347749217406976 on Oct 26, 2016.
When debate was on live I can't download either.

Bit rate of the video is 3.5 m/s,but bit rate is 2.4 M/s in that video that I downloaded from http://www.bloomberg.com/politics/videos/2016-09-27/the-complete-first-2016-presidential-debate

@ugukkylbklaom
Copy link
Author

@ugukkylbklaom ugukkylbklaom commented Sep 30, 2016

http://www.uspresidentialelectionnews.com/2016-debate-schedule/2016-presidential-debate-schedule/

YEAH same ADDRESS https://mobile.twitter.com/i/live/778347749217406976 bloomberg share live source with twitter.

ON Wednesday, October 19, 2016, THE HOST IS Chris Wallace !!!

@ugukkylbklaom
Copy link
Author

@ugukkylbklaom ugukkylbklaom commented Oct 1, 2016

@yan12125
exotic encryption? I didn't watch live on https://mobile.twitter.com/i/live/778347749217406976
I extracted the m3u8 address. Then I tried to download it. I failed. I tried to use vlc. I failed. But I successfully watch live using this https://www.hlsplayer.net/ it works.

I hope this may help

@yan12125
Copy link
Collaborator

@yan12125 yan12125 commented Oct 1, 2016

ffmpeg and mpv can't handle this m3u8, either, so I call it "exotic".

Does hlsplayer.net work for finished debates? I paste https://hlslive-llc.med1.m.lv.twimg.com/ls01/twitter/2016/09/27/TWITTER_VIDEO_bloomberg_debate_1_20160927/3500K/3500_slide_twv1.m3u8 to it and nothing happens after clicking "Play".

@ugukkylbklaom
Copy link
Author

@ugukkylbklaom ugukkylbklaom commented Oct 1, 2016

NO. It worked only when the debate was on LIve

@yan12125
Copy link
Collaborator

@yan12125 yan12125 commented Oct 1, 2016

Got it. So we can only wait.

@ugukkylbklaom
Copy link
Author

@ugukkylbklaom ugukkylbklaom commented Oct 5, 2016

@yan12125 hey, Vice presidential debate is on live.

proxychains4 youtube-dl -v https://hlslive-llc.med2.m.lv.twimg.com/ls01/twitter/2016/10/05/TWITTER_VIDEO_bloomberg_debate_2_20161005/3500K/3500_slide_twv1.m3u8
[proxychains] config file found: /usr/local/Cellar/proxychains-ng/4.7/etc/proxychains.conf
[proxychains] preloading /usr/local/Cellar/proxychains-ng/4.7/lib/libproxychains4.dylib
[proxychains] DLL init
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-v', u'https://hlslive-llc.med2.m.lv.twimg.com/ls01/twitter/2016/10/05/TWITTER_VIDEO_bloomberg_debate_2_20161005/3500K/3500_slide_twv1.m3u8']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.10.02
[proxychains] DLL init
[debug] Python version 2.7.9 - Darwin-14.5.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 3.0.1-tessus
[debug] Proxy map: {}
[generic] 3500_slide_twv1: Requesting header
[proxychains] Strict chain  ...  127.0.0.1:1080  ...  hlslive-llc.med2.m.lv.twimg.com:443  ...  OK
[generic] 3500_slide_twv1: Downloading m3u8 information
[proxychains] Strict chain  ...  127.0.0.1:1080  ...  hlslive-llc.med2.m.lv.twimg.com:443  ...  OK
[debug] Invoking downloader on u'https://hlslive-llc.med2.m.lv.twimg.com/ls01/twitter/2016/10/05/TWITTER_VIDEO_bloomberg_debate_2_20161005/3500K/3500_slide_twv1.m3u8'
[download] Destination: 3500_slide_twv1-3500_slide_twv1.mp4
[debug] ffmpeg command line: ffmpeg -y -headers 'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip, deflate
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/47.0 (Chrome)
' -i https://hlslive-llc.med2.m.lv.twimg.com/ls01/twitter/2016/10/05/TWITTER_VIDEO_bloomberg_debate_2_20161005/3500K/3500_slide_twv1.m3u8 -c copy -f mp4 -bsf:a aac_adtstoasc file:3500_slide_twv1-3500_slide_twv1.mp4.part
[proxychains] DLL init
ffmpeg version 3.0.1-tessus Copyright (c) 2000-2016 the FFmpeg developers
  built with Apple LLVM version 6.0 (clang-600.0.57) (based on LLVM 3.5svn)
  configuration: --cc=/usr/bin/clang --prefix=/opt/ffmpeg --as=yasm --extra-version=tessus --enable-avisynth --enable-fontconfig --enable-gpl --enable-libass --enable-libbluray --enable-libfreetype --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopus --enable-libschroedinger --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libvidstab --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzmq --enable-version3 --disable-ffplay --disable-indev=qtkit --disable-indev=x11grab_xcb
  libavutil      55. 17.103 / 55. 17.103
  libavcodec     57. 24.102 / 57. 24.102
  libavformat    57. 25.100 / 57. 25.100
  libavdevice    57.  0.101 / 57.  0.101
  libavfilter     6. 31.100 /  6. 31.100
  libswscale      4.  0.100 /  4.  0.100
  libswresample   2.  0.101 /  2.  0.101
  libpostproc    54.  0.100 / 54.  0.100
[proxychains] Strict chain  ...  127.0.0.1:1080  ...  hlslive-llc.med2.m.lv.twimg.com:443  ...  OK
[proxychains] Strict chain  ...  127.0.0.1:1080  ...  media-api.m.lv.twimg.com:443  ...  OK
Unable to open key file https://media-api.m.lv.twimg.com/programs/38136099/keys/id/20190054
[proxychains] Strict chain  ...  127.0.0.1:1080  ...  hlslive-llc.med2.m.lv.twimg.com:443  ...  OK
[hls,applehttp @ 0x7fd6f9801000] Error when loading first segment 'https://hlslive-llc.med2.m.lv.twimg.com/ls01/twitter/2016/10/05/TWITTER_VIDEO_bloomberg_debate_2_20161005/3500K/02/20/45.ts'
https://hlslive-llc.med2.m.lv.twimg.com/ls01/twitter/2016/10/05/TWITTER_VIDEO_bloomberg_debate_2_20161005/3500K/3500_slide_twv1.m3u8: Invalid data found when processing input


ERROR: ffmpeg exited with code 1
  File "/Library/Frameworks/Python.framework/Versions/2.7/bin/youtube-dl", line 11, in <module>
    sys.exit(main())
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/youtube_dl/__init__.py", line 445, in main
    _real_main(argv)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/youtube_dl/__init__.py", line 435, in _real_main
    retcode = ydl.download(all_urls)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/youtube_dl/YoutubeDL.py", line 1791, in download
    url, force_generic_extractor=self.params.get('force_generic_extractor', False))
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/youtube_dl/YoutubeDL.py", line 705, in extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/youtube_dl/YoutubeDL.py", line 751, in process_ie_result
    return self.process_video_result(ie_result, download=download)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/youtube_dl/YoutubeDL.py", line 1435, in process_video_result
    self.process_info(new_info)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/youtube_dl/YoutubeDL.py", line 1699, in process_info
    success = dl(filename, info_dict)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/youtube_dl/YoutubeDL.py", line 1641, in dl
    return fd.download(name, info)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/youtube_dl/downloader/common.py", line 354, in download
    return self.real_download(filename, info_dict)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/youtube_dl/downloader/external.py", line 43, in real_download
    self.get_basename(), retval))
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/youtube_dl/downloader/common.py", line 162, in report_error
    self.ydl.report_error(*args, **kargs)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/youtube_dl/YoutubeDL.py", line 568, in report_error
    self.trouble(error_message, tb)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/youtube_dl/YoutubeDL.py", line 530, in trouble
    tb_data = traceback.format_list(traceback.extract_stack())

https://hlslive-llc.med2.m.lv.twimg.com/ls01/twitter/2016/10/05/TWITTER_VIDEO_bloomberg_debate_2_20161005/3500K/3500_slide_twv1.m3u8

https://www.hlsplayer.net/
https://i.imgur.com/Tah8Mz5.png

@yan12125
Copy link
Collaborator

@yan12125 yan12125 commented Oct 5, 2016

I was sleeping then lol
Does Twitter have a list of current live streams like https://www.youtube.com/channel/UC4R8DWoMoI7CAwX8_LjQHig?

@ugukkylbklaom
Copy link
Author

@ugukkylbklaom ugukkylbklaom commented Oct 9, 2016

@yan12125 @remitamine
HEY, dude. the second debate is coming

Sunday, October 9, 2016
Second presidential debate
Time: 9pm ET (8pm CT, 7pm MT, 6pm PT)

live streaming: https://mobile.twitter.com/i/live/778347749217406976

@montanaflynn
Copy link

@montanaflynn montanaflynn commented Oct 11, 2016

@yan12125

There's a live stream going on right now:

https://twitter.com/i/live/783724492577054720

I copied the HTML source code from the page:

https://gist.github.com/montanaflynn/61db8f0560388ed2bd9e8755bf669925

@yan12125
Copy link
Collaborator

@yan12125 yan12125 commented Oct 12, 2016

Chasing others' live streams is impractical due to time zone difference. Is there a way for ordinary Twitter users to create live streams?

@montanaflynn
Copy link

@montanaflynn montanaflynn commented Oct 12, 2016

Unfortunately I don't think so.

I copied the source code for the HTML page, is there anything else that would be helpful?

@yan12125
Copy link
Collaborator

@yan12125 yan12125 commented Oct 12, 2016

If you know the use how to use debuggers, a dump of HLS key requests would be useful.

@mreweilk
Copy link

@mreweilk mreweilk commented Oct 14, 2016

Noticed this issue when trying to watch the NFL game: https://twitter.com/i/live/768633364911788032

When you press play the page will make a request to "https://twitter.com/i/live_video_stream/status/786690271966670848?client=web" which will have a json respondse that includes a url that looks like this "https://t.lv.twimg.com/live_video_stream/authorized_status/786690271966670848/LIVE_PUBLIC/CuriomDUMAAiisH?url=https://hlslive-llcg5.med2.m.lv.twimg.com/ls01/twitter/2016/10/14/TWITTER_VIDEO_cbs_broncos_at_chargers_20161014/master_wired_web_twv1.m3u8&ctx=7_786690271966670848:blahblahblahverylongbase64&evt=38143199"

When the player requests this url it will set a Authorization cookie which is needed to grab the key.
The key request itself looks like this:

GET /programs/38143199/keys/id/20192878 HTTP/1.1
Host: media-api.m.lv.twimg.com
Connection: keep-alive
Origin: https://twitter.com
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.59 Safari/537.36
Accept: */*
DNT: 1
Referer: https://twitter.com/i/videos/live_video/768633364911788032?embed_source=clientlib&player_id=0&rpc_init=1
Accept-Encoding: gzip, deflate, sdch, br
Accept-Language: en-US,en;q=0.8
Cookie: Authorization=eyJraWQiOiJw....

I think this should be enough information to make this work. It seems some programs insert ads so maybe those could be ignored somehow?

@odeke-em
Copy link

@odeke-em odeke-em commented Dec 23, 2016

So Twitter now supports live for anyone https://twitter.com/twitter/status/809035848138178562 and https://blog.twitter.com/2016/go-live-on-twitter. Now we'll have more test streams.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants
You can’t perform that action at this time.