Skip to content

/ youtube-dl

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

When downloading YouTube live video, download from very beginning #21255

Open
fireattack opened this issue May 30, 2019 · 10 comments
Open

When downloading YouTube live video, download from very beginning #21255

fireattack opened this issue May 30, 2019 · 10 comments
Labels
request

Comments

@fireattack
Copy link

@fireattack fireattack commented May 30, 2019 •
edited

Checklist

  • I'm reporting a feature request
  • I've verified that I'm running youtube-dl version 2019.05.20
  • I've searched the bugtracker for similar feature requests including closed ones

Description

Currently, when downloading an ongoing live stream video on YouTube, youtube-dl will start from the current time.

I was wondering if we can add a way to download it from the very beginning instead. (I can confirm it is technical-wise possible at least, the video sources are available in m3u8 files etc.).

This way, the user can get the full video once the stream is finished. Otherwise, at least the first few seconds will be missing since you can't really start downloading before the live starts.

Of course, you can start the download when the live is done, which typically will solve this problem. However, some uploaders delete their live stream immediately after premiere, makes it somewhat difficult (also if you wait too long, you can't get HLS version any more, only re-compressed version.)

Thanks.
@fireattack fireattack added the request label May 30, 2019
@fireattack
Copy link
Author

@fireattack fireattack commented May 30, 2019 •
edited

I did a quick test myself again.

  1. I use webcam to start a quick live stream (by using https://www.youtube.com/livestreaming/) at https://www.youtube.com/watch?v=12REdrcaP1w.
  2. I waited for 1-2 min and then start to download it with youtube-dl.
  3. I waited a few minutes, and then terminated my stream. The youtube-dl download session ended later automatically.

Part of the log:

[hls,applehttp @ 00000000004c3500] Opening 'https://r1---sn-n0qqxoapo3-jaal.googlevideo.com/videoplayback/id/12REdrcaP1w.0/itag/96/source/yt_live_broadcast/requiressl/yes/ratebypass/yes/live/1/goi/160/sgoap/gir%3Dyes%3Bitag%3D140/sgovp/gir%3Dyes%3Bitag%3D137/hls_chunk_host/r1---sn-n0qqxoapo3-jaal.googlevideo.com/playlist_type/DVR/ei/V1jwXJvgFM_5igSbkIaYAw/gcr/us/initcwndbps/12270/mm/44/mn/sn-n0qqxoapo3-jaal/ms/lva/mv/m/pl/22/keepalive/yes/mt/1559255077/disable_polymer/true/ip/75.111.128.184/ipbits/0/expire/1559276727/sparams/ip,ipbits,expire,id,itag,source,requiressl,ratebypass,live,goi,sgoap,sgovp,hls_chunk_host,playlist_type,ei,gcr,initcwndbps,mm,mn,ms,mv,pl/signature/4A019262A7752F272786E5CA8B0A65680D1B855C.3E1B47BD68DB818609D3E6C57B3DE4FD350640DF/key/dg_yt0/playlist/index.m3u8/sq/293/goap/clen%3D22961%3Blmt%3D1559255013176245/govp/clen%3D64154%3Blmt%3D1559255013176251/dur/1.000/file/seg.ts' for reading
[https @ 0000000003c5be00] Opening 'https://manifest.googlevideo.com/api/manifest/hls_playlist/id/12REdrcaP1w.0/itag/96/source/yt_live_broadcast/requiressl/yes/ratebypass/yes/live/1/goi/160/sgoap/gir%3Dyes%3Bitag%3D140/sgovp/gir%3Dyes%3Bitag%3D137/hls_chunk_host/r1---sn-n0qqxoapo3-jaal.googlevideo.com/playlist_type/DVR/ei/V1jwXJvgFM_5igSbkIaYAw/gcr/us/initcwndbps/12270/mm/44/mn/sn-n0qqxoapo3-jaal/ms/lva/mv/m/pl/22/dover/11/keepalive/yes/mt/1559255077/disable_polymer/true/ip/75.111.128.184/ipbits/0/expire/1559276727/sparams/ip,ipbits,expire,id,itag,source,requiressl,ratebypass,live,goi,sgoap,sgovp,hls_chunk_host,playlist_type,ei,gcr,initcwndbps,mm,mn,ms,mv,pl/signature/4A019262A7752F272786E5CA8B0A65680D1B855C.3E1B47BD68DB818609D3E6C57B3DE4FD350640DF/key/dg_yt0/playlist/index.m3u8' for reading
[https @ 00000000039345c0] Cannot reuse HTTP connection for different host: r5---sn-q4fl6nlz.googlevideo.com:-1 != r1---sn-n0qqxoapo3-jaal.googlevideo.com:-1
[hls,applehttp @ 00000000004c3500] keepalive request failed for 'https://r1---sn-n0qqxoapo3-jaal.googlevideo.com/videoplayback/id/12REdrcaP1w.0/itag/96/source/yt_live_broadcast/requiressl/yes/ratebypass/yes/live/1/goi/160/sgoap/gir%3Dyes%3Bitag%3D140/sgovp/gir%3Dyes%3Bitag%3D137/hls_chunk_host/r1---sn-n0qqxoapo3-jaal.googlevideo.com/playlist_type/DVR/ei/V1jwXJvgFM_5igSbkIaYAw/gcr/us/initcwndbps/12270/mm/44/mn/sn-n0qqxoapo3-jaal/ms/lva/mv/m/pl/22/keepalive/yes/mt/1559255077/disable_polymer/true/ip/75.111.128.184/ipbits/0/expire/1559276727/sparams/ip,ipbits,expire,id,itag,source,requiressl,ratebypass,live,goi,sgoap,sgovp,hls_chunk_host,playlist_type,ei,gcr,initcwndbps,mm,mn,ms,mv,pl/signature/4A019262A7752F272786E5CA8B0A65680D1B855C.3E1B47BD68DB818609D3E6C57B3DE4FD350640DF/key/dg_yt0/playlist/index.m3u8/sq/294/goap/clen%3D22981%3Blmt%3D1559255013176256/govp/clen%3D64017%3Blmt%3D1559255013176258/dur/1.000/file/seg.ts', retrying with new connection: Invalid argument
[hls,applehttp @ 00000000004c3500] Opening 'https://r1---sn-n0qqxoapo3-jaal.googlevideo.com/videoplayback/id/12REdrcaP1w.0/itag/96/source/yt_live_broadcast/requiressl/yes/ratebypass/yes/live/1/goi/160/sgoap/gir%3Dyes%3Bitag%3D140/sgovp/gir%3Dyes%3Bitag%3D137/hls_chunk_host/r1---sn-n0qqxoapo3-jaal.googlevideo.com/playlist_type/DVR/ei/V1jwXJvgFM_5igSbkIaYAw/gcr/us/initcwndbps/12270/mm/44/mn/sn-n0qqxoapo3-jaal/ms/lva/mv/m/pl/22/keepalive/yes/mt/1559255077/disable_polymer/true/ip/75.111.128.184/ipbits/0/expire/1559276727/sparams/ip,ipbits,expire,id,itag,source,requiressl,ratebypass,live,goi,sgoap,sgovp,hls_chunk_host,playlist_type,ei,gcr,initcwndbps,mm,mn,ms,mv,pl/signature/4A019262A7752F272786E5CA8B0A65680D1B855C.3E1B47BD68DB818609D3E6C57B3DE4FD350640DF/key/dg_yt0/playlist/index.m3u8/sq/294/goap/clen%3D22981%3Blmt%3D1559255013176256/govp/clen%3D64017%3Blmt%3D1559255013176258/dur/1.000/file/seg.ts' for reading
frame= 5400 fps= 30 q=-1.0 Lsize=   13487kB time=00:02:59.97 bitrate= 613.9kbits/s speed=1.01x
video:10515kB audio:2870kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.756449%
[ffmpeg] Downloaded 13810230 bytes
[download] 100% of 13.17MiB in 02:59

The video file I downloaded is only 3min:

General
Complete name                            : D:\test-12REdrcaP1w.mp4
Format                                   : MPEG-4
Format profile                           : Base Media
Codec ID                                 : isom (isom/iso2/avc1/mp41)
File size                                : 13.2 MiB
Duration                                 : 3 min 0 s
Overall bit rate                         : 614 kb/s
Writing application                      : Lavf58.20.100

The stream, if recorded from very beginning, should be around 4:55 as you can see in above link.
@fireattack
Copy link
Author

@fireattack fireattack commented May 30, 2019 •
edited

Also by simply looking at the network traffic, you can see youtube-dl will only fetch the new video segments everytime one comes out (instead of trying to catch up all the available segments from the very beginning, which would have a much higher download speed).
@fireattack
Copy link
Author

@fireattack fireattack commented May 30, 2019 •
edited

Another thing I want to mention is that if you stop the download manually by ctrl-C it will generate a broken (merged) mp4 file. So you have to wait for the stream to end itself, which makes testing hard.

This doesn't seem to happen if you download a normal video from YouTube: you can always interrupt the download and the part file (after rename) can be played just fine.
@ealgase
Copy link
Contributor

@ealgase ealgase commented Jun 3, 2019

Another thing I want to mention is that if you stop the download manually by ctrl-C it will generate a broken (merged) mp4 file. So you have to wait for the stream to end itself, which makes testing hard.

This doesn't seem to happen if you download a normal video from YouTube: you can always interrupt the download and the part file (after rename) can be played just fine.

Cancelling with CTRL+C works for me.
@fireattack
Copy link
Author

@fireattack fireattack commented Jun 3, 2019 •
edited

Another thing I want to mention is that if you stop the download manually by ctrl-C it will generate a broken (merged) mp4 file. So you have to wait for the stream to end itself, which makes testing hard.
This doesn't seem to happen if you download a normal video from YouTube: you can always interrupt the download and the part file (after rename) can be played just fine.

Cancelling with CTRL+C works for me.

Did you try on a ongoing live stream? It works on normal video but not live stream here.

I just tried again with this (currently ongoing): https://www.youtube.com/watch?v=QWTNIDZw608

The result is the same. I'm on Windows 7 64 bit with newest FFMPEG.

Logs

D:\>youtube-dl https://www.youtube.com/watch?v=QWTNIDZw608 -v
[debug] System config: []
[debug] User config: ['--write-description', '--write-info-json', '-i', '-4', '-o', '%(upload_date)s %(title)s-%(extractor)s-%(id)
s.%(ext)s']
[debug] Custom config: []
[debug] Command-line args: ['https://www.youtube.com/watch?v=QWTNIDZw608', '-v']
[debug] Encodings: locale cp936, fs mbcs, out cp936, pref cp936
[debug] youtube-dl version 2019.05.20
[debug] Python version 3.4.4 (CPython) - Windows-7-6.1.7601-SP1
[debug] exe versions: ffmpeg 4.1.3, ffprobe 4.1.3
[debug] Proxy map: {}
[youtube] QWTNIDZw608: Downloading webpage
[youtube] QWTNIDZw608: Downloading video info webpage
[youtube] QWTNIDZw608: Downloading m3u8 information
[youtube] QWTNIDZw608: Downloading MPD manifest
[debug] Default format spec: best/bestvideo+bestaudio
[info] Writing video description to: 20190603 Euronews English Live-youtube-QWTNIDZw608.description
[info] Writing video description metadata as JSON to: 20190603 Euronews English Live-youtube-QWTNIDZw608.info.json
[debug] Invoking downloader on 'https://manifest.googlevideo.com/api/manifest/hls_playlist/id/QWTNIDZw608.0/itag/95/source/yt_live_broadcast/requiressl/yes/ratebypass/yes/live/1/goi/160/sgoap/gir%3Dyes%3Bitag%3D140/sgovp/gir%3Dyes%3Bitag%3D136/hls_chunk_host/r4---sn-n0qqxoapo3-jaal.googlevideo.com/playlist_type/DVR/ei/rnf1XN7gL8PADvW3ulg/gcr/us/initcwndbps/14880/mm/44/mn/sn-n0qqxoapo3-jaal/ms/lva/mv/m/pl/22/dover/11/keepalive/yes/mt/1559590708/disable_polymer/true/ip/75.111.128.184/ipbits/0/expire/1559612430/sparams/ip,ipbits,expire,id,itag,source,requiressl,ratebypass,live,goi,sgoap,sgovp,hls_chunk_host,playlist_type,ei,gcr,initcwndbps,mm,mn,ms,mv,pl/signature/8B4CA4AFEE8D643727BF21EDE5097841DD9E61A7.87DE91740939E27D5DB59FF0F9C372C3ECBE9052/key/dg_yt0/playlist/index.m3u8'
[download] Destination: 20190603 Euronews English Live-youtube-QWTNIDZw608.mp4
[debug] ffmpeg command line: ffmpeg -y -loglevel verbose -headers "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept-Language: en-us,en;q=0.5
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
" -i "https://manifest.googlevideo.com/api/manifest/hls_playlist/id/QWTNIDZw608.0/itag/95/source/yt_live_broadcast/requiressl/yes/ratebypass/yes/live/1/goi/160/sgoap/gir%3Dyes%3Bitag%3D140/sgovp/gir%3Dyes%3Bitag%3D136/hls_chunk_host/r4---sn-n0qqxoapo3-jaal.googlevideo.com/playlist_type/DVR/ei/rnf1XN7gL8PADvW3ulg/gcr/us/initcwndbps/14880/mm/44/mn/sn-n0qqxoapo3-jaal/ms/lva/mv/m/pl/22/dover/11/keepalive/yes/mt/1559590708/disable_polymer/true/ip/75.111.128.184/ipbits/0/expire/1559612430/sparams/ip,ipbits,expire,id,itag,source,requiressl,ratebypass,live,goi,sgoap,sgovp,hls_chunk_host,playlist_type,ei,gcr,initcwndbps,mm,mn,ms,mv,pl/signature/8B4CA4AFEE8D643727BF21EDE5097841DD9E61A7.87DE91740939E27D5DB59FF0F9C372C3ECBE9052/key/dg_yt0/playlist/index.m3u8" -c copy -f mp4 "file:20190603 Euronews English Live-youtube-QWTNIDZw608.mp4.part"
[ffmpeg] Interrupted by user

[ffmpeg] Downloaded 4194352 bytes

[download] 100% of 4.00MiB in 00:13

Resulted video:
20190603 Euronews English Live-youtube-QWTNIDZw608.zip

(Unplayable on any player I have, including MPV and MPC-HC).
@ealgase
Copy link
Contributor

@ealgase ealgase commented Jun 3, 2019

Try it on Linux?
@fireattack
Copy link
Author

@fireattack fireattack commented Jun 3, 2019

I don't use Linux, but I may give it a try later in VM.
@fireattack
Copy link
Author

@fireattack fireattack commented Jun 9, 2019

Try it on Linux?

Hi, just tested in a Ubuntu 19 VM. Ctrl+C indeed doesn't generate broken file like on Windows (tested on Win 7 and Win 10).

But the main issue remains: it doesn't download from the very beginning on Linux either.
@fireattack fireattack mentioned this issue Jun 9, 2019
Closed
6 of 6 tasks complete
@ealgase
Copy link
Contributor

@ealgase ealgase commented Jun 9, 2019

Yeah, it doesn't download from the beginning, I was just troubleshooting the other issue.
@Bangaio65
Copy link

@Bangaio65 Bangaio65 commented Jun 22, 2019 •
edited

I don't think you can do it with just youtube-dl, but you can using ffmpeg.

Caveats:

  • as far as I know, youtube-dl can only grab the HLS (.m3u8) manifest. Some livestreams let you rewind up to 12 hours on the website (as far as I cant tell that is using the MPEG Dash (.mpd) manifest) but only 2 or 4 hours with the HLS stream. I don't think ffmpeg can correctly parse an .mpd manifest as of yet, at least last time I checked. And I don't know if youtube-dl can even get the DASH manifest.
  • Some livestreams don't support rewinding

Instructions:

  1. Grab the HLS manifest url using youtube-dl (add -f best if you want the best quality)
    youtube-dl --no-warnings -f "best" -g -- <youtube-url>

  2. Figure out how many seconds is each segment (it's usually 2 or 5) and convert seconds to segments.
    If you're familiar with video editing, this is the time between keyframes.
    Say you want to go back 1 hour and the segment duration is 2 seconds. Then you do (1 * 60 * 60) / 2 = 1800. You need to "go back" 1800 segments on the stream.
    If you just need to go back to the start of the stream, just input a large number on the next step.

  3. Download using ffmpeg

  • add -t <seconds or HH:MM:SS> to specify a duration
  • it's recommended to save as .ts just in case the connection dies in the middle and you're left with an unusable .mp4
    ffmpeg -live_start_index -1800 -i <m3u8_url> -t 600 -c copy livestream01.ts
    or
    ffmpeg -live_start_index -1800 -i <m3u8_url> -t 600 -bsf:a aac_adtstoasc -c copy livestream01.mp4

Hope this helps.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
request
Projects
None yet
Milestone
No milestone
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
@fireattack @ealgase @Bangaio65
You can’t perform that action at this time.