[reddit] add support for downloading a subreddit as a playlist #18389

ealgase · 2018-12-04T02:34:23Z

Please follow the guide below

You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
Put an x into all the boxes [ ] relevant to your issue (like this: [x])
Use the Preview tab to see what your issue will actually look like

Make sure you are using the latest version: run `youtube-dl --version` and ensure your version is 2018.12.03. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

I've verified and I assure that I'm running youtube-dl 2018.12.03

Before submitting an issue make sure you have:

At least skimmed through the README, most notably the FAQ and BUGS sections
Searched the bugtracker for similar issues including closed ones
Checked that provided video/audio/playlist URLs (if any) are alive and playable in a browser

What is the purpose of your issue?

Bug report (encountered problems with youtube-dl)
Site support request (request for adding support for a new site)
Feature request (request for a new functionality)
Question
Other

If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

Add the -v flag to your command line you run youtube-dl with (youtube-dl -v <your command line>), copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):

(xenial)ealgase@localhost:~$ youtube-dl https://www.reddit.com/r/all -v > redditlog.txt
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'https://www.reddit.com/r/all', u'-v']
[debug] Encodings: locale UTF-8, fs UTF-8, out None, pref UTF-8
[debug] youtube-dl version 2018.12.03
[debug] Python version 2.7.12 (CPython) - Linux-3.14.0-x86_64-with-Ubuntu-16.04-xenial
[debug] exe versions: ffmpeg 2.8.15-0ubuntu0.16.04.1, ffprobe 2.8.15-0ubuntu0.16.04.1
[debug] Proxy map: {}
WARNING: Falling back on generic information extractor.
ffmpeg version 2.8.15-0ubuntu0.16.04.1 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.10) 20160609
  configuration: --prefix=/usr --extra-version=0ubuntu0.16.04.1 --build-suffix=-ffmpeg --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --cc=cc --cxx=g++ --enable-gpl --enable-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-libx264 --enable-libopencv
  libavutil      54. 31.100 / 54. 31.100
  libavcodec     56. 60.100 / 56. 60.100
  libavformat    56. 40.101 / 56. 40.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 40.101 /  5. 40.101
  libavresample   2.  1.  0 /  2.  1.  0
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.101 /  1.  2.101
  libpostproc    53.  3.100 / 53.  3.100
[hls,applehttp @ 0xf0a3e0] HLS request for url 'https://v.redd.it/jneg6lmn2y121/HLS_1_0_M.ts', offset 0, playlist 0
[h264 @ 0xf11940] Current profile doesn't provide more RBSP data in PPS, skipping
Input #0, hls,applehttp, from 'https://v.redd.it/jneg6lmn2y121/HLS_1_0_M_v4.m3u8':
  Duration: 00:00:40.00, start: 1.466667, bitrate: 0 kb/s
  Program 0 
    Metadata:
      variant_bitrate : 0
    Stream #0:0: Video: h264 (Main), 4 reference frames ([27][0][0][0] / 0x001B), yuv420p(tv, bt709), 640x334 (640x336) [SAR 3841:3840 DAR 23:12], 30 fps, 30 tbr, 90k tbn, 60 tbc
[mp4 @ 0x132eae0] Codec for stream 0 does not use global headers but container format requires global headers
Output #0, mp4, to 'file:reddit - the front page of the internet (1)-all-1.fhls-1409.mp4.part':
  Metadata:
    encoder         : Lavf56.40.101
    Stream #0:0: Video: h264, 1 reference frame ([33][0][0][0] / 0x0021), yuv420p, 640x334 (0x0) [SAR 3841:3840 DAR 23:12], q=2-31, 30 fps, 30 tbr, 90k tbn, 90k tbc
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
Press [q] to stop, [?] for help
[NULL @ 0xf11940] Current profile doesn't provide more RBSP data in PPS, skipping
[tls @ 0xf124c0] The request is invalid.
[hls,applehttp @ 0xf0a3e0] HLS request for url 'https://v.redd.it/jneg6lmn2y121/HLS_1_0_M.ts', offset 919884, playlist 0
[NULL @ 0xf11940] Current profile doesn't provide more RBSP data in PPS, skipping
    Last message repeated 1 times
[tls @ 0x130cd60] The request is invalid.
[hls,applehttp @ 0xf0a3e0] HLS request for url 'https://v.redd.it/jneg6lmn2y121/HLS_1_0_M.ts', offset 1840896, playlist 0
[NULL @ 0xf11940] Current profile doesn't provide more RBSP data in PPS, skipping    
    Last message repeated 1 times
[tls @ 0x132f900] The request is invalid.
[hls,applehttp @ 0xf0a3e0] HLS request for url 'https://v.redd.it/jneg6lmn2y121/HLS_1_0_M.ts', offset 2521456, playlist 0
[NULL @ 0xf11940] Current profile doesn't provide more RBSP data in PPS, skipping
    Last message repeated 1 times
[tls @ 0xf127e0] The request is invalid.
[hls,applehttp @ 0xf0a3e0] HLS request for url 'https://v.redd.it/jneg6lmn2y121/HLS_1_0_M.ts', offset 3285676, playlist 0
[NULL @ 0xf11940] Current profile doesn't provide more RBSP data in PPS, skipping    
    Last message repeated 1 times
[tls @ 0xf11760] The request is invalid.
[hls,applehttp @ 0xf0a3e0] HLS request for url 'https://v.redd.it/jneg6lmn2y121/HLS_1_0_M.ts', offset 4096708, playlist 0
[NULL @ 0xf11940] Current profile doesn't provide more RBSP data in PPS, skipping
    Last message repeated 1 times
[tls @ 0x17b83c0] The request is invalid.
[hls,applehttp @ 0xf0a3e0] HLS request for url 'https://v.redd.it/jneg6lmn2y121/HLS_1_0_M.ts', offset 4750572, playlist 0
[NULL @ 0xf11940] Current profile doesn't provide more RBSP data in PPS, skipping    
    Last message repeated 1 times
[tls @ 0x132f980] The request is invalid.
No more output streams to write to, finishing.
frame= 1200 fps=623 q=-1.0 Lsize=    4439kB time=00:00:39.90 bitrate= 911.3kbits/s    
video:4424kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.325707%
Input file #0 (https://v.redd.it/jneg6lmn2y121/HLS_1_0_M_v4.m3u8):
  Input stream #0:0 (video): 1200 packets read (4530459 bytes); 
  Total: 1200 packets (4530459 bytes) demuxed
Output file #0 (file:reddit - the front page of the internet (1)-all-1.fhls-1409.mp4.part):
  Output stream #0:0 (video): 1200 packets muxed (4530459 bytes); 
  Total: 1200 packets (4530459 bytes) muxed
ffmpeg version 2.8.15-0ubuntu0.16.04.1 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.10) 20160609
  configuration: --prefix=/usr --extra-version=0ubuntu0.16.04.1 --build-suffix=-ffmpeg --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --cc=cc --cxx=g++ --enable-gpl --enable-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-libx264 --enable-libopencv
  libavutil      54. 31.100 / 54. 31.100
  libavcodec     56. 60.100 / 56. 60.100
  libavformat    56. 40.101 / 56. 40.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 40.101 /  5. 40.101
  libavresample   2.  1.  0 /  2.  1.  0
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.101 /  1.  2.101
  libpostproc    53.  3.100 / 53.  3.100
[hls,applehttp @ 0xc4c440] HLS request for url 'https://v.redd.it/jneg6lmn2y121/HLS_AUDIO_64_K.ts', offset 0, playlist 0
[aac @ 0x1062e00] ChannelElement 1.0 missing 
Input #0, hls,applehttp, from 'https://v.redd.it/jneg6lmn2y121/HLS_AUDIO_64_K_v4.m3u8':
  Duration: 00:00:39.89, start: 1.400000, bitrate: 0 kb/s
  Program 0 
    Metadata:
      variant_bitrate : 0
    Stream #0:0: Audio: aac (HE-AACv2) ([15][0][0][0] / 0x000F), 44100 Hz, stereo, fltp, 61 kb/s
[mp4 @ 0x10667a0] Codec for stream 0 does not use global headers but container format requires global headers
Output #0, mp4, to 'file:reddit - the front page of the internet (1)-all-1.fhls-77.mp4.part':
  Metadata:
    encoder         : Lavf56.40.101
    Stream #0:0: Audio: aac ([64][0][0][0] / 0x0040), 44100 Hz, stereo, 61 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
Press [q] to stop, [?] for help
[tls @ 0xc54520] The request is invalid.
[hls,applehttp @ 0xc4c440] HLS request for url 'https://v.redd.it/jneg6lmn2y121/HLS_AUDIO_64_K.ts', offset 48504, playlist 0
[tls @ 0x1063820] The request is invalid.
[hls,applehttp @ 0xc4c440] HLS request for url 'https://v.redd.it/jneg6lmn2y121/HLS_AUDIO_64_K.ts', offset 95880, playlist 0
[tls @ 0xc548a0] The request is invalid.
[hls,applehttp @ 0xc4c440] HLS request for url 'https://v.redd.it/jneg6lmn2y121/HLS_AUDIO_64_K.ts', offset 144008, playlist 0
[tls @ 0x1063b00] The request is invalid.  62.9kbits/s    
[hls,applehttp @ 0xc4c440] HLS request for url 'https://v.redd.it/jneg6lmn2y121/HLS_AUDIO_64_K.ts', offset 192136, playlist 0
[tls @ 0x1063b00] The request is invalid.  62.8kbits/s    
[hls,applehttp @ 0xc4c440] HLS request for url 'https://v.redd.it/jneg6lmn2y121/HLS_AUDIO_64_K.ts', offset 240264, playlist 0
[tls @ 0x1063a00] The request is invalid.
[hls,applehttp @ 0xc4c440] HLS request for url 'https://v.redd.it/jneg6lmn2y121/HLS_AUDIO_64_K.ts', offset 288204, playlist 0
[tls @ 0x1063b00] The request is invalid.
[hls,applehttp @ 0xc4c440] HLS request for url 'https://v.redd.it/jneg6lmn2y121/HLS_AUDIO_64_K.ts', offset 335956, playlist 0
[tls @ 0x1063d40] The request is invalid.  62.8kbits/s    
No more output streams to write to, finishing.
size=     313kB time=00:00:40.12 bitrate=  63.9kbits/s    
video:0kB audio:308kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 1.691089%
Input file #0 (https://v.redd.it/jneg6lmn2y121/HLS_AUDIO_64_K_v4.m3u8):
  Input stream #0:0 (audio): 864 packets read (320993 bytes); 
  Total: 864 packets (320993 bytes) demuxed
Output file #0 (file:reddit - the front page of the internet (1)-all-1.fhls-77.mp4.part):
  Output stream #0:0 (audio): 864 packets muxed (314945 bytes); 
  Total: 864 packets (314945 bytes) muxed
ERROR: unable to download video data: HTTP Error 403: Forbidden
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1908, in process_info
    success = dl(filename, info_dict)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1847, in dl
    return fd.download(name, info)
  File "/usr/local/bin/youtube-dl/youtube_dl/downloader/common.py", line 364, in download
    return self.real_download(filename, info_dict)
  File "/usr/local/bin/youtube-dl/youtube_dl/downloader/http.py", line 341, in real_download
    establish_connection()
  File "/usr/local/bin/youtube-dl/youtube_dl/downloader/http.py", line 109, in establish_connection
    ctx.data = self.ydl.urlopen(request)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2211, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 473, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 556, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 403: Forbidden
...
<end of log>

Note: output depends on subreddit. Sometimes no links will be found, sometimes a few m3u8 files will be found by the generic extractor.

If the purpose of this issue is a site support request please provide all kinds of example URLs support for which should be included (replace following example URLs by yours):

Single video: Already supported
Playlist: https:/reddit.com/r/all

Note that youtube-dl does not support sites dedicated to copyright infringement. In order for site support request to be accepted all provided example URLs should not violate any copyrights.

Description of your issue, suggested solution and other information

Reddit has many large communities for sharing videos, and it's very useful to download a lot of them at once. Currently, solutions for this involve individually downloading videos with youtube-dl, and automatically creating a list of links with other tools.

The ideal way for this to work would be to scrape every link (back to the very first page of the subreddit), then put them together as a playlist. Also, it would ideally skip unsupported links (as there would be many).

This has already been suggested before, in #2955, and ended up getting closed in 2017. However, there wasn't a lot of discussion about the issue, and I feel that a subreddit extractor would still be useful to youtube-dl.

th3illu · 2020-05-05T12:46:34Z

is it working yet? i have tried to paste subreddit but it downloads only 4 videos at the time, even with the playlist commands

ealgase changed the title ~~Add support for downloading a subreddit as a playlist~~ [reddit] add support for downloading a subreddit as a playlist Dec 4, 2018

ytdl-org / youtube-dl

[reddit] add support for downloading a subreddit as a playlist #18389

[reddit] add support for downloading a subreddit as a playlist #18389

ealgase commented Dec 4, 2018

th3illu commented May 5, 2020

ytdl-org / youtube-dl

Join GitHub today

GitHub is where the world builds software

[reddit] add support for downloading a subreddit as a playlist #18389

[reddit] add support for downloading a subreddit as a playlist #18389

Comments

ealgase commented Dec 4, 2018

Please follow the guide below

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2018.12.03. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

Before submitting an issue make sure you have:

What is the purpose of your issue?

If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

If the purpose of this issue is a site support request please provide all kinds of example URLs support for which should be included (replace following example URLs by yours):

Description of your issue, suggested solution and other information

th3illu commented May 5, 2020

Essential cookies

Always active

Analytics cookies

Make sure you are using the latest version: run `youtube-dl --version` and ensure your version is 2018.12.03. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.