Playlist handling is a bit of a mess (formerly: Suggestions for Advanced Usage)

[ChrisK2]

Hi, I'm from the mpv-player team and we are currently working on replacing libquvi with youtube-dl. While trying to make some advanced features work, we ran into some problems and hope you can help us with those. See this gist for the current progress.

Let me know if you'd rather have this in separate issues.

Playlist handling is a bit of a mess

We'd also like to support playlists, but the way ytdl handles those currently is a bit inconvenient. Instead of fetching the info for all videos in a playlist at once (which may take quite a while, also the URLs may be invalid be the time you want to play them), we'd prefer if there was a way to just get the page URLs to the individual videos (ideally as a JSON array of "unresolved" URLs), so that we then can run ytdl for each video again when we actually want to play it.

–––––––––––––––––––––––––––––––––––––

Automatically extract ASS subs when available (Moved to #4019)

Subtitles are written to disk with -j (See #3036)

We use the -j option to get the video info in JSON, however, when requesting subtitles as well, those are also still written to disc (cwd), which ends up cluttering the users directory. Since the JSON data contains the subtitles, there is no need to write them to disk as well.

Video URL sometimes has escaped & (See #4014)

The video URL we get from the JSON data may have & escaped as &, although there is no need for this (this may be specific to the Crunchyroll Extractor).

[DarrenCattle]

Playlist handling would be nice to manipulate similar to an array. Say I only want the first 10 songs of a youtube mix, I should be able to add a -n 10, that way I don't have to make a new playlist everytime I want to save parts of a mix. I guess you can just CTRL-C once you hit 10 songs as well..

[phihag]

@DarrenCattle Please file a separate issue, but make sure you read about the playlist options first.

[phihag]

Thanks, it's great to hear that youtube-dl is being used. As documented in our FAQ, please file multiple issues for multiple issues though. @jaimeMF has already moved the third one to #4014. I'm just going to answer the last one. I do think the others are good suggestions though, especially when combined with an actual test case.

In youtube-dl 2014.10.24, there is a new --flat-playlist option (only really useful with -j at the moment):

$ youtube-dl  test:youtube:playlist -j --flat-playlist
{"extractor": "youtube:playlist", "_type": "url", "ie_key": "Youtube", "url": "bV9L5Ht9LgY", "extractor_key": "YoutubePlaylist", "playlist": "ytdl test PL", "playlist_index": 1, "n_entries": 3, "webpage_url": "https://www.youtube.com/playlist?list=PLwiyx1dc3P2JR9N8gQaQN_BCvlSlap7re", "id": "bV9L5Ht9LgY", "webpage_url_basename": "playlist"}
{"extractor": "youtube:playlist", "_type": "url", "ie_key": "Youtube", "url": "FXxLjLQi3Fg", "extractor_key": "YoutubePlaylist", "playlist": "ytdl test PL", "playlist_index": 2, "n_entries": 3, "webpage_url": "https://www.youtube.com/playlist?list=PLwiyx1dc3P2JR9N8gQaQN_BCvlSlap7re", "id": "FXxLjLQi3Fg", "webpage_url_basename": "playlist"}
{"extractor": "youtube:playlist", "_type": "url", "ie_key": "Youtube", "url": "tU3Bgo5qJZE", "extractor_key": "YoutubePlaylist", "playlist": "ytdl test PL", "playlist_index": 3, "n_entries": 3, "webpage_url": "https://www.youtube.com/playlist?list=PLwiyx1dc3P2JR9N8gQaQN_BCvlSlap7re", "id": "tU3Bgo5qJZE", "webpage_url_basename": "playlist"}

[jaimeMF]

The second problem is already reported in #3036.
I'm a mpv user, so I'll try it as soon as I can.

[ChrisK2]

Okay, thanks a lot! I'll make a new issue for the first one.

Shouldn't webpage_url and webpage_url_basename refer to the individual video instead of back to the playlist?

[phihag]

Oh, you' re right, I was applying the playlist info by accident, fixed:

$ youtube-dl  test:youtube:playlist -j --flat-playlist
{"url": "bV9L5Ht9LgY", "_type": "url", "ie_key": "Youtube", "id": "bV9L5Ht9LgY"}
{"url": "FXxLjLQi3Fg", "_type": "url", "ie_key": "Youtube", "id": "FXxLjLQi3Fg"}
{"url": "tU3Bgo5qJZE", "_type": "url", "ie_key": "Youtube", "id": "tU3Bgo5qJZE"}

[ChrisK2]

Can you include the full URL to each video? Otherwise we'd have to figure out how to build one ourself for each service. Or can ytdl somehow make directly use of the supplied info?

[phihag]

These URIs are valid for youtube-dl:

$ youtube-dl bV9L5Ht9LgY -e
Download Youtube Playlist with Youtube-dl

If you need a webpage URL, this will not be possible with --flat-playlist.

[ChrisK2]

Mh, but the vimeo playlist extractor returns full URLs. Does this depend on what the service delivers in the playlists?

[phihag]

Yes, each extractor is free to define its own URI scheme (which does not have to follow the actual URI schemes outside of youtube-dl, case in point). Most extractors will just use the video URLs, but they are certainly free to use any other form of URI (some will use a scheme akin to youtube:bV9L5Ht9LgY). The only thing that is guaranteed is that you can plug this URI into youtube-dl. That should suffice for your purpose, shouldn't it?

[ChrisK2]

Okay, not ideal, but we can make it work.

Independently from this, I would recommend though to at least always specify the Extractor, to ensure that the ID gets passed on correctly in case two sites use a similar/identical ID scheme.

Still thanks for your help!

[phihag]

Why is this not ideal for you? Can you elaborate on your usage scenario?

With the exception of the generic extractor (which matches everything), URI schemes are disjunct, so a collision can never happen.

Note that the extractor is specified right there in the ie_key field, although that is optional here.

[ChrisK2]

See the script from the gist. What our script gets from the user will be a string, which may be a path to a local file, a HTTP/FTP/etc URL directly to a video or to a video site. At this point we can only tell that we should ask youtube-dl to lookup our String, if it starts with "http://" or "https://". If we get a playlist from youtube-dl back (we already have to do some guesswork to tell if what we got from youtube-dl is video info or a playlist), we want to insert the playlist entries into mpv's internal playlist, so that for each video ytdl is called again to fetch the video info for the video that should now be played. But since what we get from the ytdl-playlist may not be a URL that starts with http(s)://, our script will have no idea that it may be supposed to ask ytdl for the individual video info now.

Due to this, we'd have to prefix the playlist-entries we get from youtube-dl with something like ytdl:// before sending them to mpv's playlist, to tell our script that the incoming string is supposed to go to ytdl. If we always got full URLs, this wouldn't be necessary.

So yeah, this is not really a big problem. It's just "not ideal", but we can make it work.

[phihag]

Yes, the ytdl: prefix seems like a good option then. Note that the premise that users only input URLs starting with http: or https: is faulty for youtube-dl, since some of our users do input YouTube video IDs and some features are unusable without allowing arbitrary URLs. Also mind that some services do not generate URLs (say, they have everything on a single page) or that that the video URL may be the same for all videos (say, the video is some strange embedded stuff, so the URL would be http://service.com/player.js), or not present in the first place(the test: URL schema as well as the various search and "my videos" extractors come to mind). But if you don't advertise theses features to your users, I understand that you don't need that.

Sorry, I noted the gist just now. It looks like you have embedded many workarounds. I would prefer to have bugs filed against youtube-dl so that we can improve the situation for all users of youtube-dl.

What comes to mind:

• The URLs generated by youtube-dl are final. There is no need to go replacing &. There was a bug in crunchyroll which has been fixed now.
• You have already filed #4019 for subtitle format handling similar to video format handling.
• With youtube-dl 2014.10.25 or newer, pass in -J instead of -j to get the whole JSON in one line. Look at the _type key to figure out what to do next.
• If you have found a situation where the return code is not 0, but the output is not valid JSON, please report this.
• Do your users always want --no-playlist? I mean, you have to come down one way or the other, but afair YouTube hides the playlist URLs for some playlists so that users will only get the URLs with videos and playlists.

[ChrisK2]

Thanks for your input and help. I've updated the script now to reflect the lastest changes (I was just waiting for the situation to stabilise). The workarounds were removed as far as possible and playlists are now supported.

Regarding --no-playlist: On the internet you often get youtube links like these lately; https://www.youtube.com/watch?v=mCLFmspLqNU&list=PLED-ZERSwDDnmcs1Ad6vnB-NQ1Y3eQe97&index=2 which point to a specific video, but also the playlist the video comes from. When such a link is passed to ytdl (without --no-playlist), ytdl goes into playlist mode and gets us the full playlist instead of the video info. We then have no idea which video of that playlist is even supposed to be played. If the user inputs a link to a specific video, they probably expect that video to be played, and not the first video of the playlists it comes from. Hence we use --no-playlist to keep ytdl from fetching playlists unless we hand it an actual playlist.