Download files from own playlist

[davidhedlund]

Please implement these features in youtube-dl:

Example of playlist file "Antares - Exodus.txt":
Antares - The Exodus
Antares - Mysticism
Antares - Aurum Solis
Antares - The 4th Dimension
Antares - Dreaming Universe
Antares - Mount Meru
Antares - Astral Plane
Antares - Sun Sanctuary

#!/bin/bash

# youtube-dl-pl-sh is a youtube-dl Bash extension to download Youtube files from text files.

# This file is released to the public domain.
# Created by David Englund

rm *.html

for x in $(find . -name "*.txt"); do


  file_full=$(echo $x | awk -F/ '{print $NF}');
  file=$(echo $x | sed -e "s|.txt||");


################################################
# Find links, Youtube links are outdated regualry so it is pointless to collect them

mkdir $file

  while read line; do

    # At the moment M4A (MP4) "audio only" are only supported on Youtube
    # Download youtube files, primarily "audio only" to not damage SSD drives, and to save time by not having to convert video to audio. Convert proprietary formats to free ones.
    # This might run faster but on the other hand, youtube handles different formats (eg .webm, .m4a) which make this inconvenient.
    # for i in $(cat list-id); do youtube-dl $i.ogg --title $i; ffmpeg -i $i -vn -acodec libvorbis; done

    # context-data-item clearfix = Ads
    # __title__ = Dummy

    # &search_sort=video_view_count (that is https://www.youtube.com/results?search_query=$line&search_sort=video_view_count) will repeat videos with many views and not show it for the unique track.

    # Youtube file facts: M4A files have unknown bitrate.
    # find . -name '*.m4a' -exec sh -c 'ffmpeg -i "$0" -sameq "${0%%.mp4}.ogg"' {} \; # Converting M4A files to OGG like this makes them about 10 times larger in file size

    # Sound quality log:

    # Use: SAVEIFS=$IFS; IFS=$(echo -en "\n\b"); for i in $(find . -type f); do tid=$(mplayer -vo null -ao null -identify -frames 0 "$i" | grep --max-count="1" "BITRATE" | sed "s|ID_AUDIO_BITRATE=||"); if [ $tid -lt 128000  ]; then echo $i >> foo.txt; fi; done; IFS=$SAVEIFS
    #2014-01-21: About 50% of OGG files extracted from WEBM had lower bitrate than 128 kb/s.

# Run detox later
artist="${line%% -*}";
artist_url=$(echo $artist | sed "s|\&|%26|g; s|\/|%26|g; s|!||g; s|?||g;");
artist_html=$(echo $artist | sed "s|\&|\&|g; s|\/|\&|g; s|'|\'|g; s|!||g; s|?||g;");
artist_unix=$(echo $artist | sed "s|\/|\&|g;");
track="${line##*- }";
track_url=$(echo $track | sed "s|\&|%26|g; s|\/|%26|g; s|!||g; s|?||g;");
track_html=$(echo $track | sed "s|\&|\&|g; s|\/|\&|g; s|'|\'|g; s|!||g; s|?||g;");
track_unix=$(echo $track | sed "s|\/|\&|g;");



wget --output-document=1.html "https://www.youtube.com/results?search_query=$artist_url+-+$track_url"; cp -a 1.html 2.html; sed -i '/__title__/d; /full album/Id; /playlist/d; /context-data-item clearfix/d;' 2.html; grep "data-context-item-id=" 2.html | grep -i "$track_html" >> 3.html; cat 3.html >> metadata.log;
id=$(cp -a 3.html 4.html; sed -i "s|data-context-item-id=\"|\ndata-context-item-id=|g; s|\" |\n|g; s|\"|\n|g;" 4.html; grep data-context-item-id 4.html > 5.html; sed -i "s|data-context-item-id=||g; s|\"||" 5.html);


downloaded="no";

while read linex; do

  if [ $downloaded == "no" ]; then


    cd $file
     # Bug: --extract-audio does not work with -o: youtube-dl $linex -o "$line - %(id)s.ogg" -f 43 --extract-audio --no-overwrites
    youtube-dl $linex --id -f 43 --extract-audio --no-overwrites --abort-on-error && mv $linex.ogg "$artist_unix - $track_unix - $linex.ogg" && downloaded="yes"; # --no-overwrites only works with default filenames so we have to include the Youtube ID in the filename make it unique (as done by default with youtubes own filename)
    #   youtube-dl $linex -f 43 --abort-on-error && exit
    cd ..

echo $linex > x

  fi

done <5.html



#detox $x; # Could not be used directly after "mv" above

rm *.html

  done <"$x"

done >> sessions.log

[jaimeMF]

It seems that you can do that with youtube-dl 'ytsearch1:Antares The Exodus' (you can use all the normal options to select format, extract audio ...).
I don't really understand if you are requesting something else, could you explain what do you exactly want?

[sahutd]

are you suggesting that we run queries on youtube based on the search text in the file?
or
the urls are present in the txt(which does not seem the case) ?

[davidhedlund]

On 2014-01-22 12:58, Saimadhav wrote:

are you suggesting that we run queries on youtube based on the search
text in the file?

Yes.

[jaimeMF]

Maybe the request is too specific, you can use the --batch-file for that. You just need to write the "urls" in urls.txt (for example):

ytsearch1:Antares The Exodus
ytsearch1:Antares - Mysticism

And then you can just use youtube-dl --batch-file 'urls.txt' (with all the options you need). You could write a simple script that writes urls.txt from the contents of Antares - Exodus.txt.

[davidhedlund]

On 2014-01-22 14:20, Jaime Marquínez Ferrándiz wrote:

Maybe the request is too specific, you can use the |--batch-file| for
that. You just need to write the "urls" in |urls.txt| (for example):

|ytsearch1:Antares The Exodus
ytsearch1:Antares - Mysticism
|

And then you can just use |youtube-dl --batch-file 'urls.txt'| (with
all the options you need). You could write a simple script that writes
|urls.txt| from the contents of |Antares - Exodus.txt|.

—
Reply to this email directly or view it on GitHub
#2193 (comment).

How do I quit youtube-dl once it have started to download?

[davidhedlund]

On 2014-01-22 14:20, Jaime Marquínez Ferrándiz wrote:

Maybe the request is too specific, you can use the |--batch-file| for
that. You just need to write the "urls" in |urls.txt| (for example):

|ytsearch1:Antares The Exodus
ytsearch1:Antares - Mysticism
|

And then you can just use |youtube-dl --batch-file 'urls.txt'| (with
all the options you need). You could write a simple script that writes
|urls.txt| from the contents of |Antares - Exodus.txt|.

—
Reply to this email directly or view it on GitHub
#2193 (comment).

How do I exclude "full album" results in:

while read line; do youtube-dl "ytsearch1:$line" -o "$line - %(id)s.ogg"
-f 43 --extract-audio --no-overwrites --abort-on-error; done <
Top_tracks.txt

[davidhedlund]

On 2014-01-22 14:20, Jaime Marquínez Ferrándiz wrote:

Maybe the request is too specific, you can use the |--batch-file| for
that. You just need to write the "urls" in |urls.txt| (for example):

|ytsearch1:Antares The Exodus
ytsearch1:Antares - Mysticism
|

And then you can just use |youtube-dl --batch-file 'urls.txt'| (with
all the options you need). You could write a simple script that writes
|urls.txt| from the contents of |Antares - Exodus.txt|.

—
Reply to this email directly or view it on GitHub
#2193 (comment).

My personal opinion is that a nifty one-line like this should be
included as an example in the man page:

while read line; do youtube-dl "ytsearch1:$line" -o "$line - %(id)s.ogg"
-f 43 --extract-audio --no-overwrites --abort-on-error --max-filesize
30m; done < Top_tracks.txt

Can you forward that?

[phihag]

As @jaimeMF said, your batch file is a really complicated way of writing

#!/bin/sh

for x in $(find . -name "*.txt"); do
    youtube-dl --batch-file "$x" --default-search ytsearch: -o "$(echo $x | sed -e s,.txt,,)/%(playlist)s.%(ext)s" -f 43 --extract-audio --no-overwrites

with the newest version of youtube-dl (2014.01.22.3).

I'm also puzzled of the choice of 43 as the quality, as that is a video format. Although the wearability of modern SSDs is so good that you're unlikely to ever have to worry about it, why download a video format when audio formats are available? A sensible value would be -f bestaudio/43/best. For most YouTube videos, that would mean that recoding is not necessary.

Also, this comment is almost certainly dead wrong:

# Bug: --extract-audio does not work with -o: youtube-dl $linex -o "$line - %(id)s.ogg" -f 43 --extract-audio --no-overwrites

Your output format is obviously defective (you want %(ext)s in there), but I see no reason why it shouldn't work, and no bug entry here. It works just fine for me:

$ youtube-dl --extract-audio  -o '%(id)s.ogg' BaW_jenozKc
[youtube] Setting language
[youtube] BaW_jenozKc: Downloading webpage
[youtube] BaW_jenozKc: Downloading video info webpage
[youtube] BaW_jenozKc: Extracting video information
[download] Destination: BaW_jenozKc.ogg
[download] 100% of 154.06KiB in 00:00
[ffmpeg] Destination: BaW_jenozKc.m4a
Deleting original file BaW_jenozKc.ogg (pass -k to keep)

[davidhedlund]

Webm contains OGG which respect freedom contrary to MP*

You are right in everything else in this email.

On 2014-01-22 15:03, Philipp Hagemeister wrote:

As @jaimeMF https://github.com/jaimeMF said, your batch file is a
really complicated way of writing

|#!/bin/sh

for x in $(find . -name "*.txt"); do
youtube-dl --batch-file "$x" --default-search ytsearch: -o "$(echo $x | sed -e s,.txt,,)/%(playlist)s.%(ext)s" -f 43 --extract-audio --no-overwrites
|

with the newest version of youtube-dl (2014.01.22.3).

I'm also puzzled of the choice of 43 as the quality, as that is a
video format. Although the wearability of modern SSDs is so good that
you're unlikely to ever have to worry about it, why download a video
format when audio formats are available? A sensible value would be |-f
bestaudio/43/best|. For most YouTube videos, that would mean that
recoding is not necessary.

Also, the comment is almost certainly ddead wrong

|# Bug: --extract-audio does not work with -o: youtube-dl $linex -o "$line - %(id)s.ogg" -f 43 --extract-audio --no-overwrites
|

Your output format is obviously defective (you want |%(ext)s| in
there), but I see no reason why it shouldn't work, and no bug entry
here. It works just fine for me:

|$ youtube-dl --extract-audio -o '%(id)s.ogg' BaW_jenozKc
[youtube] Setting language
[youtube] BaW_jenozKc: Downloading webpage
[youtube] BaW_jenozKc: Downloading video info webpage
[youtube] BaW_jenozKc: Extracting video information
[download] Destination: BaW_jenozKc.ogg
[download] 100% of 154.06KiB in 00:00
[ffmpeg] Destination: BaW_jenozKc.m4a
Deleting original file BaW_jenozKc.ogg (pass -k to keep)
|

—
Reply to this email directly or view it on GitHub
#2193 (comment).

[jaimeMF]

Some videos have an audio only version in webmwith itags 171 and 172. you could use -f 172/171/43 to request them.

[davidhedlund]

On 2014-01-22 14:20, Jaime Marquínez Ferrándiz wrote:

Maybe the request is too specific, you can use the |--batch-file| for
that. You just need to write the "urls" in |urls.txt| (for example):

|ytsearch1:Antares The Exodus
ytsearch1:Antares - Mysticism
|

And then you can just use |youtube-dl --batch-file 'urls.txt'| (with
all the options you need). You could write a simple script that writes
|urls.txt| from the contents of |Antares - Exodus.txt|.

—
Reply to this email directly or view it on GitHub
#2193 (comment).

Can I "ytsearch1:Antares - Mysticism" at the end of line to ensure that
no remix is included in the search (eg Antares - Mysticism (SOME ARTIST
REMIX))?

[phihag]

If you want webm and other free formats, simply pass in --prefer-free-formats. There is really no need to pass in any other format specification then.

You can use --reject-title REMIX as well as simply include -remix in the search query.

[davidhedlund]

On 2014-01-22 15:11, Jaime Marquínez Ferrándiz wrote:

Some videos have an audio only version in |webm|with itags |171| and
|172|. you could use |-f 172/171/43| to request them.

—
Reply to this email directly or view it on GitHub
#2193 (comment).

"from Philipp Hagemeister

If you want webm and other free formats, simply pass in
|--prefer-free-formats|. There is really no need to pass in any other
format specification then.

You can use |--reject-title REMIX| as well as simply include |-remix| in
the search query.

—
Reply to this email directly or view it on GitHub
#2193 (comment).

"

Would a mixture of your suggestions be ideal for saving disk space,
time, and freedom, or should I stick with one of the solutions (if yes,
which one?)?

[davidhedlund]

How do I do this with --prefer-free-formats?

for x in $(find . -name "*.txt"); do
youtube-dl --batch-file "$x" --default-search ytsearch: -o "$(echo
$x | sed -e s,.txt,,)/%(playlist)s.%(ext)s" --prefer-free-formats
--extract-audio --no-overwrites --ignore-errors --max-filesize 30m
done

That downloads M4A instead of WEBM. What am I doing wrong?

On 2014-01-22 15:03, Philipp Hagemeister wrote:

As @jaimeMF https://github.com/jaimeMF said, your batch file is a
really complicated way of writing

|#!/bin/sh

for x in $(find . -name "*.txt"); do
youtube-dl --batch-file "$x" --default-search ytsearch: -o "$(echo $x | sed -e s,.txt,,)/%(playlist)s.%(ext)s" -f 43 --extract-audio --no-overwrites
|

with the newest version of youtube-dl (2014.01.22.3).

I'm also puzzled of the choice of 43 as the quality, as that is a
video format. Although the wearability of modern SSDs is so good that
you're unlikely to ever have to worry about it, why download a video
format when audio formats are available? A sensible value would be |-f
bestaudio/43/best|. For most YouTube videos, that would mean that
recoding is not necessary.

Also, the comment is almost certainly ddead wrong

|# Bug: --extract-audio does not work with -o: youtube-dl $linex -o "$line - %(id)s.ogg" -f 43 --extract-audio --no-overwrites
|

Your output format is obviously defective (you want |%(ext)s| in
there), but I see no reason why it shouldn't work, and no bug entry
here. It works just fine for me:

|$ youtube-dl --extract-audio -o '%(id)s.ogg' BaW_jenozKc
[youtube] Setting language
[youtube] BaW_jenozKc: Downloading webpage
[youtube] BaW_jenozKc: Downloading video info webpage
[youtube] BaW_jenozKc: Extracting video information
[download] Destination: BaW_jenozKc.ogg
[download] 100% of 154.06KiB in 00:00
[ffmpeg] Destination: BaW_jenozKc.m4a
Deleting original file BaW_jenozKc.ogg (pass -k to keep)
|

—
Reply to this email directly or view it on GitHub
#2193 (comment).

[davidhedlund]

On 2014-01-22 15:15, Philipp Hagemeister wrote:

If you want webm and other free formats, simply pass in
|--prefer-free-formats|. There is really no need to pass in any other
format specification then.

You can use |--reject-title REMIX| as well as simply include |-remix|
in the search query.

—
Reply to this email directly or view it on GitHub
#2193 (comment).

youtube-dl XCHrWyMpPhM --prefer-free-formats # Downloads M4A instead of WEBM

youtube-dl XCHrWyMpPhM -F # Shows WEBM that contains OGG (free)

Is this a bug?

[phihag]

Everything is working as intended. The default preference for audio extracting (bestaudio/best) with --prefer-free-formats is (simplified)

webm audio
audio
webm HQ video + audio
HQ video + audio
webm video + audio
video + audio

Therefore, if there's no webm audio format, you'll get mp4 audio. If you pass in -f 172/171/43/webm/bestaudio/best, you're changing that to

webm audio
webm HQ video + audio
webm video + audio
audio
HQ video + audio
video + audio

Which one you prefer is up to you - would you rather avoid non-free formats at all cost or load a far smaller audio-only file in a non-free format?

[davidhedlund]

FSF would certainly say that |-f 172/171/43/webm/ is the only way to go
as the rest is non-free. Can you please make a new option
"--require-free-formats" that run ||-f 172/171/43/webm/|

By the way, what is the last "webm" in |-f 172/171/43/webm/ for? Isn't
43 for webm?|

On 2014-01-22 17:30, Philipp Hagemeister wrote:

Everything is working as intended. The default preference for audio
extracting (|bestaudio/best|) with |--prefer-free-formats| is (simplified)

|webm audio
audio
webm HQ video + audio
HQ video + audio
webm video + audio
video + audio
|

Therefore, if there's no webm audio format, you'll get mp4 audio. If
you pass in |-f 172/171/43/webm/bestaudio/best|, you're changing that to

|webm audio
webm HQ video + audio
webm video + audio
audio
HQ video + audio
video + audio
|

Which one you prefer is up to you - would you rather avoid non-free
formats at all cost or load a far smaller audio-only file in a
non-free format?

—
Reply to this email directly or view it on GitHub
#2193 (comment).

[phihag]

See #2196 for --require-free-formats. In short, I do think that virtually all users would prefer to get a video (which can be played by open-source software like mplayer and vlc just fine) than get no video.

43 is just one of the webm formats on YouTube, and will match webm on other sites.

[davidhedlund]

On 2014-01-22 19:53, Philipp Hagemeister wrote:

See #2196 #2196 for
|--require-free-formats|. In short, I do think that virtually all
users would prefer to get a video (which can be played by open-source
software like mplayer and vlc just fine) than get no video.

43 is just one of the |webm| formats on YouTube, and will match webm
on other sites.

—
Reply to this email directly or view it on GitHub
#2193 (comment).

Thanks for everything. I guess this is what I wanted after all.

#!/bin/sh

for x in $(find . -name "*.txt"); do
youtube-dl --batch-file "$x" --default-search ytsearch: -o "$(echo $x |
sed -e s,.txt,,)/%(playlist)s - %(id)s.%(ext)s" -f webm/ogg
--extract-audio --no-overwrites --max-filesize 30m -i
done

Please share =)

[davidhedlund]

youtube-dl --batch-file "Afgin - Old Is Gold" --default-search ytsearch:
-o "$(echo "Afgin - Old Is Gold" | sed -e s,.txt,,)/%(playlist)s -
%(id)s.%(ext)s" -f webm/ogg --extract-audio --no-overwrites
--max-filesize 30m -i

This will download "Afgin - Old Is Gold (Part 2)", not "Afgin - Old Is
Gold". How do I search exact phrase and case insensetive?

[phihag]

youtube-dl doesn't implement any special kind of search, it's just relaying the YouTube search. Therefore, you may be able to use tricks from the YouTube search (maybe -part or so). I believe the YouTube search to be case-insensitive.

[davidhedlund]

On 2014-01-22 21:07, Philipp Hagemeister wrote:

youtube-dl doesn't implement any special kind of search, it's just
relaying the YouTube search. Therefore, you may be able to use tricks
from the YouTube search (maybe |-part| or so). I believe the YouTube
search to be case-insensitive.

—
Reply to this email directly or view it on GitHub
#2193 (comment).

How do I define to search only type "video" (eg not "playlist") are
included in youtube-dl?

[davidhedlund]

On 2014-01-22 21:07, Philipp Hagemeister wrote:

youtube-dl doesn't implement any special kind of search, it's just
relaying the YouTube search. Therefore, you may be able to use tricks
from the YouTube search (maybe |-part| or so). I believe the YouTube
search to be case-insensitive.

—
Reply to this email directly or view it on GitHub
#2193 (comment).

How can I make this try next youtube hit as I receive "ERROR: requested
format not available":

youtube-dl https://www.youtube.com/watch?v=-sN9ATB7juU -f webm/ogg

[phihag]

See #2203 for an answer to that.