What are yt-dlp's error codes?

[victormmr-github]

Checklist

• I'm asking a question and not reporting a bug or requesting a feature
• I've looked through the README
• I've verified that I'm running yt-dlp version 2022.06.29 (update instructions) or later (specify commit)
• I've searched the bugtracker for similar questions including closed ones. DO NOT post duplicates
• I've read the guidelines for opening an issue

Please make sure the question is worded well enough to be understood

I have to download multiple videos from youtube (a list of approximately 30 URLs) and I have written a script reading these youtube URLs from a text file and launching the yt-dlp in a loop for each download. But I know, sometimes downloads of video files, when being performed with this utility, fail, but being repeated after some time this download succeeds. All I need is a way the yt-dlp utility could notify the calling script of an error that took place during download or processing downloaded files and the requested video not being downloaded and saved to my hard drive. If a mistake of that kind occured the calling script could write a message to a text log, and after the script had terminated (with successful and failed downloads) I could analyze the logfile manually and start the unlucky downloads repeatedly.

What's the method of the yt-dlp utility to notify the calling script or binary program about the errors that took place during the video download? How my script can differ successful downloads from failed ones? Of course one could redirect, parse and analyze the screen output of the utility, but it's a complicated task and this output is ineligible for machine analysis. And I can't sit all the time and watch the output of multiple yt-dlp copies - the fetch will take many hours. Could anybody help me resolve my problem? Has it any acceptable solution?

Provide verbose output that clearly demonstrates the problem

• Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
• Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output

No response

[pukkandan]

The exit code is 0 if there was no error

[victormmr-github]

OK! And if an error occured while an attempt to download a file, what would be the exit code of the yt-dlp utility?

[pukkandan]

Here are the current exit codes:

  0 = Everything is successful
100 = yt-dlp must restart for update to complete
101 = Download cancelled by --max-downloads etc
  2 = Error in user-provided options
  1 = Any other error

Since 1 represents a broad range of errors, we may split this into multiple error codes in the future. The other error codes are guaranteed to continue as-is in future versions.

For deciding whether to retry, you just need to check that the error is not in 0,2,101

[victormmr-github]

Thank you very much for your answers, pukkandan!
Actually I 've got all the information I needed.

Due to my question, I have a small feature request. Where can I publish it if it's interesting to you?

[pukkandan]

If it is tightly related to this question, you can just ask here. Otherwise, you can open a new issue with the "feature request" template

[victormmr-github]

1. It seems the yt-dlp utility doesn't have a manpage. Counter to it, its mother utility, youtube-dl has a manpage. I think, it's a serious flaw. Such a complicated multifunctional utility should have a good thorough man.

On the other hand, some useful convenient documentation is available through "yt-dlp --help" call. It's good and this option must be preserved. A good thorough manpage and "yt-dlp --help" option could supplement each other. "yt-dlp --help" could contain a circumcised version of man, while "man yt-dlp" could have the full detailed description of the program and all of its options and keys.

Yet another variant is possible. The manpage and the --help option output could be precise copies of each other? They could show in facsimile the same text. Still two ways of showing quite the same text. In this case there is no need of creating two different documentations, one complete and thorough and the other one brief. You'll have to maintain the single text, and it will be much easier. And this text will be shown by two different commands. Still I'm sure, a good manpage is necessary for such a useful program.

2. Before asking you a question about error information returned to the calling script by yt-dlp I looked into the youtube-dl and yt-dlp documentation for the exit codes, returned by these utilities. I searched for the information in

man youtube-dl
youtube-dl --help
yt-dlp --help

but nothing appropriate was found there. The failure of video download reported as a nonzero exit code, returned to the calling script, is an obvious decision. Amost all Unix/Linux utilities behave this way. However this aspect was not reflected in the documentation, neither youtube-dl nor yt-dlp. The guides of both programs contain no section dedicated to exit codes of these programs. It's a serious omission, in respect that all Unix/Linux command line utilities mans have a section, describing exit codes of corresponding utilities. By habit, this section is called "Exit Status". It's a very important section for script writers, without knowledge of these codes it's impossible to write reliable scripts capable of processing and correcting erroneous situations. Here is an example of such a section from another famous file fetcher, Wget:

EXIT STATUS
Wget may return one of several error codes if it encounters problems.

   0   No problems occurred.

   1   Generic error code.

   2   Parse error---for instance, when parsing command-line options, the
       .wgetrc or .netrc...

   3   File I/O error.

   4   Network failure.

   5   SSL verification failure.

   6   Username/password authentication failure.

   7   Protocol errors.

   8   Server issued an error response.

   With the exceptions of 0 and 1, the lower-numbered exit codes take
   precedence over higher-numbered ones, when multiple types of errors are
   encountered.

   In versions of Wget prior to 1.12, Wget's exit status tended to be
   unhelpful and inconsistent. Recursive downloads would virtually always
   return 0 (success), regardless of any issues encountered, and non-
   recursive fetches only returned the status corresponding to the most
   recently-attempted download.

I think, something like that yt-dlp must have. All exit codes must be listed and a detailed explanation what each code does mean must be given.

Otherwise there is no sense in producing error codes by your Python utility, no one knows, no one is aware of, and thus no one may take advantage of. It's a sort of a secret exit code, one can learn only after getting deeply familiar with your Python source. I'm sure, a good thorough "Exit Status" section must be created in your documentation.

You've expressed a thought, it would be good to split the existing small number of error codes, each representing a wide range of errors, into a large number of exit codes. I think it's a good idea. The more detailed information of errors, that have happened, is, the better is for programmer or a script writer. More chances to analyze and understand them. The single rule must be, if the files were finally downloaded successfully and without mistakes, and they were successfully merged, the exit code must be equal to 0. This is a sign, that no repeated download of this video is needed.

3. I have some considerations on making a log file.

On the one hand, a script can analyze the exit code of the terminated yt-dlp utility, and according to this exit code returned it can form and output a message to the text file. In this case all text message generation will be implemented in the calling script. On the other hand, we can entrust this task to the utility itself.

We cannot simply redirect the yt-dlp output to the text file in the manner like

yt-dlp > log.txt 2>&1

as yt-dlp output contains lot's of terminal control sequences. When printing log.txt on a terminal via cat, there is an illusion we are having a correct text, but if we open this file in any text editor we'll see what it actually is.

Thus I have a suggestion. We could implement several options which force the yt-dlp program to create such logs. Here is a sketch of these options.

I.

-l <filename>
--logfile <filename>

Make a simple brief log and write it to the file.

Create a file <filename> if nonexistent or open an existing file and truncate it to zero length. In the terminal stage of the program execution before exiting write a simple brief notification of the video fetching operation, performed by the program. In case of success the notification may contain the text like this:

Video by the URL <video_URL> was downloaded successfully.
The resulting file is <file_name>.

In case of error of any kind and download failure the text can be like this:

Video by the URL <video_URL> failed to download.
The server interrupted connection.
No video file was created.

Video by the URL <video_URL> failed to download.
The server returned an error status <http_error_code>.
No video file was created.

Video by the URL <video_URL> not capable of downloading.
Not enough free space on your hard drive (548 MB required, 259 MB free)
No video file was created.

II.

-L <filename>
--appendlog <filename>

Make a simple brief log and append it to the end of a file.

The same as I, but if the file exists it's not cleared, it's not truncated, however the log message is appended to its end.

In more details, if the file does not exist it is created with zero length and the message is put to it. If the file yet exists, it is opened, it's not truncated, its contents is preserved and the message is appended to its end. The message is quite the same as in case I.

III.

-m <filename>
--termlog <filename>

Duplicate the contents of the terminal output to the file.

Create a file if nonexistent or open an existing file and truncate it to zero length. In the terminal stage of the program execution before exiting write the same message into it as was printed to the terminal, but without all these terminal control sequences. No ncurses, no colours, just plain text, which repeats precisely the terminal output of the program.

IV.

-M <filename>
--appendterm <filename>

Duplicate the contents of the terminal output and append it to the end of a file.

The same as I, but if the file exists it's not cleared, it's not truncated, however the log message is appended to its end.

In more details, if the file does not exist it is created with zero length and the message is put to it. If the file yet exists, it is opened, it's not truncated, its contents is preserved and the message is appended to its end. The message is quite the same as in case III.

In case of II and IV we need some delimitor string to delimit messages from each other. Various versions are possible, and I don't know which one is better. We can delimit neighbouring messages with one empty string, in C-like notation "\n". We can delimit them with a pair of empty strings, i. e. "\n\n". We can separate them from each other with a string of 80 dashes - "--------------------------------------------------------------------------------". We can devide them with a string of 80 underscores - "________________________________________________________________________________". We can add an empty string before these dashes and another one after them. Then we'll get this "\n--------------------------------------------------------------------------------\n" or that "\n________________________________________________________________________________\n". We can think out something else, different from the variants mentioned above. Various ways are possible, the main idea is two adjacent log messages must not merge with each other.

I think my designations are not good and well-turned, I've thought them out on the run. You may choose letters other then -l,-L, -m and -M, and words different from --logfile, --appendlog, --termlog and --appendterm, I think they are bad.

On the other hand, it seems to me the idea of log generating by the yt-dlp to be good. It could simplify the task of creating a log and rid the programmer or a script writer from analyzing exit codes of the yt-dlp utility and composing numerous text messages, from all these numerous case ... esac constructs. It could save lot's of time to a bash script writer, since all logging functionality would be concluded in the utility itself and called with a simple key, followed by the name of the text file. After the script had finished its job, the user could open the logfile in any text editor, look it through and decide, whether there were some errors and does he need to refetch some of his files, whose download was unsuccessfull.

Excuse me for being too verbose. I just wanted to state my proposal in an easy to understand way. Anyway, it's a view of a common user of this utility. Perhaps yt-dlp developers will find some of my suggestions to be interesting.

[pukkandan]

That is a lot of different questions and feature requests. It would have been better to open a separate issue for each

It seems the yt-dlp utility doesn't have a manpage. Counter to it, its mother utility, youtube-dl has a manpage. I think, it's a serious flaw. Such a complicated multifunctional utility should have a good thorough man.

We do have a manpage. It just might not have been registered on your system based on how you installed yt-dlp. If you want to manually register it, the manpage is in yt-dlp.tar.gz (https://github.com/yt-dlp/yt-dlp#release-files)

The failure of video download reported as a nonzero exit code, returned to the calling script, is an obvious decision. Amost all Unix/Linux utilities behave this way. However this aspect was not reflected in the documentation, neither youtube-dl nor yt-dlp. The guides of both programs contain no section dedicated to exit codes of these programs. It's a serious omission, in respect that all Unix/Linux command line utilities mans have a section, describing exit codes of corresponding utilities. By habit, this section is called "Exit Status". It's a very important section for script writers, without knowledge of these codes it's impossible to write reliable scripts capable of processing and correcting erroneous situations. Here is an example of such a section from another famous file fetcher, Wget:

Would you like to create a PR adding this to Readme? (The manpage is autogenerated from readme)

You've expressed a thought, it would be good to split the existing small number of error codes, each representing a wide range of errors, into a large number of exit codes. I think it's a good idea. The more detailed information of errors, that have happened, is, the better is for programmer or a script writer. More chances to analyze and understand them. The single rule must be, if the files were finally downloaded successfully and without mistakes, and they were successfully merged, the exit code must be equal to 0. This is a sign, that no repeated download of this video is needed.

See #2947

We cannot simply redirect the yt-dlp output to the text file in the manner like

yt-dlp > log.txt 2>&1

as yt-dlp output contains lot's of terminal control sequences. When printing log.txt on a terminal via cat, there is an illusion we are having a correct text, but if we open this file in any text editor we'll see what it actually is.

yt-dlp should detect it is not writing to console and omit the color codes. If that is not working, open a bug report with example and relevant verbose logs (You can also use --no-color as a workaround)

Thus I have a suggestion. We could implement several options which force the yt-dlp program to create such logs. Here is a sketch of these options.
...

I don't understand why you need a new "brief" log. Explain to me why the current logging levels are insufficient

-M
--appendterm

Duplicate the contents of the terminal output and append it to the end of a file.

For a fixed filename, this is the same as shell redirect and there is no point in adding such a feature directly. However, there is merit in such a feature if we are to allow writing the different log levels (progress, verbose, info, warning, error, traceback) to different files. But sadly, this is not trivial to implement and the request can be tracked in #2668

PS: There is also no need to have separate appending and truncating options. User can always delete the file before running yt-dlp if needed

[victormmr-github]

Pukkandan, thank you for your answers. Several topics were touched on, so let's discuss them in series.

Concerning the manpage. Yes, there is a manpage in your project, I have made sure of it's presence, and it's big enough. I have settled it to my Debian system. First I found the "yt-dlp.1" file, then I compressed it with gzip, and afterall I put it to the /usr/share/man directory. It really works, when called with man yt-dlp.

Still there is a problem. Initially I installed the yt-dlp the simplest way, described in your Readme instruction.

su -
wget https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp -O /usr/local/bin/yt-dlp
chmod a+rx /usr/local/bin/yt-dlp

Thus no manpage was set up. Now I installed this manpage manually. However there is a problem. The yt-dlp utility is capable of update by invoking it with the -U option - ytdlp -U. Still, I am assured this won't update the manpage. That is if a new release of yt-dlp is published with a revised man and I update my own version to it, only the Python binary will be updated, but the man will stay unchanged. So what I'll get will be the new executable binary with the old man, corresponding to the old release of the utility. Aside of the Python byte-code binary and a manpage, your package contains autocomplete for bash, zsh and fish, a very nice feature. When I copied your executable to my bin directory, obviously these autocomplete files weren't set up to my system. Even if I add this file for bash shell manually, it won't update anymore. Updating man and autocomplete each time after the Python binary was updated is too tedious.

So my question is. Is there a chance to install the yt-dlp program in such a way, that not only the main binary but all auxiliary files, applying to it, were installed to it? And is there a way to perform update, so that all these auxiliary files were updated too to the newer version?

I'll be greatful to you for your help!

[pukkandan]

Use PIP or another package manager

[victormmr-github]

Another question. You wrote:

yt-dlp should detect it is not writing to console and omit the color codes. If that is not working, open a bug report with example and relevant verbose logs (You can also use --no-color as a workaround)

Please try this:

yt-dlp https://www.youtube.com/watch?v=C9O28ne6bG8 > log.txt 2>&1

I tried it on Debian. What I got in the file log.txt was a text with lots of carriage returns. In text editors like Emacs or vim they are visible as ^M.

Try after another line

yt-dlp --no-color https://www.youtube.com/watch?v=C9O28ne6bG8 > log_1.txt 2>&1

The result in my case was quite the same: plenty of lines with carriage returns, that rewrite the line repeatedly. It's obvious this output is ineligible for logging into a file.

[pukkandan]

huh? Why is carriage return problematic? It should just be treated like newline in text files... Or, use --newline. I thought you were talking about other control characters

[victormmr-github]

huh? Why is carriage return problematic? It should just be treated like newline in text files... Or, use --newline. I thought you were talking about other control characters

Because carriage return in a log is a nonsense. It's used to erase the current line and replace its contents with a new one, but it's never present in any text files including files of program output journals. It's an element of screen visualization like other control sequences and it has nothing to do in a text file.

P.S.
Carriage return is really treated as a newline in MAC OS, but it's from quite another opera. In that operating system CR serves as LF in Unices and Linux.

[pukkandan]

Did you not read the rest of my comment? or did that not work?

Or, use --newline. I thought you were talking about other control characters

You also never responded to the rest of my questions. So I am closing this. While I entertained your mass of unrelated questions in this one issue, this makes it difficult to track the flow of the conversation over a long term. I suggest you open a new issue for each feature request/question if you need further clarification on them