Can't issue any downloads at all right now

[GlassedSilver]

This has been the case in a previous version already, I think it started with 3.5, but I chalked it up to possibly a temporary issue. Looking back and actually reading the error code I might have a little bit of actual issue with the software itself. I think what's happening is that youtubedl-material is passing a broken or bad argument? Idk

Here's the log (freshly updated as you can see):

2020-04-13T09:00:57.265Z INFO: Beginning migration: 3.5->3.6+
2020-04-13T09:00:57.297Z INFO: 3.5->3.6+ migration complete!
2020-04-13T09:00:57.300Z INFO: YoutubeDL-Material v3.6 started on PORT 17442
2020-04-13T09:10:20.619Z ERROR: ERROR: u'' is not a valid URL. Set --default-search "ytsearch" (or run youtube-dl "ytsearch:" ) to search YouTube
(node:1) UnhandledPromiseRejectionWarning: Error: Command failed with exit code 1: /app/node_modules/youtube-dl/bin/youtube-dl -o video/%(uploader)s_[%(channel_id)s]/%(upload_date)s_-_%(title)s_[%(id)s].mp4 -f bestvideo[ext=mp4]+bestaudio[ext=m4a]/mp4 --write-info-json --print-json --download-archive video/merged.txt --write-sub --write-thumbnail --no-call-home --embed-thumbnail --sub-format srt/ass/best --sub-lang en,eng,de,ger,deu http://www.youtube.com/watch?v=J-2VSNPID2g
at makeError (/app/node_modules/execa/lib/error.js:56:11)
at handlePromise (/app/node_modules/execa/index.js:114:26)
at processTicksAndRejections (internal/process/task_queues.js:94:5)
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:1) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

This is when using regular YoutubeDL-Material on the WebUI in Docker. No extension, bookmarklet or API used.

Is the docker maybe missing a dependency?

[Tzahi12345]

Is the docker maybe missing a dependency?

Might be that a package called AtomicParsley is missing which only gets invoked when the argument --embed-thumbnail is passed, at least that's what happens on my end. But the error I got was a bit different:

ERROR: AtomicParsley was not found. Please install.

Could you try the download without the --embed-thumbnail argument? Alternatively, just for testing, does downloading the video without additional args help? Maybe that will help us narrow down what might be causing this error.

What confuses me about the error you have is it seems like it's a parsing error for the URL. But looking at the URL, it seems completely fine! Hm. Let me know if removing the custom args helps.

[GlassedSilver]

Is the docker maybe missing a dependency?

Might be that a package called AtomicParsley is missing which only gets invoked when the argument --embed-thumbnail is passed, at least that's what happens on my end. But the error I got was a bit different:

ERROR: AtomicParsley was not found. Please install.

Could you try the download without the --embed-thumbnail argument? Alternatively, just for testing, does downloading the video without additional args help? Maybe that will help us narrow down what might be causing this error.

What confuses me about the error you have is it seems like it's a parsing error for the URL. But looking at the URL, it seems completely fine! Hm. Let me know if removing the custom args helps.

Turns out that it's the subtitles messing this up. After disabling the subtitle downloads it passed fine. However, I really need subtitle downloading for a bunch of weeby videos (like vtubers). If you could fix that and being able to embed the thumbnail that'd be amazing! :D

Other than this it is working.

[Tzahi12345]

What's interesting is the error message you got is for when a URL does not exist at all.

One other thing that caught my attention when investigating this error is this excerpt from a StackOverflow answer:

Note: Enclose all arguments with ' ' for parsing safety.

When I looked back at the youtube-dl commands that YoutubeDL-Material runs, there are no apostrophes in sight. This might be what is causing the parsing error. Just a guess, but I haven't been able to reproduce your exact error so it's hard to know for sure.

One way you could test this (if you have the package installed) is by running the youtube-dl command on your own, one with and one without apostraphes:

Without:
youtube-dl -o video/%(uploader)s_[%(channel_id)s]/%(upload_date)s_-_%(title)s_[%(id)s].mp4 -f bestvideo[ext=mp4]+bestaudio[ext=m4a]/mp4 --write-info-json --print-json --download-archive video/merged.txt --write-sub --write-thumbnail --no-call-home --embed-thumbnail --sub-format srt/ass/best --sub-lang en,eng,de,ger,deu http://www.youtube.com/watch?v=J-2VSNPID2g

With:
youtube-dl -o 'video/%(uploader)s_[%(channel_id)s]/%(upload_date)s_-_%(title)s_[%(id)s].mp4' -f 'bestvideo[ext=mp4]+bestaudio[ext=m4a]/mp4' --write-info-json --print-json --download-archive video/merged.txt --write-sub --write-thumbnail --no-call-home --embed-thumbnail --sub-format 'srt/ass/best' --sub-lang 'en,eng,de,ger,deu' http://www.youtube.com/watch?v=J-2VSNPID2g

If the second one works, then we're in business.

[GlassedSilver]

Without apostrophes:
sh: syntax error: unexpected "("

With apostrophes:
ERROR: AtomicParsley was not found. Please install.

If we always wrap commands with apostrophes this would also help with the issue of not being able to have spaces in the path or filename right now. :)

Edit: Omitting the embedding of thumbnails again worked fine. Video with subs downloaded a-ok and all is dandy. Still would love the embedded thumbnail, so maybe stay on that? Other than that we have debugged this I think. :)

[Tzahi12345]

If we always wrap commands with apostrophes this would also help with the issue of not being able to have spaces in the path or filename right now. :)

Yup, it would kill two birds with one stone (well, hopefully two).

Omitting the embedding of thumbnails again worked fine. Video with subs downloaded a-ok and all is dandy. Still would love the embedded thumbnail, so maybe stay on that?

I'll play around with it a bit, but I'll maybe have AtomicParsley be an optional dependency for embedding thumbnails if that actually fixes the issue. For Docker I can even update the existing 3.6/latest image so it will be included.

Anyways, thanks for the debugging help! I'll keep this issue open until I get a fix out for both.

[Tzahi12345]

Hey! Just wanted to let you I've fixed the AtomicParsley bug (it will now be included in future Docker releases), and I've reworked the custom args to support filenames with spaces (related to #37).

Custom args (and global custom args) are now delimited by double commas (,,) rather than a space (relevant commit). This prevents the latter part of a file with a space in its name from being considered a new arg. It's double commas rather than a single comma so subtitle args, or any other args that use a comma will still work.

That should fix all the issues here. Let me know if anything else comes up!

[GlassedSilver]

Any reason you didn't use apostrophes as delimiters? Either way, will test asap, thanks so much!

[Tzahi12345]

No problem! About the apostraphes, forgot to address that: in my testing, I found that sometimes the apostraphes were being included as part of the arg. To prevent any errors by forcing their encapsulation, I just decided to use a double comma delimiter.

[GlassedSilver]

Cool, that's very thorough of you and good to hear!

Any ETA on v3.7 or v4.0? Just so I know when I can expect my latest stable build container to take over and to evaluate the CA again? (someone else already posted an unRAID CA, but I still intend to give this my own take on it and stand by my word of bridging unRAID forums to here and doing support and anything else that I can help out with.

[Tzahi12345]

Good question! I'd say 3 days give or take a day.

I was thinking of doing a release-candidate but I think I'll just use that time testing it a bit more. There's the nightly Docker releases anyways which serve that purposes.

We're almost there! I wanna make sure it's as bug-free as possible. All that's left is improvements to the args modifier. I want to make the args chips so it's more visually appealing and some other usability improvements. After that I gotta translate, package the app, and write the release notes.

I'll be changing strategies for the next few months to be more focused on bug fixes and usability and less focused on features. It's mostly related to me starting an internship this summer so I'll probably have slightly less time to invest. We'll see. Anyways, that's pretty much where I'm at on 4.0 and beyond.

[GlassedSilver]

Sounds wonderful, can't wait :) I do dig your prioritization as well, the chips will definitely go a long way, youtube-dl finally getting maneuverable settings has been long overdue. Especially in combination with running a centralized instance and many ways to interface and automate... 2020 sucks overall, but it's a damn good time for selfhosters/datahoarders who like to minimize their dependencies. :)

[GlassedSilver]

Hmmm, on latest nightly:

2020-05-04T03:58:20.872Z ERROR: Command failed with exit code 1: /app/node_modules/youtube-dl/bin/youtube-dl --dump-json -f best -o video/%(uploader)s [%(channel_id)s]/%(upload_date)s - %(title)s [%(id)s].%(ext)s -f best[ext=mp4] --write-info-json --print-json --download-archive video/merged.txt --no-call-home --write-thumbnail --write-info-json http://www.youtube.com/watch?v=-wd2_A8nIaM

Still fails :/

[Tzahi12345]

I think I still need to update the frontend files (it requires a simple build command, I'll do it within the next couple hours). Once that's done, make sure your args are delimited by the double commas. I'll also test your command myself and see if the results are successful.

[Tzahi12345]

I was able to get it working, as long as the args are being delimited by the double commas (I assume you're using the custom args in the advanced downloader):

-o,,video/%(uploader)s [%(channel_id)s]/%(upload_date)s - %(title)s [%(id)s].%(ext)s,,-f,,best[ext=mp4],,--write-info-json,,--print-json,,--download-archive,,video/merged.txt,,--no-call-home,,--write-thumbnail,,--write-info-json

Docker nightly is updated with the built frontend files, so you should be good to test it as well.

[GlassedSilver]

Getting this now :S

2020-05-05T08:19:47.383Z ERROR: Command failed with exit code 2: /app/node_modules/youtube-dl/bin/youtube-dl --dump-json -f best -o video/%(uploader)s [%(channel_id)s]/%(upload_date)s - %(title)s [%(id)s].%(ext)s -f best[ext=mp4] --write-info-json --print-json --download-archive video/merged.txt --no-call-home --write-thumbnail --write-info-json http://www.youtube.com/watch?v=-wd2_A8nIaM

[Tzahi12345]

I can't seem to reproduce this particular error. Let me know if I did everything right:

1. Set global args in settings to --no-call-home,,--write-thumbnail,,--write-info-json
2. Enable archive mode
3. Set custom output in advanced downloader to video/%(uploader)s [%(channel_id)s]/%(upload_date)s - %(title)s [%(id)s].%(ext)s

For me, the video downloads fine. Exit code 2 from youtube-dl (which you received) means there was a failure parsing args. This could be that the spaces in the file names are causing the args parsing to fail. I'm a bit hesitant to believe it since it works just fine on my end, but I do most of my testing on Windows and it would be strange that a Linux container errors when running on a Linux host but not a Windows host. It should all be the same, no?

You could verify this by adding underscores to the file name (not permanently, just for testing) and seeing if it works. If so, then spaces are for sure the culprit and I'll have to do some more investigating. Otherwise, maybe youtube-dl doesn't like some other arg in there. Maybe it's the --write-info-json being listed twice?

[GlassedSilver]

I can't seem to reproduce this particular error. Let me know if I did everything right:

1. Set global args in settings to `--no-call-home,,--write-thumbnail,,--write-info-json`

2. Enable archive mode

3. Set custom output in advanced downloader to `video/%(uploader)s [%(channel_id)s]/%(upload_date)s - %(title)s [%(id)s].%(ext)s`

For me, the video downloads fine. Exit code 2 from youtube-dl (which you received) means there was a failure parsing args. This could be that the spaces in the file names are causing the args parsing to fail. I'm a bit hesitant to believe it since it works just fine on my end, but I do most of my testing on Windows and it would be strange that a Linux container errors when running on a Linux host but not a Windows host. It should all be the same, no?

You could verify this by adding underscores to the file name (not permanently, just for testing) and seeing if it works. If so, then spaces are for sure the culprit and I'll have to do some more investigating. Otherwise, maybe youtube-dl doesn't like some other arg in there. Maybe it's the --write-info-json being listed twice?

TIL I'm quite the potato... I hadn't changed my arg delimiters in Settings/Downloader LOL

I am incredibly sorry. Oops

Side question: would it be possible to set a custom output path for subscriptions? I would like to have the same hierarchy in my subscriptions folder as in the videos one. :)

[Tzahi12345]

I am incredibly sorry. Oops

Haha it's always the little things, no worries :) Glad we could get that working.

Side question: would it be possible to set a custom output path for subscriptions? I would like to have the same hierarchy in my subscriptions folder as in the videos one. :)

It would probably be relatively easy to implement, maybe an optional input when subscribing where you can set the custom output path. The only complication is that the path would need to be relative to the {subscriptions folder}/{'channels' or 'playlists'}/{subscription name} folder. In the future, the videos won't need to get stored in that folder (related to this issue) as the file information will instead get stored on the database in the same way non-subscription videos get stored now.

Whenever I refactor subscription videos so they get to be stored in the database, I'll make it so the subscription custom path will be used if available.

[GlassedSilver]

I am incredibly sorry. Oops

Haha it's always the little things, no worries :) Glad we could get that working.

Side question: would it be possible to set a custom output path for subscriptions? I would like to have the same hierarchy in my subscriptions folder as in the videos one. :)

It would probably be relatively easy to implement, maybe an optional input when subscribing where you can set the custom output path. The only complication is that the path would need to be relative to the {subscriptions folder}/{'channels' or 'playlists'}/{subscription name} folder. In the future, the videos won't need to get stored in that folder (related to this issue) as the file information will instead get stored on the database in the same way non-subscription videos get stored now.

Whenever I refactor subscription videos so they get to be stored in the database, I'll make it so the subscription custom path will be used if available.

Whenever we talk I smile. :) Great news! If you want I can create an issue ticket for this to make it easier to track. :)

[Tzahi12345]

It's always a pleasure :) And about creating an issue, sure! That would be helpful.

Btw, I just released 4.0, so check that out when you can! Haven't noticed any bugs, but if you find any lmk and I'll be on it

[GlassedSilver]

Cool! Will test once my unRAID system is done testing parity. (I don't have cache drives, so performance is wonky during parity checks when also running all the other tasks)