Help setting this up

[Brancliff]

Hey! I'm running trying to set this up on a QNAP NAS. It's running QTS's Docker Station (I'm not sure if any of this will be important information). Anyways, I pulled the Docker-compose file and just changed the locations for the files it'll pull. When I navigate to the URL for the server, the browser just displays this json:

{
"name" : "305f54c99168",
"cluster_name" : "docker-cluster",
"cluster_uuid" : "B270t1WITSWJ1m1ooeqA1Q",
"version" : {
"number" : "7.14.1",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "66b55ebfa59c92c15db3f69a335d500018b3331e",
"build_date" : "2021-08-26T09:01:05.390870785Z",
"build_snapshot" : false,
"lucene_version" : "8.9.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}

I also got a TON of messages in the console terminal, I'm not sure what any of them mean. I think I might actually have more of this, the terminal might've cleared away some of it when I was checking on something else. It's attached here --> https://pastebin.com/kggPyasL Some of these are about my currently high disk usage - funnily enough, a lot of those are YouTube videos, and I'd love to move them onto Tubearchivist as soon as I can get it working.

[bbilly1]

I think that you aren't opening the Tube Archivist interface but the database. If you haven't changed anything from the defaults in the docker-comopse file, Tube Archivist serves the interface on port 8000.

It looks like elasticsearch starts up fine, as the welcome json you posted in your questions is a good sign, despite the high disk usage message, this might not be a problem right now, but at some point it will. Tube Archivist also downloads artwork from youtube, that will need space, then the index itself too...

[Brancliff]

Oh, that's exactly it! Thanks a lot! That makes a lot of sense, actually - Qnap's Container Station will pop up with a "link" icon to bring you to a container's page if it has one, and archivist-es links to port 9200 while archivist itself links to port 8000. Some more questions:

• When downloading an individual video, I got a server error (500). What's that about? I tried another individual download that was added to the queue fine-- is it on a video-by-video basis? Here's the link if that'll help (it's jams, and timely as well --> https://www.youtube.com/watch?v=UTZTzyaY7k4 )

• Anything I can do if the downloader queue gets stuck? I've ran the container for about an hour and it went smoothly but got caught on this one video. The status message on the download page just says "Downloading: [video title]". It's only 10 seconds long, so I know it's not still downloading or anything. Here's its link: https://www.youtube.com/watch?v=0MttD7_8SMc I tried hitting "Download Now", but that didn't help. Actually, I also tried hitting ignore, but after the video was added to the ignore list, it went back to saying "Now Downloading: [the video it was stuck on]". I hit "download now" on another video, and that worked, but then it went back to this ignored video.

• I think I goofed up the folder mounting. I downloaded a few videos, but I didn't see them in the folder they were supposed to be in, so I think I messed that up. My NAS lets me make "shared folders" like home/public/web and such. I have one called "Appdata" where container stuff goes. What would I type for, say, Appdata > Tubearchivist > youtube? I think when I used the docker-compose, I changed it to say something like this:

  volumes:

• ./volumes/tubearchivist/media:/appdata/tubearchivist/youtube

[bbilly1]

There was an issue with the "download now" button in a previous version. Make sure you are running the latest version, I've just published v0.0.3, you should be able to see the version number in the footer of the interface if you are running the latest version. Sometimes it is a good idea to delete old container versions, so things don't get mixed up. I'm not sure how QNAP handles that, I'm sure you'll find something in their documentation.

If the download get stuck, you can delete the partial file in cache/download. But if you don't have an volume there, you'll have to access the container over the command line. I'm working on a more dynamic download queue that could handle that directly from the interface.

When you create a volume in docker everything after the ':' is where the folder goes within the container, you can't change that. So that needs to be ':/youtube' and ':/cache' as described in the readme, otherwise things break. The part in front of the ':' is wherever the files will get stored on your host system, this you can choose however you like.

So I think, If I were you I'd start over with this, set the volumes correctly, then things should work out as expected.

[Brancliff]

Hey! Thanks for the help. I've changed up my docker-compose, but now I'm no longer able to get to the WebGUI. I'm having a problem with Elasticsearch now. I'll give you my docker-compose and what I got from Elasticsearch's terminal.

Docker-compose: https://pastebin.com/keR0Jkse
Elasticsearch: https://pastebin.com/pVeAJLMD

I'm still not sure if I have the folder mounting down-- What you said makes sense to me, but these errors make me wonder if I still messed it up somehow. Could I get a second opinion on the yml too?

[bbilly1]

This looks like the access denied issue as described in the readme. You can see, it's the exact same message at the bottom there.

[Brancliff]

Thank you! That was it-- I was so intimidated by the huge amount of logs that I didn't actually think about reading them...

I've set everything back up, and it turned out I actually did miss the YAML anyway, even though that wasn't the problem. Turns out I had to add /share/CE_CACHEDEV1_DATA/ before the shared folders.

So, I'm up and running at the GUI again! But, when trying to download the same test videos again, I got stuck on the same video. But, it's weird-- upon checking the cache > download folder, it seemed fine? There were three files, two webms and an mp4, and the mp4 looked right. Not sure if the sound was there, this video doesn't have any. Is there any way to just add them to the database then, or does it not work like that?

For the meantime, I deleted the temporary files. But I tried redoing the download, and it just got stuck again. What should I do if the delete doesn't fix the problem? I am on version 0.0.3 now, if that helps- I can see it on the footer.

Also, here's the video if that ends up meaning anything either - https://www.youtube.com/watch?v=0MttD7_8SMc

[bbilly1]

The URL you have provided fails for me too. When you try to download the video outside of Tube Archivist, you can confirm that it fails there too. Most likely an issue with that the video doesn't have any audio stream so when yt-dlp and ffmpeg try to combine things together...

That's a limitation of yt-dlp and ffmpeg and I have seen multiple issues raised on their platform. That's also what you see on your filesystem, the two webm files, one for video and one for audio and a broken mp4 file. So delete them from the cache folder and you should be able to continue, just not with that video.

Then if you need to learn more about how docker volumes work, they have great documentation about that.

[Brancliff]

Oh thank goodness-- if the video is out for you too, that means I didn't mess something up. I'm shocked that one of the test videos that I just randomly picked happened to be a lemon. I don't plan on downloading any more silent videos in the future, so it's not a big deal. Is that something that the yt-dlp devs would be able to fix? Or maybe like a timeout function so it knows to give up if its not going anywhere

I think we're almost done here, I've got a few final questions before I start moving everything over to this:

• 500 server error: Sometimes I get this error when trying to download a channel or video. What does it mean, and what should I do about it? Sometimes it'll go away if I switch link formats - like if I change a shortened youtu.be link to its regular link--
• YT-DLP: The default settings are great here - the only thing I want is to embed the thumbnail as the icon for the file while getting the best video/audio. (It'll be helpful if I end up organizing things a different way in the future) After all, Tubearchivist gets the thumbnails anyway. But, I can't get this functionality to work. The documentation says the command for this is "--embed-thumbnail". I put that in the "download format" options and just outputs this error:

[2021-09-25 14:57:29,680: WARNING/ForkPoolWorker-4] ERROR: [youtube] 2tdiKTSdE9Y: Requested format is not available
[2021-09-25 14:57:29,681: WARNING/ForkPoolWorker-4] retry failed download: 2tdiKTSdE9Y
[2021-09-25 14:57:29,681: WARNING/ForkPoolWorker-4]
[pid: 38|app: 0|req: 538/538] 10.0.0.134 () {44 vars in 858 bytes} [Sat Sep 25 19:57:30 2021] GET /downloads/progress => generated 17 bytes in 8 msecs (HTTP/1.1 200) 5 headers in 157 bytes (1 switches on core 0)

And thankfully I think I can understand most of what this means, but- in terms of yt-dlp, did I mess something up? I've also tried removing one or both of the dashes at the beginning of --embed-thumbnail, but that hasn't worked out either

• PyFilesystem: I saw this as one of the important planned features in the readme - what does it mean? I know there's a link to it that explains it, but it went way over my head. It seems really advanced, but I'm just wondering how it'll change the user experience

Thanks so much for all of the help you've been in addition to making this! Tubearchivist is a really cool idea and a lot of fun!

[bbilly1]

For the 500 errors, that happens when the parsing of the links failes. For example if you are trying do download a YouTube username and not a YouTube Channel. In the latest release v0.0.4 I have added some more checks so it doesn't show the blank error page.

The option for --embed-thumbnail is currently not implemented yet. But definitely useful, I have added it into the roadmap.

The PyFilesystem is just something I'm looking into to provide additional options on how the media folder gets stored. That won't change anything of how it works now.