Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[New instance] https://invidious.nerdvpn.de #241

Closed
7 tasks done
Sommerwiesel opened this issue May 22, 2022 · 22 comments · Fixed by #254
Closed
7 tasks done

[New instance] https://invidious.nerdvpn.de #241

Sommerwiesel opened this issue May 22, 2022 · 22 comments · Fixed by #254
Assignees

Comments

@Sommerwiesel
Copy link
Contributor

Sommerwiesel commented May 22, 2022

URL

https://invidious.nerdvpn.de

Mandatory checks

  • Instance has a domain name
  • Instance is served over HTTPS
  • Statistics (/api/v1/stats) are enabled
  • Instance is properly configured (including the mandatory post install configuration)
  • Instance has an automatic hourly restart setup

Maintainer chart

  • Ensure that my instance is up to date (less than one month old)
  • Ensure a proper uptime of my instance (around 90%)

Host country

Germany

Man in the Middle

None

Source code URL

https://github.com/Sommerwiesel/invidious

Analytics

None

Additionnal informations

  • The only source code changes as of 05-22-2022 are css changes and there are no plans in changing anything else.
  • The instance is open for new user registrations.
  • The banner text is german only.
  • The server has a planned downtime of up to 15min every day around 1am (UTC) for backups to an encrypted external storage and updates.
  • The hourly restart is done via cron at 42 * * * *
@Sommerwiesel Sommerwiesel changed the title New public instance <invidious.nerdvpn.de> [New instance] invidious.nerdvpn.de May 22, 2022
@Sommerwiesel
Copy link
Contributor Author

@unixfox unixfox changed the title [New instance] invidious.nerdvpn.de [New instance] https://invidious.nerdvpn.de Jun 1, 2022
@unixfox
Copy link
Member

unixfox commented Jun 1, 2022

Added on uptime robot: https://stats.uptimerobot.com/89VnzSKAn/791917306

@Sommerwiesel
Copy link
Contributor Author

Sommerwiesel commented Jun 27, 2022

I have a question about the hourly restart.
I understand why it's neccessary to circumvent the inevitable "invidious breaking down" after a while.
However, I find it hard to believe it's the best solution to this.

Can someone explain what exactly happens when invidious crashes?
Do we get any kind of status/error/log?
Is there some way to monitor this and only restart invidious the moment such a crash happens?

I have full root access to my server, surely there is some better way a cronjob/monitor can watch the invidious service/website/whatever and react accordingly.
I use the official docker image with a custom systemd unit starting it.

The problem I have with the hourly restart is, that it also breaks active sessions. I often watch videos which are longer than an hour and this results in the video breaking multiple times because of the restart.

Also, what if invidious breaks only 15min after the last restart, that would leave the service with a downtime of 45min until the next.

I would really like to implement a better solution that "just" a hourly restart.

Thanks for any input.

@unixfox
Copy link
Member

unixfox commented Jun 27, 2022

Nowadays if you don't have use_quic enabled invidious will never really crash but it can become sluggish due to some unknown memory leaks: iv-org/invidious#1438

This is why we recommend restarting it every hour because a public instance will receive way more traffic than a private one and will become sluggish more quickly.

If you don't want to break "active sessions", you can also run two invidious processes at the same time behind a reverse proxy and just restart one at a time. Do note that it's not normal if videojs (the player of invidious) doesn't retry if a request fails due to invidious being offline for a short period of time, you should probably open a bug report for that on invidious.

@SamantazFox
Copy link
Member

SamantazFox commented Jun 27, 2022

Can someone explain what exactly happens when invidious crashes?

I have absolutely no clue, sorry :/ If you find out why, please open an issue with your findings !

Do we get any kind of status/error/log? Is there some way to monitor this and only restart invidious the moment such a crash happens?

I have full root access to my server, surely there is some better way a cronjob/monitor can watch the invidious service/website/whatever and react accordingly. I use the official docker image with a custom systemd unit starting it.

Nope.

The phenomenon is weird in many aspects. It's not as simple as "invidious breaks" like your program have a segfault. Invidious becomes sluggish (slow to respond) as the time goes on, and you get more occurences of weird things, like slow response times, subscriptions not refreshing anymore, database queries failing, etc..

I don't know if it's related to the language itself (maybe the internal Crystal scheduler doesn't handle that many fibers?), to one of the libraries (shards) that we're using, to the garbage compiler, or to our own code. There are so many things that I don't fully understand in the chain that could cause those problems.

As unifox said, one big source of problems was QUIC, which is a monster (it embeds its own SSL library). Since we changed the default behavior from "use it" to "not use it", it also reduced the amount of crashes encountered by users / instance admins.

I would really like to implement a better solution that "just" a hourly restart.

Thanks for any input.

I'd love to, too, but at the moment I don't have the knowledge nor the time to dig in depth in valgrind, strace, or other memory analysis tools usage.

Technically, invidious can run fine for multiple hours (generally > 6h on a medium sized instance) but in order to reduce the amount of problems (= issues) we have to handle, we decided that an 1h restart was mandatory. This indeed helped with a lot of common problems.

Until a definitive fix is found, that requirement will stay.

@Sommerwiesel
Copy link
Contributor Author

Hello there,
thanks a lot for the detailed answer and explaining why the whole thing isn't that simple.
Huge kudos for taking your time writing this 👍🏻

As of now, I'll just accept this weird bug and keep the hourly restart active.

Have a nice week everyone :)

@TheFrenchGhosty
Copy link
Member

TheFrenchGhosty commented Jul 1, 2022

@Sommerwiesel Instance is properly configured and has good uptime.

Please confirm that you indeed have auto-restart setup?

If yes, are you ready for it to be added to the instance list?

PS: Some, in my opinion, issues with your CSS:

  • the RSS button isn't visible on the channel/playlist page
  • the "video/playlist/community" category "switcher" isn't visible on the channel page
  • the "watch on youtube" button isn't visible anywhere (on purpose I guessing) except on a playlist page

If you explicitly want those "issues", it's not a problem.

@Sommerwiesel
Copy link
Contributor Author

Sommerwiesel commented Jul 1, 2022

@TheFrenchGhosty Yes, please add my instance to the public list.

I confirm, I have a hourly cronjob restarting invidious active and I actively monitor the successful execution of said cronjob.

Thanks for the feedback. The "video/playlist/community" category "switcher" not being visible is a bug in the css, I will fix it asap. The rest is as intended.

A pleasure working with you guys/gals and have a nice weekend :)

@FuccDucc
Copy link

FuccDucc commented Dec 5, 2022

This instance should be removed again, due to the situation described in these links:

libredirect/browser_extension#555
Sommerwiesel/invidious#6

I hope that Invidious takes a stance against such behavior (discrimination on the basis of country/nationality/ethnicity)
I doubt that the instance would be admitted for listing should they have requested it today, with the same situation going on

// Edit: i noticed the discussion at places, like: #149 (comments at the very bottom, including what Ammako called him bringing it up with them), and especially just Sommerwiesel/invidious#6 after which i believe and understand what the instance operator is saying. Looks resolved to me

@Sommerwiesel
Copy link
Contributor Author

Sommerwiesel commented Dec 5, 2022

Cheers,
the whole thing goes a little deeper than that and I explained here what happened and why I had to take "some" measures to prevent me suddenly emptying my whole pockets for this instance...
However, what I did was very wrong and it turned out to be more discriminative than actually help the situation.
I apologize for the whole mess. I never wanted to discriminate. It was a heat-of-the-moment decision, which originated from me getting a warning and a huge invoice from my hosting provider.

I will never do somthing like this again i.e. block anyone out who wants to use the instance legitimately and I hope this whole mess doesn't lead to my instance getting removed.

It was a really dumb mistake for me to react like this. I'm sorry...

@unixfox
Copy link
Member

unixfox commented Jan 6, 2023

@Sommerwiesel

Please double-check that you have configured hmac_key in the config file or INVIDIOUS_CONFIG then restart the instance.

You must set a random generated value for the parameter hmac_key:! On Linux, you can generate it using the command pwgen 20 1.

@Sommerwiesel
Copy link
Contributor Author

Done

@Sommerwiesel
Copy link
Contributor Author

Sommerwiesel commented Feb 15, 2023

Hello, I'm currently getting hit with DMCA claims and a warning from my hosting provider that they'll shut down the server. Until the issue has been dealt with, I'm forced to stop the instance.

I've already sent the DMCA template and time will show if this can get resolved in a good way. After all, I'm technically not doing anything illegal. It's just the same DMCA-bullying that plagues many YouTube channels.

Expect my instance to be down until 02-20-2023.

I will switch to a maintenance page later, but right now 502 Bad Gateway will have to suffice.

Sorry.

@unixfox
Copy link
Member

unixfox commented Feb 15, 2023

Sad to see that many instances providers have been getting DMCA requests lately...

I do have plans to submitting a tutorial on how to hide your servers behind a proxy server hosted at a provider that don't care about DMCA requests or understand what invidious is.

For the time being, I'll remove your instance from the list. I hope your provider will understand that invidious is not a threat.

@unixfox
Copy link
Member

unixfox commented Feb 15, 2023

Detailed comment on what will be documented related to hiding the servers: #348

@Sommerwiesel
Copy link
Contributor Author

Btw, I have not forgotten about the matrix room. Can you send me an invite?

@unixfox
Copy link
Member

unixfox commented Feb 15, 2023

Please join the matrix room and ping me there: https://matrix.to/#/#invidious:matrix.org

@unixfox
Copy link
Member

unixfox commented Jun 3, 2023

Hello,

We temporarily removed your instance in order to avoid the Invidious users to use an instance that doesn't work for the moment.

Once you have fixed the issue (The video returned by YouTube isn't the requested one. (WEB client) (VideoNotAvailableException)) on your instance, please ask to get your instance back in the list.

We are fully aware that this issue is not your fault, but for the moment we can't do anything than filtering in the list the instances that works from the instances that don't. Again, like said previously, we are working on the problem, and all progress will be posted in #3822.

Thank you for your understanding.

@Sommerwiesel
Copy link
Contributor Author

Sommerwiesel commented Oct 22, 2023

@unixfox Please add my instance back to the public list. Ipv6 connectivity has been fixed now.

Moreover, can you add the Tor version to the list as well?
jemgkaq2xibfu37hm2xojsxoi7djtwb25w6krhl63lhn52xfzgeyc2ad.onion

It's been up for almost a month now and has already been used by visitors from Russia.

Thanks :)

unixfox added a commit that referenced this issue Oct 31, 2023
also reformat some things

related #241 (comment)
@Sommerwiesel
Copy link
Contributor Author

@unixfox
Changed my onion domain.
PR: #511

@Sommerwiesel
Copy link
Contributor Author

Changed hoster from Germany to Ukraine after German hoster prooved too problematic with DMCA requests.

#515

@bugmaschine
Copy link
Collaborator

Should be merged now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants