Talk with HTTP/2

[tiagoefreitas]

The installer says Talk doesn't work with HTTP/2, but this can be fixed according to this :

nextcloud/spreed#602 (comment)

Its a configuration issue, so it may not be fixed automatically just by upgrading Talk.

[enoch85]

As @Schnookeuz pointed out, using the mpm_event worker in combination with php-fpm is the recommended way of running Nextcloud Talk.

Problem is, we don't run php-fpm. The VM is built and designed to work for everything. Adding HTTP/2 in general may break other things. That's why we made it a choice and not per default in the installer scripts. But if you as a end user want to run HTTP/2, then it's up to you.

I already use HTTP/2 in my Nginx Reverse Proxy so I don't need it on the backend. That's one of the reasons it's harder for me to try it live. But if you find a way to activate it without breaking anything else, then be my guest and write a PR. 👍

Would be awesome, since HTTP/2 is better.

[shadoxx]

@enoch85
I can confirm that running Apache 2.4 with php-fpm doesn't break anything else related to Nextcloud. I've ran a heavily used production instance for more than six months in this configuration without issue. It is light on resources and blazing fast.
I'd be happy to make a PR once I've understood how the VM is built. It looks like the script located in the root of the repo called nextcloud_install_production.sh does the heavy lifting for getting everything configured?

[enoch85]

@shadoxx Thanks for your offer. I'm not convinced though. Same as HTTP/2 I don't know of any of the scripts will break on the change to php-fpm and I don't have time to test all the scripts.

You are very welcome with a PR though. I understand if it will take time as you need to check all the scripts and all the lines.

It looks like the script located in the root of the repo called nextcloud_install_production.sh does the heavy lifting for getting everything configured?

Yes that's correct.

The main design is that everything is built on the install script and the setup script. That's what this VM started with. Since then everything is divided into smaller pieces, and a while ago we also added lib.sh where all the $vars and functions are stored so that they can be fetched upon every load.

[shadoxx]

You are very welcome with a PR though. I understand if it will take time as you need to check all the scripts and all the lines.

Understood. So, to test this script as is, do I just spin up a fresh Ubuntu VM and run it? I just learned about this project through the reference to my comment in another issue. After a cursory look, it does look like it'll take me a bit of time to get everything working. Definitely do-able though.

[enoch85]

Definitely do-able though.

That's great news!

Can yo tell me what the biggest difference between php-fpm and the Apache module (cli) is?

[shadoxx]

I think the differences have been explained better elsewhere, so I will quote a better source: https://serverfault.com/a/645765

mod_php is running a PHP as Apache module - that is PHP request is run under Apache process with everything that goes with it - Apache processes are defined by Apache configuration, PHP is run with Apache permission etc.

PHP-FPM is PHP's FastCGI implementation; PHP-FPM runs as a standalone FastCGI server and Apache connects to the server using Apache's module, usually mod_fcgid or mod_fastcgi; I personally think this is much better than running as mod_php, but it depends on your requirements and is also a little more complex; in this configuration, permission, processes related stuff & everything is run under PHP configuration, PHP user etc. & Apache connects to PHP as to a server; in this configuration it is also possible to have pool of PHP servers and to have PHP server on physically different machine than Apache. They say this is almost as fast as using Apache's module and there are benefits of better control over PHP configuration.

So, basically:

CGI, Fast-CGI are protocols; CGI is slow, Fast-CGI is much faster
mod_php (with underscore) and PHP-FPM are two basic ways how to run PHP
mod_SuPHP is similar to mod_php but can change the user/group that the process runs under

tl;dr Improved security, better scaleability, lighter on resources.

[enoch85]

tl;dr Improved security, better scaleability, lighter on resources.

And more complex, harder to debug for Linux noobs which this VM is aiming for - even though it's used by Linux PROs as well.

If you can give me a POC that works better than the original one, then let's do it. We might even use a different branch and release one PHP-FPM version just to get it tested thoroughly before we put it in master.

[enoch85]

OK, I've looked at some videos and did some reading - digging into the differences. Apparently PHP-FPM is recommended for Apache 2.4 which is used in the VM. So, I'm fine with changing.

@shadoxx Please move forward with your PR. Thank you very much!

[enoch85]

Oh, and btw. To answer your question:

do I just spin up a fresh Ubuntu VM and run it?

Yes, when I test something new I usually does it on a Digitalocean VPS, or localhost on a local VM.

So basically, just install Ubuntu 18.04, once installed do a snapshot so that you can revert the installation process when you test the scripts. Make changes and revert to a fresh install of Ubuntu everytime you want to test your changes. Make sure to include all the scripts and choose all the extras once you do to make sure nothing breaks.

Also, a recommendation would be to check all the scripts, and search for anything with PHP or Apache in it, to see where it's used or set.

[enoch85]

@shadoxx Actually, if you start a PR we can work on it together. I have nothing planned for this weekend so I can help you with all the changes.

[tiagoefreitas]

The collaboration going on looks good 👍

If php-fpm is the recommended way of running Nextcloud Talk, and this is the "official" vm, it makes sense to use the best possible configuration.

I installed with HTTP/2 because I don't need talk video for now and thought a future update of talk would solve the issue.

But now I see it a vm upgrade is needed, do you think you will release one that does all the necessary changes, or a complete reinstall of another branch will be necessary?
In that case I may as well disable HTTP/2 for now, I can check the code that enabled it, or is it better to reinstall to do that?

Is there a recommended way to move all user data to another install in case I need to start fresh later?
Just getting started and want to minimize future changes and downtime.

Thank you

[enoch85]

@tiagoefreitas Well, you have to keep in mind that the Nextcloud VM isn't mainly made to work with Talk, there are so many other scripts and factors that needs to be weighed in. If PHP-FPM breaks 10 scripts but solves one problem, then I wouldn't go forward, but now it seems like it's recommended anyhow by Apache since it's the new way of dealing with PHP in general.

The change to PHP-FPM will happen in the whole VM, and not just for Talk, meaning that it will require some effort to test it properly before put in production. I'm thinking we'll make a seperate branch called "php-fpm" or similar so that we can test everything several times before we put it in master.

With the upcoming release of Nextcloud 14, adding PHP-FPM to the VM would be a major change and suitable for a major version bump.

Then we have backwards compability to keep in mind as well. How do we ensure that ppl running Nextcloud 12 won't break anything by using the update script for example?

So with all that said, there are a lot to think of when making a change like this. It affects so many users since this VM is downloaded about 150 times per day.

Is there a recommended way to move all user data to another install in case I need to start fresh later?
Just getting started and want to minimize future changes and downtime.

Just do a regular import and export of the DB and data if you want to start fresh, noting too complicated. It's documented here and here.

[enoch85]

@shadoxx Started to work on a separate branch. Feel free to dig in!