Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add config option to disable http requests in System Checks to prevent server crashes with mod_security #17589

Closed
BigBerny opened this issue May 19, 2021 · 44 comments · Fixed by #18063
Assignees
Labels
Bug For errors / faults / flaws. Regression Indicates a feature used to work in a certain way but it no longer does even though it should.
Milestone

Comments

@BigBerny
Copy link

When opening the Admin area since 4.3 it shows a few more issues like "Required Private Directories" and a new database issue (can't see it right now, since server is down). When I open System Check to find out more about it, the page doesn't load and the whole server is not available anymore (ERR_CONNECTION_REFUSED). I can't even access Plesk.

It's reproducible and didn't happen in 4.2. Which log might be useful to find out what's going wrong?

Expected Behavior

Opening System Check should open it

Current Behavior

Opening System check crashes server.

Steps to Reproduce (for Bugs)

Open System check

Your Environment

Can't access it right now but what I know:

  • Matomo Version: 4.3
  • PHP Version: 8.0
  • Server Operating System:
  • Additionally installed plugins:
@BigBerny BigBerny added the Potential Bug Something that might be a bug, but can't be reproduced (yet). label May 19, 2021
@Findus23
Copy link
Member

Hi,

No matter how big of a bug there is in Matomo, it should never be able to bring down your PHP server.
I'd recommend you to look into the php log (and also your webserver logs or php-fpm logs) to find out more about what is going wrong.

@BigBerny
Copy link
Author

Hmm... What could it be then? I just have these lines in de log:
[Wed May 19 13:07:13.555599 2021] [authz_core:error] [pid 1007] [client xxx.xxx.xxx.xxx:58208] AH01630: client denied by server configuration: /var/www/vhosts/domain.app/s.domain.app/tmp/
[Wed May 19 13:07:13.567639 2021] [authz_core:error] [pid 1004] [client xxx.xxx.xxx.xxx:58212] AH01630: client denied by server configuration: /var/www/vhosts/domain.app/s.domain.app/tmp/empty
[Wed May 19 13:07:13.588946 2021] [authz_core:error] [pid 1006] [client xxx.xxx.xxx.xxx:58218] AH01630: client denied by server configuration: /var/www/vhosts/domain.app/s.domain.app/lang/en.json
[Wed May 19 13:07:21.143921 2021] [authz_core:error] [pid 1006] [client xxx.xxx.xxx.xxx:58454] AH01630: client denied by server configuration: /var/www/vhosts/domain.app/s.domain.app/tmp/
[Wed May 19 13:07:21.154434 2021] [authz_core:error] [pid 1005] [client xxx.xxx.xxx.xxx:58458] AH01630: client denied by server configuration: /var/www/vhosts/domain.app/s.domain.app/tmp/empty
[Wed May 19 13:07:21.175353 2021] [authz_core:error] [pid 1003] [client xxx.xxx.xxx.xxx:58464] AH01630: client denied by server configuration: /var/www/vhosts/domain.app/s.domain.app/lang/en.json

@Findus23
Copy link
Member

Hi,

Which web server and which PHP are you using (php-fpm?)?

If you use php-fpm, you might want to look into its error logs (/var/log/php7.4-fpm.log or similar). Maybe there are just not enough child processes to handle the incomming requests.

@BigBerny
Copy link
Author

BigBerny commented May 19, 2021

Apache and I tried, PHP 8.0 nginx and FPM, and PHP 7.4 FPM. I don't think there are more logs but I have to check. Child processes could be the issue but why should this always happen when I open System Check?

@sgiehl
Copy link
Member

sgiehl commented May 19, 2021

the system check actually tries to fetch some files/directories of Matomo via http request. If those directories are not "protected" and the requests are processed by the webserver / php this can cause some additional processes.

@BigBerny
Copy link
Author

But then how to fix it? Is this something Matomo should fix by not running all requests at the same time?

@diosmosis
Copy link
Member

Hi @BigBerny, the check only runs one http request at a time, which means its probably the first one that fails. I'm guessing you're using shared hosting and can't change the web server config? We could disable the individual check, if you'd like to do that, add this file /path/to/matomo/config/config.php with the following contents:

<?php

use Piwik\Plugins\Diagnostics\Diagnostic\RequiredPrivateDirectories;

return [
    'diagnostics.disabled' => array(
        DI\get(RequiredPrivateDirectories::class),        
    ),
];

Though if you're using shared hosting, it might also be helpful to talk to them about this, since Matomo can initiate HTTP requests like this in other places.

@patermatzka
Copy link

same problem here...
running it on plesk, even followed the official install guide to the line...
I can change the webserver-config, but what should I add there?

@HCl-not-HCi
Copy link
Contributor

HCl-not-HCi commented Aug 8, 2021

Same problem here.
When trying to access the system check page, the following happens:

2021-08-08 01:49:41	Error     403  GET /config/config.ini.php HTTP/1.0                                                                                                        
2021-08-08 01:49:41	Error     403  GET /tmp/ HTTP/1.0                                                                                                                         
2021-08-08 01:49:41	Error     403  GET /tmp/empty HTTP/1.0                                                                                                                    
2021-08-08 01:49:41	Error     403  GET /tmp/cache/tracker/matomocache_general.php HTTP/1.0                                                                                    
2021-08-08 01:49:41	Error     403  GET /lang/en.json HTTP/1.0                                                                                                                 
2021-08-08 01:49:41	Access    200  GET /index.php?module=Installation&action=systemCheckPage&idSite=1&period=day&date=yesterday&showtitle=1&random=8985 HTTP/1.0              
2021-08-08 01:49:41	Error          AH01630: client denied by server configuration: /var/www/vhosts/example.com/subdomain.example.com/config/config.ini.php                    
2021-08-08 01:49:41	Error          AH01630: client denied by server configuration: /var/www/vhosts/example.com/subdomain.example.com/tmp/                                     
2021-08-08 01:49:41	Error          AH01630: client denied by server configuration: /var/www/vhosts/example.com/subdomain.example.com/tmp/empty                                
2021-08-08 01:49:41	Error          AH01630: client denied by server configuration: /var/www/vhosts/example.com/subdomain.example.com/tmp/cache/tracker/matomocache_general.php
2021-08-08 01:49:41	Error          AH01630: client denied by server configuration: /var/www/vhosts/example.com/subdomain.example.com/lang/en.json                             
2021-08-08 01:49:45	Access    200  GET /index.php?module=Installation&action=getEmptyPageForSystemCheck HTTP/1.0                                                              

It seems like I sometimes even get the error when opening the settings where the system check is embedded into a panel:

system check error

After that I have to wait 10 minutes until I can access the website again...

The code that goes into the config.php above did work but I think that's not the right solution (#17589 (comment)).

The child process settings are configured with ondemand (that's what Plesk shows; Plesk doesn't allow me to modify these settings), so that should not be the problem as far as I understand.

pm.max_children       5
pm.max_requests 
pm                    ondemand
pm.start_servers      1
pm.min_spare_servers  1
pm.max_spare_servers  1

The same thing also happened when opening the DSGVO/GDPR tools.

@tsteur tsteur added Bug For errors / faults / flaws. Regression Indicates a feature used to work in a certain way but it no longer does even though it should. and removed Potential Bug Something that might be a bug, but can't be reproduced (yet). labels Aug 8, 2021
@tsteur tsteur added this to the 4.5.0 milestone Aug 8, 2021
@tsteur
Copy link
Member

tsteur commented Aug 8, 2021

Marking this for now as a regression for us to look into it if there's anything we can do better as the system check should ideally always work.

@wi-wissen
Copy link

I had the sam issue. But use php 7.4.
I missed the last step disabling ModSecurity (https://matomo.org/faq/how-to-install/how-do-i-install-matomo-with-plesk/)

@BigBerny
Copy link
Author

And this helped? Unfortunately, I don't have access to this setting. I can ask the Hoster maybe but before it worked properly.

@wi-wissen
Copy link

Yes, that had solved the problem for me.

@geekdenz geekdenz self-assigned this Sep 10, 2021
@geekdenz
Copy link
Contributor

@BigBerny Could you please check if following the FAQ above helps you? Please ensure to look carefully for the setting or asking your host. If following the FAQ does not help or there is no such setting for you and the host didn't help you either, please let us know.

I suspect, given all the above, we could have a setting that indicates plesk/modsecurity or better, disables some system checks when modsecurity is detected before it crashes and indicates that as a warning. To test this we would either need to install Apache+ModSecurity ourselves or have you test some code for us remotely.

Better yet would be of course, that we avoid this. How @Findus23 points out, Matomo should not be able to crash the webserver in any case, but I guess if there is a bug that puts it into an infinite loop it is something we will want to fix.

However, most of the above is speculative.

In the first instance, please double check the FAQ and whether this can be resolved by using the documentation or otherwise human level. There are too many different configurations we would have to write code for when it is more desirable to concentrate our efforts on high impact, new features and other prominent bugs.

I hope you understand and appreciate @BigBerny that we want to provide the best possible product in as little time as possible. If you have any further ideas or concerns, please don't hesitate to reach out again. Thanks for raising this, as it does affect people.

@BigBerny
Copy link
Author

I checked the FAQ and the only thing which we didn't do is to disable mod_security. I asked our hoster to do that. Don't know if it's possible though since it's shared hosting.

One more thing: It didn't happen in 4.2 so there must be something new which leads to the problem.

@BigBerny
Copy link
Author

The hoster was fast and gave me access to the setting. Unfortunately even after disbling it and restart System Check, it crashed the server and it's offline now...

When opening the admin part it says red "Required Private Directories" by the way.

@BigBerny
Copy link
Author

BigBerny commented Sep 13, 2021

The server is up again. I ran "./console core:create-security-files" from cronjobs in the meantime. And now I just have a yellow warning:
image

I did start full System Check again but it does crash again.

@HCl-not-HCi
Copy link
Contributor

I had a conversation with my hoster ("IP-Projects" in Germany) and they told me that ModSecurity was already disabled (but they did not give me access to toggle the setting).
They also could not help me with the problem and do not have more information (they just told me that the .htaccess was not compatible with Apache 2.4 which is like the first search result in google - I hope, you devs checked that already as I'm no expert regarding .htaccess files ;)).

Btw, the Apache server does not crash (at least in my case), it's more like a temporary IP block which lasts around 10 to 15 minutes (in my case) and results in a permanent block after trying many times. I asked my hoster whether they know something about this but I'm not sure whether I will get an answer as they already told me to ask the software manufacturer for help.

My configuration should be the same as written in the FAQ except for the Cron job which should not be relevant for this problem.

@geekdenz
Copy link
Contributor

geekdenz commented Sep 13, 2021

Thanks @HCl-not-HCi and @BigBerny for your quick replies.

I checked and the file doing the Private Directories check was introduced between 4.2.1 and 4.3.1 which also introduces the curl command.

Could you both try running this script on your hosting and report if it causes the hanging or other problems?

https://gist.github.com/geekdenz/63c820a004c59100612c9bd81b7fc73c

Please replace the string REPLACE_WITHYOUR_DOMAIN.com with your domain and then send the output unless there are secrets you don't want to share in which case, please describe the output or obfuscate any secrets.

I suspect your host might not resolve the domain correctly locally. If that is the case, we cannot fix the cause with Matomo. However, we could add a setting to work around this problem.

@geekdenz
Copy link
Contributor

Actually, just thinking about it and looking at some code, you could use a forward proxy to possibly work around this problem, if the domain is the problem:
https://matomo.org/faq/troubleshooting/faq_121/

Note however, it might have an impact on other things in the Matomo installation as well.

@BigBerny
Copy link
Author

BigBerny commented Sep 14, 2021

@geekdenz How can I run it? If I upload it to the matomo directory and access it (e.g. https://s.mydomain.com/curl.php) I get "403 Forbidden".

Edit: If I just add use echo "test"; in the script, it outputs "test" correctly.

Edit2: The error log says:
[Tue Sep 14 21:22:43.703352 2021] [authz_core:error] [pid 25389] [client XXX.XXX.XXX.XXX:45694] AH01630: client denied by server configuration: /var/www/vhosts/mydomain.com/s.mydomain.com/config/config.ini.php

@geekdenz
Copy link
Contributor

@BigBerny Thanks for trying this.

403 is actually expected and not wrong in some configurations of the server. What do you mean by "it is crashing"? That you get a 403 Forbidden or that you cannot access it for a while because the request hangs?

403 means that the resource is not allowed to be accessed which can be achieved by configuring the web server and does not mean it crashed. It might mean it is configured in .htaccess files.

If it crashes, could you try accessing with your browser:

Paste the source (obtained by right-clicking in the page and selecting View Source), check if there are any secrets and obfuscate them, into something like https://gist.github.com/ and posting the links here?

If it crashes anywhere above, I think it is a problem from your hosting side. Please let us know how it goes, so we can potentially rule out a bug in Matomo. Good luck! 👍

@HCl-not-HCi
Copy link
Contributor

@geekdenz For me, the script also throws a 403 (including a custom error page) as expected when accessing config.ini.php (similar to the log excerpt I pasted in #17589 (comment)).

Execept for the log entries and the 403 response, nothing else happens for me when running your script (even when holding F5 in the browser to initiate many of these requests).
But when triggering the system check in matomo, my IP gets blocked by the host (the host confirmed that they block clients who try to access forbidden content multiple times).
I tried to reproduce getting blocked without using the system check, but it's pretty random and I cannot reproduce it correctly (I also have to wait every time I was successful^^). I had most luck by spamming the /lang/en.json endpoint (i.e., holding F5 to reload the page continuously). I will try to do some more tests.

All in all, I can say that the problem is most likely a result of too many invalid requests sent by the system check which result in 403 responses. Too many 403 responses result in an IP block in my case.

@HCl-not-HCi
Copy link
Contributor

Just tried it one more time in my browser:

  1. Many requests to /config/config.ini.php using your script: No problem
  2. Many requests to /tmp: IP block

But still no evidence as I tried 1. just before 2.

@BigBerny
Copy link
Author

The reason I thought the server crashes was that I also can't access Plesk anymore via https://userid.hosttech.eu but I noticed I still can via https://userid.hosttech.eu:8443

It seems I'm blocked from accessing the server (also my other subdomains are affected), except Plesk over port 8443.

So it seems to be an IP block.

@BigBerny
Copy link
Author

BigBerny commented Sep 14, 2021

It could be Fail2Ban: https://docs.plesk.com/en-US/onyx/administrator-guide/server-administration/protection-against-brute-force-attacks-fail2ban.73381/
Unfortunately, I don't have access to this setting to try to disable it or whitelist my IP.

@geekdenz
Copy link
Contributor

In that case I agree with @tsteur and a configuration setting to simply not do the HTTP requests suffices.

@geekdenz
Copy link
Contributor

If the user has this problem at installation, they could just add the setting to config/global.ini.php.

@geekdenz
Copy link
Contributor

@BigBerny and @HCl-not-HCi

Could you please download these files manually in their raw versions and copy them to the corresponding directories on your hosting:

https://github.com/matomo-org/matomo/blob/m-17589-4.4.1/config/global.ini.php
to config/global.ini.php
https://github.com/matomo-org/matomo/blob/m-17589-4.4.1/plugins/Diagnostics/Diagnostic/RequiredPrivateDirectories.php
to plugins/Diagnostics/Diagnostic/RequiredPrivateDirectories.php

Also, please change

disable_http_diagnostics = false

to

disable_http_diagnostics = true

and let us know if this solved the problem for you, so we can apply it to our main branch and update the documentation.

@geekdenz
Copy link
Contributor

FYI, did you try what @diosmosis pointed out above?

This issue seems to relate
#17430

@BigBerny
Copy link
Author

Patch works 👍

@BigBerny
Copy link
Author

I just don't know if it should be enabled by default. If all Plesk users by default are affected, many will have problems installing it.

@geekdenz
Copy link
Contributor

I wonder if Plesk or whatever is causing this issue could be detected and the check disabled then.

Could you try to run this script on your server and share whether it reveals any information that could be due to Plesk or related to the issue?

Again, be careful, secrets might be revealed and only share what you believe is OK.

https://gist.github.com/geekdenz/fd8c72fc037d79f306d3a06e5e4784c4

@geekdenz
Copy link
Contributor

WARNING! May reveal secrets, so be careful with sharing!

Try this instead:

<pre><?php
echo '$GLOBALS = ';
print_r($GLOBALS);

@HCl-not-HCi
Copy link
Contributor

Then there's the case where the system report actually seems to crash see
image

this might be because of a time out configured in Apache or similar. Say max request time is configured to 30 seconds. Be great to ideally find out if this is due to such a request timeout or whether there is actually an error that isn't caught yet. It looks like we're catching any error during the HTTP request so the problem might be a timeout issue.

@tsteur The error I got there was most likely the same problem as when opening the system check, because I also got blocked and the log contained the same errors (but this error occcurs only sometimes, most likely older system check results get cached).

@geekdenz Your fix is working for me too 👍

I don't think trying to detect Plesk (if even possible) to disable the http check, is a 100% safe solution. As @BigBerny pointed out, this is probably Fail2ban which might not be enabled in all Plesk instances, and much worse, it could be installed in other environments where Plesk is not used.

Nevertheless, I tried the script above. But I don't think there is anything helpful in there:

Output (some parts changed because of secrets)
$GLOBALS = Array
(
    [_GET] => Array
        (
        )

    [_POST] => Array
        (
        )

    [_COOKIE] => Array
        (
            [MATOMO_SESSID] => abc
        )

    [_FILES] => Array
        (
        )

    [_SERVER] => Array
        (
            [USER] => user
            [HOME] => /var/www/vhosts/example.com
            [SCRIPT_NAME] => /issue-tests/server2.php
            [REQUEST_URI] => /issue-tests/server2.php
            [QUERY_STRING] => 
            [REQUEST_METHOD] => GET
            [SERVER_PROTOCOL] => HTTP/1.0
            [GATEWAY_INTERFACE] => CGI/1.1
            [REMOTE_PORT] => 40554
            [SCRIPT_FILENAME] => /var/www/vhosts/example.com/statstest.example.de/issue-tests/server2.php
            [SERVER_ADMIN] => [no address given]
            [CONTEXT_DOCUMENT_ROOT] => /var/www/vhosts/example.com/statstest.example.de
            [CONTEXT_PREFIX] => 
            [REQUEST_SCHEME] => https
            [DOCUMENT_ROOT] => /var/www/vhosts/example.com/statstest.example.de
            [REMOTE_ADDR] => <IP address 1>
            [SERVER_PORT] => 443
            [SERVER_ADDR] => <IP address 2>
            [SERVER_NAME] => statstest.example.de
            [SERVER_SOFTWARE] => Apache
            [SERVER_SIGNATURE] => 
Apache Server at statstest.example.de Port 443


            [PATH] => /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
            [HTTP_COOKIE] => MATOMO_SESSID=abc
            [HTTP_SEC_FETCH_USER] => ?1
            [HTTP_SEC_FETCH_SITE] => none
            [HTTP_SEC_FETCH_MODE] => navigate
            [HTTP_SEC_FETCH_DEST] => document
            [HTTP_UPGRADE_INSECURE_REQUESTS] => 1
            [HTTP_DNT] => 1
            [HTTP_ACCEPT_ENCODING] => gzip, deflate, br
            [HTTP_ACCEPT_LANGUAGE] => de,en-US;q=0.7,en;q=0.3
            [HTTP_ACCEPT] => text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
            [HTTP_USER_AGENT] => Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:92.0) Gecko/20100101 Firefox/92.0
            [HTTP_CONNECTION] => close
            [HTTP_X_ACCEL_INTERNAL] => /internal-nginx-static-location
            [HTTP_X_REAL_IP] => <IP address 1>
            [HTTP_HOST] => statstest.example.de
            [proxy-nokeepalive] => 1
            [HTTPS] => on
            [PASSENGER_DOWNLOAD_NATIVE_SUPPORT_BINARY] => 0
            [PASSENGER_COMPILE_NATIVE_SUPPORT_BINARY] => 0
            [UNIQUE_ID] => unique id
            [FCGI_ROLE] => RESPONDER
            [PHP_SELF] => /issue-tests/server2.php
            [REQUEST_TIME_FLOAT] => 1631709261.1153
            [REQUEST_TIME] => 1631709261
        )

    [_REQUEST] => Array
        (
        )

    [_ENV] => Array
        (
        )

    [GLOBALS] => Array
 *RECURSION*
)

@geekdenz
Copy link
Contributor

Would it be worth creating a release 4.4.2 @tsteur ?

Also, I merged into 4.x-dev, but it is not general Diagnostic, because we know it works as is. I think it is good enough though because it resolves a corner case for users using Plesk or Fail2Ban or similar.

We should probably create an FAQ.

@tsteur
Copy link
Member

tsteur commented Sep 15, 2021

Would it be worth creating a release 4.4.2 @tsteur ?

Not sure I understand for what?

Next planned release will be 4.5.0. Possibly in around 2 or 2.5 weeks time

@geekdenz
Copy link
Contributor

@tsteur I branched off 4.4.1 here:

#18001

So, that fix has no release. I guess we could just have users wait for the 4.5.0 release if it will happen in 2-3 weeks.

@moquin
Copy link

moquin commented May 13, 2022

I was having this same problem. Whenever I ran a diagnostic I lost access to the website. I am running on Plesk and found in Fail2ban it had banned my ip. Turned Fail2Ban off and everything runs normally.

@MatomoForumNotifications

This issue has been mentioned on Matomo forums. There might be relevant details there:

https://forum.matomo.org/t/ip-address-blocked-after-opening-matomo-err-connection-timed-out/48045/2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug For errors / faults / flaws. Regression Indicates a feature used to work in a certain way but it no longer does even though it should.
Projects
None yet