Apache SSI Enabled hangs Blitz CP interface

[adrianjean]

I'm having a strange issue.

When I enable Apache Server Side Includes a2enmod include, and turn SSI on in the Blitz CP interface (after updating my vhost to process html/php +includes), subsequent refreshes of the Blitz CP interface hang the server. I have to restart Apache to unlock it.

I check the running processes on the server and see there's nothing taking up the CPU.

I wonder, if when SSI is checked on, the SSI "confirmation tip" feature is somehow locking the server while it's trying to check if SSI is available, while SSI is enabled?

I also noticed that when I do manage to enable SSI, my site shows in the browser as if it's loading a binary file instead of actual HTML — I turned off gzip in Blitz CP, thinking it was related, but no effect.

I am using a few calls to craft.blitz.includeCached() to bring in cached content, so not caching the whole site.

Apache: 2.4.5
PHP: 8.2.11
MySQL 8.0.30
Linux 5.15.0-87-generic

CraftCMS: Pro 4.5.9
Blitz: 4.6.0

[bencroker]

Can you do a test to confirm that SSI has been enabled and is working by manually including an SSI tag and pointing it to a test.html file?

<!--#include virtual="/test.html" -->

[adrianjean]

Welp... interestingly — I just updated Apache to 2.4.58 and a few other libs (seemingly unrelated to include), and put that test SSI tag into the page and it works fine. Now everything works just fine!

No idea what the issue was. Closing this question.

[adrianjean]

Update: everything works, except the Blitz CP interface still hangs.
I'm using PHP-FPM if that makes a difference.

The Blitz CP interface loads once, but then refreshes of the page seem to hang.

[adrianjean]

It almost looks like the Blitz CP interface is redirecting back to the main settings page but hangs the browser.

I tried it on 2 other Craft sites. One on the same server, and another on a different server, and all the Blitz installs seem to have this issue.

[bencroker]

Hmm, this must be something to do with your environment setup, as you’re the first person to report this. I’ve attempted a fix in 709d84c, though not sure if this will make any difference.

You can test this by running composer require "putyourlightson/craft-blitz:bugfix/ssi-detection as 4.7.1".

Is your CP site available at the root domain, so at https://domain.com/admin as opposed to https://domain.com/subpath/admin?

[adrianjean]

Sadly, I tried the fix and doesn't seem to help. :(
CP is available at the root domain. https://domain.com/admin

If mod_include is enabled, I'm unable to load the Blitz CP.
Accessing it, locks up the site, and whatever browser I use seems to increase in CPU use as it tries to load the page.
However, top doesn't show any process working on the server, so it feels like the issue is at the browser level.
I tried it on MS Edge running on a remote windows box (Browserstack) and same behaviour.

Tried it on 2 different servers running the same CraftCMS version, but each with slightly different versions of Apache (2.4.58 and 2.4.52), and and PHP (8.2.x and the other 8.1.x)

CraftCMS logs record nothing because the browser locks up before writing to the log file.
Apache logs show the page request for the Blitz CP page, but that's it.

I might be completely wrong about this, but is there a way to temporarily disable/comment out the code for the SSI detection to see if that's even the issue?

[adrianjean]

Another update.
So I left the browser window open and after about 5 minutes, the Blitz CP eventually does load.
So I turned on the "debug toolbar" and refreshed the page to see if I could get any more info. Here's some additional info:

1. Once the Blitz CP loads, it says that SSI is NOT enabled (even tho it is), so I feel like there is something happening here.
2. Total processing time: 300,404 ms
3. Peak memory was 63.083 MB... fairly "normal" for CP pages, and the box has plenty of memory and CPU to go around.
4. php memory_limit set to 512MB
5. The "Profiling Timeline" in the debug toolbar has a consistent chart for all 5 minutes, as opposed to a slow ramp up for most CP pages. So it's doing something, and it's stuck on it.

[bencroker]

Just to confirm, have you enabled SSI by following the instructions at https://putyourlightson.com/plugins/blitz#ssi-enabled?

Also, in what specific scenarios does the CP load without any delays and in what does it not?

[bencroker]

I might be completely wrong about this, but is there a way to temporarily disable/comment out the code for the SSI detection to see if that's even the issue?

You can remove the SSI include in the settings page to test what difference it makes.

craft-blitz/src/templates/_settings.twig

Line 343 in baeffba

<!--#include virtual="{{ detectSsiActionUrl }}" -->

[adrianjean]

I do have all the redirects in place - and I should say, on the front end, SSI all works totally fine now.

It's now just the Blitz CP interface that's locking up.

I'll try disabling the code for the Blitz CP SSO setting to see if it has an effect.

[adrianjean]

I might be completely wrong about this, but is there a way to temporarily disable/comment out the code for the SSI detection to see if that's even the issue?

You can remove the SSI include in the settings page to test what difference it makes.

craft-blitz/src/templates/_settings.twig

Line 343 in baeffba

<!--#include virtual="{{ detectSsiActionUrl }}" -->

THIS DOES have a positive effect! Removing the SSI, allows the Blitz CP to load without issue.

I noticed that {{ detectSsiActionUrl }} outputs:

/?action=blitz/settings/detect-ssi

And the SSI include code is:

<!--#include virtual="{{ detectSsiActionUrl }}" -->

Which means when rendered is:

<!--#include virtual="/?action=blitz/settings/detect-ssi" -->

This is now beyond my knowledge, but could the URL being a query be an issue? But then I checked and that URL does return your js constant:

<script>
const blitzSsiDetected = true;
</script>

Could the javascript be somehow looping and not getting what it needs and then just times out after 5 mins?
Aside from that everything seems to work in the individual parts, but when assembled, it locks things up.

[adrianjean]

I changed that include line to a test file and it loads the Blitz CP just fine:

<!--#include virtual="/test.html" -->

So there's something about including the javascript that's loading and hanging the page.

[bencroker]

With the change in 709d84c, detectSsiActionUrl should return a CP URI, like /admin?action=blitz/settings/detect-ssi. Can you confirm this is the case?

Can you also test a few variations of the include URI?

/index.php/?action=blitz/settings/detect-ssi
/_includes/?action=blitz/settings/detect-ssi

craft-blitz/src/templates/_settings.twig

Line 343 in baeffba

<!--#include virtual="{{ detectSsiActionUrl }}" -->

[adrianjean]

Applying the change in 709d84c

Causes {{detectSsiActionUrl}} to output: /admin?action=blitz/settings/detect-ssi

I tried both paths you suggest — hard coding them into the _settings.twig file on the include line:

Example: <!--#include virtual="/index.php/?action=blitz/settings/detect-ssi" -->

Here's the strange thing....
If I quit the browser, and open it again, and visit the Blitz CP, it loads fine (sometimes after restarting php-fpm). If I try to load the Blitz CP again (like refreshing the page, or clicking on the CP icon again for it), it hangs.

So it seems on "second visit" of the Blitz CP page, it hangs. Is it possible that somehow the const blitzSsiDetected = true;
is persistent and causing an issue on reload of the Blitz CP page?

Thank you so much for hanging in here on this with me.

[bencroker]

Can you try a using a different web browser? The fact that restarting the browser sometimes solves this might point to a client-side issue rather than a server-side one.

I don’t see how or why a JavaScript variable would cause issues for the browser.

[adrianjean]

That was my first thought as well - so I tried it in Firefox, Safari, and even on MS Edge in an off-site Windows VM to rule out my home network. All behaves the same way. First load is fine, second load hangs for five or so minutes and comes back showing that SSI is not available (even tho it is)

I even tried it on 2 other CraftCMS installs - one on the same LAMP server and one on another LAMP server.

Seems odd but must be something I am doing... can't think of what tho.

Maybe there is no solution and I can just remember to comment out the BlitzCP SSI tag in the settings template when there is an plugin update.

Selfishly ... I could ask for a non JavaScript way to report to users that SSI is enabled or not, but sounds like I'm an edge case so maybe wait and see if anyone else has these issues.

[bencroker]

Selfishly ... I could ask for a non JavaScript way to report to users that SSI is enabled or not, but sounds like I'm an edge case so maybe wait and see if anyone else has these issues.

Have you determined that the JS code is the cause of the issue?

[adrianjean]

So this was a good question — I investigated and it seems it's not related to JS code.
Some things I've noticed:

• {{ detectSsiActionUrl }} — when run by itself, outputs: /admin?action=blitz/settings/detect-ssi

• That URL outputs this JS code: <script>const blitzSsiDetected = true;</script>

• I took the JS code and put it into a test.html file and replaced the SSI include in _settings.twig with one that points to that html file... — and.... it works!!

• I even thought, well maybe the twig code wasn't being executed before the SSI tag was needed by the server, so I did the following, which required that twig be interpreted to output the filename. and it works as well!

{% set urlFile = '/test.html' %}
<!--#include virtual="{{ urlFile }}" -->

So it's not the JS code that's causing the issue. Then I thought....

• The twig tag {{detectSsiActionUrl}}(and all related functions) are the issue: <!--#include virtual="{{ detectSsiActionUrl }}" -->.

• Is it possible the twig tag {{detectSsiActionUrl}} is not being interpreted/output before the SSI tag needs to be actioned by the server?

• Could the SSI tag be cached in some way and instead of outputting the URL to include, it outputs nothing, or is not interpreted as twig when the BlitzCP is refreshed?

• Does the {{detectSsiActionUrl}} have to be a function that returns a query string URL... or can it just be a static HTML template in the plugin that has the JS code?

[bencroker]

It’s very unlikely that the Twig tag is at fault. I’m almost certain that it is a server config issue.

Can you show me the Apache config you used to enabled SSI? I assume you followed the instructions at https://putyourlightson.com/plugins/blitz#ssi-enabled?

[adrianjean]

Here's my vhost config.
In addition I enabled the include module for Apache: a2enmod include
Feels all normal, and includes work! In fact the website is totally fine, except for the BlitzCP.

<IfModule mod_ssl.c>
<VirtualHost *:443>

  ServerName website.ca
  DocumentRoot /srv/www/website.ca/web/

  <Directory /srv/www/website.ca/web/>
    Options -Indexes +FollowSymLinks +MultiViews +Includes +ExecCGI
    AddOutputFilter INCLUDES .html .php
    AllowOverride All
    Require all granted
  </Directory>

  ErrorLog /srv/logs/website.ca-error.log
  CustomLog /srv/logs/website.ca-access.log combined

  RewriteEngine On

  #- Blitz cache rewrite for gzip files
  RewriteCond %{DOCUMENT_ROOT}/cache/blitz/%{HTTP_HOST}/%{REQUEST_URI}/%{QUERY_STRING}/index.html.gz -s
  RewriteCond %{HTTP:Accept-encoding} gzip
  RewriteCond %{REQUEST_METHOD} GET
  RewriteCond %{QUERY_STRING} !token= [NC]
  RewriteRule .* /cache/blitz/%{HTTP_HOST}/%{REQUEST_URI}/%{QUERY_STRING}/index.html.gz [L]

  #- AddEncoding allows you to have certain browsers uncompress information on the fly.
  AddEncoding gzip .gz

  #- Serve correct content types, and prevent mod_deflate double gzip.
  RewriteRule \.html\.gz$ - [T=text/html,E=no-gzip:1]

  #- Blitz cache rewrite
  RewriteCond %{DOCUMENT_ROOT}/cache/blitz/%{HTTP_HOST}/%{REQUEST_URI}/%{QUERY_STRING}/index.html -s
  RewriteCond %{REQUEST_METHOD} GET
  RewriteCond %{QUERY_STRING} !token= [NC]
  RewriteRule .* /cache/blitz/%{HTTP_HOST}/%{REQUEST_URI}/%{QUERY_STRING}/index.html [L]

  SSLCertificateFile /etc/ssl/website.ca/fullchain.pem
  SSLCertificateKeyFile /etc/ssl/website.ca/privkey.pem
  Include /etc/ssl/options-ssl-apache.conf

</VirtualHost>
</IfModule>

[adrianjean]

Doing more testing today.
It's almost as if the php/twig code is not executed fast enough for the include to work — which sounds crazy, but on first load after a php-fpm/apache restart, maybe the code would execute a fraction slower on "first" run. But then on refresh, code execution is much faster and maybe php/twig isn't quick enough to respond to build the URL for the SSI before it's executed.

I now feel like I have idea what I am talking about anymore...

But ... when I do this:

<!--#include virtual="/test.html" --> it works on refreshes. This file has the same code as your public function actionDetectSsi() The only difference is that it's "hard coded" so no php/twig needs to be executed to generate this.

and yet these don't work on refresh:

• <!--#include virtual="{{ detectSsiActionUrl }}" -->
• <!--#include virtual="https://website.ca/admin?action=blitz/settings/detect-ssi" -->

So whereas the test.html file is already rendered and ready to include, the /admin?action= URL query maybe is "slower" (?) and doesn't respond in time for the SSI to execute.... a total guess at this point.

If all this script is meant to do is just prove that SSI works, could it not be just a static file in the blitz plugin that has the JS code? Tho.. it's a far less sexy solution than the public function approach you have.

Not sure what else I can try. Aside from declaring my own function that overrides the Blitz function of {{detectSsiActionUrl}} which could then point to a "real" file on my site.... I remember doing this kind of thing back in my Drupal days... but this is sadly beyond my CraftCMS knowledge. But maybe it's the same issue just a different form of it.

[bencroker]

My guess is that this specific server-side include is causing an endless redirect due to the action URL format /?action=blitz/settings/detect-ssi.

Looking at your Apache configuration, where is the redirect to index.php that is required for Craft to work? Specifically, what is contained in https://github.com/craftcms/craft/blob/main/web/.htaccess

[adrianjean]

My guess is that this specific server-side include is causing an endless redirect due to the action URL format /?action=blitz/settings/detect-ssi.

Interesting -- could be!?

Looking at your Apache configuration, where is the redirect to index.php that is required for Craft to work? Specifically, what is contained in https://github.com/craftcms/craft/blob/main/web/.htaccess

The redirect to index.php is in the /web/.htaccess file. The Apache config I shared is my vhost config file.

[bencroker]

Please share your .htaccess file too.

[adrianjean]

Here's my /web/.htaccess file:

<IfModule mod_rewrite.c>
    RewriteEngine On

    # Send would-be 404 requests to Craft
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteCond %{REQUEST_URI} !^/(favicon\.ico|apple-touch-icon.*\.png)$ [NC]
    RewriteRule (.+) index.php?p=$1 [QSA,L]
</IfModule>

[bencroker]

I’m not sure what to suggest. I could add a config setting to prevent detecting whether SSI is enabled, but only as a last resort.

Are you able to replicate this in a local development environment, and if so, can you set up a repo for me to test with?

[adrianjean]

No no — I wouldn't ask to make a check box only for me.

Yup I'll spool up a dev environment that you can access — I'll try it with a vanilla install of CraftCMS and Blitz and go from there. Give me a day or so.

[bencroker]

Any update on this, @adrianjean?

[bencroker]

I just released 4.8.0 with a small improvement to the SSI detection process. It would be great if you could test and let me know the general status of this issue.

[adrianjean]

Apologies — was trying to get that dev server going and got pulled into squashing other bugs on the site.
Thanks for keeping on this — I just tried the update and the issue sadly remains :(

I'll get that dev server up so you can see what's going on — that might help.

[bencroker]

Sounds good!

[adrianjean]

Ok!... I managed to isolate the problem... but I still have no idea how/what it could be. Here's what I did:

1. I setup a fresh, brand new server identical to the problem server, but using nothing from the problem server in the setup so it was fresh, but the same.
2. I installed a fresh, vanilla install of CraftCMS + Blitz to see if the issue showed up. That way I could tell if it's a server config issue, or a CraftCMS+Blitz issue. Everything worked as expected!!

Next:

1. I tarred up one of the problem websites, and installed it on the new server.
2. The problem was there, but different... the page didn't hang (like the others) but the SSI check could not "see" that SSI was enabled. So the <!--#include virtual="{{ detectSsiActionUrl }}" --> would not hang, but it would always report SSI is not enabled. Yet, when I created a "real file" <!--#include virtual="/test.html" --> with the exact same javascript inside to the detectSsiActionUrl script — it is able to see that SSI is enabled.

Next:

1. On the original problem server, I created a new vanilla install of CraftCMS+Blitz — I want to see if there is something special about this server — turns out — the vanilla install works perfectly. Ugh...
2. I cloned the composer.json from the problem server into this new vanilla server and turned on all the plugins one by one and checked to see if there was some kind of interaction between a combination of plugins I had been using. Nothing — everything worked fine.
3. I then exported a project config from the problem website, and installed it on to this vanilla, working test site and BAM the problem shows up!!

Next - just to confirm:

1. I created a totally vanilla install of CraftCMS
2. Copied the plugin config from the problem composer.json to the vanilla composer.json
3. composer upgrade — to get the plugins
4. Installed and applied the project config from the problem site to the vanilla site
   5. BAM! — the problem appears again!!

SO... somehow, on a few sites now — all the ones I tested, I managed to configure them in such a way that it is creating this issue — but I have no clue now what it could be.

Could I send you the problem project config and the list of plugins from composer.json to see if this happens to you as well?

[bencroker]

A database dump and the composer files would be great, to support@putyourlightson.com

[adrianjean]

Sent!

[bencroker]

Thanks. That works just fine for me in a DDEV environment.

[bencroker]

I just made a tweak in 0cc030a, which is very unlikely to help, but may still be worth a shot.

[bencroker]

I’ve also added a new config setting that will allow you to override the SSI tag format, which may help with troubleshooting

// Default value:
'ssiTagFormat' => '<!--#include virtual="{uri}" -->',

// For troubleshooting:
'ssiTagFormat' => '<!--Placeholder for an SSI tag -->',

You can test this by running composer require "putyourlightson/craft-blitz:dev-feature/ssi-tag-format as 4.9.1".

[adrianjean]

Update:

0cc030a — sadly doesn't have an effect. The include statement itself is where it fails.

Configuration setting:
When I use the default setting, issue is still there. BUT I am able to override it using the new config setting (better than editing the code every time there is a blitz update):

• I create /web/blitz-detect.html
• Copy your <script>blitzSsiDetected = true;</script> into it.
• Set that as the config 'ssiTagFormat' => '<!--#include virtual="/blitz-test.ssi" -->', — and it all works!!

I think with the config setting as a workaround, I'd say this issue is resolved.

[adrianjean]

Small note: The dev-feature/ssi-tag-format as 4.9.1 breaks the includeCached feature.

[bencroker]

Please elaborate.

[adrianjean]

Ah — ok I was confused about the ssiTagFormat config setting.
I spun up a local ddev as well and everything seems to work fine there.
At this point I've run out of clues how to isolate this any further to find a solution. There may just not be a fix for this.

[bencroker]

Yeah, it’s a real mystery. I’ve added a new detectSsiEnabled config setting in 042901c which setting to false will prevent that check in the CP.

[bencroker]

Released in 4.9.1.

[adrianjean]

Amazing — thank you SO much. If I every find out what's going on I'll update this here.