opcache crashes on high volume IIS servers (Windows)

[juongithub]

Description

There is no way to reproduce the bug with code.
The sneakiest thing is that everything works as it should, but opcache fails when the server load increases beyond a certain threshold.
I assume, but of course I'm not sure, because of the shared memory. This assumption derives from the fact that when it happens all the PHP (cgi) instances close unexpectedly (opcache's shared memory should be the only thing connecting them). I also tried to change the opcache memory model, but nothing changes.
Furthermore, in 99% of the cases no error is visible on the Windows events, but obviously on the IIS user side it shows a 500 error.

The only way I've found to reproduce the bug is to overload a test server (with the same configuration as the production server), after some time (usually a few minutes), all PHP instances close and occasionally I see an error similar to this:

Faulting application name: php-cgi.exe, version: 8.0.28.0, time stamp: 0x63eb7b04
Faulting module name: php_opcache.dll, version: 8.0.28.0, time stamp: 0x63eb7e4f
Exception code: 0xc0000005
Fault offset: 0x0000000000168011
Faulting process id: 0x1f2c
Faulting application start time: 0x01d9d3b2249e086e
Faulting application path: /path/to/php-cgi.exe
Faulting module path: /path/to/php_opcache.dll
Report Id: de8f6365-8a29-4f20-b3cf-d2d160d39508
Faulting package full name: 
Faulting package-relative application ID: 

I got proof that the cause is linked to opcache overloading the same server after disabling opcache. The server becomes (obviously) very slow, but no PHP instances crash even after 1 hour.

PHP Version

Both TS and NTS, PHP 8.0.28 + PHP 8.0.29 + (PHP 8.0.?)

Operating System

Windows Server 2022 Datacenter

[juongithub]

I just completed another test, if I set opcache.file_cache_only = 1 opcache doesn't crash which would seem to confirm that the problem is in shared memory management.

[sagentac]

This sounds similar to the problem we are facing with Apache: #11549
Are there any news on this topic?

[juongithub]

Unfortunately no updates from our side. The issue continues to generate 500 errors randomly. Could one of the IIS failed request logs be useful? In most logs, the error appears to occur at the end of the request.

[Jan-E]

Did you try adding a mmap_base? For 64-bit PHP's the values are different. See the old comments by @pierrejoye here:
https://github.com/php/php-src/blob/PHP-8.2/ext/opcache/shared_alloc_win32.c#L208

With PHP 8.1 TS (loaded as mod_php) I am using the opcache config below. I haven't seen opcache crashes for months, on a low traffic server.

[opcache]
opcache.mmap_base=0x0000700000000000
opcache.cache_id="xp81.x64"
opcache.memory_consumption=256
opcache.interned_strings_buffer=64
opcache.max_accelerated_files=32531
opcache.enable_cli=0
opcache.validate_permission=1
opcache.use_cwd=1
opcache.revalidate_path=1
opcache.enable_file_override=1
opcache.validate_timestamp=1
opcache.revalidate_freq=60
opcache.file_cache_fallback=1
opcache.file_cache="c:\\temp\\php81.x64"
opcache.error_log="c:\\temp\\opcache.php81.x64.log"
opcache.jit=off

Some stats:

Cache hits: 2971901
Cache misses: 4602
Used memory: 137760648
Free memory: 130299080
Wasted memory: 375728
Interned Strings Used memory: 6958520
Interned Strings Free memory: 43372640
Cached scripts: 4127
Cached keys: 4127
Max keys: 32531

[juongithub]

Did you try adding a mmap_base? For 64-bit PHP's the values are different. See the old comments by @pierrejoye here:

Already tried, I tested with all the available models and with various configurations.
At the moment the only thing that seems to work is file_cache_only=1, but the random 500s have decreased, not disappeared.

[juongithub]

#11549 With opcache.file_cache_only=1 the it crashes once a day (Some random dll, not opcache). Today it was icuuc71.dll. With opcache.file_cache_only=0, the opcache.dll continuously crashes for a few minutes (5-10), and then heals itself.

@sagentac This is the exact behavior I experienced.

[juongithub]

@Jan-E following your suggestion, I'm trying the following opcache configuration, it seems to work better, although the 500 errors are not completely gone

[opcache]
opcache.mmap_base = 0x0000700000000000
opcache.enable = 1
opcache.enable_cli = 0
opcache.validate_timestamps = 1
opcache.max_accelerated_files = 32531
opcache.revalidate_freq = 300
opcache.save_comments = 0
opcache.huge_code_pages = 1
opcache.memory_consumption = 1024
opcache.interned_strings_buffer = 128
opcache.revalidate_path = 1
opcache.file_cache_fallback = 1
opcache.file_cache = "/path/to/opcache"
opcache.error_log = "/path/to/opcache_error.log"
opcache.enable_file_override = 1
opcache.validate_permission = 1
opcache.use_cwd = 1

opcache.jit = 1205
opcache.jit_buffer_size = 1024M

[Jan-E]

opcache.jit is known to have issues on Windows. The developer of the Wampserver has the same experience:
https://www.apachelounge.com/viewtopic.php?p=42199#42199
The performance gains do not weigh up to the error 500 crashes IMHO.

opcache.huge_code_pages = 1 is not supported on Windows
https://www.php.net/manual/en/opcache.configuration.php#ini.opcache.huge_code_pages

[juongithub]

opcache.jit is known to have issues on Windows. The developer of the Wampserver has the same experience:

Honestly at the moment I don't see any difference between enabling it or not

opcache.huge_code_pages = 1 is not supported on Windows

I know, but thanks for the tip

[Jan-E]

opcache.jit is known to have issues on Windows. The developer of the Wampserver has the same experience:

Honestly at the moment I don't see any difference between enabling it or not

If you do not see any performance differences either, better turn jit off. It complicates the inner workings of opcache.

opcache.huge_code_pages = 1 is not supported on Windows

I know, but thanks for the tip

I really would not know how opcache handles the situation when huge_code_pages are enabled on an OS that does not have huge pages. Does it assume that huge pages exist? Then errors are bound to happen.

[Jan-E]

opcache.memory_consumption = 1024
Wild guess: 1024 is on a boundary. The wincache extension once had problems with 1024, while 1022 went OK. You might try to set the memory consumption to 960 (15/16 of 1024).