sporadic opcache clear fails, file not found. But the file is present inside the release folder #722

Xippo · 2023-03-15T15:00:08Z

Expected Behavior

successful opcache clear over the specific php file

Actual Behavior

Sometimes and sometimes we got the follow error during the deployment:

<html><body>
<!--StartFragment-->

15-Mar-2023 15:28:35 | domain.de (TYPO3 CMS) TYPO3\Surf\Task\Php\WebOpcacheResetExecuteTask
-- | --
15-Mar-2023 15:28:35 |  
15-Mar-2023 15:28:35 | Warning: file_get_contents(https://domain.de/surf-opcache-reset-19058a0687ad66b0ecfd3b9f1fd231578489f3d7675b07ec7c3b4f929f515f47.php): failed to open stream: HTTP request failed! HTTP/1.1 404 Not Found
15-Mar-2023 15:28:35 | in /tmp/vendor/typo3/surf/src/Domain/Filesystem/Filesystem.php on line 35
15-Mar-2023 15:28:35 | Executing PHP opcache reset script at "https://domain.de/surf-opcache-reset-19058a0687ad66b0ecfd3b9f1fd231578489f3d7675b07ec7c3b4f929f515f47.php" did not return expected result
15-Mar-2023 15:28:35 | Stage cleanup

<!--EndFragment-->
</body>
</html>

sometimes it works fine and the file is executed, when not and we see this inside the log we can call the url and the file is there and the cache is flushed.
I was wondering if the tasks are all in sync, because if they are not and, for example, the copy process from the server from transfer to the release folder is not yet complete, then it could of course happen that at that moment the file is not yet there.

Perhaps there will also be a delay due to the updating of the labels. In order to cause less traffic, we will probably transfer this to the shared data anyway.

Maybe some one of you has some idea.

I have also seen approaches that would probably also solve the problem:
https://gist.github.com/jonaseberle/9bfc29b3af726c41b3cd9cec088fc722

Specifications

Surf Version: TYPO3 Surf 3.3.5
Application: TYPO3 11
PHP Version: PHP 7.4
Platform: Bamboo
Environment (CI): Linux/Mittwald
Deployment configuration:

<?php
/** @var \TYPO3\Surf\Domain\Model\Deployment $deployment */

$node = new \TYPO3\Surf\Domain\Model\Node('domain.de');
$node
    ->setHostname($node->getName())
    ->setOption('username', 'xxx')
    ->setOption('phpBinaryPathAndFilename', '/usr/bin/php');

$application = new \TYPO3\Surf\Application\TYPO3\CMS();
$application
    ->setDeploymentPath('/html/domain.de/v11')
    ->setOption('baseUrl', 'https://domain.de')
    ->setOption('webDirectory', 'web')
    ->setOption('symlinkDataFolders', ['fileadmin'])
    ->setOption('repositoryUrl', 'ssh://git@bitbucket.somedomain.de:7999/xxx.git')
    ->setOption('keepReleases', 1)
    ->setOption('composerCommandPath', 'composer')
    ->setOption('context', 'Production/Live/Mittwald')
    ->setOption('rsyncExcludes', [
        '.ddev',
        '.surf',
        '.git',
        'var/*',
        $application->getOption('webDirectory') . '/fileadmin',
        'composer.json',
        'composer.lock'
    ])
    ->addSymlinks(
        [
            'var/charset' => '../../../shared/Data/var/charset',
            'var/lock' => '../../../shared/Data/var/lock',
            'var/log' => '../../../shared/Data/var/log',
            'var/session' => '../../../shared/Data/var/session',
        ]
    )
    ->setOption(TYPO3\Surf\Task\TYPO3\CMS\FlushCachesTask::class . '[arguments]', [])
    //->addSymlink($application->getOption('webDirectory') . '/typo3conf/LocalConfiguration.php', '../../../../shared/Configuration/LocalConfiguration.php')
    ->addNode($node);
$deployment
    ->addApplication($application)
    ->onInitialize(
        function () use ($deployment, $application) {
            $deployment->getWorkflow()
                ->defineTask('InstallLanguagesTask', \TYPO3\Surf\Task\ShellTask::class, [
                    'command' => [
                        'cd {releasePath}',
                        'TYPO3_CONTEXT="Production/Live/Mittwald" /usr/bin/php vendor/bin/typo3cms language:update de'
                    ],
                    'ignoreErrors' => false,
                    'logOutput' => true
                ])
                ->beforeTask(\TYPO3\Surf\Task\TYPO3\CMS\SetUpExtensionsTask::class, \TYPO3\Surf\Task\TYPO3\CMS\CompareDatabaseTask::class, $application)
                ->afterTask(\TYPO3\Surf\Task\TYPO3\CMS\SetUpExtensionsTask::class, 'InstallLanguagesTask', $application)
                ->beforeStage('transfer', \TYPO3\Surf\Task\Php\WebOpcacheResetCreateScriptTask::class, $application)
                ->afterStage('switch', \TYPO3\Surf\Task\Php\WebOpcacheResetExecuteTask::class, $application);
        }
    );

Thx for your awesome work 👍

//edit we have removed the InstallLanguagesTask but the error shows up anyway

The text was updated successfully, but these errors were encountered:

simonschaufi · 2023-03-16T08:58:28Z

I consider this a duplicate to #221

Xippo · 2023-03-17T09:29:53Z

jep, seems so.
The interesting thing here is that we have just upgraded to TYPO3 11. So we went to version 3 of surf. The deployments with surf 2 always managed to constantly clear the opcache. Maybe there has been a change in the way symlinks are set. The problem seems to go in this direction:
https://forge.typo3.org/issues/99355

on our stage where mittwald "performance plus" is not active (so OPcache seems to be not active) we have no trouble. PHP is running over fpm.

We have now also tested the cachetool (https://gist.github.com/jonaseberle/9bfc29b3af726c41b3cd9cec088fc722), but here too access to the file via the domain is not (always) possible.

sabbelasichon · 2023-04-01T08:49:03Z

@Xippo You can use a static file shipped with the same identifier in every release. That´s how i fix this issue for myself. Use a very long and obscure identifier name. In that case security by obscurity is the way to go.

Xippo · 2023-04-06T08:00:28Z

The static file was my first attempt to fix the error, unfortunately without success. For some reason, the server needs some time until the symlink change is really there, or it almost seems as if a call has to be made in the frontend first so that the new link is active. However, this may be due to Mittwald and its "Performance Plus", although this should actually only be the extra OPcache, as I always understand it. We have now installed a smoke test after the switch, which takes 2-3 seconds, but then it works with the call of the static opcache clear file.

$smokeTestOptions = [
    'url' => $application->getOption('baseUrl'),
    'remote' => true
];

$deployment
    ->addApplication($application)
    ->onInitialize(
        function () use ($deployment, $application, $smokeTestOptions) {
            $deployment->getWorkflow()
                ->defineTask('TYPO3\\Surf\\Task\\Php\\WebOpcacheResetExecuteTask',
                    'TYPO3\\Surf\\Task\\Php\\WebOpcacheResetExecuteTask', [
                        'scriptIdentifier' => 'someFileName'
                    ]
                )
                //this smoketest alias curl call for the domain is important, it seems to fix paths
                // on server side (cached by performance plus of mittwald -> or better some tech inside)
                // so the task WebOpcacheResetExecuteTask can execute
                ->defineTask('SmokeTest', \TYPO3\Surf\Task\Test\HttpTestTask::class, $smokeTestOptions)
                ->beforeTask(\TYPO3\Surf\Task\TYPO3\CMS\SetUpExtensionsTask::class, \TYPO3\Surf\Task\TYPO3\CMS\CompareDatabaseTask::class, $application)
                //->beforeStage('transfer', \TYPO3\Surf\Task\Php\WebOpcacheResetCreateScriptTask::class, $application)
                ->afterStage('switch', 'SmokeTest', $application)
                ->afterTask('SmokeTest', \TYPO3\Surf\Task\Php\WebOpcacheResetExecuteTask::class, $application)
                ->afterTask(\TYPO3\Surf\Task\Php\WebOpcacheResetExecuteTask::class, TYPO3\Surf\Task\TYPO3\CMS\FlushCachesTask::class, $application);

        }
    );

sabbelasichon closed this as completed Apr 1, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sporadic opcache clear fails, file not found. But the file is present inside the release folder #722

sporadic opcache clear fails, file not found. But the file is present inside the release folder #722

Xippo commented Mar 15, 2023 •

edited

simonschaufi commented Mar 16, 2023

Xippo commented Mar 17, 2023

sabbelasichon commented Apr 1, 2023

Xippo commented Apr 6, 2023

sporadic opcache clear fails, file not found. But the file is present inside the release folder #722

sporadic opcache clear fails, file not found. But the file is present inside the release folder #722

Comments

Xippo commented Mar 15, 2023 • edited

Expected Behavior

Actual Behavior

Specifications

simonschaufi commented Mar 16, 2023

Xippo commented Mar 17, 2023

sabbelasichon commented Apr 1, 2023

Xippo commented Apr 6, 2023

Xippo commented Mar 15, 2023 •

edited