Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Executing grunt refresh or grunt exec:all in production mode, leads to error "Unable to retrieve deployment version of static files from the file system." #29570

Closed
1 of 5 tasks
Eddcapone opened this issue Aug 17, 2020 · 18 comments
Labels
Issue: ready for confirmation Reported on 2.3.5-p2 Indicates original Magento version for the Issue report. stale issue

Comments

@Eddcapone
Copy link

Eddcapone commented Aug 17, 2020

Preconditions (*)

  1. Magento 2.3.5-p2

Steps to reproduce (*)

  1. Make the folder pub your project root
  2. Put magento in production mode. php bin/magento deploy:mode:set production
  3. Create a theme.
  4. Setup grunt.
  5. Execute grunt exec:all

Expected result (*)

  1. It should compile.

Actual result (*)

  1. You will get Unable to retrieve deployment version of static files from the file system. and the command fails and the file pub/static/deployed_version.txt gets deleted.

This error is getting logged:

[2020-08-17 05:25:31] report.CRITICAL: Can not load static content version. [] []
[2020-08-17 05:25:31] report.ERROR: Unable to retrieve deployment version of static files from the file system.
#0 vendor/magento/framework/App/View/Deployment/Version.php(67): Magento\Framework\App\View\Deployment\Version->readValue('production')
#1 vendor/magento/module-theme/Model/Url/Plugin/Signature.php(87): Magento\Framework\App\View\Deployment\Version->getValue()
#2 vendor/magento/module-theme/Model/Url/Plugin/Signature.php(65): Magento\Theme\Model\Url\Plugin\Signature->renderUrlSignature()
#3 vendor/magento/framework/Interception/Interceptor.php(146): Magento\Theme\Model\Url\Plugin\Signature->afterGetBaseUrl(Object(Magento\Store\Model\Store\Interceptor), 'https://relaunc...', 'static', $
#4 vendor/magento/framework/Interception/Interceptor.php(153): Magento\Store\Model\Store\Interceptor->Magento\Framework\Interception\{closure}('static', false)
#5 generated/code/Magento/Store/Model/Store/Interceptor.php(143): Magento\Store\Model\Store\Interceptor->___callPlugins('getBaseUrl', Array, Array)
#6 vendor/magento/framework/Url.php(482): Magento\Store\Model\Store\Interceptor->getBaseUrl('static', false)
#7 vendor/magento/framework/View/Asset/Repository.php(284): Magento\Framework\Url->getBaseUrl(Array)
#8 vendor/magento/framework/View/Asset/Repository.php(233): Magento\Framework\View\Asset\Repository->getFallbackContext('static', NULL, 'frontend', 'company/fresh', 'de_DE')
#9 vendor/magento/module-developer/Console/Command/SourceThemeDeployCommand.php(172): Magento\Framework\View\Asset\Repository->createAsset('css/styles-m.le...', Array)
#10 vendor/symfony/console/Command/Command.php(255): Magento\Developer\Console\Command\SourceThemeDeployCommand->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Con$
#11 vendor/magento/framework/Interception/Interceptor.php(58): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output$
#12 vendor/magento/framework/Interception/Interceptor.php(138): Magento\Developer\Console\Command\SourceThemeDeployCommand\Interceptor->___callParent('run', Array)
#13 vendor/magento/framework/Interception/Interceptor.php(153): Magento\Developer\Console\Command\SourceThemeDeployCommand\Interceptor->Magento\Framework\Interception\{closure}(Object(Symfony\Componen$
#14 generated/code/Magento/Developer/Console/Command/SourceThemeDeployCommand/Interceptor.php(104): Magento\Developer\Console\Command\SourceThemeDeployCommand\Interceptor->___callPlugins('run', Array,$
#15 vendor/symfony/console/Application.php(1000): Magento\Developer\Console\Command\SourceThemeDeployCommand\Interceptor->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Componen$
#16 vendor/symfony/console/Application.php(271): Symfony\Component\Console\Application->doRunCommand(Object(Magento\Developer\Console\Command\SourceThemeDeployCommand\Interceptor), Object(Symfony\Comp$
#17 vendor/magento/framework/Console/Cli.php(115): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput$
#18 vendor/symfony/console/Application.php(147): Magento\Framework\Console\Cli->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#19 bin/magento(23): Symfony\Component\Console\Application->run()
#20 {main} [] []

Please provide Severity assessment for the Issue as Reporter. This information will help during Confirmation and Issue triage processes.

  • Severity: S0 - Affects critical data or functionality and leaves users without workaround.
  • Severity: S1 - Affects critical data or functionality and forces users to employ a workaround.
  • Severity: S2 - Affects non-critical data or functionality and forces users to employ a workaround.
  • Severity: S3 - Affects non-critical data or functionality and does not force users to employ a workaround.
  • Severity: S4 - Affects aesthetics, professional look and feel, “quality” or “usability”.
@m2-assistant
Copy link

m2-assistant bot commented Aug 17, 2020

Hi @Eddcapone. Thank you for your report.
To help us process this issue please make sure that you provided the following information:

  • Summary of the issue
  • Information on your environment
  • Steps to reproduce
  • Expected and actual results

Please make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, please, add a comment to the issue:

@magento give me 2.4-develop instance - upcoming 2.4.x release

For more details, please, review the Magento Contributor Assistant documentation.

Please, add a comment to assign the issue: @magento I am working on this


⚠️ According to the Magento Contribution requirements, all issues must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting.

🕙 You can find the schedule on the Magento Community Calendar page.

📞 The triage of issues happens in the queue order. If you want to speed up the delivery of your contribution, please join the Community Contributions Triage session to discuss the appropriate ticket.

🎥 You can find the recording of the previous Community Contributions Triage on the Magento Youtube Channel

✏️ Feel free to post questions/proposals/feedback related to the Community Contributions Triage process to the corresponding Slack Channel

@magento-engcom-team magento-engcom-team added the Issue: Format is valid Gate 1 Passed. Automatic verification of issue format passed label Aug 17, 2020
@ghost ghost added this to Ready for QA in Community Backlog Aug 17, 2020
@Eddcapone Eddcapone changed the title Executing grunt refresh or grunt exec:all leads to error "Unable to retrieve deployment version of static files from the file system." Executing grunt refresh or grunt exec:all in production mode, leads to error "Unable to retrieve deployment version of static files from the file system." Aug 17, 2020
@Eddcapone
Copy link
Author

I figured out that grunt exec calls this code: grunt --force clean:company_fresh. This calls the script dev/tools/grunt/configs/clean.js.

It looks like this script is deleting the file pub/static/deployed_content.txt and so the errors Unable to retrieve deployment version of static files from the file system and Can not load static content version. are showing.

So I modified the script and removed all lines where "deployedVersion" is showing.

Original:

/**
 * Copyright © Magento, Inc. All rights reserved.
 * See COPYING.txt for license details.
 */

'use strict';

var themes = require('../tools/files-router').get('themes'),
    _      = require('underscore');

var themeOptions = {};

_.each(themes, function(theme, name) {
    themeOptions[name] = {
        "force": true,
        "files": [
            {
                "force": true,
                "dot": true,
                "src": [
                    "<%= path.tmp %>/cache/**/*",
                    "<%= combo.autopath(\""+name+"\", path.pub ) %>**/*",
                    "<%= combo.autopath(\""+name+"\", path.tmpLess) %>**/*",
                    "<%= combo.autopath(\""+name+"\", path.tmpSource) %>**/*",
                    "<%= path.deployedVersion %>"
                ]
            }
        ]
    };
});

var cleanOptions = {
    "var": {
        "force": true,
        "files": [
            {
                "force": true,
                "dot": true,
                "src": [
                    "<%= path.tmp %>/cache/**/*",
                    "<%= path.tmp %>/generation/**/*",
                    "<%= path.tmp %>/log/**/*",
                    "<%= path.tmp %>/maps/**/*",
                    "<%= path.tmp %>/page_cache/**/*",
                    "<%= path.tmp %>/tmp/**/*",
                    "<%= path.tmp %>/view/**/*",
                    "<%= path.tmp %>/view_preprocessed/**/*"
                ]
            }
        ]
    },
    "pub": {
        "force": true,
        "files": [
            {
                "force": true,
                "dot": true,
                "src": [
                "dot": true,
                "src": [
                    "<%= path.pub %>frontend/**/*",
                    "<%= path.pub %>adminhtml/**/*",
                    "<%= path.deployedVersion %>"
                ]
            }
        ]
    },
    "styles": {
        "force": true,
        "files": [
            {
                "force": true,
                "dot": true,
                "src": [
                    "<%= path.tmp %>/view_preprocessed/**/*",
                    "<%= path.tmp %>/cache/**/*",
                    "<%= path.pub %>frontend/**/*.less",
                    "<%= path.pub %>frontend/**/*.css",
                    "<%= path.pub %>adminhtml/**/*.less",
                    "<%= path.pub %>adminhtml/**/*.css",
                    "<%= path.deployedVersion %>"
                ]
            }
        ]
    },
    "markup": {
        "force": true,
        "files": [
            {
                "force": true,
                "dot": true,
                "src": [
                    "<%= path.tmp %>/cache/**/*",
                    "<%= path.tmp %>/generation/**/*",
                    "<%= path.tmp %>/view_preprocessed/html/**/*",
                    "<%= path.tmp %>/page_cache/**/*"
                ]
            }
        ]
    },
    "js": {
        "force": true,
        "files": [
            {
                "force": true,
                "dot": true,
                "src": [
                    "<%= path.pub %>**/*.js",
                    "<%= path.pub %>**/*.html",
                    "<%= path.pub %>_requirejs/**/*",
                    "<%= path.deployedVersion %>"
                ]
            }
        ]
    }
};

module.exports = _.extend(cleanOptions, themeOptions);

Fix:

/**
 * Copyright © Magento, Inc. All rights reserved.
 * See COPYING.txt for license details.
 */

'use strict';

var themes = require('../tools/files-router').get('themes'),
    _      = require('underscore');

var themeOptions = {};

_.each(themes, function(theme, name) {
    themeOptions[name] = {
        "force": true,
        "files": [
            {
                "force": true,
                "dot": true,
                "src": [
                    "<%= path.tmp %>/cache/**/*",
                    "<%= combo.autopath(\""+name+"\", path.pub ) %>**/*",
                    "<%= combo.autopath(\""+name+"\", path.tmpLess) %>**/*",
                    "<%= combo.autopath(\""+name+"\", path.tmpSource) %>**/*",
                ]
            }
        ]
    };
});

var cleanOptions = {
    "var": {
        "force": true,
        "files": [
            {
                "force": true,
                "dot": true,
                "src": [
                    "<%= path.tmp %>/cache/**/*",
                    "<%= path.tmp %>/generation/**/*",
                    "<%= path.tmp %>/log/**/*",
                    "<%= path.tmp %>/maps/**/*",
                    "<%= path.tmp %>/page_cache/**/*",
                    "<%= path.tmp %>/tmp/**/*",
                    "<%= path.tmp %>/view/**/*",
                    "<%= path.tmp %>/view_preprocessed/**/*"
                ]
            }
        ]
    },
    "pub": {
        "force": true,
        "files": [
            {
                "force": true,
                "dot": true,
                "src": [
                "dot": true,
                "src": [
                    "<%= path.pub %>frontend/**/*",
                    "<%= path.pub %>adminhtml/**/*",
                ]
            }
        ]
    },
    "styles": {
        "force": true,
        "files": [
            {
                "force": true,
                "dot": true,
                "src": [
                    "<%= path.tmp %>/view_preprocessed/**/*",
                    "<%= path.tmp %>/cache/**/*",
                    "<%= path.pub %>frontend/**/*.less",
                    "<%= path.pub %>frontend/**/*.css",
                    "<%= path.pub %>adminhtml/**/*.less",
                    "<%= path.pub %>adminhtml/**/*.css",
                ]
            }
        ]
    },
    "markup": {
        "force": true,
        "files": [
            {
                "force": true,
                "dot": true,
                "src": [
                    "<%= path.tmp %>/cache/**/*",
                    "<%= path.tmp %>/generation/**/*",
                    "<%= path.tmp %>/view_preprocessed/html/**/*",
                    "<%= path.tmp %>/page_cache/**/*"
                ]
            }
        ]
    },
    "js": {
        "force": true,
        "files": [
            {
                "force": true,
                "dot": true,
                "src": [
                    "<%= path.pub %>**/*.js",
                    "<%= path.pub %>**/*.html",
                    "<%= path.pub %>_requirejs/**/*",
                ]
            }
        ]
    }
};

module.exports = _.extend(cleanOptions, themeOptions);

Now if I call grunt refresh && php bin/magento setup:static-content:deploy && php bin/magento setup:static-content:deploy de_DE then it works. So why is the clean script deleting the file by default and causing the bug? It only happens in production mode.

@luccame
Copy link

luccame commented Aug 17, 2020

From: https://devdocs.magento.com/guides/v2.4/frontend-dev-guide/tools/using_grunt.html

Prerequisites

Make sure that you set your Magento application to the developer or default mode.

@Eddcapone
Copy link
Author

Eddcapone commented Aug 17, 2020

@luccame Thx. Then how are we supposed to compile our theme in production mode?
Are we supposed to compile on a develop instance and deploy the generated content to the production server?
How is the normal workflow supposed to look like?

@ihor-sviziev
Copy link
Contributor

@Eddcapone
for production you should use bin/magento setup:static-content:deploy command. More info here https://devdocs.magento.com/guides/v2.4/config-guide/cli/config-cli-subcommands-static-view.html#config-cli-subcommands-staticview

@mrtuvn
Copy link
Contributor

mrtuvn commented Aug 17, 2020

i believe in production we should not use grunt . Every changes need to update by ultimate commands di compile and deploy static

@Eddcapone
Copy link
Author

@Eddcapone
for production you should use bin/magento setup:static-content:deploy command. More info here https://devdocs.magento.com/guides/v2.4/config-guide/cli/config-cli-subcommands-static-view.html#config-cli-subcommands-staticview

But this does not compile your themes LESS files, so it makes no sense?

@Eddcapone
Copy link
Author

Eddcapone commented Aug 18, 2020

i believe in production we should not use grunt . Every changes need to update by ultimate commands di compile and deploy static

The commands bin/magento setup:di:compile and bin/magento setup:static-content:deploy are not compiling your themes LESS files, so how are we supposed to compile the Theme without using grunt?

EDIT: 2021, this is wrong. bin/magento setup:static-content:deploy does compile your theme, it just did not worked out for me back then, because errors where showing which were not showing when using grunt refresh. More infos here.

@ihor-sviziev
Copy link
Contributor

ihor-sviziev commented Aug 18, 2020 via email

@hostep
Copy link
Contributor

hostep commented Aug 18, 2020

@Eddcapone: you can use https://github.com/baldwin-agency/magento2-module-less-js-compiler which uses nodejs to compile less code instead of php, which - in the early days of Magento was much more performant, but recently the scales have tipped. But the most important feature of that module, is that the outputted css code should be 100% the same between local development (using grunt) and production (using setup:static-content:deploy) which is not the case if you use the build-in php less compiler which Magento ships with.

@Eddcapone
Copy link
Author

Eddcapone commented Aug 18, 2020

i believe in production we should not use grunt . Every changes need to update by ultimate commands di compile and deploy static

The commands bin/magento setup:di:compile and bin/magento setup:static-content:deploy are not compiling your themes LESS files, so how are we supposed to compile the Theme without using grunt?

I just tested it and php bin/magento maintenance:enable && php bin/magento setup:upgrade && php bin/magento setup:di:compile && php bin/magento setup:static-content:deploy && php bin/magento setup:static-content:deploy de_DE && php bin/magento maintenance:disable actually does compile your Theme. It works.
But I still don't understand why grunt only works for default/developer mode? It just makes no sense for me.
I applied my fix and I will use grunt in production, until someone gives a valid reason why we should not use grunt in production.

@ihor-sviziev
Copy link
Contributor

ihor-sviziev commented Aug 18, 2020 via email

@Eddcapone
Copy link
Author

Eddcapone commented Aug 18, 2020

Php compiler gives a bit different result time to time, that’s the main reason - you might see the issues that no one else see

On Tue, 18 Aug 2020 at 13:09 Eddcapone @.***> wrote: i believe in production we should not use grunt . Every changes need to update by ultimate commands di compile and deploy static The commands bin/magento setup:di:compile and bin/magento setup:static-content:deploy are not compiling your themes LESS files, so how are we supposed to compile the Theme without using grunt? I just tested it and bin/magento setup:di:compile acutally does compile your Theme. It works. But I still don't understand why grunt only works for default/developer mode? It just makes no sense for me. I applied my fix and I will use grunt in production, until someone gives a valid reason why we should not use grunt in production. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#29570 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOJOUIVA2VYTDL4C6M4ZW3SBJHO3ANCNFSM4QBKB63Q .

So grunt uses the php compiler? Then which compiler is used by setup:di:compile? Is it working better than grunt compiler?

@hostep
Copy link
Contributor

hostep commented Aug 18, 2020

@Eddcapone

  1. setup:di:compile has nothing to do with generating static assets
  2. setup:upgrade also implicitly executes setup:static-content:deploy (unless you specify the --keep-generated flag)
  3. the main reason for wanting to use setup:static-content:deploy on production is because it does a lot more than simply compiling less code. It's also merging/bundling/minifying css and javascript and html if you have those settings enabled. It's also responsible for generating javascript translations and might do some more tasks. I don't know if grunt can also do all these tasks by default in Magento? I've discussed this before in another ticket.

If you want steps for a production deploy, here are the basic ones which we perform:

composer install --optimize-autoloader --prefer-dist --no-dev
npm install --production # only needed if you need nodejs dependencies on your build/production server
php bin/magento deploy:mode:set --skip-compilation production
php bin/magento setup:di:compile
composer dump-autoload --optimize --no-dev
touch pub/static/deployed_version.txt # to work around an old bug https://github.com/magento/magento2/issues/6426 which is not clear if it was fixed or not yet
php bin/magento setup:static-content:deploy --theme=Custom/theme nl_NL fr_FR en_US ... # depending on which languages you need in the frontend
php bin/magento setup:static-content:deploy --theme=Magento/backend en_US ... # depending on which languages you need in the backend
php bin/magento setup:upgrade --keep-generated

These are just basic steps, we do a lot of complex other things due to how our workflow is setup, but hopefully this can put you in the correct direction 🙂

@Eddcapone
Copy link
Author

I made a color change in a less file of my theme and then executed php bin/magento setup:static-content:deploy && php bin/magento cache:flush but the change was not visible after it.

I always have to call the whole set of commands to make it work:

php bin/magento maintenance:enable && php bin/magento setup:upgrade && php bin/magento setup:di:compile && php bin/magento setup:static-content:deploy && php bin/magento setup:static-content:deploy de_DE && php bin/magento maintenance:disable

@hostep
Copy link
Contributor

hostep commented Aug 18, 2020

That's because you start from dirty directories, Magento uses previously generated files in their SCD procedures and doesn't always overwrite them. If you start from a clean environment (like we always do with every single new deploy to production) it should work.

In your case, please try removing these files before running setup:static-content:deploy:

  • rm -R var/view_preprocessed/*
  • rm -R pub/static/*

(the * is important and intentionally, so you won't accidentally remove the .htaccess files in those directories which is important to keep if you use Apache as your webserver)

@Eddcapone
Copy link
Author

Eddcapone commented Aug 18, 2020

I see thank you. I just tried php bin/magento maintenance:enable && grunt clean && php bin/magento setup:static-content:deploy && php bin/magento setup:static-content:deploy de_DE && php bin/magento maintenance:disable instead.

That worked.

@sidolov sidolov added this to Ready for Grooming in Low Priority Backlog Sep 24, 2020
@sidolov sidolov added this to Ready for Confirmation in Issue Confirmation and Triage Board Oct 20, 2020
@ghost ghost removed this from Ready for QA in Community Backlog Oct 20, 2020
@ghost ghost removed this from Ready for Grooming in Low Priority Backlog Oct 20, 2020
@ghost ghost added Issue: ready for confirmation and removed Issue: Format is valid Gate 1 Passed. Automatic verification of issue format passed labels Oct 20, 2020
@magento-engcom-team magento-engcom-team added the Reported on 2.3.5-p2 Indicates original Magento version for the Issue report. label Nov 13, 2020
@stale
Copy link

stale bot commented Jan 29, 2021

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 14 days if no further activity occurs. Is this issue still relevant? If so, what is blocking it? Is there anything you can do to help move it forward? Thank you for your contributions!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Issue: ready for confirmation Reported on 2.3.5-p2 Indicates original Magento version for the Issue report. stale issue
Projects
None yet
Development

No branches or pull requests

6 participants