How are we supposed to deploy changes to production with zero downtime

[Eddcapone]

Summary (*)

Lets say you have two environments. A dev and a production. Now on dev make a change in your theme or in a module, and push the theme/module into production and execute the commands:

 php bin/magento setup:upgrade && php bin/magento setup:di:compile && php bin/magento setup:static-content:deploy de_DE en_US

Now reload the frontend a second after firing up the command and you will get:

There has been an error processing your request

How are you supposed to bring the change to your production environment, without any downtime?

Examples (*)

• See text above.

Proposed solution

• Please show how we are supposed to bring changes to production without downtime. How is the intended workflow look like?

Is my workflow below good or bad and how should it be done instead?

WORKFLOW

I setup my project like this:

1. Dev (Developer)
2. Staging (Production)
3. Live (Production)

---

1. On Dev I am developing. If I am done with my development, then I push it via git to the branch master.

2. Then I switch to Staging where the branch master is also checked out, and execute git pull.

Now I execute my commands to deploy static files. I am executing grunt refresh but you can also use

php bin/magento setup:upgrade && php bin/magento setup:di:compile && php bin/magento setup:static-content:deploy && php bin/magento setup:static-content:deploy de_DE

(de_DE for german shops...) if you don't use grunt.

Important: Make sure that your production branch is not ignoring the folders pub/static/ and generated/.

Now commit the generated files to the branch production and push the branch. Then switch to master branch again.

3. On your live environment where your branch production is checked out, execute git pull and then clear the cache with php bin/magento c:c.
   Your new changes should be live now.

EDIT: It won't work this way, since you also have to execute php bin/magento setup:upgrade in production, which will cause downtime.

---

Please provide Severity assessment for the Issue as Reporter. This information will help during Confirmation and Issue triage processes.

• Severity: S0 - Affects critical data or functionality and leaves users with no workaround.
• Severity: S1 - Affects critical data or functionality and forces users to employ a workaround.
• Severity: S2 - Affects non-critical data or functionality and forces users to employ a workaround.
• Severity: S3 - Affects non-critical data or functionality and does not force users to employ a workaround.
• Severity: S4 - Affects aesthetics, professional look and feel, “quality” or “usability”.

[m2-assistant]

Hi @Eddcapone. Thank you for your report.
To help us process this issue please make sure that you provided the following information:

• Summary of the issue
• Information on your environment
• Steps to reproduce
• Expected and actual results

Please make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, please, add a comment to the issue:

@magento give me 2.4-develop instance - upcoming 2.4.x release

For more details, please, review the Magento Contributor Assistant documentation.

Please, add a comment to assign the issue: @magento I am working on this

---

• Join Magento Community Engineering Slack and ask your questions in #github channel.

⚠️ According to the Magento Contribution requirements, all issues must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting.

🕙 You can find the schedule on the Magento Community Calendar page.

📞 The triage of issues happens in the queue order. If you want to speed up the delivery of your contribution, please join the Community Contributions Triage session to discuss the appropriate ticket.

🎥 You can find the recording of the previous Community Contributions Triage on the Magento Youtube Channel

✏️ Feel free to post questions/proposals/feedback related to the Community Contributions Triage process to the corresponding Slack Channel

[mrtuvn]

IMO this is more related with infastructure how to delivery code than related with magento problem
In real case process more complicated than normal deploy
zero-downtime = you can not run these commands that you posted on live system. So basically you need to have pre-production system deploy on that and apply to real live production

[Eddcapone]

"So basically you need to have pre-production system deploy on that and apply to real live production"

Like described in my workflow example?

[ihor-sviziev]

Hi @Eddcapone,
You have to separate build preparation and production itself.
More details here: https://devdocs.magento.com/guides/v2.4/config-guide/deployment/
In general it should look like this:

For example you can look at https://github.com/jalogut/magento2-deployer-plus that supports zero downtime deployments

Are you using magento cloud, some magento compatible hosting or developing infra yourself?

—-
Technically you could run all the commands on your development env, but you still need to execute some commands on your production env as shown on image above, but it’s not recommended solution

[m2-assistant]

Hi @ihor-sviziev. Thank you for working on this issue.
In order to make sure that issue has enough information and ready for development, please read and check the following instruction: 👇

• 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).

  Details

  If the issue has a valid description, the label Issue: Format is valid will be added to the issue automatically. Please, edit issue description if needed, until label Issue: Format is valid appears.

• 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue. If the report is valid, add Issue: Clear Description label to the issue by yourself.

• 3. Add Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

• 4. Verify that the issue is reproducible on 2.4-develop branch

  Details

  - Add the comment @magento give me 2.4-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.4-develop branch, please, add the label Reproduced on 2.4.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!

• 5. Add label Issue: Confirmed once verification is complete.

• 6. Make sure that automatic system confirms that report has been added to the backlog.

[Eddcapone]

@ihor-sviziev Thank you. We are hosted on Rackspeed.

I have a few questions:

1. Stop queue workers (EE only)

Where did you read about this, it is not described in the documentation entry which you have posted. How is it done? How do I know if the queue workers are even running atm?

2. Update/Import configurations

I guess this is done by calling php bin/magento setup:upgrade --keep-generated ?
If I execute this command, then my site stops loading until the command run through, so I will have downtime.

3. Set sensitive values

Can you please explain what this means in this context?

[ihor-sviziev]

1. About message queue in general described in https://devdocs.magento.com/guides/v2.4/config-guide/mq/manage-message-queues.html
   It is available not only in EE since magento 2.3. By default magento runs message queue consumers automatically with a cron job, but you can customize this behavior if needed.
   Usually we’re killing all message queue processes just before pulling new code to production infra.

2. Yes, it’s correct, you need to run php bin/magento setup:upgrade --keep-generated.
   If you have any issues - please provide any logs and I’ll be able to provide typical reason for such behavior

3. As a build system don’t have any db - we need to provide it some basic values, like websites, locales, used themes, is is generated by php bin/magento app:config:dump and could be adjusted if needed.
   I found quite good example here: https://blog.e-zest.com/magento-2-pipeline-deployment-with-minimum-downtime

[ihor-sviziev]

Hi @Eddcapone,
As I understood - my responses fixed your issues? Can we close the issue?

[Eddcapone]

Not yet. I have some questions left.

1. How to kill/stop/pause queue workers. It is not described in the documentation entry.

2. If I understand it correctly, then we should not make any changes in production only in development.
   We are supposed to dump the config with php bin/magento app:config:dump and push the resulting file app/etc/config.php to the repository (e.g. git) and then pull this file in production. But this file contains sensitive data like password for Elasticsearch, this data should not be pushed to git. How are we supposed to handle sensitive data?

[ihor-sviziev]

About killing the cron jobs and message queues - you can see how magento does it in their deployment tool named ece tools
Here is how they detect which processes need to be killed https://github.com/magento/ece-tools/blob/9d201d782d976012ab4a90d9dbb6a4a21999da8c/src/Util/BackgroundProcess.php#L51
And here is how they actually killing them:
https://github.com/magento/ece-tools/blob/9d201d782d976012ab4a90d9dbb6a4a21999da8c/src/Util/BackgroundProcess.php#L77

About dumping sensitive data - you could remove whole section “system”, it should work without it

[Eddcapone]

1. But the scripts you have posted are about Magento Cloud, how is it done in Magento Enterprise?
2. It makes no sense to remove the whole section "system", most settings are stored here. Then I would need to configure everything in production again, but in the documentation it says that we should not make any changes in production.

Production system
Your live store. You should make minimal configuration changes here and no changes to:
websites
stores
store views
products
product view settings
catalog
categories
category view settings.
You should make all those types of changes in your development system.

[ihor-sviziev]

@Eddcapone

1. we're doing the same way on Magento OS, I believe for EE it should be the same.
2. Basically these settings mostly needed for the "build" phase as there are no DB at all. For build phase you don't really need elasticsearch params.
   By default magento dumps all configs, including "system", while as for me - there should be only needed params, as when there settings are inside config.php - you can't change them in the admin when it's needed. If it's not a problem for you - you could just remove "sensitive" data from the config.php yourself.

I think these config params should be marked as "system specific", if they're not - seems like it's a bug. https://devdocs.magento.com/guides/v2.4/config-guide/prod/config-reference-sens.html

[Eddcapone]

"we're doing the same way on Magento OS, I believe for EE it should be the same."
I still don't understand how you are doing it. Do you call a command or a script to stop queue workers? How is it done?

[ihor-sviziev]

We just using custom bash script that does the same commands as I listed above

…

On Fri, 4 Sep 2020 at 15:21 Eddcapone ***@***.***> wrote: "we're doing the same way on Magento OS, I believe for EE it should be the same." I still don't understand how you are doing it. Do you call a command or a script to stop queue workers? How is it done? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#29819 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAOJOUKEGVDDWYYKCICQAQLSEDLVLANCNFSM4QQVEWQA> .

[Eddcapone]

Oh... so there is not even something official from magento?
What if I just not stop the queue workers?

[ihor-sviziev]

I think it’s just not documented.
If you’ll not stop it - potentially you could have multiple workers running in parallel that will eat your server's memory, I don’t think there might be some other issues

[Eddcapone]

One last question. In the diagram you posted, it says "Stop Queue Workers". But are we supposed to start them again at some point or does this happen automatically? And do you mind to share your bash script for stopping queue workers?

[hostep]

@Eddcapone, there is a feature request pending to introduce an official way to stop queue consumers, but it's not implemented yet. See "Problem 2" (A solution to "Problem 1" will make its introduction in Magento 2.4.1 btw, but it's not 100% implemented like I wanted it too, but it should be sufficient for most people, I haven't found time yet to properly report what's wrong with it yet, I'll try to do this one day, but I'm really busy)

For now, if you haven't changed the default configuration, after you killed the consumers using some script, they will spawn again by the cron system.

See here for some script we've been using to kill them: davidalger/capistrano-magento2#133 (comment)

[onlinebizsoft]

@ihor-sviziev 1. In your "deploy" phase, you have enabled the maintenance mode then disable it at the end of the process so I understand that "maintenance" mode is equal as downtime
2. I can't see the step where you do "setup:upgrade" ?

[ihor-sviziev]

@onlinebizsoft, basically setup upgrade (with keep generated flag) is running on the add/update configuration phase on production environment.
And yes, when the maintenance mode enabled - you will have downtime. As I said before, there is a way to skip setup upgrade when no changes in configuration and not data/schema upgrades. You can see example of such implementation in the deployer (I posted link to it in past)

[onlinebizsoft]

@ihor-sviziev in my opinion, we can't call that as zero downtime which misleads to readers. I think we should have something like this installed in the system
https://github.com/zepgram/module-zero-downtime-deployment

Then we don't need enable/disable maintenance during "setup:upgrade".

[ihor-sviziev]

@onlinebizsoft in case when we don't have any config changes as data/schema upgrades - we shouldn't enable/disable maintenance as well as running setup upgrade. That means that your visitors will immediately see a new version w/o any interruptions.

This mechanism was introduced in order to prevent having not expected issues. I'm not recommend disabling it without really knowing what you're doing.