Prevent cache collisions in multisite db-update tasks

[lcatlett]

Fixes #3158 subtasks

Changes proposed:

• Adds utility script for generating unique drush cache directory per site / environment
• Creates and sets unique drush cache directory per site process for db update tasks on ACSF
• Removes redundant cache clears resetting site context
• Adds BLT update hook

Steps to replicate the issue:

the original issue can be replicated by running a site update on ACSF. When the db-update Factory hook is executed, updates and config impport will fail because the site context is being reset by cache race conditions:

[ded-000|10677] - The db-update.sh script has completed: Completed in 27 seconds. Exit code: 0. Standard out: Found installation profile profile1.
sandboxacsf.01live: Running BLT deploy tasks on commercialus.sandboxacsf.acsitefactory.com domain in 01live environemnt on the sandboxacsf subscription.
> drupal:config:import
Executing pre-config-import target hook...
No pre-config-import configured.
 �[37;46;1m[info]�[39;49;22m Executing: mysql --defaults-file=/mnt/tmp/sandboxacsf01live/drush_mmFoqe --database=sandboxacsdb216364 --host=dbmaster-1028 --port=3306 --silent  < /mnt/tmp/sandboxacsf01live/drush_8GA1Hh > /dev/null
 �[37;46;1m[info]�[39;49;22m Executing: mysql --defaults-file=/mnt/tmp/sandboxacsf01live/drush_zNbUU7 --database=sandboxacsdb216364 --host=dbmaster-1028 --port=3306 --silent  < /mnt/tmp/sandboxacsf01live/drush_0NVDab > /dev/null
 �[37;46;1m[success]�[39;49;22m 'drush' cache was cleared.

�[33mIn EntityTypeManager.php line 133:�[39m
�[37;41m                                                               �[39;49m
�[37;41m  [Drupal\Component\Plugin\Exception\PluginNotFoundException]  �[39;49m
�[37;41m  The "" entity type does not exist.                           �[39;49m
�[37;41m                        ...nt/www/html/sandboxacsf01live/docroot
[ExecStack] Done in 0.002s
[Acquia\Blt\Robo\Tasks\DrushTask] Running /mnt/www/html/sandboxacsf01live/vendor/bin/drush @self cc drush --uri=commercialus-site.sandboxdigital.net --no-interaction -v --ansi in /mnt/www/html/sandboxacsf01live/docroot
[Acquia\Blt\Robo\Tasks\DrushTask] Done in 0.767s
[Acquia\Blt\Robo\Tasks\DrushTask] Running /mnt/www/html/sandboxacsf01live/vendor/bin/drush @self cache-rebuild --uri=commercialus-site.sandboxdigital.net --no-interaction -v --ansi in /mnt/www/html/sandboxacsf01live/docroot
[Acquia\Blt\Robo\Tasks\DrushTask]  Exit code 1  Time 8.993s
[error]  Failed to import configuration! 
[error]  Command `drupal:config:import ` exited with code 1. 
 [error]  Drupal\Core\Entity\EntityStorageException: Exception thrown while performing a schema update. SQLSTATE[22004]: Null value not allowed: 1138 Invalid use of NULL value: ALTER TABLE {taxonomy_term_field_data} CHANGE `status` `status` TINYINT NOT NULL; Array
(
)
 in Drupal\Core\Entity\Sql\SqlContentEntityStorage->wrapSchemaException() (line 1481 of /mnt/www/html/sandboxacsf01live/docroot/core/lib/Drupal/Core/Entity/Sql/SqlContentEntityStorage.php). 
 [success] Cache rebuild complete.
 [success] Finished performing updates.

Steps to verify the solution:

1. Deploy PR to ACSF environment
2. Run site update at admin/gardens/site-update/update, selecting the branch or tag of this PR
3. Validate that db-update hook in site update executes without errors. To test via the command line as if the task was run by the factory, run /mnt/www/html/sandbox.01live/factory-hooks/db-update/db-update.sh' 'sandbox' '01live' '[db_role]' [uri]', substituting your site, environment, database role, and uri as needed.

[lcatlett]

@lpeabody this is the first refactored PR to replace #3006 and you can see the status of the remaining tasks in #3158. I will have the additional PRs available to test as soon as possible.

[seanhamlin]

I have a PR up for our shared ACSF customer with @lpeabody. Will hopefully get a test release out today to test the BLT 9.2.0 upgrade + this PR patch.

EDIT

The PR appears to work for this customer. I had to merge the script db-update.sh into our existing db-update script (as we need several customizations to it). The 17 sites on ACSF DEV appeared to work fine.

WIP log (customer name redacted):

[staging-1032|26500] - The 000-updates.sh script has completed: Completed in 29 seconds. Exit code: 0. Standard out: Found installation profile lightning.
Generated temporary drush cache directory: /mnt/tmp/customer.01dev/drush_tmp_cache/ce2a12cd556c6e8af6fcb73451eafd1d.
customer.01dev: Running BLT deploy tasks on professionalau.dev-customer.acsitefactory.com domain in 01dev environment on the customer subscription.
> drupal:config:import
 �[37;46;1m[info]�[39;49;22m Executing: mysql --defaults-file=/mnt/tmp/customer01dev/drush_IkWW9U --database=customeracsdb123106 --host=staging-1032 --port=3306 --silent  < /mnt/tmp/customeracsf01dev/drush_9idUR9 > /dev/null
 �[37;46;1m[info]�[39;49;22m Executing: mysql --defaults-file=/mnt/tmp/customer01dev/drush_JaAuNr --database=customeracsdb123106 --host=staging-1032 --port=3306 --silent  < /mnt/tmp/customer01dev/drush_tR1HtG > /dev/null
 �[37;46;1m[success]�[39;49;22m 'drush' cache was cleared.
 �[37;46;1m[success]�[39;49;22m Cache rebuild complete.
 �[37;46;1m[info]�[39;49;22m Executing: mysql --defaults-file=/mnt/tmp/customer01dev/drush_c8mLQV --database=customeracsdb123106 --host=staging-1032 --port=3306 --silent  < /mnt/tmp/customer01dev/drush_Zc49lA > /dev/null
 �[37;4...guration import strategy defined, falling back to running the database updates only.
[Acquia\Blt\Robo\Tasks\DrushTask] Running /mnt/www/html/customer01dev/vendor/bin/drush @self cc drush --uri=professionalau-dev-site.customer.com --no-interaction -v --ansi in /mnt/www/html/customeracsf01dev/docroot
[Acquia\Blt\Robo\Tasks\DrushTask] Done in 1.21s
[Acquia\Blt\Robo\Tasks\DrushTask] Running /mnt/www/html/customeracsf01dev/vendor/bin/drush @self cache-rebuild --uri=professionalau-dev-site.customer.com --no-interaction -v --ansi in /mnt/www/html/customeracsf01dev/docroot
[Acquia\Blt\Robo\Tasks\DrushTask] Done in 11.863s
[Acquia\Blt\Robo\Tasks\DrushTask] Running /mnt/www/html/customeracsf01dev/vendor/bin/drush @self updb --uri=professionalau-dev-site.customer.com --no-interaction -v --ansi in /mnt/www/html/customer01dev/docroot
[Acquia\Blt\Robo\Tasks\DrushTask] Done in 17.206s
[Acquia\Blt\Robo\Tasks\DrushTask] Running /mnt/www/html/customer01dev/vendor/bin/drush @self cache-rebuild --uri=professionalau-dev-site.customer.com --no-interaction -v --ansi in /mnt/www/html/customer01dev/docroot
[Acquia\Blt\Robo\Tasks\DrushTask] Done in 27.49s
[info] modules.01dev.enable is not set.
[info] modules.01dev.uninstall is not set.

[ba66e77]

@lcatlett, I filed lcatlett#11 against your branch with a couple minor tweaks. Those aside, I'm not seeing areas of concern with the code here, but I'm still testing things out.

A couple notes:

1. I do not have a setup which will allow me to fully test this out on ACSF. I'm taking on faith that you and @seanhamlin have done that testing.
2. This does not address non-ACSF multisite installations.
3. I'm not sure about the changes to src/Robo/Commands/Setup/ConfigCommand.php. We've beat that question back and forth a few times now and even though they should not be needed we've seen replicable cases where removing them triggered problems. That's what I'm testing at the moment.

[ba66e77]

I have not been able to trigger issues stemming from the removal of the drush cr's so, with the other caveats noted in my previous comment, I'll give this a thumbs up.

Now that the 10.x branch is the default, we should be creating PRs against it and backporting to earlier versions. However, given we don't have anyone able to do a full live test on 10.x and the resolution of this and related PRs is a priority, I think it's reasonable to make an exception to process in this case.

[lcatlett]

@ba66e77 thanks for the PR - I just merged. The approach in this specific PR was not intended to work on standard multisite since it uses Factory Hooks and ACE uses Cloud Hooks, so I can create an issue to implemement similar functionality on ACE regarding the DRUSH_PATHS_CACHE_DIRECTORY config. The alternative would be to configure this value for every BLT drush command, but that seems like overkill.

Regarding the ConfigCommand.php changes, I realize those changes may seem substantial, however at the least, the drush cc drush is redundant and what is causing a lot of challenges since it is already run by the parent db-update Factory Hook process prior to the custom dp-update.sh firing, though I am not sure if this is the case with Cloud Hooks on ACE. Additionally, drush cc drush does not pass the site, root, and uri contexts which negates the benefits derived from setting DRUSH_PATHS_CACHE_DIRECTORY in the parent process.

Regarding the drush cr, this was likely added to ConfigCommand when we didn't know a lot about the internals of deploys in D8. A heavy-handed cache flush or cc all was a common step in D7 deploys because we did not yet have the ability to cache the service container or plugin discovery mechanisms, and were forced to use config management such as Features which bypassed core caching. In looking at past issues related to deploying BLT config and code updates, it looks like most would not actually be resolved using current drupal core or current BLT config.

I would argue that we should actually be encouraging developers to enable caching locally since even when caching is turned off, cache flushes are still required to refresh plugins, config, and classes. Using a full cr to accomplish this requires a full bootstrap of Drupal locally which makes development unnecessarily slow without a lot of benefit. Additionally, on the remote upstream environment where caching is presumaly enabled, a cr does not actually refresh those particular cache objects in most cases unless you are using an alternative cache backend.

At any rate, the drush cc drush is an absolute must to remove, and if the cr is not removed completely, then BLT should only invalidate the cache objects required for refreshing configuration in ConfigCommand, which in this case I believe is the service container, plugin cache, and config bins, but that should likely be in a dedicated PR and discussed separately due to the urgency of this one.