Update acquia.yaml.example #3209

greggmarshall · 2021-09-01T19:09:42Z

We noticed the ddev pull acquia was significantly slower than a direct ddev blt drupal:sync, but drupal:sync started failing. Two suggested optimizations:

In the db pull, we switched from acli remote:drush to the local drush (using it for the files pull), which allows using the --gzip option and downloads directly into db.sql.gz. On our relatively large DB (about 300 mb) this change reduced the time from about 23 minutes to 15 minutes (using Linux time command)
In the files_pull, we added an --exclude-paths='styles:css:js' to not bother with what are usually Drupal's aggregation temporaries. We actually added a project specific directory with PDFs to reduce transfer size. This change reduce the time from about 66 minutes to 43 minutes (there are A LOT of files in the project). A better optimization would be to not copy the files into the .ddev/.downloads directory then have ddev copy it into the Drupal site directory, but instead do a direct rsync to the Drupal site directory which would avoid a lot of files being copied after the first time but that would require changes to the Golang..... When we tested that option outside of the pull the rsync time went down to 15 minutes.

The Problem/Issue/Bug:

How this PR Solves The Problem:

Manual Testing Instructions:

Automated Testing Overview:

Release/Deployment notes:

rfay

Thanks for this. We'll see if it can pass the tests. I'm worried that you have custom aliases set up.

rfay · 2021-09-01T19:29:54Z

pkg/ddevapp/dotddev_assets/providers/acquia.yaml.example

@@ -42,15 +42,14 @@ db_pull_command:
    # set -x   # You can enable bash debugging output by uncommenting
    ls /var/www/html/.ddev >/dev/null # This just refreshes stale NFS if possible
    pushd /var/www/html/.ddev/.downloads >/dev/null
-    acli remote:drush -n ${project_id} -- sql-dump --extra-dump=--no-tablespaces >db.sql
-    gzip db.sql
+    drush @${project_id} sql-dump -n --extra-dump=--no-tablespaces --gzip >db.sql.gz


Doesnt' this assume you have custom aliases set up for the remote?

I suppose it does, but doesn't the acli remote:aliases:download in the auth_command download them?

If it's just relying on the ones that acli downloads, it will work fine. If you project has custom aliases, it won't work when they're not there.

Actually it must work since the files_pull_command below, line 52/53, is (and was) using the same alias to get the files.

Unfortunately the circleci tests "normal_build_and_test", which are the only ones that run the TestAcquiaPull, didn't run, I don't know why. Maybe because you're not "team" member on there? I can check out your PR and run it manually.

I can see one possible issue. After doing a ddev pull acquia, I tried running ddev drush sa and got [success] No site aliases found.

Then I tried running ddev exec acli remote:aliases:download -n to see the output

Drush changed how aliases are defined in Drush 9. Drush 8 aliases are PHP-based, while Drush 9+ aliases are YAML-based. Cloud Platform Drush aliases installed into /home/gregg/.drush/sites Drush 9+ does not automatically read aliases from this directory. Run drush core:init to ensure these aliases are discovered. For more details, see https://github.com/drush-ops/drush/blob/master/examples/example.site.yml

So I'm guessing I'll need to add drush core:init -y and probably source ~/.bashrc so the aliases downloaded show up

rfay

I experimented with this for quite a while and was not able to get it to work with anything I tried. Always get

        	            	drush @${project_id} sql-dump -n --extra-dump=--no-tablespaces --gzip >db.sql.gz
        	            	)], err='exit status 1', stdout='', stderr=' [preflight] The alias @eeamoreno.dev could not be found.'

That was after I had changed it to look in the alias path with drush @${project_id} sql-dump --alias-path=~/.drush -n --extra-dump=--no-tablespaces --gzip >db.sql.gz

Maybe there's something I was doing wrong, but it doesn't pass tests and I'm unable to get it to work fiddling with it. Seems the drush alias isn't found. Perhaps your Acquia environment has aliases that another one doesn't?

greggmarshall · 2021-09-09T02:52:25Z

That's very possible. I'm pretty sure we had the aliases in the repository. But i am still confused why the drush command in line 51 works then, its using the same alias from what I can see...

drush -r docroot rsync --alias-path=~/.drush -q -y @${project_id}:%files ./files

rfay · 2021-09-09T02:56:33Z

maybe rsync works with it and sql-dump doesn't?

greggmarshall · 2021-09-09T03:00:22Z

I have no idea. Shall we just cancel this then?

rfay · 2021-09-09T03:03:31Z

If you would be willing to take another shot at it and use the test Acquia credentials I could give them to you privately. You'd want to make ddev pull acquia work manually and also make TestAcquiaPull work with those credentials.

I'm quite sure you know vastly more about drush aliases and remotes and Acquia than I do, so it's a great honor to have you working on this, and if you could get this solved it would be great. So let me know if you'd like the credentials. I'll be happy to give them to you privately in slack or whatever.

rfay · 2021-09-09T23:05:26Z

I may be wrong but...

ddev pull and ddev push do not work with mutagen enabled #3237

I may have had mutagen enabled globally when testing this. I'll recheck when that goes in or before.

greggmarshall · 2021-09-10T00:33:26Z

Ok, if not, send me the credentials via DM on Slack and I'll try to figure out why it works in one case but not another. I might need help understanding how to run the tests.

We noticed the ddev pull acquia was significantly slower than a direct ddev blt drupal:sync, but drupal:sync started failing. Two suggested optimizations: 1. In the db pull, we switched from acli remote:drush to the local drush (using it for the files pull), which allows using the --gzip option and downloads directly into db.sql.gz. On our relatively large DB (about 300 mb) this change reduced the time from about 23 minutes to 15 minutes (using Linux time command) 2. In the files_pull, we added an --exclude-paths='styles:css:js' to not bother with what are usually Drupal's aggregation temporaries. We actually added a project specific directory with PDFs to reduce transfer size. This change reduce the time from about 66 minutes to 43 minutes (there are A LOT of files in the project). A better optimization would be to not copy the files into the .ddev/.downloads directory then have ddev copy it into the Drupal site directory, but instead do a direct rsync to the Drupal site directory which would avoid a lot of files being copied after the first time but that would require changes to the Golang..... When we tested that option outside of the pull the rsync time went down to 15 minutes. Update acquia.yaml.example Added `drush core:init` and `source ~/.bashrc` to the auth_command to make the aliases downloaded by `acli remote:aliases:download` to Drush Also changed debugging suggestion to `ddev exec acli auth:login` since acli doesn't have an auth:info command.

rfay

This looks like a significant improvement to me, thanks so much!

rfay reviewed Sep 1, 2021

View reviewed changes

rfay requested changes Sep 9, 2021

View reviewed changes

greggmarshall force-pushed the patch-1 branch from d392f64 to 1bed5e9 Compare September 13, 2021 19:09

greggmarshall force-pushed the patch-1 branch from 05c505f to 361d2a6 Compare September 13, 2021 19:51

greggmarshall and others added 2 commits September 13, 2021 16:13

Removed the drush core:init and use --alias-path instead.

cab3a78

Dummy commit to push circleci

e979fcf

rfay approved these changes Sep 14, 2021

View reviewed changes

rfay merged commit 59d496b into ddev:master Sep 17, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update acquia.yaml.example #3209

Update acquia.yaml.example #3209

greggmarshall commented Sep 1, 2021 •

edited by gitpod-io bot

rfay left a comment

rfay Sep 1, 2021

greggmarshall Sep 1, 2021

rfay Sep 1, 2021

greggmarshall Sep 1, 2021

rfay Sep 1, 2021

greggmarshall Sep 13, 2021

rfay left a comment

greggmarshall commented Sep 9, 2021

rfay commented Sep 9, 2021

greggmarshall commented Sep 9, 2021

rfay commented Sep 9, 2021

rfay commented Sep 9, 2021 •

edited

greggmarshall commented Sep 10, 2021

rfay left a comment

Update acquia.yaml.example #3209

Update acquia.yaml.example #3209

Conversation

greggmarshall commented Sep 1, 2021 • edited by gitpod-io bot

The Problem/Issue/Bug:

How this PR Solves The Problem:

Manual Testing Instructions:

Automated Testing Overview:

Related Issue Link(s):

Release/Deployment notes:

rfay left a comment

Choose a reason for hiding this comment

rfay Sep 1, 2021

Choose a reason for hiding this comment

greggmarshall Sep 1, 2021

Choose a reason for hiding this comment

rfay Sep 1, 2021

Choose a reason for hiding this comment

greggmarshall Sep 1, 2021

Choose a reason for hiding this comment

rfay Sep 1, 2021

Choose a reason for hiding this comment

greggmarshall Sep 13, 2021

Choose a reason for hiding this comment

rfay left a comment

Choose a reason for hiding this comment

greggmarshall commented Sep 9, 2021

rfay commented Sep 9, 2021

greggmarshall commented Sep 9, 2021

rfay commented Sep 9, 2021

rfay commented Sep 9, 2021 • edited

greggmarshall commented Sep 10, 2021

rfay left a comment

Choose a reason for hiding this comment

greggmarshall commented Sep 1, 2021 •

edited by gitpod-io bot

rfay commented Sep 9, 2021 •

edited