fix: build new ddev-webserver to get fixed lagoon-sync, rework Acquia for #4760

containers/ddev-webserver/Dockerfile

@@ -22,7 +22,7 @@ ADD ddev-webserver-etc-skel /
 RUN /sbin/mkhomedir_helper www-data
 
 RUN apt-get -qq update
-RUN DEBIAN_FRONTEND=noninteractive apt-get -qq install -y -o Dpkg::Options::="--force-confold" --no-install-recommends --no-install-suggests -y libcap2-bin locales-all supervisor
+RUN DEBIAN_FRONTEND=noninteractive apt-get -qq install -y -o Dpkg::Options::="--force-confold" --no-install-recommends --no-install-suggests -y libcap2-bin locales-all pv supervisor
 
 # Arbitrary user needs to be able to bind to privileged ports (for nginx and apache2)
 RUN setcap CAP_NET_BIND_SERVICE=+eip /usr/sbin/nginx

docs/content/users/providers/acquia.md

@@ -9,22 +9,24 @@ DDEV’s Acquia integration pulls database and files from an existing project in
 1. Get your Acquia API token from *Account Settings* → *API Tokens*.
 2. Make sure you’ve added your SSH key to your Acquia account in *Account Settings* → *SSH Keys*.
 3. Run [`ddev auth ssh`](../usage/commands.md#auth-ssh). (Typically once per DDEV session, not every pull.)
-4. In `~/.ddev/global_config.yaml`, add or update the [`web_environment`](../configuration/config.md#web_environment) section with the API keys:
+4. In `~/.ddev/global_config.yaml` (or the project `config.yaml`), add or update the [`web_environment`](../configuration/config.md#web_environment) section with the API keys:
 
    ```yaml
    web_environment:
    - ACQUIA_API_KEY=xxxxxxxx
    - ACQUIA_API_SECRET=xxxxx
    ```
 
-5. Copy `.ddev/providers/acquia.yaml.example` to `.ddev/providers/acquia.yaml`.
-6. Update the `project_id` and database corresponding to the environment you want to work with.
-   - If you have `acli` installed, you can run: `acli remote:aliases:list`.
-   - Or, on the Acquia Cloud Platform navigate to the *Environments* page, click on the header, and look for the *SSH URL* line. For example, `project1.dev@cool-projects.acquia-sites.com` uses project ID `project1.dev`.
-7. Your project must include Drush. Run `ddev composer require drush/drush` if it isn’t there already.
-8. Run [`ddev restart`](../usage/commands.md#restart).
-9. Use `ddev pull acquia` to pull the project database and files.
-10. Optionally use `ddev push acquia` to push local files and database to Acquia. Be aware that [`ddev push`](../usage/commands.md#push) is a command that can potentially damage your production site, so we don’t recommend using it.
+5. In the project `.ddev/config.yaml` add the `ACQUIA_ENVIRONMENT_ID` environment variable:
+
+   ```yaml
+   web_environment:
+   - ACQUIA_ENVIRONMENT_ID=yoursite.dev
+   ```
+
+6. Run [`ddev restart`](../usage/commands.md#restart).
+7. Use `ddev pull acquia` to pull the project database and files.
+8. Optionally use `ddev push acquia` to push local files and database to Acquia. Be aware that [`ddev push`](../usage/commands.md#push) is a command that can potentially damage your production site, so we don’t recommend using it.
 
 ## Usage
 

docs/content/users/providers/index.md

@@ -2,19 +2,21 @@
 
 DDEV offers hosting provider integration and sample integrations for [Pantheon](https://pantheon.io), [Platform.sh](https://platform.sh) and [Acquia](https://www.acquia.com) hosting, along with other examples.
 
-Each DDEV project includes [example recipes](https://github.com/ddev/ddev/tree/master/pkg/ddevapp/dotddev_assets/providers) in its `.ddev/providers` directory, which you can use and adapt however you’d like.
+DDEV provides ready-to-go integrations for Platform.sh, Acquia, and Lagoon in every project, see the .ddev/providers directory. These can be used as is, or they can be modified as you see fit (but remove the `#ddev-generated` line so DDEV doesn't replace them with the defaults).
 
-DDEV provides the `pull` command with whatever recipes you have configured. For example, `ddev pull acquia` if you have created `.ddev/providers/acquia.yaml`.
+In addition, each project includes [example recipes](https://github.com/ddev/ddev/tree/master/pkg/ddevapp/dotddev_assets/providers) for Pantheon, Git, local files, and `rsync` in its `.ddev/providers` directory, which you can use and adapt however you’d like.
+
+DDEV provides the `pull` command with whatever recipes you have configured. For example, `ddev pull platform` is available by default, and `ddev pull pantheon` is available if you have created `.ddev/providers/pantheon.yaml`.
 
 DDEV also provides the `push` command to push database and files to upstream. This is very dangerous to your upstream site and should only be used when appropriate. We don’t even recommended implementing the push stanzas in your YAML file, but it’s there if it suits your workflow.
 
 Each provider recipe is a YAML file that can have whatever name you want. The examples are mostly named after the hosting providers, but they could be named `upstream.yaml` or `live.yaml`, so you could `ddev pull upstream` or `ddev pull live`. If you wanted different upstream environments to pull from, you could name one “prod” and one “dev” and `ddev pull prod` and `ddev pull dev`.
 
-[Example recipes](https://github.com/ddev/ddev/blob/master/pkg/ddevapp/dotddev_assets/providers/) are provided for [Acquia](https://github.com/ddev/ddev/blob/master/pkg/ddevapp/dotddev_assets/providers/acquia.yaml.example), [Local files](https://github.com/ddev/ddev/blob/master/pkg/ddevapp/dotddev_assets/providers/localfile.yaml.example) (like Dropbox, for example), [Pantheon](https://github.com/ddev/ddev/blob/master/pkg/ddevapp/dotddev_assets/providers/pantheon.yaml.example), [Platform.sh](https://github.com/ddev/ddev/blob/master/pkg/ddevapp/dotddev_assets/providers/platform.yaml), and [rsync](https://github.com/ddev/ddev/blob/master/pkg/ddevapp/dotddev_assets/providers/rsync.yaml.example). We know you’ll find improvements to these examples and will have lots to contribute for other hosting providers, and we look forward to your contributions as pull requests here or in [ddev-contrib](https://github.com/ddev/ddev-contrib).
+[Recipes](https://github.com/ddev/ddev/blob/master/pkg/ddevapp/dotddev_assets/providers/) are provided for [Acquia](https://github.com/ddev/ddev/blob/master/pkg/ddevapp/dotddev_assets/providers/acquia.yaml), [Local files](https://github.com/ddev/ddev/blob/master/pkg/ddevapp/dotddev_assets/providers/localfile.yaml.example) (like Dropbox, for example), [Pantheon](https://github.com/ddev/ddev/blob/master/pkg/ddevapp/dotddev_assets/providers/pantheon.yaml.example), [Platform.sh](https://github.com/ddev/ddev/blob/master/pkg/ddevapp/dotddev_assets/providers/platform.yaml), and [rsync](https://github.com/ddev/ddev/blob/master/pkg/ddevapp/dotddev_assets/providers/rsync.yaml.example). We know you’ll find improvements to these examples and will have lots to contribute for other hosting providers, and we look forward to your contributions as pull requests here or in [ddev-contrib](https://github.com/ddev/ddev-contrib).
 
 Each provider recipe is a file named `<provider>.yaml` and consists of several mostly-optional stanzas:
 
-* `environment_variables`: Environment variables will be created in the web container for each of these during pull or push operations. They’re used to provide context (project ID, environment name, etc.) for each of the other stanzas.
+* `environment_variables`: Environment variables will be created in the web container for each of these during pull or push operations. They’re used to provide context (project ID, environment name, etc.) for each of the other stanzas. This stanza is not used in more recent hosting integrations, since providing the environment variables in config.yaml or via `ddev pull xxx --environment=VARIABLE=value` is preferred.
 * `db_pull_command`: A script that determines how DDEV should obtain a database. Its job is to create a gzipped database dump in `/var/www/html/.ddev/.downloads/db.sql.gz`. This is optional; if nothing has to be done to obtain the database dump, this step can be omitted.
 * `db_import_command`: (optional) A script that imports the downloaded database. This is for advanced usages like multiple databases. The default behavior only imports a single database into the `db` database. The [localfile example](https://github.com/ddev/ddev/blob/master/pkg/ddevapp/dotddev_assets/providers/localfile.yaml.example) uses this technique.
 * `files_pull_command`: A script that determines how DDEV can get user-generated files from upstream. Its job is to copy the files from upstream to `/var/www/html/.ddev/.downloads/files`. If nothing has to be done to obtain the files, this step can run `true`.

pkg/ddevapp/config.go

@@ -672,7 +672,7 @@ func (app *DdevApp) FixObsolete() {
 	}
 
 	// Remove old provider/*.example as we migrate to not needing them.
-	for _, providerFile := range []string{"platform.yaml.example"} {
+	for _, providerFile := range []string{"acquia.yaml.example", "platform.yaml.example"} {
 		providerFilePath := app.GetConfigPath(filepath.Join("providers", providerFile))
 		err := os.Remove(providerFilePath)
 		if err == nil {
@@ -1345,7 +1345,7 @@ func PrepDdevDirectory(app *DdevApp) error {
 		return err
 	}
 
-	err = CreateGitIgnore(dir, "**/*.example", ".dbimageBuild", ".dbimageExtra", ".ddev-docker-*.yaml", ".*downloads", ".global_commands", ".homeadditions", ".importdb*", ".sshimageBuild", ".venv", ".webimageBuild", ".webimageExtra", "apache/apache-site.conf", "commands/.gitattributes", "commands/db/mysql", "commands/host/launch", "commands/web/xdebug", "commands/web/live", "config.local.y*ml", "db_snapshots", "import-db", "import.yaml", "mutagen/mutagen.yml", "mutagen/.start-synced", "nginx_full/nginx-site.conf", "postgres/postgresql.conf", "providers/lagoon.yaml", "providers/platform.yaml", "sequelpro.spf", "settings/settings.ddev.py", fmt.Sprintf("traefik/config/%s.yaml", app.Name), fmt.Sprintf("traefik/certs/%s.crt", app.Name), fmt.Sprintf("traefik/certs/%s.key", app.Name), "xhprof/xhprof_prepend.php", "**/README.*")
+	err = CreateGitIgnore(dir, "**/*.example", ".dbimageBuild", ".dbimageExtra", ".ddev-docker-*.yaml", ".*downloads", ".global_commands", ".homeadditions", ".importdb*", ".sshimageBuild", ".venv", ".webimageBuild", ".webimageExtra", "apache/apache-site.conf", "commands/.gitattributes", "commands/db/mysql", "commands/host/launch", "commands/web/xdebug", "commands/web/live", "config.local.y*ml", "db_snapshots", "import-db", "import.yaml", "mutagen/mutagen.yml", "mutagen/.start-synced", "nginx_full/nginx-site.conf", "postgres/postgresql.conf", "providers/acquia.yaml", "providers/lagoon.yaml", "providers/platform.yaml", "sequelpro.spf", "settings/settings.ddev.py", fmt.Sprintf("traefik/config/%s.yaml", app.Name), fmt.Sprintf("traefik/certs/%s.crt", app.Name), fmt.Sprintf("traefik/certs/%s.key", app.Name), "xhprof/xhprof_prepend.php", "**/README.*")
 	if err != nil {
 		return fmt.Errorf("failed to create gitignore in %s: %v", dir, err)
 	}

pkg/ddevapp/ddevapp_test.go

@@ -225,8 +225,8 @@ var (
 		// 12: drupal10
 		{
 			Name:                          "TestPkgDrupal10",
-			SourceURL:                     "https://ftp.drupal.org/files/projects/drupal-10.1.1.tar.gz",
-			ArchiveInternalExtractionPath: "drupal-10.1.1/",
+			SourceURL:                     "https://ftp.drupal.org/files/projects/drupal-10.1.6.tar.gz",
+			ArchiveInternalExtractionPath: "drupal-10.1.6/",
 			FilesTarballURL:               "https://github.com/ddev/ddev_test_tarballs/releases/download/v1.1/drupal10-files.tgz",
 			DBTarURL:                      "https://github.com/ddev/ddev_test_tarballs/releases/download/v1.1/drupal10-alpha6.sql.tar.gz",
 			FullSiteTarballURL:            "",

pkg/ddevapp/dotddev_assets/providers/acquia.yaml

@@ -0,0 +1,71 @@
+#ddev-generated
+# Acquia provider configuration.
+
+# To use this configuration,
+
+# 1. Get your Acquia API token from your Account Settings->API Tokens.
+# 2. Make sure your ssh key is authorized on your Acquia account at Account Settings->SSH Keys
+# 3. `ddev auth ssh` (this typically needs only be done once per ddev session, not every pull).
+# 4. Add / update the web_environment section in ~/.ddev/global_config.yaml
+#    or your project config.yamlwith the API keys:
+#    ```yaml
+#    web_environment:
+#    - ACQUIA_API_KEY=xxxxxxxx
+#    - ACQUIA_API_SECRET=xxxxx
+#    ```
+# 5. Add the ACQUIA_ENVIRONMENT_ID environment variable to your project config.yaml, for example:
+#    ```yaml
+#    web_environment:
+#    - ACQUIA_ENVIRONMENT_ID=project1.dev
+#   - On the Acquia Cloud Platform you can find this out by navigating to the environments page,
+#     clicking on the header and look for the "SSH URL" line.
+#     Eg. `project1.dev@cool-projects.acquia-sites.com` would have a project ID of `project1.dev`
+# 6. `ddev restart`
+# 7. Use `ddev pull acquia` to pull the project database and files.
+# 8. Optionally use `ddev push acquia` to push local files and database to Acquia. Note that `ddev push` is a command that can potentially damage your production site, so this is not recommended.
+
+# Debugging: Use `ddev exec acli command` and `ddev exec acli auth:login`
+
+# Instead of setting the environment variables in configuration files, you can use
+# `ddev pull acquia --environment=ACQUIA_ENVIRONMENT_ID=yourproject.dev` for example
+
+auth_command:
+  command: |
+     set -eu -o pipefail
+     if [ -z "${ACQUIA_API_KEY:-}" ] || [ -z "${ACQUIA_API_SECRET:-}" ]; then echo "Please make sure you have set ACQUIA_API_KEY and ACQUIA_API_SECRET in ~/.ddev/global_config.yaml" && exit 1; fi
+      if [ -z "${ACQUIA_ENVIRONMENT_ID:-}" ] ; then echo "Please set ACQUIA_ENVIRONMENT_ID via config.yaml or with '--environment=ACQUIA_ENVIRONMENT_ID=xxx'" && exit 1; fi
+      ssh-add -l >/dev/null || ( echo "Please 'ddev auth ssh' before running this command." && exit 1 )
+      acli -n auth:login -n --key="${ACQUIA_API_KEY}" --secret="${ACQUIA_API_SECRET}"
+
+db_pull_command:
+  command: |
+    set -eu -o pipefail
+    # xargs here just trims whitespace
+    # We could use an easier technique when https://github.com/acquia/cli/issues/1629 is resolved
+    # just using `acli pull:db ${ACQUIA_ENVIRONMENT_ID}`
+    echo "Using ACQUIA_ENVIRONMENT_ID=${ACQUIA_ENVIRONMENT_ID}"
+    set -x   # You can enable bash debugging output by uncommenting
+    db_dump=$(acli pull:db ${ACQUIA_ENVIRONMENT_ID} --no-interaction --no-import | tail -2l | xargs)
+    ls /var/www/html/.ddev >/dev/null # This just refreshes stale NFS if possible
+    cp ${db_dump} /var/www/html/.ddev/.downloads/db.sql.gz
+
+files_import_command:
+  command: |
+    # set -x   # You can enable bash debugging output by uncommenting
+    set -eu -o pipefail
+    acli -n pull:files ${ACQUIA_ENVIRONMENT_ID}
+
+# push is a dangerous command. If not absolutely needed it's better to delete these lines.
+db_push_command:
+  command: |
+    set -eu -o pipefail
+    export ACLI_DB_HOST=db ACLI_DB_NAME=db ACLI_DB_USER=db ACLI_DB_PASSWORD=db
+    set -x   # You can enable bash debugging output by uncommenting
+    acli push:db ${ACQUIA_ENVIRONMENT_ID} --no-interaction
+
+# push is a dangerous command. If not absolutely needed it's better to delete these lines.
+files_push_command:
+  command: |
+    # set -x   # You can enable bash debugging output by uncommenting
+    set -eu -o pipefail
+    acli push:files ${ACQUIA_ENVIRONMENT_ID} --no-interaction