Availability of environment variables from .env file in user-defined commands

[dahaupt]

Since latest version v1.17.0, environment variables declared in an .env file are not available anymore within custom ddev commands.

We are been using a custom command import-production-data, which pulls database and assets of remote location:

#!/bin/bash

## Description: Fetch and import production data
## Usage: import-production-data
## Example: "ddev import-production-data"

# save current database
ddev snapshot

# fetch and import production database
ssh $SSH_CREDENTIALS "mysqldump -u$PRODUCTION_DB_USER -p$PRODUCTION_DB_PASSWORD -h$PRODUCTION_DB_HOSTNAME -l --opt --skip-extended-insert $PRODUCTION_DB_DATABASE | gzip -9 -c" > production-database.sql.gz && \
ddev import-db --src=production-database.sql.gz && \
rm production-database.sql.gz

# rsync fileadmin from production
rsync -ah --progress --exclude '_processed_' $SSH_CREDENTIALS:$PATH_FILEADMIN $DDEV_APPROOT/$DDEV_DOCROOT/fileadmin/

# Update database schema
ddev typo3cms database:updateschema

# Clear all caches
ddev typo3cms cache:flush

Since we didn't want to add sensitive data like the database password to git, we added it ($PRODUCTION_DB_DATABASE) to our local .env file. It worked before in version v1.16.7, but was probably never intended to be supported at all.

I guess we should switch to the new hosting provider integration (shipped with v1.17.0) to implement our functionality without the need for the production database password at all (e.g. using only SSH access).

Looking forward for any help and opinions!

The custom command was copied from this gist and slightly adjusted:
https://gist.github.com/peterkraume/6646acca841274516c41a3644010bbae

[rfay]

ddev has never had any support for .env files. docker-compose might have had some kind of change...

[rfay]

Interesting, here's the answer: Embedding files (like default command files) in the ddev binary used to be done using packr2, but it was a manual and error-fraught process. When go v1.16 was released, we were able to use the new embed capability, far simpler, no extra step to preprocess files. Well, it turns out that the packr2 package required gobuffalo/envy, which unconditionally loaded .env files, so that's why you saw the root .env file get processed in v1.16. It was removed in 868e2ea

[rfay]

I'm open to explicit handling of .env files, although I'd think we'd want it in .ddev/.env, not sure. I think it could be nice. Unfortunately, most people need the environment variables processed into the web environment, which would require a bit more work. Open to feature requests and PRs.

[h1nds1ght]

@rfay

Hi, just a quick question: If I understand right this seems to be about a .env support for host commands, which was temporarily available by accident? Actually it would be quite nice to have this! I just found the .env support for containers, but I have also some use cases for a host .env. Would it be possible to implement a .env in $HOME/.ddev and [project]/.ddev which is then available to host + containers?

[rfay]

This is all about .env for in-container commands. Nobody has ever mentioned host commands :)

[h1nds1ght]

OK, sorry I just thought that this would be a command running on host, which picked up (by accident) the .env here. Maybe I misunderstood. Nevertheless it would be nice to have this...

[mandrasch]

Hey y'all, just stumbled about this!

I'm planning to use provider integration https://ddev.readthedocs.io/en/stable/users/providers/provider-introduction/ via rsync (https://github.com/drud/ddev/blob/master/pkg/ddevapp/dotddev_assets/providers/rsync.yaml.example).

I now have the same challenge as @dahaupt:

Where could I secretly store the SSH oder database password? .ddev/config.yaml and web environment variables (https://ddev.readthedocs.io/en/stable/users/extend/customization-extendibility/#providing-custom-environment-variables-to-a-container) are very cool, but config.yaml is stored in the git project => Not a good idea obviously.

Is there a current solution? Does not need to be an .env file, can be another solution (another file) as well, as long as it can be ignored in git.

Thanks in advance!

[rfay]

Mostly the easiest technique is to put things in the web_environment in the global config, where it's typically not in the project. I'm sure there are other techniques.

[dahaupt]

Hey @mandrasch, we have configured our provider so that variables from the .env file are no longer necessary. Authentication is done with SSH key authentication. For the export of the database, we use a command on the remote server, which already takes into account the database credentials from the .env file there (in our case this command is provided by the console of TYPO3).

Our db_pull_command looks like this:

ssh user@host "db-export-command | gzip -9 -c" > .ddev/.downloads/db.sql.gz

One advantage here is that the specification of the database credentials locally is not necessary at all. A disadvantage might be that every developer has to store his SSH public key on the server.

[mandrasch]

A nice, makes perfect sense! Thanks for the quick reply and example, @dahaupt & @rfay ! 👍 👍 👍

[lhridley]

A followup on this thread.

Docker Compose will read environment variables stored in a .env file. Unless you specifically configure the location of the .env file in the docker-compose file (which for DDEV is located in the .ddev directory), the .env file does need to be located in the same directory as the docker-compose YAML file(s) being used.

We use this method as follows with DDEV:

• Place the environment variables keys and values in a .env file located in the .ddev hidden directory for the project (example: RECAPTCHA_V3_SITE_KEY=jibberishvalueforexample)
• Map the environment variable references in the /.ddev/config.yaml file for the project under the web_environment key as follows (example shown):
  - RECAPTCHA_V3_SITE_KEY=${RECAPTCHA_V3_SITE_KEY}
• Restart DDEV, and check the availability of the environment variables in the web service, dumping with printenv or echoing out individual variables should verify that the values in the .ddev/.env file are available for use by the web service.

Hope this was helpful!

[riconeitzel]

funny … now that I'm trying to find a proper place for it, I found it: https://ddev.readthedocs.io/en/stable/users/extend/customization-extendibility/#web_environment

The only question to @lhridley is: did it work WITHOUT the additional config.yaml changes? The configuration looks like it works just by creating the .env file.

[rfay]

DDEV supports .ddev/.env file as explained in https://ddev.readthedocs.io/en/latest/users/extend/customization-extendibility/#providing-custom-environment-variables-to-a-container, and there are several other ways.

[lhridley]

@riconeitzel It may work, we ran into some issues where Drupal wasn't reading the environment variabes with getenv() unless we mapped them. I didn't have that issue when I was using docker-compose, but I did have it when we moved to DDEV.

[lhridley]

@riconeitzel Update: It will work (I just tested it), but if you map them you will get warnings if your .env file does not contain the mapped environment variables. We adopted that to ensure that the developers added all of the values needed into their .env file (we trained them to read the messages output when starting DDEV and how to solve that problem, it cut down on the number of questions when we onboarded a new team member to a project).

we did not commit the .env file to the code repo, but we would include a env.example file in every project, but even then some of the devs would forget about providing all of the environment variables. This just made it a little more bullet proof.

We also used that method for mapping our COMPOSER_AUTH credentials file (we stored the path to a developer's file in
this environment variable, and added that to their bash profile) into the DDEV containers so that our private repo credentials were automatically available for Composer in the DDEV web container.

[riconeitzel]

perfect explanation! Thank you very much! :-)