Add no_project_mount option to omit the web mount for app code (experiment with mutagen)

[cweagans]

The Problem/Issue/Bug:

It is extremely difficult to experiment with new methods of getting files into the web container. docker-compose doesn't allow removing things in the base docker-compose file, so any attempt to override the mount point, whether or not a volume is mounted at all, etc doesn't actually lead to success.

How this PR Solves The Problem:

I added a new option to config.yml that lets people just opt out of the web mount in question. Since this is realistically the only mount people are going to care about getting rid of for testing new file sync methods, I didn't take it further than that.

Manual Testing Instructions:

Add no_project_mount: true to a project config.yaml, then start the project. Inside the web container, you should see /var/www/phpstatus.php, /var/www/html/index.html, and /var/www/html/docroot/ which should contain the default php files from the container.

I've also been testing mutagen. The process for doing that is to install Mutagen (brew install mutagen-io/mutagen/mutagen), then run mutagen daemon start, and then mutagen sync create . docker://ddev-[ddev project name]-web/var/www/html --sync-mode=two-way-resolved --name=[ddev project name]. Wait for the initial sync to complete (somewhere on the order of 20 seconds depending on how large your project is -- you can view progress by running mutagen sync list [ddev project name]). Once it has synced, you don't need to do anything else. Mutagen will just handle things for you in the background.

Automated Testing Overview:

I added a test to essentially perform the manual instructions above. Specifically, it's looking for /var/www/html/index.html, which is unlikely to exist on any of the PHP applications that ddev targets.

Related Issue Link(s):

Release/Deployment notes:

[cweagans]

Well the test here is definitely causing problems. Is there a straightforward way to run one test? I can't remember exactly how that works. Not too keen on tying up my machine for an hour.

[rfay]

See https://github.com/drud/ddev/blob/master/docs/developers/building-contributing.md for how to run a single test. It's also easy in GoLand.

[rfay]

Artifacts for this build are in https://circleci.com/gh/drud/ddev/23730#artifacts/containers/0 - you have to drill down under "artifacts" just below the "test summary" header.

[cweagans]

I'm not super clear on why the circle CI tests are failing. The failures in the output don't seem to be related to this PR -- are those tests flaky or is there a real problem that needs some attention from my end?

[rfay]

I just restarted that one test again. It looks to me like the failures this time are different from last time. Not related to your change.

[rfay]

• I'd like a better name than OmitAppWebMount. Let's brainstorm. NoProjectMount? SkipProjectMount?
• We always also add a ddev config flag for each config.yaml entry.

[rfay]

Oh, I think those tests are failing because your test didn't clean up after itself maybe? It left stuff unmounted? You don't need to WriteConfig() I don't believe, in the first place. But if you do, you have to clean it up. Or else have it in its own directory.

[cweagans]

I'd like a better name than OmitAppWebMount. Let's brainstorm. NoProjectMount? SkipProjectMount?

I have no real preference. NoProjectMount is just fine if you're happy with that -- I went with Omit for consistency with OmitContainers, but I don't have strong feelings on this one.

We always also add a ddev config flag for each config.yaml entry.

Noted. Will add in the next day or so.

Oh, I think those tests are failing because your test didn't clean up after itself maybe? It left stuff unmounted? You don't need to WriteConfig() I don't believe, in the first place. But if you do, you have to clean it up. Or else have it in its own directory.

Ah, got it. The testing here was a little confusing. It seems like there's quite a bit of magic happening somewhere and I've never really been able to wrap my head around it. I think the config needs written because otherwise, the project won't start without the app mount. Cleaning it up should be easy enough. I'll take a crack at that as well.

[cweagans]

@rfay 4 new commits above should resolve all of your notes! Please let me know if anything else is needed here -- very excited to get this in :)

[rfay]

Could you please add more to the manual testing instructions @cweagans - maybe explain what you're doing, or a simple demo of how one would want to use an alternate mount.

[rfay]

If it's OK, I'm going to rebase this to current and fiddle with the test a little bit. Forgot that the regular NFS test uses findmnt -T which is better than what I suggested and you used here.

[rfay]

This is looking fine to me. I added some minor commits to simplify the test and omit more from volumes.

I tested with no_project_mount: true and docker-compose.manualmount.yaml:

version: '3.6'

services:
  web:
    volumes:
    - "..:/var/www/html:rw"

[rfay]

It does look like this docker-compose issue relates, and there may be a technique without this PR, docker/compose#5203

[cweagans]

I've been through the whole chain of PRs linked from that one. There really isn't a way to remove something from the original list of volumes as far as I can see.

[rfay]

I think this is the authoritative "no": docker/compose#3729

[cweagans]

Yep. I also added some manual testing instructions for Mutagen to the issue body if you want to give that a try. It's pretty easy to set up.

[Mactory]

Thank you for this PR, I was just looking for exactly this functionality. Our use-case: We are developing a wordpress theme. I would like to only have the actual theme files in my project directory and mounted and pre-install the wordpress source in the docker image. This change will allow me to do that via the Dockerfile and docker-compose.local.yaml .
Looking forward to the release!

[rfay]

This is already in ddev v1.14.0-rc1, available in the edge channels of homebrew and chocolatey. We'd love to have you test it out... and then write up what you did!

[Mactory]

Great! I'll give it a try and report back 👍

[rfay]

I tried this with mutagen on Windows 10 Home with Docker (edge) Desktop 2.2.3.0 and it worked fine for a casual test:

no_project_mount: true
hooks:
  post-start:
  - exec-host: mutagen sync create . docker://ddev-$DDEV_SITENAME-web/var/www/html
      --sync-mode=two-way-resolved --name=$DDEV_SITENAME
  - exec-host: 'while ! mutagen sync list $DDEV_SITENAME | grep "Status: Watching for changes"; do echo "waiting for mutagen sync..."; sleep 2; done  '
  post-stop:
  - exec-host: mutagen sync terminate $DDEV_SITENAME
  pre-start:
  - exec-host: mutagen daemon start
  - exec-host: mutagen sync terminate $DDEV_SITENAME >/dev/null || true
use_dns_when_possible: true

I'm not completely sure the mutagen terminate is the whole deal, but I do note that mutagen is perfectly happy to create two sessions with the same name.

Casual performance test is wonderful.

[Mactory]

Looks good so far. I am still working on the Dockerfile. I have a quick question: During the build phase of the docker image, are there any environment variables available? Specifically the DDEV_SITENAME variable would be of interest to me. Thank you for your help.

EDIT:
Never mind: I don't actually need the project name at build time... I can simply do without

[rfay]

I wouldn't use a Dockerfile if I were you, rather a docker-compose.manualmounts.yaml - and most of these will be available in docker-compose.*.yaml, https://ddev.readthedocs.io/en/stable/users/extend/custom-commands/#environment-variables-provided

[Mactory]

I use a combination of both. A Dockerfile which installs Bedrock in the container and a docker-compose file that mounts my theme in the right place. I thought I would need the Sitename during build time to configure bedrocks, but i am now using Hooks to set the variables post-start

[Mactory]

Ok, i finished my setup and did a quick test yesterday. Seems to be working fine. Here is what I did.

Goal

We are developing a wordpress Theme. We only have the theme folder checked into our git and I would like to use ddev for the development. So far I needed to install bedrock into the Project root folder to get wordpress running. I would then checkout my git folder into the corresponding themes folder of bedrock. This works, but prevents me from adding the .ddev config directly to my project. I would like to have wordpress installed in the docker image and only mount the contents of my project folder directly into the correct folder of the wordpress structure.

Use the new no_project_mount option

I added the line no_project_mount: true to my config.yaml in order to prevent my project folder from being mounted directly into the webserver's root.

Add Wordpress to the Docker image

I created a .ddev/web-build/Dockerfile that uses composer to install bedrock into /var/bedrock and symlink bedrock's web folder to the webserver's root directory:

ARG BASE_IMAGE=drud/ddev-webserver:v1.13.1
FROM $BASE_IMAGE
ARG uid
ARG gid
RUN cd /var \
    && composer create roots/bedrock \
    && chown -R $uid:$gid /var/bedrock \
    && rm -rf /var/www/html \
    && ln -s /var/bedrock/web /var/www/html \
    &&  mkdir -p /var/bedrock/web/app/themes/

Use docker-compose to insert the project folder at the correct mountpoint.

I created a custom .ddev/docker-compose.mount.yaml that mounts the project folder as theme folder into the bedrock installation and add a docker volume for the dynamic content (plugins, uploads, etc) of wordpress.

version: '3.6'

volumes:
  wp-content:

services:
  web:
    volumes:
      - "wp-content:/var/bedrock/web/app"
      - '../:/var/bedrock/web/app/themes/${DDEV_SITENAME:-theme}:ro'

Use DDEV Hooks to set the bedrock environment

Finally I needed to pass the projects URL to the bedrock installation. For this, I added a post-start hook to my config.yaml:

hooks:
    post-start:
      - exec: 'echo "DB_NAME=db

        DB_USER=db 

        DB_PASSWORD=db 

        DB_HOST=db 

        WP_HOME=https://${VIRTUAL_HOST}" > /var/bedrock/.env'
      - exec: 'echo "WP_ENV=''development'' 

        WP_SITEURL="${WP_HOME}/wp" 

        WP_DEBUG_LOG=/path/to/debug.log 

        # Generate your keys here: https://roots.io/salts.html

        AUTH_KEY=''generateme''

        SECURE_AUTH_KEY=''generateme''

        LOGGED_IN_KEY=''generateme''

        NONCE_KEY=''generateme''

        AUTH_SALT=''generateme''

        SECURE_AUTH_SALT=''generateme''

        LOGGED_IN_SALT=''generateme''

        NONCE_SALT=''generateme''" >> /var/bedrock/.env'

Ideally the generateme values should of course be replaced by actual individual keys.

This setup does what I want.

Possible Improvements

The no_project_mount option works nicely for the given procedure, an even easier way could be to simply be able to specify a project_mountpoint in the config.yaml. In my case, this would still not eliminate the docker-compose.mount.yaml since I would need to add the volume for the dynamic data.

[rfay]

Yay, glad it works, and thanks for the report!