Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add no_project_mount option to omit the web mount for app code (experiment with mutagen) #2162

Merged
merged 9 commits into from Apr 13, 2020

Conversation

cweagans
Copy link
Contributor

@cweagans cweagans commented Apr 8, 2020

The Problem/Issue/Bug:

It is extremely difficult to experiment with new methods of getting files into the web container. docker-compose doesn't allow removing things in the base docker-compose file, so any attempt to override the mount point, whether or not a volume is mounted at all, etc doesn't actually lead to success.

How this PR Solves The Problem:

I added a new option to config.yml that lets people just opt out of the web mount in question. Since this is realistically the only mount people are going to care about getting rid of for testing new file sync methods, I didn't take it further than that.

Manual Testing Instructions:

Add no_project_mount: true to a project config.yaml, then start the project. Inside the web container, you should see /var/www/phpstatus.php, /var/www/html/index.html, and /var/www/html/docroot/ which should contain the default php files from the container.

I've also been testing mutagen. The process for doing that is to install Mutagen (brew install mutagen-io/mutagen/mutagen), then run mutagen daemon start, and then mutagen sync create . docker://ddev-[ddev project name]-web/var/www/html --sync-mode=two-way-resolved --name=[ddev project name]. Wait for the initial sync to complete (somewhere on the order of 20 seconds depending on how large your project is -- you can view progress by running mutagen sync list [ddev project name]). Once it has synced, you don't need to do anything else. Mutagen will just handle things for you in the background.

Automated Testing Overview:

I added a test to essentially perform the manual instructions above. Specifically, it's looking for /var/www/html/index.html, which is unlikely to exist on any of the PHP applications that ddev targets.

Related Issue Link(s):

Release/Deployment notes:

@cweagans
Copy link
Contributor Author

cweagans commented Apr 8, 2020

Well the test here is definitely causing problems. Is there a straightforward way to run one test? I can't remember exactly how that works. Not too keen on tying up my machine for an hour.

@rfay
Copy link
Member

rfay commented Apr 8, 2020

See https://github.com/drud/ddev/blob/master/docs/developers/building-contributing.md for how to run a single test. It's also easy in GoLand.

@rfay
Copy link
Member

rfay commented Apr 9, 2020

Artifacts for this build are in https://circleci.com/gh/drud/ddev/23730#artifacts/containers/0 - you have to drill down under "artifacts" just below the "test summary" header.

@cweagans
Copy link
Contributor Author

cweagans commented Apr 9, 2020

I'm not super clear on why the circle CI tests are failing. The failures in the output don't seem to be related to this PR -- are those tests flaky or is there a real problem that needs some attention from my end?

@rfay
Copy link
Member

rfay commented Apr 9, 2020

I just restarted that one test again. It looks to me like the failures this time are different from last time. Not related to your change.

Copy link
Member

@rfay rfay left a comment

  • I'd like a better name than OmitAppWebMount. Let's brainstorm. NoProjectMount? SkipProjectMount?
  • We always also add a ddev config flag for each config.yaml entry.

pkg/ddevapp/ddevapp_test.go Outdated Show resolved Hide resolved
@rfay
Copy link
Member

rfay commented Apr 9, 2020

Oh, I think those tests are failing because your test didn't clean up after itself maybe? It left stuff unmounted? You don't need to WriteConfig() I don't believe, in the first place. But if you do, you have to clean it up. Or else have it in its own directory.

@cweagans
Copy link
Contributor Author

cweagans commented Apr 10, 2020

I'd like a better name than OmitAppWebMount. Let's brainstorm. NoProjectMount? SkipProjectMount?

I have no real preference. NoProjectMount is just fine if you're happy with that -- I went with Omit for consistency with OmitContainers, but I don't have strong feelings on this one.

We always also add a ddev config flag for each config.yaml entry.

Noted. Will add in the next day or so.

Oh, I think those tests are failing because your test didn't clean up after itself maybe? It left stuff unmounted? You don't need to WriteConfig() I don't believe, in the first place. But if you do, you have to clean it up. Or else have it in its own directory.

Ah, got it. The testing here was a little confusing. It seems like there's quite a bit of magic happening somewhere and I've never really been able to wrap my head around it. I think the config needs written because otherwise, the project won't start without the app mount. Cleaning it up should be easy enough. I'll take a crack at that as well.

@cweagans
Copy link
Contributor Author

cweagans commented Apr 10, 2020

@rfay 4 new commits above should resolve all of your notes! Please let me know if anything else is needed here -- very excited to get this in :)

@rfay
Copy link
Member

rfay commented Apr 12, 2020

Could you please add more to the manual testing instructions @cweagans - maybe explain what you're doing, or a simple demo of how one would want to use an alternate mount.

@rfay
Copy link
Member

rfay commented Apr 13, 2020

If it's OK, I'm going to rebase this to current and fiddle with the test a little bit. Forgot that the regular NFS test uses findmnt -T which is better than what I suggested and you used here.

@rfay rfay force-pushed the omit-web-mount-option branch from b92a144 to 8802e8c Compare Apr 13, 2020
rfay
rfay approved these changes Apr 13, 2020
Copy link
Member

@rfay rfay left a comment

This is looking fine to me. I added some minor commits to simplify the test and omit more from volumes.

I tested with no_project_mount: true and docker-compose.manualmount.yaml:

version: '3.6'

services:
  web:
    volumes:
    - "..:/var/www/html:rw"

@rfay
Copy link
Member

rfay commented Apr 13, 2020

It does look like this docker-compose issue relates, and there may be a technique without this PR, docker/compose#5203

@cweagans
Copy link
Contributor Author

cweagans commented Apr 13, 2020

I've been through the whole chain of PRs linked from that one. There really isn't a way to remove something from the original list of volumes as far as I can see.

@rfay
Copy link
Member

rfay commented Apr 13, 2020

I think this is the authoritative "no": docker/compose#3729

@cweagans
Copy link
Contributor Author

cweagans commented Apr 13, 2020

Yep. I also added some manual testing instructions for Mutagen to the issue body if you want to give that a try. It's pretty easy to set up.

@rfay rfay changed the title Add option to omit the web mount for app code Add option to omit the web mount for app code (experiment with mutagen) Apr 13, 2020
@rfay rfay changed the title Add option to omit the web mount for app code (experiment with mutagen) Add no_project_mount option to omit the web mount for app code (experiment with mutagen) Apr 13, 2020
@rfay rfay merged commit 7d9a3f9 into drud:master Apr 13, 2020
10 checks passed
@cweagans cweagans deleted the omit-web-mount-option branch Apr 13, 2020
@Mactory
Copy link

Mactory commented Apr 14, 2020

Thank you for this PR, I was just looking for exactly this functionality. Our use-case: We are developing a wordpress theme. I would like to only have the actual theme files in my project directory and mounted and pre-install the wordpress source in the docker image. This change will allow me to do that via the Dockerfile and docker-compose.local.yaml .
Looking forward to the release!

@rfay
Copy link
Member

rfay commented Apr 14, 2020

This is already in ddev v1.14.0-rc1, available in the edge channels of homebrew and chocolatey. We'd love to have you test it out... and then write up what you did!

@Mactory
Copy link

Mactory commented Apr 14, 2020

Great! I'll give it a try and report back 👍

@rfay rfay mentioned this pull request Apr 14, 2020
@rfay
Copy link
Member

rfay commented Apr 15, 2020

I tried this with mutagen on Windows 10 Home with Docker (edge) Desktop 2.2.3.0 and it worked fine for a casual test:

no_project_mount: true
hooks:
  post-start:
  - exec-host: mutagen sync create . docker://ddev-$DDEV_SITENAME-web/var/www/html
      --sync-mode=two-way-resolved --name=$DDEV_SITENAME
  - exec-host: 'while ! mutagen sync list $DDEV_SITENAME | grep "Status: Watching for changes"; do echo "waiting for mutagen sync..."; sleep 2; done  '
  post-stop:
  - exec-host: mutagen sync terminate $DDEV_SITENAME
  pre-start:
  - exec-host: mutagen daemon start
  - exec-host: mutagen sync terminate $DDEV_SITENAME >/dev/null || true
use_dns_when_possible: true

I'm not completely sure the mutagen terminate is the whole deal, but I do note that mutagen is perfectly happy to create two sessions with the same name.

Casual performance test is wonderful.

@Mactory
Copy link

Mactory commented Apr 15, 2020

Looks good so far. I am still working on the Dockerfile. I have a quick question: During the build phase of the docker image, are there any environment variables available? Specifically the DDEV_SITENAME variable would be of interest to me. Thank you for your help.

EDIT:
Never mind: I don't actually need the project name at build time... I can simply do without

@rfay
Copy link
Member

rfay commented Apr 15, 2020

I wouldn't use a Dockerfile if I were you, rather a docker-compose.manualmounts.yaml - and most of these will be available in docker-compose.*.yaml, https://ddev.readthedocs.io/en/stable/users/extend/custom-commands/#environment-variables-provided

@Mactory
Copy link

Mactory commented Apr 15, 2020

I use a combination of both. A Dockerfile which installs Bedrock in the container and a docker-compose file that mounts my theme in the right place. I thought I would need the Sitename during build time to configure bedrocks, but i am now using Hooks to set the variables post-start

@Mactory
Copy link

Mactory commented Apr 16, 2020

Ok, i finished my setup and did a quick test yesterday. Seems to be working fine. Here is what I did.

Goal

We are developing a wordpress Theme. We only have the theme folder checked into our git and I would like to use ddev for the development. So far I needed to install bedrock into the Project root folder to get wordpress running. I would then checkout my git folder into the corresponding themes folder of bedrock. This works, but prevents me from adding the .ddev config directly to my project. I would like to have wordpress installed in the docker image and only mount the contents of my project folder directly into the correct folder of the wordpress structure.

Use the new no_project_mount option

I added the line no_project_mount: true to my config.yaml in order to prevent my project folder from being mounted directly into the webserver's root.

Add Wordpress to the Docker image

I created a .ddev/web-build/Dockerfile that uses composer to install bedrock into /var/bedrock and symlink bedrock's web folder to the webserver's root directory:

ARG BASE_IMAGE=drud/ddev-webserver:v1.13.1
FROM $BASE_IMAGE
ARG uid
ARG gid
RUN cd /var \
    && composer create roots/bedrock \
    && chown -R $uid:$gid /var/bedrock \
    && rm -rf /var/www/html \
    && ln -s /var/bedrock/web /var/www/html \
    &&  mkdir -p /var/bedrock/web/app/themes/

Use docker-compose to insert the project folder at the correct mountpoint.

I created a custom .ddev/docker-compose.mount.yaml that mounts the project folder as theme folder into the bedrock installation and add a docker volume for the dynamic content (plugins, uploads, etc) of wordpress.

version: '3.6'

volumes:
  wp-content:

services:
  web:
    volumes:
      - "wp-content:/var/bedrock/web/app"
      - '../:/var/bedrock/web/app/themes/${DDEV_SITENAME:-theme}:ro'

Use DDEV Hooks to set the bedrock environment

Finally I needed to pass the projects URL to the bedrock installation. For this, I added a post-start hook to my config.yaml:

hooks:
    post-start:
      - exec: 'echo "DB_NAME=db

        DB_USER=db 

        DB_PASSWORD=db 

        DB_HOST=db 

        WP_HOME=https://${VIRTUAL_HOST}" > /var/bedrock/.env'
      - exec: 'echo "WP_ENV=''development'' 

        WP_SITEURL="${WP_HOME}/wp" 

        WP_DEBUG_LOG=/path/to/debug.log 

        # Generate your keys here: https://roots.io/salts.html

        AUTH_KEY=''generateme''

        SECURE_AUTH_KEY=''generateme''

        LOGGED_IN_KEY=''generateme''

        NONCE_KEY=''generateme''

        AUTH_SALT=''generateme''

        SECURE_AUTH_SALT=''generateme''

        LOGGED_IN_SALT=''generateme''

        NONCE_SALT=''generateme''" >> /var/bedrock/.env'

Ideally the generateme values should of course be replaced by actual individual keys.

This setup does what I want.

Possible Improvements

The no_project_mount option works nicely for the given procedure, an even easier way could be to simply be able to specify a project_mountpoint in the config.yaml. In my case, this would still not eliminate the docker-compose.mount.yaml since I would need to add the volume for the dynamic data.

@rfay
Copy link
Member

rfay commented Apr 16, 2020

Yay, glad it works, and thanks for the report!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants