Use caching filesystem to improve webserving performance

[rfay]

The Problem/Issue/Bug:

ddev webserving behavior is slower than we'd like on macOS and Windows because we're serving from a docker bind-mount, and on those platforms it's not as good as it ought to be. Linux is fine.

How this PR Solves The Problem:

Use a fork of cweagans/bg-sync to sync between the bind-mounted filesystem and a much faster docker native volume. bgsync uses a container with unison running on it.

This is experimental and currently tested only on macOS, as there are filesystem problems on Windows.

Manual Testing Instructions:

• Make sure you have the latest version of the bgsync container with docker pull drud/ddev-bgsync:20181117_bgsync
• In .ddev/config.yaml set webcache_enabled: true
• ddev start. Do things. Verify behavior. See how fast it is. Try to figure out how to make it unreliable.

Automated Testing Overview:

• For macOS we run the entire test suite.
• A number of tests had to be adapted to wait for sync before continuing.
• I would like to see standalone tests for ddev-bgsync container, based on BATS probably.

Related Issue Link(s):

Release/Deployment notes:

[rfay]

This approach (with the fork from @cweagans at https://github.com/rfay/docker-bg-sync/tree/20181117_use_arbitrary_user) makes the webserver behavior really, really snappy. On macOS twice as fast. On Windows much more than that.

But it's probably unworkable because the startup time is outrageous. On macOS it takes unison more than 2 minutes to rummage through a base Drupal 8 site, and until it's through with the startup, behavior isn't predictable in either direction. On Windows, it can easily take 30 minutes to rummage through a D8 site before it's really ready.

Color me sad.

[andrewfrench]

I tried to get startup time statistics, but the initial ddev start failed on several new (empty) projects and after a Docker restart:

λ time ddev start
Starting p...
Creating ddev-ssh-agent ... done

ssh-agent container is running: If you want to add authentication to to the ssh-agent container, run 'ddev auth ssh' to enable your keys.
You must manually add the following entry to your hosts file:
127.0.0.1 p.ddev.local
Or with root/administrative privileges execute 'ddev hostname p.ddev.local 127.0.0.1'
Creating volume "p-mariadb" with default driver
Creating volume "ddev-p_webdir" with default driver
Creating volume "ddev-p_unisondir" with default driver
Creating ddev-p-bgsync ... done
Creating ddev-p-db     ... done
Creating ddev-p-web    ... done
Creating ddev-p-dba    ... done

Network ddev_default is external, skipping
Creating ddev-router ... done

Failed to start p: bgsync container failed: log=touch: /source/.bgsync_healthcheck/healthcheck.1542747947: No such file or directory
Sync is not working after 2 seconds
, err=container /ddev-p-bgsync unhealthy: touch: /source/.bgsync_healthcheck/healthcheck.1542747947: No such file or directory
Sync is not working after 2 seconds


real    2m46.886s
user    0m0.015s
sys     0m0.015s

Docker version:

Client: Docker Engine - Community
 Version:           18.09.0
 API version:       1.39
 Go version:        go1.10.4
 Git commit:        4d60db4
 Built:             Wed Nov  7 00:47:51 2018
 OS/Arch:           windows/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.0
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.4
  Git commit:       4d60db4
  Built:            Wed Nov  7 00:55:00 2018
  OS/Arch:          linux/amd64
  Experimental:     false

Win10Pro 1803

[rfay]

I saw that once as well. Not sure what is the problem there. Thanks for looking at it.

[andrewfrench]

Upgrading an existing project to use the sync container looks good to me so far, but I'm running into issues with a fresh project installation using Composer on MacOS 10.14.

$ ddev config

< set webcache_enabled: true >

$ ddev start 
$ ddev composer create drupa-composer/drupal-project:8.x-dev

... (normal download/installation log output) ...

Failed to purge the temporary install directory /Users/andrew/Code/cms
/ppp/.tmp_ddev_composer_create_ykZZoR: open /Users/andrew/Code/cms/ppp
/.tmp_ddev_composer_create_ykZZoR: no such file or directory

...followed by a more-or-less empty project directory:

$ tree -a 

.
├── .bgsync_healthcheck
└── .ddev
    ├── .gitignore
    ├── config.yaml
    ├── docker-compose.yaml
    └── import-db

3 directories, 3 files

[rfay]

@andrewfrench I made a small change to the ddev composer create code to do the move on the docker volume if running webcache, so this works for me with the trivial composer examples in the test. (And that test now passes).

However, I think we should meet about the future of this PR today, see if we think it has life. It adds complexity everywhere it touches. As in the fix for ddev composer create. But wow, is it a happy thing when it's working. It's also really, really hard to write reliable tests because the caching completion is nondeterministic. There might be a way to make it more deterministic, be able to watch the log for a particular thing's sync. But I'm not optimistic.

[andrewfrench]

The recent updates to this re: ddev composer create seem to be working well, and I have no problem releasing this "experimentally" as we discussed to let interested users try this out and let us know if the speed improvements are worth the added complexity. The test improvements (when complete) will definitely be helpful.