Named Volume Performance Feature #1460
Comments
dustinleblanc
added
task
feature
|
@dustinleblanc so have some initial stuff over in https://github.com/lando/lando/tree/hyperdrive-motivator the syntax is as follows name: my-lando-app
excludes:
- vendor
- modules/contribAnd requires a We also could consider adding support for this sort of syntax as well, eg exclude all of X except Y, but that might be better as a separate issue. name: my-lando-app
excludes:
- modules/contrib
- '!modules/contrib/search_api' |
|
@dustinleblanc dropping a note here:
|
|
@serundeputy... adding you to the party here. Dustin and I need a third opinion, can you...
Then let's meet and see if we can get the without excluding lando drush status: 20s
lando drush cr: 1m30sexcluding lando drush status: 1.3s
lando drush cr: 19sSo one thing i did notice @dustinleblanc (and @serundeputy), was restarting docker seemed to be necessary to get these gains to show up. No idea why but until i did that i saw "good" performance improvements but not "great"(eg the above) ones |
|
In order to get the performance increase I had to restart docker as @pirog also indicated. dgreatsans exclude lando drush st
real 1m56.403s
user 0m0.907s
sys 0m0.287s
lando drush cr:
real 2m39.110s
user 0m0.816s
sys 0m0.280sexcluding lando drush st:
real 0m3.855s
user 0m0.657s
sys 0m0.318s
lando drush cr:
real 0m15.643s
user 0m0.724s
sys 0m0.175spesans exclude lando drush st:
real 0m1.328s
user 0m0.457s
sys 0m0.507s
lando drush cc all:
real 3m0.532s
user 0m0.808s
sys 0m0.183sexcluding lando drush st:
real 0m3.120s
user 0m0.628s
sys 0m0.163s
lando drush cc all:
real 1m16.717s
user 0m0.699s
sys 0m0.313s |
|
Alright gents i've injected the coaxium into the motivator and TURBO MODE is now experimentally/unofficially in Lando This is currently an "unsupported/undocumented" feature but i've got a PR queued up for when we want to put a ring on it And the relevant documentation for it: I'm going to ping people over at #763 to see if we can find any pioneers who want to try this out |
|
@dustinleblanc @pirog @serundeputy .lando.yml for my pantheon wordpress recipe.... excludes:
- vendor
- web/wp-content/uploads
- web/wp-content/themes/wholesalesolar/node_modules
- web/wp-content/themes/wholesalesolar/vendorOther observations:
Overall I'd say this is a huge win. We could probably exclude even more files since we are using a composer based wordpress install. Assuming we don't need our code editors to search wp core from the host, we could exclude web/wp as well. EXCELLENT PROGRESS! Thank You. I'll roll this out to some other devs and see how it goes. |
|
On RC13 comparing no excludes -> excludes I get: So pretty good on that front. Interestingly, site install was taking more like 300 seconds on RC1, so I feel like there have been other improvements along the way too. Thanks! 🚀 🚀 🚀 👍 These were my excludes: |
|
About 10 minutes in, things are going really fast, but I'm wishing I didn't exclude |
|
The example is meant to illustrate how the configuration works. It is not
meant as recommended settings, although I’m guessing they probably will
work well for most people.
It’s important to note that whatever you choose to exclude is going to be
contingent on how fast your computer is, how many files you are excluding
and whether you need to sync those files from the host to begin with or
whether they can be installed as part of a build step which would eliminate
the need for some part of the initial sync in favor of an automated npm
install or composer install. I believe all of these considerations are
documented.
Regardless, it’s going to be up to you to determine the best settings for
you and your team and there should be sufficient mechanisms in place to get
good performance without a 10 minute boot up.
For some perspective, on some of our largest sites where we are syncing
hundreds of megabytes initially we never see boot up times that exceed a
minute or two. Subsequent rebuilds are considerably faster.
…On Wed, Feb 27, 2019 at 10:01 PM Daniel Nitsche ***@***.***> wrote:
About 10 minutes in, things are going really fast, but I'm wishing I
didn't exclude modules/contrib. Considering the number of contrib modules
and files in each of those modules, I not sure it's worth including this
path in the example documentation.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1460 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAri0JnSMj4SNDzr4tOBE-38JY1GdfNrks5vR0ZxgaJpZM4a52Sy>
.
|
|
For me,
|
|
Does excluding the directories mean the autocompletition in IDE also won't work? |
|
@tormi i also ran into this earlier this morning. what do you have for my excluded directories are showing |
|
@vasikgreif possibly, i think it would depend on whether the code you exclude exists on your host before you exclude it or not. There are definitely consequences to excluding code and we aren't 100% sure of what they all are at this point hence why we've only pinged this thread about the feature and haven't released it generally yet. |
|
Also seeing permissions related stuff going on... ➜ pl-wss-dot-com (master) ✗ lando ssh --service=sage
node@faa5180fbdd4:/app$ cd $LANDO_MOUNT/web/wp-content/themes/wholesalesolar
node@faa5180fbdd4:/app/web/wp-content/themes/wholesalesolar$ yarn install
yarn install v1.13.0
[1/5] Validating package.json...
[2/5] Resolving packages...
[3/5] Fetching packages...
info fsevents@1.2.4: The platform "linux" is incompatible with this module.
info "fsevents@1.2.4" is an optional dependency and failed compatibility check. Excluding it from installation.
[4/5] Linking dependencies...
warning " > stylelint-webpack-plugin@0.10.5" has incorrect peer dependency "webpack@^1.13.2 || ^2.7.0 || ^3.11.0 || ^4.4.0".
error An unexpected error occurred: "EACCES: permission denied, mkdir '/app/web/wp-content/themes/wholesalesolar/node_modules/abbrev'".
info If you think this is a bug, please open a bug report with the information provided in "/app/web/wp-content/themes/wholesalesolar/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
node@faa5180fbdd4:/app/web/wp-content/themes/wholesalesolar$Seems like we don't have proper perms in the node_modules folder for some reason: node@faa5180fbdd4:/app/web/wp-content/themes/wholesalesolar$ cd node_modules/
node@faa5180fbdd4:/app/web/wp-content/themes/wholesalesolar/node_modules$ ls -al
total 4
drwxr-xr-x 2 www-data www-data 4096 Feb 27 23:01 .
drwxr-xr-x 23 www-data www-data 736 Feb 28 16:42 ..
node@faa5180fbdd4:/app/web/wp-content/themes/wholesalesolar/node_modules$ touch t
touch: cannot touch 't': Permission deniedMay have something to do with the directory already being there (or not) on rebuild? |
|
I don’t think the directory being there already matters.
If any of you all have installed lando from source there is a commit in
master that seems to address some of perm issues we are seeing with these
excluded directories.
Will include it in the next version but we aren’t scheduled to do a release
for two weeks, barring any critical bugs that surface beforehand
…On Thu, Feb 28, 2019 at 11:48 AM Chad Lancour ***@***.***> wrote:
Also seeing permissions related stuff going on...
➜ pl-wss-dot-com (master) ✗ lando ssh --service=sage
***@***.***:/app$ cd $LANDO_MOUNT/web/wp-content/themes/wholesalesolar
***@***.***:/app/web/wp-content/themes/wholesalesolar$ yarn install
yarn install v1.13.0
[1/5] Validating package.json...
[2/5] Resolving packages...
[3/5] Fetching packages...
info ***@***.***: The platform "linux" is incompatible with this module.
info ***@***.***" is an optional dependency and failed compatibility check. Excluding it from installation.
[4/5] Linking dependencies...
warning " > ***@***.***" has incorrect peer dependency "webpack@^1.13.2 || ^2.7.0 || ^3.11.0 || ^4.4.0".
error An unexpected error occurred: "EACCES: permission denied, mkdir '/app/web/wp-content/themes/wholesalesolar/node_modules/abbrev'".
info If you think this is a bug, please open a bug report with the information provided in "/app/web/wp-content/themes/wholesalesolar/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
***@***.***:/app/web/wp-content/themes/wholesalesolar$
Seems like we don't have proper perms in the node_modules folder for some
reason:
***@***.***:/app/web/wp-content/themes/wholesalesolar$ cd node_modules/
***@***.***:/app/web/wp-content/themes/wholesalesolar/node_modules$ ls -al
total 4
drwxr-xr-x 2 www-data www-data 4096 Feb 27 23:01 .
drwxr-xr-x 23 www-data www-data 736 Feb 28 16:42 ..
***@***.***:/app/web/wp-content/themes/wholesalesolar/node_modules$ touch t
touch: cannot touch 't': Permission denied
May have something to do with the directory already being there (or not)
on rebuild?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1460 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAri0G1H0JLqaML3EMp3pCOzrUVqcbfUks5vSAhfgaJpZM4a52Sy>
.
|
|
@pirog , I have
Fresh site install (
initially and The ownership of
after using |
|
Hitting a composer error when excluding my core directory, on windows if it makes a difference
|
|
Thees docs havent been released yet because the feature is still
experimental but they provide some background
https://github.com/lando/lando/pull/1500/files#diff-9d47c78a8401d598dd434000aa5115e4R126
…On Tue, Aug 13, 2019 at 2:58 PM mrweiner ***@***.***> wrote:
Hitting a composer error when excluding my core directory, on windows if
it makes a difference
λ lando composer update drupal/core
1/7: http://repo.packagist.org/p/provider-latest$afb01c74abd36587b21f8c2236bf0e570db1793b24dbe21c56d18bbd6cfd6b96.json
2/7: http://repo.packagist.org/p/provider-2019-04$540f80defa19af1bbf855afaadc3cc6e4ee04fb54bb071b4df627a615e0b4632.json
3/7: http://repo.packagist.org/p/provider-2016$7361df6435a0942610c6619b5cbaaf350007adbcf0cd928f8e4207dda7c6d337.json
4/7: http://repo.packagist.org/p/provider-2019-07$9e91e76798beafedf23ccf56725d3d088278192fb48ee96c8b2267c0a73664a3.json
5/7: http://repo.packagist.org/p/provider-2018$0785964fbd05606fdd81679ebe8fde72adc8d1b7cf506b830ddba7166527eefe.json
6/7: http://repo.packagist.org/p/provider-2017$bcdd918fa4125f012aa3ab5e3a1775f80c13cc6e5cb9f1b6ea19ef07cf462d01.json
7/7: http://repo.packagist.org/p/provider-2019-01$75cb18f70539666b6897b83eacf219b97779e88b01ffff05278aa31230d5062c.json
Finished: success: 7, skipped: 0, failure: 0, total: 7
Gathering patches for root package.
- Removing drupal/core (8.7.6)
Removing package drupal/core so that it can be re-installed and re-patched.
[RuntimeException]
Could not delete core:
lando ssh then trying rm -rf core throws rm: cannot remove 'core': Device
or resource busy
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1460?email_source=notifications&email_token=AAFOFUFHSQOX5746RO5PKCDQEL77BA5CNFSM4GXHMSZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4GUTCY#issuecomment-520964491>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAFOFUF4WYVUF7QXFLVRSELQEL77BANCNFSM4GXHMSZA>
.
|
|
Ah okay. Does that essentially mean that excluded directories cannot be updated/re-installed with composer? Do we need to completely destroy and restart the machine in order to run updates? |
|
yeah unfortunately docker wont let you delete an entire mounted directory and composer tries to delete an entire mounted directory when it updates... so this is always going to be a struggle. what i've done in this situation is
its not pretty but i think that is the best (only?) workaround for now. definitely down for other suggestions if anyone's got them! |
|
@pirog not sure if this is at all viable, but do you think it'd be possible to setup some sort of symlink internally so that the mounted directory itself doesn't need to be removed? I guess the idea would be that a separate "copy" of the vendor folder ( |
|
@mrweiner maybe im not understanding exactly what you are describing but i feel like you'd end up with the same problem in that model as well. Even if you didn't my guess is something like that would be very hard to implement eg it would likely be brittle and possibly cause more problems than it fixed. Sadly, im not sure there is really a great way to deal with this mismatch between what docker needs and what composer needs beyond documenting a workaround. definitely open to other ideas though! |
|
Not sure if it's been reported before, but I didn't recall seeing this. It seems that on an entirely fresh start for a project, the Turbo Mode doesn't "trigger". Because I had a mismatched Docker version I uninstalled Lando and Docker for Mac and reinstalled everything. My projects had to be set up from scratch, obiously. The named volumes weren't created, though, I needed to issue a rebuild to get things to work again. I don't think this is expected behaviour, right? |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions and please check out this if you are wondering why we auto close issues. |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions and please check out this if you are wondering why we auto close issues. |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions and please check out this if you are wondering why we auto close issues. |
|
Fwiw, tested this a few days ago while running Docker 2.2.0.4 (patched version from #2052 (comment)) before downgrading back to 2.2.0.3 and the sync time appears to be exponentially faster with the update applied. Syncd ~400mil Possible that it's coincidence and something else was going on but it seemed noteworthy. Vendor seems to be syncing at ~142kb/s which seems terribly slow but I don't know enough to say why this might be. EDIT: Welp, in the end, failure |
|
@mrweiner that is quite interesting! I wonder what would account for the major different between 2.2.0.3 and 2.2.0.4, but I won't look a gift horse in the mouth if we get mo powa baby |
|
@dustinleblanc I like the part where he says mo powa baby |
|
To confirm, as of now changes made inside the container for an excluded path like |
|
@amaisano its a 0-way street. Host and container are completely decoupled. The exception is on the first start/rebuild when the host files will be synced into the container to make sure you are starting from a consistent place. Another caveat is if you are running on a Docker Desktop for Mac that supports mutagen eg the latest Docker Desktop Edge and running on a newer version of Lando then Lando will mutagen sync the excluded directories which means you should get performance gains without sacrificing the ability to edit files on either side and have those edits propagate to "the other side". |
Is this any time I run |
|
the first
|
|
I'd love to have some way to disable the automatic syncing in the beginning and have some way to do a manual sync from container to host. I have the following use case: So I need to run the npm install in the container. Right now my solution is
Whenever I need to rebuild the container, I repeat the steps above This seems like a little too much overhead and stuff to think about. As I don't really care if excluded folders of the container are in a consistent state with the host when the containers are being built. I thought about having a command that spins up the same container with the node_modules (just for this example, it can be whatever folder) folder from the host being mounted to a different path on the container and then doing a rsync to sync the contents and then kill the new container again. This could be called manually whenever I want to sync the content back to my host system. Just as an Idea |
As a Mac OS / Windows Lando user,
I want to selectively mount paths within my application in named volumes,
So that I can increase my development speed by bypassing a significant portion of the speed penalty imposed by Docker's underlying filesharing mechanisms.
Notes from today's discussion:
.lando.ymlThe text was updated successfully, but these errors were encountered: