Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A NEED FOR SPEED #763

Open
dustinleblanc opened this issue Mar 15, 2018 · 104 comments

Comments

Projects
None yet
@dustinleblanc
Copy link
Member

commented Mar 15, 2018

Lando is amazing, but we're never going to make the Kessel run fast enough when we have to fileshare all that vendor code in mounted volumes. We should start exploring if there is a HYPERDRIVE mode we could use that keeps vendor code outside the volume mounts for as many frameworks as we can and make it togglable so we don't scare people.

@pirog

This comment has been minimized.

Copy link
Member

commented Mar 15, 2018

+1

we should add a toggle for node_modules as well so we can complete the run in less than 12 parsecs

@uberhacker

This comment has been minimized.

Copy link
Contributor

commented Mar 15, 2018

Oh, so now you agree including node_modules is a good idea? ;)

@pirog

This comment has been minimized.

Copy link
Member

commented Mar 15, 2018

@uberhacker this is something completely different

@uberhacker

This comment has been minimized.

Copy link
Contributor

commented Mar 15, 2018

I wish node_modules would just disappear. The best way you can improve your speed is get off Mac and start using Linux instead. :-P

@pirog

This comment has been minimized.

Copy link
Member

commented Mar 15, 2018

for this specific concern eg file sharing slowdowns you are absolutely right about linux.

@dustinleblanc we should def consider the OS if we decide to try something fancy here.

@dustinleblanc

This comment has been minimized.

Copy link
Member Author

commented Mar 15, 2018

I think that in the end, whatever solution will be marginally faster on linux too, so long as it hurts nobody, I think its fine to be OS agnostic.

@uberhacker this is actually 180 degrees opposite our discussion earlier 😆 we're discussing finding a way to ensure the vendored code (/vendor in php, node_modules for node, whatevers for others) doesn't get shared to the host OS because volume mounting is slow. The idea here would be to install deps only in the container and ensure the application can access them. After some improvements in Git clone ops we were encouraged to make ALLTHESPEEDS as much as we can

@nterbogt

This comment has been minimized.

Copy link

commented Mar 15, 2018

+1

1 similar comment
@vincenzo

This comment has been minimized.

Copy link
Contributor

commented Mar 23, 2018

+1

@pirog

This comment has been minimized.

Copy link
Member

commented Mar 23, 2018

I don't think we need to tie these together in an epic but the only currently actionable issue we have to help with this broader initiative is over at #760

It might be worth spit balling additional ideas here so when the time comes we are ready to roll with some improvements. I'm open to pretty much any suggestion to improve performance as long as it doesnt require SIGNIFICANT alteration to our upstream dependencies eg: implementing our own file sharing solution is not something that is going to happen.

It also is worth reviewing previous issues that were similar to this so we don't rehash things that have already been discussed:
#518
#561

@jraviotta

This comment has been minimized.

Copy link

commented Apr 1, 2018

My understanding of docker is that the aggregation more smaller containers is more efficient than fewer larger containers. Is it worth splitting larger muti-component apps into discreet containers? For example, for a drupal recipe, building separate micro services for behat, mink, selenium, gulp, and whatever else gets crammed into the php container with composer.

@pirog

This comment has been minimized.

Copy link
Member

commented Apr 1, 2018

@jraviotta this is already what we do and also not really what we are talking about in this thread.

@dustinleblanc

This comment has been minimized.

Copy link
Member Author

commented Apr 1, 2018

No, I don't think that's going to have any real impact on what we're discussing here. We don't actually install any of those dependencies in default containers, users add them to whatever service they decide to, typically divided by runtime. We already encourage using Gulp in a node server, behat in your php service (as its a project Composer dependency) and running your headless browser in it's own container (Selenium in your scenario, but could be Phantom or Chrome Driver).

This is talking mostly about finding ways to offload code to non shared directories to improve speed by relying less on volumes (which are slow on mac/win).

@jraviotta

This comment has been minimized.

Copy link

commented Apr 1, 2018

Ok. Thanks for the clarification. I'm still learning. 😁

@tylerssn

This comment has been minimized.

Copy link

commented May 25, 2018

Has docker-sync been considered? My understanding is that all project files will be synced into the container on-change rather than being mounted and read from the host system.

@pirog

This comment has been minimized.

Copy link
Member

commented May 25, 2018

@tylerssn we used a very similar mechanism to docker-sync in Kalabox which was sort of the precursor to Lando.

While it definitely CAN be a lot faster for SOME THINGS on SOME MACHINES we found that at scale (eg thousands of thousands of users) it ended up being

  1. Not super reliable eg the underlying daemon would crash a lot
  2. Not suitable for all use cases eg youd wait for minutes after running something like npm install for all your files to show up in the container
  3. A huge drain on time, eg we spent like 40% of our total support and dev time handling "file sharing" related issues.

So it's highly unlikely we use something else unless it does not have any of the above three issues. Currently i know of no such solution. I also think we are better served here to let Docker continue to improve their solution while we focus on the things that make Lando great.

@dustinleblanc

This comment has been minimized.

Copy link
Member Author

commented May 25, 2018

Yeah to piggy back on what @pirog said. I think the goal of this initiative is to do what we can WITHOUT messing with the filesync/mounting setup. Our current thoughts are to see if we can keep as much code as humanly possible outside of the volume mounts to avoid the performance loss. This was sparked by a revelation of how much faster we made git clones by cloning into a non-volume mounted directory. We want to see if we can use that strategy for vendor code as that would likely help immensely with speed.

On the actual filesyncing level, I think tweaks we want to make will be things that stick as close to the vanilla docker-compose behavior as we can. The goal is to get wins without the cost of support.

@tylerssn

This comment has been minimized.

Copy link

commented May 25, 2018

Is it feasible to keep the current Lando mount setup but create an in-container project directory replica that maintains a two-way sync with the mounted directory? This replicated synced directory would be what NGINX and PHP commands such as phpunit reference.

Something like the following would run within the container:

while inotifywait -r -e modify,create,delete /directory; do
    rsync -avz /directory /target
done

This would give the system a non-mounted reference to the files, however, it would:

  • require inotifywait to be installed within the container
  • require rsync to be installed within the container
  • potentially double project disk space usage
  • potentially simplify Lando backwards compatibility
    • i.e. if mount_sync: true and service rebuilt the $LANDO_WEBROOT would be updated as well as www-data users default login path
@tylerssn

This comment has been minimized.

Copy link

commented May 25, 2018

Agh, seeing your response @dustinleblanc now, I think we can throw out my suggestion.

@pirog

This comment has been minimized.

Copy link
Member

commented May 25, 2018

@tylerssn this is almost exactly what we did for Kalabox (except we used Unison) and i feel fairly confident saying that it wasn't worth it.

@jbertoen

This comment has been minimized.

Copy link
Contributor

commented Jun 11, 2018

What about using named volumes for the vendor and node_modules dir?

This way we can share those volumes between containers without the unnecessary overhead on syncing it and duplicating the files between those volumes.

Because currently if you do composer install on an appserver, you sync those files back to your host OS. And that will sync it back to all the other containers again, like your nginx and nodejs containers. Right?

Mapping a named volume, the composer install would create the files only in the named volume, mounted to one or more containers. Making those instantly available.

Another, if it works, more preferable approach would be one named volume for app, shared across multiple containers.
docker/compose#4675

In docker compose v2 there was an option available called volumes_from. With this option you could target a named volume from a container which could be mapped to a path on the host system.
https://docs.docker.com/compose/compose-file/compose-file-v2/#volumes_from
In docker compose v3 this is removed and it is focused completely on named volume. Which lando already uses for solr, mysql, and other data storage.
But maybe this could work:
docker/compose#4675 (comment)
Which would allow a named volume to be bound locally if I understand it correctly.

@hanoii

This comment has been minimized.

Copy link
Contributor

commented Jun 19, 2018

@pirog what wasn't worth it about unison? I keep getting back to dockerized solutions from time to time and right now seems that unison looks like the most performant solution. I was pointed to Docksal, which I tried. The performance (benchmarking just a call to drush cr on both is very similar, but they provide an option to use unison, and with that it goes almost to half of it. Still 2x my native performance on osx.

@pirog

This comment has been minimized.

Copy link
Member

commented Jun 19, 2018

@hanoii if you are legitimately interested in knowing there is a considerable amount of data over at
https://github.com/kalabox/kalabox/issues?q=is%3Aissue+file+syncing+is%3Aclosed+sort%3Acomments-desc

@hanoii

This comment has been minimized.

Copy link
Contributor

commented Jun 19, 2018

thinking out loud, and I am sure you might have thought about this, but wouldn't it make sense to find a way to do it backwards, share the container FS on the host (thinking sshfs or the like). It's probably counter intuitive but it might be something to explore, or maybe is a terrible idea :)

@pirog thanks, I am legitimately interested - will try to go over the details.

@pirog

This comment has been minimized.

Copy link
Member

commented Jun 19, 2018

@hanoii we've tried various solutions like that, in both directions and then were able to get feedback from thousands of users. At the end of the day it wasn't something most users had a pleasant experience with unless they knew what was going on behind the scenes. i think the comments above at
#763 (comment) and #763 (comment) still stand.

That said, i think there are two potentially actionable pathways we can take in the near term that come with acceptable maintainability considerations:

  1. Provide some config mechanism to target volume mounting (eg dont share vendor or only share mycode). @dustinleblanc has done some initial benchmarking here and found some significant performance gains
  2. Use the nfs that ships with (at least) Docker for Mac. This comes with a considerably more complex implementation so while its possible we do it at some point it's unlikely to happen before we finish our testing infrastructure which is a BIG task.
@dustinleblanc

This comment has been minimized.

Copy link
Member Author

commented Jun 19, 2018

My testing thus far on using named volumes per @jbertoen's suggestion has been great for performance but surfaced some quirky issues:

  1. If you use PHPStorm or another IDE that allows you to dive in an explore code inside your dependencies, it can't find that code inside the container, making these features unusable in some circumstances.
  2. When trying to mount Drupal's core library, I got a good speed boost, but it unfortunately causes some errors when I run composer install/update on one of our D8 sites built with Composer.

In my case, I went for ALLTHETHINGS and mounted vendor, web/core, web/modules/contrib, and web/themes/contrib in named volumes. I was able to see composer install times cut in half using this method and noticeably faster page loads and Drush interactions.

I think this is worth exploring as optionable mounts, because it makes development of these types of apps on OSX livable.

@jbertoen

This comment has been minimized.

Copy link
Contributor

commented Jun 19, 2018

What about copying the contents of vendor dirs to a volume (on start). And on post-composer, post-npm, post-yarn event sync back to osx?
I personally do not use composer, npm and yarn commands that often to have issues with some lag at those moments. Its mostly the development of custom applications and functional use in the browser that slows me down because of these performance issues

@jbertoen

This comment has been minimized.

Copy link
Contributor

commented Jun 19, 2018

Or what about reversing the way we look at volume mapping. Instead of mounting everything in sync. What about copying the whole /app folder by default in a single named volume., and configure which specific directories to keep in sync on your host os such as osx.
Than on events such as npm install, composer install, etc. Choose directories to sync back.

@jbertoen

This comment has been minimized.

Copy link
Contributor

commented Jun 19, 2018

Or might be even faster to just map/bind the composer.json and composer.lock file. Run a composer install locally to make the vendor dir on your host the same as on your volume

@dmsmidt

This comment has been minimized.

Copy link

commented Jun 19, 2018

I know that with Lagoon from Amazee the vendor folder is not synced constantly and only when building the containers it is put in the container. So a rebuild is needed after a composer.json change. This is no problem because it doesn't happen so often. But it can be done nicer.

I don't like to do it twice, in the container and locally just for the IDE. Also the php version mismatch between the host and client can result in problematic dependency resolution of vendor packages.

So I would prefer that the install command is run inside the container. And that the vendor can automatically be synced to the host once in a while. This can be done after a start or after any composer command in the container via a little wrapping logic around 'lando composer '

@jbertoen you are a little idea stealing bad boy, love 😋

@jbertoen

This comment has been minimized.

Copy link
Contributor

commented Jun 19, 2018

Don't know what you are talking about @dmsmidt? Do we know each other 😜

I do agree with the double composer install overhead, was more meant to be as open minded as possible to all options.

Other options is to look at docker compose v2, instead of the v3 lando uses currently. With v2 the volumes_from might work. That would map the same volumes from container X within another container.
Do not know for sure if it reuses volumes, or duplicates them though.

@dmsmidt, I could not vind the whole "copy the vendor folder logic" at the amazee.io setup. Where is that done?

@pirog

This comment has been minimized.

Copy link
Member

commented Apr 1, 2019

UNSTALE

@stale stale bot removed the wontfix label Apr 1, 2019

@tylerssn

This comment has been minimized.

Copy link

commented Apr 9, 2019

Did this carry into RC14? The below setup does not prevent file syncs in vendor and node_modules after rebuilding Lando and restart docker.

excludes:
  - vendor
  - '!vendor/moninglobal'
  - node_modules

Edit: Addressed in #763 (comment)

@pirog

This comment has been minimized.

Copy link
Member

commented Apr 9, 2019

@dustinleblanc

This comment has been minimized.

Copy link
Member Author

commented Apr 10, 2019

Noting it here is good enough @tylerssn how did you confirm that file syncing was not working?

@tylerssn

This comment has been minimized.

Copy link

commented Apr 10, 2019

lando ssh -c 'touch vendor/testfile' && ls vendor/testfile; # File listed on host. Expected it not to list.
lando ssh -c 'touch node_modules/testfile' && ls node_modules/testfile; # File listed on host. Expected it not to list.
rm vendor/testfile && lando ssh -c 'ls vendor/testfile'; # File not listed in container. Expected it to list.
rm node_modules/testfile && lando ssh -c 'ls node_modules/testfile'; # File not listed in container. Expected it to list.
@edurenye

This comment has been minimized.

Copy link
Contributor

commented Apr 14, 2019

@pirog Maybe we could create a 'lando hibernate' command that uses 'docker checkpoint' to hibernate containers and fast start them up again with a let's say... 'lando restore' command.

@ghazlewood

This comment has been minimized.

Copy link

commented May 8, 2019

Just wanted to leave a note to say that very simply adding the following to the end of my config has made lando much faster and certainly more usable:

services:
  database:
    type: mysql:5.7:delegated
excludes:
  - vendor

I'm running v3.0.0-rc.14 on MacOS 10.14.4 with Docker Desktop 2.0.0.3 (Engine 18.09.2, Compose 1.23.2, Machine 0.16.1)

@tylerssn

This comment has been minimized.

Copy link

commented May 8, 2019

In reference to #763 (comment) and #763 (comment). Just clarifying that even without !vendor/mycompany and after lando rebuild -y the directories continue to sync with my host.

recipe: lemp
excludes:
  - vendor
  - node_modules

This is on Ubuntu 18.04.2 LTS.

Edit: Addressed #763 (comment)

@pirog

This comment has been minimized.

Copy link
Member

commented May 8, 2019

@tylerssn excludes is only for macOS and Windows. This is implied in the docs but should probably be made more clear
http://pr-1500-v4h3hcq-iaimrn624qfk6.us.platform.sh/config/performance.html

@stale

This comment has been minimized.

Copy link

commented Jun 8, 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions and please check out this if you are wondering why we auto close issues.

@stale stale bot added the wontfix label Jun 8, 2019

@edurenye

This comment has been minimized.

Copy link
Contributor

commented Jun 10, 2019

unstale

@stale stale bot removed the wontfix label Jun 10, 2019

@snipebin

This comment has been minimized.

Copy link

commented Jun 25, 2019

For those struggling with performance on mac - it got so bad that I resorted to running ubuntu server on VMWare Fusion and running lando inside it and performance is great now. Downsides are VM spin up and down and I have to manually add entries to /etc/hosts

@brooke-heaton

This comment has been minimized.

Copy link

commented Jun 25, 2019

My Macbook Pro is in my closet these days. I'm on a System76 now. Exponentially faster.

@dustinleblanc

This comment has been minimized.

Copy link
Member Author

commented Jun 25, 2019

@brooke-heaton I run Pop OS on my desktop and its a dream. Linux still doesn't have comparable media creation software to what I get on my Mac and that's what keeps me holding on. Once System 76 gets that dialed, a Darter Pro / Galago Pro or its equivalent at the time will be in my shopping cart.

@dustinleblanc

This comment has been minimized.

Copy link
Member Author

commented Jun 25, 2019

@snipebin have you tried using our super secret excludes feature yet? Yeah its still slower than native Docker on a linux box, but its way, way faster than without it

@digitole

This comment has been minimized.

Copy link

commented Jun 27, 2019

@dustinleblanc First of all, thank you for experimenting with speeding up lando. I have tried the excludes feature on a drupal site, and couldn't see significant changes in lando composer install or lando drush cr unfortunately. A couple of seconds faster on processes that take minutes. (I did restart of docker, and a lando destroy).
I excluded vendor, node_modules, contrib modules and the drupal core directory.

lando version: v3.0.0-rc.17
docker version: 18.09.2

@vasikgreif

This comment has been minimized.

Copy link

commented Jun 27, 2019

For those struggling with performance on mac - it got so bad that I resorted to running ubuntu server on VMWare Fusion and running lando inside it and performance is great now. Downsides are VM spin up and down and I have to manually add entries to /etc/hosts

@snipebin I'm curious how you set up this - do you have you IDE on Mac and Lando in the virtual maschine? Do you have XDebug working?

@vasikgreif

This comment has been minimized.

Copy link

commented Jun 27, 2019

@pirog I'm trying to use the excludes feature for my WP test install, with a bunch of default plugins / themes and usually one plugin I'm working on. My current config is:

name: test
recipe: wordpress
excludes:
  - wordpress/wp-content/plugins
  - wordpress/wp-content/themes
  - "!wordpress/wp-content/plugins/woo-products-mix"
config:
  php: 7.2
  via: nginx
  database: mariadb
  conf:
    php: php/php.ini
  webroot: wordpress

I have two issues with this: my wordpress/wp-content/plugins/woo-products-mix folder includes vendor and node_module directories, which I would like to exclude. Is there a way to do that? Also, my custom php.ini file does not seem to be loaded (which does not have anything to do with the speed, but I'm curious what's wrong with my config). Any help? Thanks!

@pminf

This comment has been minimized.

Copy link

commented Jun 27, 2019

@digitole What about excluding everything which is temporary or generated by software? Like .git, .sass-cache, your IDE folder (e.g. .idea for PHPStorm) and of course Drupals files directory.

@dustinleblanc

This comment has been minimized.

Copy link
Member Author

commented Jun 27, 2019

@digitole if you're exclude vendor and core, you should see a HUGE difference. One way to check that this is working is to wipe out your vendor dir and core, etc, make sure they are empty, then boot lando and do the composer install. If you see files in the vendor dir after doing that, then the exclusion isn't working. The Docker restart thing is the only thing I know of that remedies this. Also, keep in mind, the exclusions only work after a rebuild. You'll see some very clear ascii art about Turbo mode too.

@pirog

This comment has been minimized.

Copy link
Member

commented Jun 27, 2019

@ccharlton

This comment has been minimized.

Copy link
Contributor

commented Jun 27, 2019

Re: Named Volume Performance Feature #1460

@pirog

This comment has been minimized.

Copy link
Member

commented Jun 28, 2019

Yes! Thanks @ccharlton, was on my phone and couldnt do the linkage!

#1460!

@snipebin

This comment has been minimized.

Copy link

commented Jun 28, 2019

For those struggling with performance on mac - it got so bad that I resorted to running ubuntu server on VMWare Fusion and running lando inside it and performance is great now. Downsides are VM spin up and down and I have to manually add entries to /etc/hosts

@snipebin I'm curious how you set up this - do you have you IDE on Mac and Lando in the virtual maschine? Do you have XDebug working?

@vasikgreif VScode running on macOS, ubuntu server 18.04 running on vmware fusion and lando running on the ubuntu server vm.

Xdebug has been a total PIA to setup, I'm running into connection timeout errors while trying to setup a remote ssh tunnel forwarding

@tylerssn

This comment has been minimized.

Copy link

commented Jul 17, 2019

@ Lando Team - how's is this progressing? What do you need from others in terms of testing this feature?

We have a few guys on Mac OS and working with Magento 2 is painful with file sync. If there's anything I can do to help stabilize the feature whether it be testing or otherwise, let me know.

@pirog

This comment has been minimized.

Copy link
Member

commented Jul 17, 2019

Lando is free and open source which means its run basically by volunteers in their free time. This is to say we get things done when we can and we havent had a bunch of time lately.

We gladly accept sponsorship money to push/complete specific feature. If thats something you want to do then feel free to email us. If its not then you're going to have to wait till we get to it ;)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.