Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch to using the Core local environment #17004

Merged
merged 91 commits into from Aug 30, 2019

Conversation

@pento
Copy link
Member

commented Aug 12, 2019

Description

Now that WordPress Core has its own Docker environment, it's possible to base the Gutenberg environment on that.

There are several advantages for Gutenberg, when compared to the existing environment:

  • It's cross-platform, only requiring Docker to be installed and running. It also works with Docker Toolbox, allowing all Windows users (not just those with Windows 10 Pro) to use it.
  • The WordPress environment uses Docker images that we maintain for the purpose of Core development, which allows us to to customise them as we need.
  • It ensures that tests for Core and Gutenberg run in the same environment.
  • It's a single environment, which helps avoid the naming clashes we've occasionally seen between the Core and Gutenberg environments.

More broadly for WordPress, tying the development processes of Core and Gutenberg a little closer together helps pave the way for the processes to merge over time.

Architectural Aims

This PR removes the Docker config from Gutenberg entirely (barring a minimal template to hook into the Core environment): requiring wordpress-develop to run is intentional.

There are a few reasons for requiring wordpress-develop. It's worth noting that one of my primary guides for making decisions around local-env is to lay the groundwork for a dev environment that works as easily as possible for non-devs. With that in mind:

  • Having a single docker-compose.yml file in Core, as opposed to having one in Core, one in Gutenberg, and many more in other feature plugins that will be able to use this, means that there's only set of Docker containers running. Docker can get fairly weird if you're trying to run containers with the same names, which mount volumes with the same names.
  • Having multiple docker-compose.yml files doesn't lend itself to testing plugins together: each plugin ends up with its own environment which can't talk to others.
  • Managing how plugins can mount older versions of WordPress becomes exponentially more difficult if the docker-compose.yml config needs to work for every WordPress branch (this is a requirement for local-env so that auto updates can be worked on in old branches), rather than being able to tweak the config for each branch.
  • Ideally, feature plugins shouldn't need to do much customisation of the Docker environment, beyond mounting their particular volumes. Exposing the entire docker-compose.yml file is overkill.

That said, not every Gutenberg contributor will have a WordPress source clone, or want to manage one. So, there's also a helper which will download and install a copy of the WordPress source repo, when needed.

Much like the WordPress environment, however, I've aimed to not automate everything. As we found with the Gutenberg environment, too much automation tended to make it difficult (particularly for more advanced contributors) to customise or debug their environment. For example, the scripts no longer install NVM, switch your Node version, or run npm install.

How It Works

If you don't have a WordPress repo to work with, just run npm run env install, that'll take care of downloading WordPress, building it, and connecting Gutenberg into it.

Alternatively, setting the WP_DEVELOP_DIR environment variable to your WordPress source directory, and running npm run env connect will write a docker-compose.override.yml file to the WordPress source directory, and restart the Docker containers. The Docker override file contains the appropriate volume settings for mounting Gutenberg within the WordPress environment.

Here's a complete list of the commands added in this PR:

npm run env install: Automatically downloads, builds, and installs a copy of WordPress to work with. This will be installed in the wordpress folder.

npm run env connect: If you have a WordPress respository checked out already, define the WP_DEVELOP_DIR environment variable with the path to your repository, then run this command to mount Gutenberg inside the Core environment.

npm run env start / npm run env stop: After the environment is setup, there's no need to run the install script again. the start and stop scripts will start and stop the Docker containers.

npm run env update: When a managed WP install is being used, the update script will update the WordPress install and rebuild it. This doesn't happen automatically, but the start script will show warnings if it hasn't been run for more than 7 days.

npm run env reinstall: Resets the database and re-configures WordPress again.

npm run env cli: This is a shortcut for WP-CLI, it also ensures that the CLI command is being run against src or build, depending on what you've set in LOCAL_DIR.

npm run env docker-run: This is a shortcut for docker-compose run --rm.

npm run env test-php: This will run the PHPUnit tests.

npm run env lint-php: This will run the composer lint scripts.

Testing

After both of these methods, your WordPress install will be at http://localhost:8889/wp-admin/, username is admin, and password is password.

Method A (download and configure WordPress automagically)

For testing the managed WordPress install, just run npm run env install, and watch the magic happen. 馃帀

Method B (connect to an existing wordpress-develop install)

If you want to test with an external WordPress directory, check it out, and run these commands in your WordPress directory:

nvm install
npm install
npm run build:dev
npm run env:start
npm run env:install

Then in your Gutenberg directory, run:

export WP_DEVELOP_DIR=/path/to/wordpress-develop
npm run env connect
npm run env cli plugin activate gutenberg

Reusability

These scripts are intended to be reusable by other Core feature plugins. While they may support third party development in the future, this isn't an immediate goal.

Copy link
Contributor

left a comment

The ./bin/setup-local-env.sh script is documented in a lot of places of the repo. Contributors shouldn't have to clone another repository to trigger a local environment to work on Gutenberg.

We should instead have a local environment embedded in this repository instead. Sharing the same config as Core makes sense.

.travis.yml Outdated Show resolved Hide resolved
@pento

This comment has been minimized.

Copy link
Member Author

commented Aug 13, 2019

Thank you for your initial thoughts, @youknowriad. This PR is still highly experimental, I haven't touched documentation or worked on polishing the UX, since how it all works together is likely to change as I continue to explore it.

Similarly, you're likely to come across fairly hacky bits and pieces which aren't really ready for review. (For example, Travis config is currently hacked together for looking at "what would it take to get Travis passing in this scenario".)

@youknowriad

This comment has been minimized.

Copy link
Contributor

commented Aug 13, 2019

Oh sorry @pento for being too fast. Let me know when you need feedback

@pento pento force-pushed the try/use-core-local-env branch 2 times, most recently from 23a52b1 to f017b32 Aug 14, 2019
@pento pento marked this pull request as ready for review Aug 15, 2019
@pento pento requested review from ajitbohra, gziolo, nerrad, nosolosw and ntwb as code owners Aug 15, 2019
.gitignore Show resolved Hide resolved
@pento

This comment has been minimized.

Copy link
Member Author

commented Aug 15, 2019

This does have the potential drawback of requiring Gutenberg contributors who may be unfamiliar with WordPress Core development to check out and build a copy of WordPress. I haven't written a script to automate this task, but I'm open to arguments for or against it. See the before_install section of .travis.yml for an example of the steps needed in an automated solution.

After some discussion over in the core-editor channel, there's clearly still a need for this to be automated in some form. I'll look into it tomorrow.

@pierlon

This comment has been minimized.

Copy link
Contributor

commented Aug 15, 2019

Instead of cloning wordpress-develop, could it be referenced as a submodule?

@pento

This comment has been minimized.

Copy link
Member Author

commented Aug 15, 2019

@pierlon: The Docker config only lives in the wordpress-develop repo, it isn't included in the WordPress zip download.

A shallow clone is very fast, but the downside is that it requires git to be installed, and available in the path. I'll look at grabbing the wordpress-develop zip from GitHub, as an alternative, but that has its own issues.

@pento pento referenced this pull request Aug 16, 2019
3 of 4 tasks complete
@shivapoudel

This comment has been minimized.

Copy link

commented Aug 17, 2019

@pento Not only Gutenberg plugin but if I do have custom plugin development, then how can I take care of downloading WordPress, building it, and connecting custom XYZ plugin into it utilizing @wordpress/scripts package. Is it possible to link the custom XYZ plugin dev environment on core like how Gutenberg does in this branch? Docker override YAML file has hard-coded path can that be fetched according to package.json environment for plugin dir set, so it might open the possibility of creating a template override for custom plugin development.

@pento

This comment has been minimized.

Copy link
Member Author

commented Aug 19, 2019

Thank you for reminding me, @shivapoudel! I half-implemented this, but didn't get back to it. 馃檪 5e1c8c2 adds the ability to override the default docker-compose.override.yml.template with your own, by adding this section to your package.json file:

	"wp-env": {
		"plugin-dir": "gutenberg",
		"docker-template": "./bin/docker-compose.override.yml.template"
	}
@pento pento referenced this pull request Aug 19, 2019
5 of 5 tasks complete
@shivapoudel

This comment has been minimized.

Copy link

commented Aug 19, 2019

@pento I have some notes for this PR enhancement from my side as I am working on a custom plugin and implementing these scripts wisely :)

Let me go with the flow if the second try of npm run env install is executed. I receive this error means if the folder named wordpress exists inside my plugin then also it is trying to clone in the second run. Maybe there should be existsSync check :)

Some sort of check before downloading WordPress develop source needed
image
Remember I don't want to set WP_DEVELOP_DIR environment somewhere thus intentionally wordpress directory is generated which I can gitignore inside a plugin, amazing job :)

Override Template is not correctly appended on the second run

This is my docker-compose.override.yml.template:

services:
  wordpress-develop:
    volumes:
      - %PLUGIN_MOUNT_DIR%:/var/www/${LOCAL_DIR-src}/wp-content/plugins/%PLUGIN_INSTALL_DIR%
      - %PLUGIN_MOUNT_DIR%/packages/mu-plugins:/var/www/${LOCAL_DIR-src}/wp-content/mu-plugins
      - %PLUGIN_MOUNT_DIR%/packages/addons/everest-forms-zapier:/var/www/${LOCAL_DIR-src}/wp-content/plugins/everest-forms-zapier
  php:
    volumes:
      - %PLUGIN_MOUNT_DIR%:/var/www/${LOCAL_DIR-src}/wp-content/plugins/%PLUGIN_INSTALL_DIR%
      - %PLUGIN_MOUNT_DIR%/packages/mu-plugins:/var/www/${LOCAL_DIR-src}/wp-content/mu-plugins
      - %PLUGIN_MOUNT_DIR%/packages/addons/everest-forms-zapier:/var/www/${LOCAL_DIR-src}/wp-content/plugins/everest-forms-zapier
  cli:
    volumes:
      - %PLUGIN_MOUNT_DIR%:/var/www/${LOCAL_DIR-src}/wp-content/plugins/%PLUGIN_INSTALL_DIR%
  phpunit:
    volumes:
      - %PLUGIN_MOUNT_DIR%:/var/www/${LOCAL_DIR-src}/wp-content/plugins/%PLUGIN_INSTALL_DIR%

Running npm run env install configures docker.override.yml correctly, let's see :)

version: '3.7'
services:
  wordpress-develop:
    volumes:
      - 'D:\htdocs\everest-forms:/var/www/${LOCAL_DIR-src}/wp-content/plugins/everest-forms'
      - 'D:\htdocs\everest-forms/packages/mu-plugins:/var/www/${LOCAL_DIR-src}/wp-content/mu-plugins'
      - 'D:\htdocs\everest-forms/packages/addons/everest-forms-zapier:/var/www/${LOCAL_DIR-src}/wp-content/plugins/everest-forms-zapier'
  php:
    volumes:
      - 'D:\htdocs\everest-forms:/var/www/${LOCAL_DIR-src}/wp-content/plugins/everest-forms'
      - 'D:\htdocs\everest-forms/packages/mu-plugins:/var/www/${LOCAL_DIR-src}/wp-content/mu-plugins'
      - 'D:\htdocs\everest-forms/packages/addons/everest-forms-zapier:/var/www/${LOCAL_DIR-src}/wp-content/plugins/everest-forms-zapier'
  cli:
    volumes:
      - 'D:\htdocs\everest-forms:/var/www/${LOCAL_DIR-src}/wp-content/plugins/everest-forms'
  phpunit:
    volumes:
      - 'D:\htdocs\everest-forms:/var/www/${LOCAL_DIR-src}/wp-content/plugins/everest-forms'

Since this is all fine but whenever I think about future addon development and my plugin anatomy, I can add additional add-ons directory inside packages/addons and connect it to our docker environment with npm run env connect command. Now the config is not correctly generated, look the main first line of share volumes link which is not correct :)

version: '3.7'
services:
  wordpress-develop:
    volumes:
      - 'D:\htdocs\everest-forms/packages/addons/everest-forms-zapier:/var/www/${LOCAL_DIR-src}/wp-content/plugins/everest-forms-zapier'
      - 'D:\htdocs\everest-forms/packages/mu-plugins:/var/www/${LOCAL_DIR-src}/wp-content/mu-plugins'
      - 'D:\htdocs\everest-forms/packages/addons/everest-forms-zapier:/var/www/${LOCAL_DIR-src}/wp-content/plugins/everest-forms-zapier'
  php:
    volumes:
      - 'D:\htdocs\everest-forms/packages/addons/everest-forms-zapier:/var/www/${LOCAL_DIR-src}/wp-content/plugins/everest-forms-zapier'
      - 'D:\htdocs\everest-forms/packages/mu-plugins:/var/www/${LOCAL_DIR-src}/wp-content/mu-plugins'
      - 'D:\htdocs\everest-forms/packages/addons/everest-forms-zapier:/var/www/${LOCAL_DIR-src}/wp-content/plugins/everest-forms-zapier'
  cli:
    volumes:
      - 'D:\htdocs\everest-forms:/var/www/${LOCAL_DIR-src}/wp-content/plugins/everest-forms'
  phpunit:
    volumes:
      - 'D:\htdocs\everest-forms:/var/www/${LOCAL_DIR-src}/wp-content/plugins/everest-forms'

Updated: If I delete the override file inside directory wordpress and run npm run env connect from plugin directory then this issue vanishes :)

As a custom plugin development can we achieve this?
Is there any possibilities to change the site title, email used from this script. As a custom plugin, I want to set different site title and email address. Finally, how can I set the build rather than src version of wordpress while being in the plugin directory, not in the WP develop core directory and setting up it .env file?

Just like the docker override some sort of config to configure site title, email address, etc. is appreciated from my side :)

Finally, will there be a WP version selection OR development version only as a future concise asking it here? If so will the previous version of WP includes docker-compose.yml or not :)

@pento pento force-pushed the try/use-core-local-env branch from 48816cf to 7a8e3e6 Aug 20, 2019
@shivapoudel

This comment has been minimized.

Copy link

commented Aug 20, 2019

@pento I am in a windows environment with Git bash and MySQL volume is not synced in my WordPress directory (which is inside the plugin) due to the format of this line :)

If I personally change this format into - ./mysql:/var/lib/mysql then it syncs the MySQL volume perfectly. Maybe the culprit for windows user but honestly if you have much more idea about where it syncs then it will be appreciated.

@pento

This comment has been minimized.

Copy link
Member Author

commented Aug 21, 2019

@shivapoudel: Thank you for the testing and feedback!

0b4c7f5 adds a check to see if WordPress is already installed.

Good point on old volumes staying around when you run npm run env connect after removing them from the template. I'll see what can be done about that.

Is there any possibilities to change the site title, email used from this script. As a custom plugin, I want to set different site title and email address.

I think we can explore this, but it's a lower priority at the moment. Right now, the aim is to build something that works for Core and Feature Plugin development. If it works for your own plugin development, that's awesome, but I'd like to keep the focus on adding options that Core/Gutenberg need, at least for now. 馃檪

Finally, how can I set the build rather than src version of wordpress while being in the plugin directory, not in the WP develop core directory and setting up it .env file?

It will recognise the LOCAL_DIR environment variable for this. Run export LOCAL_DIR=build before you run npm run env connect to set it all up. this will only effect your current terminal window, though, we can explore making it possible to set this for the entire project.

Finally, will there be a WP version selection OR development version only as a future concise asking it here? If so will the previous version of WP includes docker-compose.yml or not :)

There will be in the future, but WP branches currently don't have the Docker config. I'm planning on backporting it once it's a bit more stable. 馃檪

If I personally change this format into - ./mysql:/var/lib/mysql then it syncs the MySQL volume perfectly. Maybe the culprit for windows user but honestly if you have much more idea about where it syncs then it will be appreciated.

It's referring to the mysql further down in the docker-compose.yml file. I put the MySQL files in a Docker volume so that they persist between restarts, but can be easily removed (for example, if you're switching to an older version of MySQL, it's easier to work with a fresh install, rather than downgrading the data file formats).

@shivapoudel

This comment has been minimized.

Copy link

commented Aug 21, 2019

@pento Thanks you for your reply with such insight... What about the second time run of npm run env connect not storing the old volumes and generating that file correctly? After the install of the environment if I again try to run connect then docker override file is not updated correctly!

TBW storing the old volumes might not be a good decision from my side and there should be an option to store or just update the override file.

@pento

This comment has been minimized.

Copy link
Member Author

commented Aug 21, 2019

@shivapoudel: Could you clarify what's going on here? Are you able to reproduce the same problem by modifying the Gutenberg bin/docker-compose.override.yml.template file, and specifying how you would expect it to behave?

@pento pento force-pushed the try/use-core-local-env branch from 0027074 to bc69eda Aug 29, 2019
@pento pento merged commit 713807d into master Aug 30, 2019
2 checks passed
2 checks passed
pull-request-automation
Details
Travis CI - Pull Request Build Passed
Details
@pento pento deleted the try/use-core-local-env branch Aug 30, 2019
@shivapoudel

This comment has been minimized.

Copy link

commented Aug 30, 2019

@pento Since this is merged into master. Is there any way to install using npm install command. Something which I couldn't find right at the moment :)

@pento

This comment has been minimized.

Copy link
Member Author

commented Aug 30, 2019

@shivapoudel: Not yet: it'll be available when the next version of the @wordpress/scripts package is released. @gziolo might know when that's likely to happen, I'm not sure what the current plan is for increasing the frequency of package releases. 馃檪

@gziolo

This comment has been minimized.

Copy link
Member

commented Aug 30, 2019

Not yet: it'll be available when the next version of the @wordpress/scripts package is released. @gziolo might know when that's likely to happen, I'm not sure what the current plan is for increasing the frequency of package releases. 馃檪

Usually, we publish to npm every time we publish the next version of Gutenberg plugin. This change should be available in 3 weeks unless there is a demand to cherry-pick this PR and do pre-release of the next version.

@gziolo gziolo added this to the Gutenberg 6.5 milestone Aug 30, 2019
@shivapoudel

This comment has been minimized.

Copy link

commented Aug 30, 2019

@gziolo Good enough to hear that with the release of Gutenberg 6.5, this package will be updated 馃憤

@aldavigdis

This comment has been minimized.

Copy link
Contributor

commented Aug 31, 2019

For anyone, especially Linux users who are arriving here after seeing ERROR: Version in "./docker-compose.yml" is unsupported. and have managed to run the original Gutenberg Docker setup with no issue until now and have the most recent version of the Docker engine installed via the official repository, you may need to update your version of docker-compose separately as well as the official Docker repo does not include it and the one for your Linux distro is very likely outdated.

Go to https://github.com/docker/compose/releases and find the most recent non-RC release and follow the instructions.

I'm sorry for those who are package manager purists and feel like their setup is "dirty" when installing software like this, but it seems like the best way to go.

I'll follow-up on this by updating the docs as soon as I can.

@pento

This comment has been minimized.

Copy link
Member Author

commented Sep 2, 2019

Thank you for noting that @aldavigdis!

For reference, docker-compose version 1.22 is the minimum required: Fedora 29 provides this, but every other distro I looked at had older versions.

Unfortunately, it doesn't look like Docker are planning on providing DEB or RPM packages any time soon (docker/compose#2235)

@ellatrix

This comment has been minimized.

Copy link
Member

commented Sep 2, 2019

I can no longer run e2e tests locally after this merge. npm run env install -- --fast is run when setting up e2e tests and gives the following error:

Warning: The 'gutenberg' plugin could not be found.
Error: No plugins activated.

The plugin is also gone when I load http://localhost:8889/wp-admin/plugins.php.

@shivapoudel

This comment has been minimized.

Copy link

commented Sep 5, 2019

@pento Out of the context but don't get me wrong. Previously I use PHPMyAdmin image in docker-compose to access the MySQL database. Since I am new to the dockerized system if you do have way out how to access that container database using MySQL workbench or any other way which you prefer in the Windows environment during the development process will be a great help :)

@pento

This comment has been minimized.

Copy link
Member Author

commented Sep 6, 2019

@shivapoudel: You can run a PHPMyAdmin Docker container like so:

docker run -d --link wordpress_mysql_1:db --net wordpress_wpdevnet -p 8081:80 phpmyadmin/phpmyadmin

You may need to change wordpress_mysql_1 to match the name of the MySQL Docker container, check docker ps for a list of running containers.

To login to PHPMyAdmin, visit http://localhost:8081, use the username root, with password password.

@shivapoudel

This comment has been minimized.

Copy link

commented Sep 6, 2019

@pento That works like a charm. Thanks a lot!

@shivapoudel

This comment has been minimized.

Copy link

commented Sep 16, 2019

@pento Since the env LOCAL_DIR is src, I get an error during new post/page creation. I have to change that to build and execute npm run build inside wordpress directory to resolve that issue. What is the easiest way so that I don't have to change that LOCAL_DIR to successfully add new post/page?

dd32 pushed a commit to dd32/gutenberg that referenced this pull request Sep 27, 2019
@aduth

This comment has been minimized.

Copy link
Member

commented Oct 4, 2019

There's many mixed references in code between usage as npm run env:foo and npm run env foo. Using the colon variant always results in an NPM error for me:

npm ERR! missing script: env:start

Should all of these be changed to be space-separated?

npm run env:start

npm run env:install

execSync( `npm run env:cli -- --path=/var/www/${ localDir } ` + args.join( ' ' ), { cwd: env.WP_DEVELOP_DIR, stdio: 'inherit' } );

execSync( 'npm run env:stop', { cwd: env.WP_DEVELOP_DIR, stdio: 'inherit' } );

execSync( 'npm run env:start', { cwd: env.WP_DEVELOP_DIR, stdio: 'inherit' } );

execSync( 'npm run env:start', { cwd: env.WP_DEVELOP_DIR, stdio: 'inherit' } );

execSync( 'npm run env:start', { cwd: env.WP_DEVELOP_DIR, stdio: 'inherit' } );

execSync( 'npm run env:install', { cwd: env.WP_DEVELOP_DIR, stdio: 'inherit' } );

execSync( 'npm run env:stop', { cwd: env.WP_DEVELOP_DIR, stdio: 'inherit' } );

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can鈥檛 perform that action at this time.