Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add .wp-env.json support to wp-env #20002

Merged
merged 4 commits into from Feb 7, 2020
Merged

Add .wp-env.json support to wp-env #20002

merged 4 commits into from Feb 7, 2020

Conversation

@noisysocks
Copy link
Member

noisysocks commented Feb 3, 2020

Adds support to wp-env for .wp-env.json files as described in #17724 (comment).

wp-env

Specification

.wp-env.json supports three fields:

Field Type Default Description
"core" string or null null The WordPress installation to use. If null is specified, wp-env will use the latest production release of WordPress.
"plugins" string[] [] A list of plugins to install and activate in the environment.
"themes" string[] [] A list of themes to install in the environment. The first theme in the list will be activated.

Several types of strings can be passed into these fields:

Type Format Example(s)
Relative path .<path> or ~<path> "./a/directory", "../a/directory", "~/a/directory"
Absolute path /<path> or <letter>:\<path> "/a/directory", "C:\\a\\directory"
GitHub repository <owner>/<repo>[#<ref>] "WordPress/WordPress", "WordPress/gutenberg#master"

Remote sources will be downloaded into a temporary directory located in ~/.wp-env.

Examples

Latest production WordPress + current directory as a plugin

This is useful for plugin development.

{
  "core": null,
  "plugins": [
    "."
  ]
}

Latest development WordPress + current directory as a plugin

This is useful for plugin development when upstream Core changes need to be tested.

{
  "core": "WordPress/WordPress#master",
  "plugins": [
    "."
  ]
}

Local wordpress-develop + current directory as a plugin

This is useful for working on plugins and WordPress Core at the same time.

{
  "core": "../wordpress-develop/build",
  "plugins": [
    "."
  ]
}

A complete testing environment

This is useful for integration testing: that is, testing how old versions of WordPress and different combinations of plugins and themes impact each other.

{
  "core": "WordPress/WordPress#5.2.0",
  "plugins": [
    "WordPress/wp-lazy-loading",
    "WordPress/classic-editor",
  ],
  "themes": [
    "WordPress/theme-experiments"
  ]
}

Testing

  1. Check out this branch
  2. cd to a directory that contains a WordPress installation, a plugin, a theme, or a .wp-env.json file
  3. Run /path/to/gutenberg/packages/env/bin/wp-env start

Questions

  • Is the wp-env reset command useful? It has some problems, namely that it requires wp-env start to have been run at least once. Since we’re bumping the major version here it might be a good chance to replace wp-env-reset with e.g. wp-env destroy which destroys all containers and volumes so that one can run wp-env start from a clean slate. This would be useful for troubleshooting.
  • What is the tests-wordpress-phpunit container used for? I could not find where this is used and so have left it out for now.
Copy link
Contributor

epiqueras left a comment

Amazing work here!

I think you need to import types to the files where they are used for them to work.

Is the wp-env reset command useful? It has some problems, namely that it requires wp-env start to have been run at least once. Since we’re bumping the major version here it might be a good chance to replace wp-env-reset with e.g. wp-env destroy which destroys all containers and volumes so that one can run wp-env start from a clean slate. This would be useful for troubleshooting.

We don't have a wp-env reset but a wp-env destroy sounds very useful. Perhaps we should add it in a follow-up?

What is the tests-wordpress-phpunit container used for? I could not find where this is used and so have left it out for now.

It was used for PHP unit tests but we held off on it due to flakiness.

@@ -46,31 +46,22 @@ module.exports = function cli() {
yargs.usage( wpPrimary( '$0 <command>' ) );

yargs.command(
'start [ref]',

This comment has been minimized.

Copy link
@epiqueras

epiqueras Feb 3, 2020

Contributor

This needs a dev note, right?

This comment has been minimized.

Copy link
@noisysocks

noisysocks Feb 7, 2020

Author Member

I noted that this breaks backwards compatibility in packages/env/CHANGELOG.md but a agree dev note would be good, too. If nothing else it will help publicise that this tool can be used for core development.

This comment has been minimized.

Copy link
@epiqueras

epiqueras Feb 7, 2020

Contributor

👍

packages/env/lib/config.js Outdated Show resolved Hide resolved
packages/env/lib/config.js Outdated Show resolved Hide resolved
packages/env/lib/config.js Show resolved Hide resolved
packages/env/lib/env.js Outdated Show resolved Hide resolved
packages/env/test/config.js Outdated Show resolved Hide resolved
packages/env/lib/env.js Show resolved Hide resolved
packages/env/lib/env.js Outdated Show resolved Hide resolved
@epiqueras epiqueras mentioned this pull request Feb 5, 2020
1 of 1 task complete
@noahtallen

This comment has been minimized.

Copy link
Contributor

noahtallen commented Feb 5, 2020

I get this error running it locally (in a WordPress plugin with and also without a config file)

~/source/wp-calypso/apps/full-site-editing/full-site-editing-plugin(try/wordpress-env-for-tests) » ~/source/gutenberg/packages/env/bin/wp-env start                     noah.allen@Noahs-MacBook-Pro
internal/util.js:214
    throw new errors.TypeError('ERR_INVALID_ARG_TYPE', 'original', 'function');
    ^

TypeError [ERR_INVALID_ARG_TYPE]: The "original" argument must be of type function
    at Object.promisify (internal/util.js:214:11)
    at Object.<anonymous> (/Users/noah.allen/source/gutenberg/packages/env/lib/detect-directory-type.js:15:23)
    at Module._compile (module.js:653:30)
    at Object.Module._extensions..js (module.js:664:10)
    at Module.load (module.js:566:32)
    at tryModuleLoad (module.js:506:12)
    at Function.Module._load (module.js:498:3)
    at Module.require (module.js:597:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/Users/noah.allen/source/gutenberg/packages/env/lib/config.js:13:29)
@epiqueras

This comment has been minimized.

Copy link
Contributor

epiqueras commented Feb 5, 2020

@noahtallen

This comment has been minimized.

Copy link
Contributor

noahtallen commented Feb 5, 2020

Thanks, @epiqueras! Spot on. Turns out I had to rm -rf node_modules and reinstall everything with the correct node version to get it to work (neither nvm use && npm i nor nvm use && npm rebuild fixed the issue.)

@epiqueras

This comment has been minimized.

Copy link
Contributor

epiqueras commented Feb 5, 2020

@noahtallen

This comment has been minimized.

Copy link
Contributor

noahtallen commented Feb 5, 2020

I'm now getting the following output, which leads me to believe that the custom port number is not working quite right. I'm not sure if it was entirely working before though, as I had similar issues with @wordpress/scripts env with custom port numbers. (Speaking of, it'd be great to add a custom port number to .wp-env.json!)

WP_ENV_PORT=3333 ./packages/env/bin/wp-env start                                                                                              

✖ 31911d623e75f345e9ed328b9f48cff6_mysql_1 is up-to-date
Recreating 31911d623e75f345e9ed328b9f48cff6_wordpress_1 ... 
Starting 31911d623e75f345e9ed328b9f48cff6_tests-wordpress_1 ... 
Starting 31911d623e75f345e9ed328b9f48cff6_tests-wordpress_1 ... error

Recreating 31911d623e75f345e9ed328b9f48cff6_wordpress_1     ... done1 (2e501b716d5fdfed3e33bad99337702ef97588c9cabb94003ce8735b5e85d28f): Bind for 0.0.0.0:8889 failed: port is already allocated

ERROR: for tests-wordpress  Cannot start service tests-wordpress: driver failed programming external connectivity on endpoint 31911d623e75f345e9ed328b9f48cff6_tests-wordpress_1 (2e501b716d5fdfed3e33bad99337702ef97588c9cabb94003ce8735b5e85d28f): Bind for 0.0.0.0:8889 failed: port is already allocated
Encountered errors while bringing up the project.



 {
  exitCode: 1,
  err: '31911d623e75f345e9ed328b9f48cff6_mysql_1 is up-to-date\n' +
    'Recreating 31911d623e75f345e9ed328b9f48cff6_wordpress_1 ... \r\n' +
    'Starting 31911d623e75f345e9ed328b9f48cff6_tests-wordpress_1 ... \r\n' +
    'Host is already in use by another container\n' +
    '\u001b[1A\u001b[2K\rStarting 31911d623e75f345e9ed328b9f48cff6_tests-wordpress_1 ... \u001b[31merror\u001b[0m\r\u001b[1B\n' +
    'ERROR: for 31911d623e75f345e9ed328b9f48cff6_tests-wordpress_1  Cannot start service tests-wordpress: driver failed programming external connectivity on endpoint 31911d623e75f345e9ed328b9f48cff6_tests-wordpress_1 (2e501b716d5fdfed3e33bad99337702ef97588c9cabb94003ce8735b5e85d28f): Bind for 0.0.0.0:8889 failed: port is already allocated\n' +
    '\u001b[2A\u001b[2K\rRecreating 31911d623e75f345e9ed328b9f48cff6_wordpress_1     ... \u001b[32mdone\u001b[0m\r\u001b[2B\n' +
    'ERROR: for tests-wordpress  Cannot start service tests-wordpress: driver failed programming external connectivity on endpoint 31911d623e75f345e9ed328b9f48cff6_tests-wordpress_1 (2e501b716d5fdfed3e33bad99337702ef97588c9cabb94003ce8735b5e85d28f): Bind for 0.0.0.0:8889 failed: port is already allocated\n' +
    'Encountered errors while bringing up the project.\n',
  out: ''
} 

To remedy

  • Stop and remove the conflicting docker container
  • Run the command again with a custom port number (i.e. make an environment for the Gutenberg repo)
  • The port number works correctly but localhost:8889 redirects to the custom port as well.

If I then try to add a separate instance with a different port number, it also fails. So it seems that even if you specify a custom port number, something under the hood still allocates the default port (8889) for use by the original container. And then when you try to do a second wp-env instance, it thinks that 8889 has been used by something, so it also fails.

Here's the output after 1. stopping and removing all docker images. 2. creating a wp-env environment with the custom port 3333. 3. Running the command again in a different plugin with a different port number (9999):

✖ 58d3a7453e296b6fce00337462f0c1c5_mysql_1 is up-to-date
Creating 58d3a7453e296b6fce00337462f0c1c5_tests-wordpress_1 ... 
Creating 58d3a7453e296b6fce00337462f0c1c5_tests-wordpress_1 ... error
Host is already in use by another container

ERROR: for 58d3a7453e296b6fce00337462f0c1c5_tests-wordpress_1  Cannot start service tests-wordpress: driver failed programming external connectivity on endpoint 58d3a7453e296b6fce00337462f0c1c5_tesCreating 58d3a7453e296b6fce00337462f0c1c5_wordpress_1       ... done

ERROR: for tests-wordpress  Cannot start service tests-wordpress: driver failed programming external connectivity on endpoint 58d3a7453e296b6fce00337462f0c1c5_tests-wordpress_1 (c3fdce977ec32954b3af0c8d638173bcb1e9c47c3915e8d5c47a2d04957f2ad6): Bind for 0.0.0.0:8889 failed: port is already allocated
Encountered errors while bringing up the project.



 {
  exitCode: 1,
  err: '58d3a7453e296b6fce00337462f0c1c5_mysql_1 is up-to-date\n' +
    'Creating 58d3a7453e296b6fce00337462f0c1c5_tests-wordpress_1 ... \r\n' +
    'Creating 58d3a7453e296b6fce00337462f0c1c5_wordpress_1       ... \r\n' +
    'Host is already in use by another container\n' +
    '\u001b[2A\u001b[2K\rCreating 58d3a7453e296b6fce00337462f0c1c5_tests-wordpress_1 ... \u001b[31merror\u001b[0m\r\u001b[2B\n' +
    'ERROR: for 58d3a7453e296b6fce00337462f0c1c5_tests-wordpress_1  Cannot start service tests-wordpress: driver failed programming external connectivity on endpoint 58d3a7453e296b6fce00337462f0c1c5_tests-wordpress_1 (c3fdce977ec32954b3af0c8d638173bcb1e9c47c3915e8d5c47a2d04957f2ad6): Bind for 0.0.0.0:8889 failed: port is already allocated\n' +
    '\u001b[1A\u001b[2K\rCreating 58d3a7453e296b6fce00337462f0c1c5_wordpress_1       ... \u001b[32mdone\u001b[0m\r\u001b[1B\n' +
    'ERROR: for tests-wordpress  Cannot start service tests-wordpress: driver failed programming external connectivity on endpoint 58d3a7453e296b6fce00337462f0c1c5_tests-wordpress_1 (c3fdce977ec32954b3af0c8d638173bcb1e9c47c3915e8d5c47a2d04957f2ad6): Bind for 0.0.0.0:8889 failed: port is already allocated\n' +
    'Encountered errors while bringing up the project.\n',
  out: ''
} 
@epiqueras

This comment has been minimized.

Copy link
Contributor

epiqueras commented Feb 6, 2020

@noisysocks noisysocks force-pushed the add/wp-env-json branch from 81da692 to 291aecd Feb 7, 2020
@noisysocks

This comment has been minimized.

Copy link
Member Author

noisysocks commented Feb 7, 2020

Thanks for the quick feedback @enriquesanchez @epiqueras! (oops) I've rebased this PR, addressed all of your inline comments, and fixed an issue where both the development and tests environments pointed to the same database.

I think you need to import types to the files where they are used for them to work.

👍 done.

We don't have a wp-env reset but a wp-env destroy sounds very useful. Perhaps we should add it in a follow-up?

Sorry, I meant to say: Is wp-env clean useful? It has some shortfalls, and since I'm bumping the major version here, it's maybe a good opportunity to remove wp-env clean in lieu of something like wp-env destroy.

It was used for PHP unit tests but we held off on it due to flakiness.

👍 I'll leave it removed for now then.

Copy link
Contributor

epiqueras left a comment

Haha, you got the wrong Enrique.

Sorry, I meant to say: Is wp-env clean useful? It has some shortfalls, and since I'm bumping the major version here, it's maybe a good opportunity to remove wp-env clean in lieu of something like wp-env destroy.

I see clean being used to reset a testing environment and destroy being used when things break as a last troubleshooting resort so both can be useful.

I think this is good to merge now. Its a kind of PR that needs a lot of testing in the wild and we can always iterate before the next release. The foundation is strong, thanks for the work!

@noisysocks

This comment has been minimized.

Copy link
Member Author

noisysocks commented Feb 7, 2020

Haha, you got the wrong Enrique.

I think I've done this a dozen times now 😂

packages/env/lib/env.js Outdated Show resolved Hide resolved
Copy link
Contributor

noahtallen left a comment

This is looking really awesome! I hope the issue I noted isn't from yet another node problem :)

I was just thinking about this in the context of @wordpress/scripts. Since we no longer automatically link the directory from which you run the script, we can't really tell what the "main" plugin is for the environment anymore. Perhaps this won't be an issue in most cases, but if I want to run phpunit/phpcs on just my own plugin (perhaps in a CI environment), we'd need to be able to specify that somehow.

packages/env/CHANGELOG.md Show resolved Hide resolved
@epiqueras

This comment has been minimized.

Copy link
Contributor

epiqueras commented Feb 7, 2020

I was just thinking about this in the context of @wordpress/scripts. Since we no longer automatically link the directory from which you run the script, we can't really tell what the "main" plugin is for the environment anymore. Perhaps this won't be an issue in most cases, but if I want to run phpunit/phpcs on just my own plugin (perhaps in a CI environment), we'd need to be able to specify that somehow.

This doesn't change any of that. When you run phpunit/phpcs you specify a path in the host project.

@noisysocks

This comment has been minimized.

Copy link
Member Author

noisysocks commented Feb 7, 2020

Since we no longer automatically link the directory from which you run the script, we can't really tell what the "main" plugin is for the environment anymore.

If you take a look at the generated docker-compose.yml (cat ~/.wp-env/*/docker-compose.yml) you can see that the composer container is still binded to the directory that .wp-env.json was located in. We could borrow this approach for phpunit, phpcs, or any other containers that need a concept of main plugin should they ever come up. (Right now it's just composer as far as I can see.)

@noisysocks

This comment has been minimized.

Copy link
Member Author

noisysocks commented Feb 7, 2020

Thanks @tellthemachines for testing this on Windows! 😍

@noisysocks noisysocks merged commit 82c301a into master Feb 7, 2020
2 checks passed
2 checks passed
pull-request-automation
Details
Travis CI - Pull Request Build Passed
Details
@noisysocks noisysocks deleted the add/wp-env-json branch Feb 7, 2020
@github-actions github-actions bot added this to the Gutenberg 7.5 milestone Feb 7, 2020
chipsnyder added a commit that referenced this pull request Feb 7, 2020
* Add .wp-env.json support to wp-env

* @wordpress/env: Maintain version at 0.4.0. This is bumped when packages are released

* @wordpress/env: Fix copy/paste error in doc comment

* @wordpress/env: Output stderr instead of error object
@simison

This comment has been minimized.

Copy link
Member

simison commented Feb 13, 2020

Did you consider using Docker's (and Docker-compose's)'s native support for "env files"?

I would suggest defining defaults in one file and then letting optional file (or environment constants!) to overrun that.

That might solve issues with Linux and furthermore, be more familiar with anyone already used to Docker setups. Less code at our end, too so easier to maintain. Less our custom-layer specific issues (you'd be still left with Docker & Docker compose specific issues of course).

The file format would be different but that's more or less subjective if it's worse or better. Wider support for ways to input values (via environment variables & CLI arguments) is a great bonus.

Thoughts?

We have an example setup of something like this in Jetpack:

@ockham

This comment has been minimized.

Copy link
Contributor

ockham commented Feb 13, 2020

Thanks @simison for the suggestion! I wasn't aware of (or had forgotten about) Docker env files, but if they (maybe together with a static docker-compose.override.yml -- rather than a dynamically generated one) can serve a similar purpose as .wp-env.json, I'd much prefer them.

Generally, if there's an existing tool and config format, I think it's preferrable to use that, rather than defining a new format.

Relying on an existing format means

  • less maintenance burden for us
  • less figuring out of obscure bugs
  • less design needed to figure out edge cases
  • less domain-specific config formats to learn for developers using our software
@epiqueras

This comment has been minimized.

Copy link
Contributor

epiqueras commented Feb 13, 2020

This goes against the goal of wp-env being minimal in its configuration. Docker-compose files are behemoths with colossal documentation. It's not a burden we want to put on users.

The Linux bug is simply a lack of permissions to write in the user folder.

@noahtallen

This comment has been minimized.

Copy link
Contributor

noahtallen commented Feb 13, 2020

Docker env file

We did briefly discuss adding support for environment variable support here: #20158 (comment). You can see some more reasoning for why we want to expose a minimal amount of configuration. Essentially, the goal is to be able to run wp-env start from any plugin and it just works without having to set anything up.

Additionally, if you want to use wp-env start to start and stop your docker compose files, you could always add the .env and/or docker-compose.override.yml in the location where wp-env puts the configuration (~/.wp-env/hash/docker-compose.yml) and Docker would pick up those files when it runs. It wouldn't be officially supported, but it could be a workaround.

but if they (maybe together with a static docker-compose.override.yml -- rather than a dynamically generated one) can serve a similar purpose as .wp-env.json, I'd much prefer them.

There would be nothing stopping anyone from setting up their WordPress configuration in this way. :) It isn't even that much work; the basic config is pretty lightweight. If you want to use docker-compose for lots of power 💪 , that's absolutely fine. But that's not quite the goal of wp-env. If you don't want to use wp-env, you don't have to, right?

I will say that wp-env gives us some stuff that docker compose does not:

  • Download plugin/theme/wordpress source code from GitHub
  • Can link local dev sources (plugins/themes/wordpress) to the instance without any headache
  • More in the future :)

Linux issues

Back to the original issue (filesystems permissions in Linux), I wonder if we could allow users to specify the download location for wp-env source via an environment variable. If set, then wp-env would use that to download sourcecode and configurations to a different location. It seems like the type of setting that would be valuable "per user"

@epiqueras

This comment has been minimized.

Copy link
Contributor

epiqueras commented Feb 13, 2020

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

6 participants
You can’t perform that action at this time.