[WIP] Improve support for .env files and different build environments (fix #1163 #903) #1178

LinusBorg · 2017-12-16T11:51:19Z

Preface

This PR is really big, with a lot of small and bigger changes all over the place. It was quite challenging to reach some kind of consistency - and it surely has errors, mistakes and quirky code here and there, so I'm looking for extensive constructive feedback.

Something important to understand: I tried to establish two different terms: mode and build environment.

The modes are the different ways your app can run:

npm run dev ( = development mode, NODE_ENV='development')
npm run build ( = production mode, NODE_ENV='production')
npm run test ( = test mode, NODE_ENV='test')

The build environment means the physical environment where you run any of those modes:

locally while you work on the project
a CI environment
a staging environment
a QA environment
the production environment

Running in production mode in a staging environment probably requires different environment variable values than in the production environment, etc. These changes attempt to provide a way to do that in a way that is maintainable and safe.

Keep that in mind when checking out these changes and especially when reading the re-written docs, and please tell me if it succeeds in getting that across.

What this PR does

The important conceptual changes:

Consolidated the NODE_ENV for the test environment on test - we had instances of both test and testing.
added dotenv package and call it where necessary to load environment variables from a .env file:
- /config/webpack.dev.conf.js
- /config/webpack.prod.conf.js
- /test/unit/karma.conf.js
- /test/unit/setup.js
Restructured the setup of *.env.js files:
- moved the /config/*.env.jsto a subdirectory called /variables and renamed them.
- created another file /config/variables/externals.js to map general environment variables
Totally re-wrote the docs section to explain the new pattern/logic. That text can also be found further down in this text below the <hr>.
moved the logic about which file from /variables to load to /build/utils, which allowed me to move the DefinePlugin from the dev and prod configs to the base config.
Quite a lot of small changes to fine-tune this and make the code more consistent, hopefully.

Reviewing

I'll mention people possibly interested i this with the reuest of review in term of both code quality and usefullness for their use cases:

Please also review the following Documentation, both stylistically and content-wise. I'm no expert on the whole 12Factors philposphy so maybe what I wrote and argued is a whole lot of crap. If that's the case, please say so ;)

Documentation for this:

Passing Environment Variables to your application

You application can run in different modes, as documented in the chapter about Commands - development, test, production. Each of these modes might require different default values for things like:

log levels
API URLs
API access keys
...?

You can define these values in the files with the name matching the mode.

// config/variables/development.js
module.exports = {
  API_URL: 'dev.localhost:3000'
}
// config/variables/prod.env.js
module.exports = {
  API_URL: 'production.yourapp.com/api/v1'
}

If you are pulling your hair at this point because we are saving config in the code, please read on. We talk about those concerns in depth a bit further down.

You can also map to environment variables easily.

// config/variables/development.js
module.exports = {
  API_URL: process.env.API_URL
}

You will also find a file called externals.js in this folder. Use this file if to map any environment variables that you want to pass to your Vue app independent of the mode.

How are these files processed?

When you run one of the available commands, process.env.NODE_ENV will be set to one of the three modes mentioned above, and the appropriate module from /config/variables will be required. The object exported by that module will be merged with the one from externals.js. In case of duplicate entries, the former overwrites the latter.

The resulting object will be passed to webpack's DefinePlugin to make the values available in your Vue application.

How do I use these values in my application?

You can access each of these values as properties of process.env inside your app.

Example:

axios.get(process.env.API_URL + '/users').then(/*...*/)

`.env` file support

Whenever you run one of the available commands (except lint), We use dontenv to load any environment variables that you have provided with an .env file in the root directory of your project.

That means that you can pass values from your .env file to your Vue app through the variable files we introduced above.

Example:

# /.env
API_KEY=hd833nfn029373bek$02

// config/variables/externals.js
module.exports = {
  API_KEY: process.env.API_KEY
}

// inside your Vue app:
axios.get(process.env.API_URL + '/users?key=' + process.env.API_KEY).then(/*...*/)

Strategies for different build environments.

Usually you have more build environments than the three app modes (development, test, production): You might have a CI environment where you run tests, a staging environment where you run a built version of your app for your team to test, but with a staging API backend, a QA environment that holds a "copy" of your prod for debugging purposes and so on. In some of those environments you only run npm run test, in others only npm run build and in some you run both, or even npm run dev. The bottom line is, each app's setup is different, each team's requirements are different.

How can you configure those different environments and still keep configuration flexible and secure?

The answer for this template is environment variables configured in an .env file, namespaced by "mode".

Here's how this could look for a "staging" environment:

// /.env
API_KEY_TEST=hd833nfn029373bek$02
API_KEY_PROD=hd833nfn029373bek$02

// /config/variables/test.js
module.exports = {
  API_KEY: process.env.API_KEY_TEST
}
// /config/variables/production.js
module.exports = {
  API_KEY: process.env.API_KEY_PROD
}

Now when you run npm run test, inside of your Vue app, process.env.API_KEY will match the value of API_KEY_TEST in your .env file, when you run npm run build, it will match API_KEY_PROD.

In this example both have the same value, because you want to run the build command with the same API_KEY as the test command, but they can be different in other environments, like when you develop locally:

// /.env
API_URL_DEV=localhost:8080 // dev-server (proxying to a test-backend)
API_URL_TEST=localhost:4001 // test-backend

// /config/variables/development.js
module.exports = {
  API_KEY: process.env.API_URL_DEV
}
// /config/variables/test.js
module.exports = {
  API_KEY: process.env.API_URL_TEST
}

In your app, you can access process.env.API_KEY and will get the correct value in each mode.

A word on 'seperation of config from code' according to "The 12 Factor App"

The 12 Factor App has a section about configuration values that recommends to extract all such values into environment variables, and never save them inside code - like we allow you do in the files mentioned above.

The main concerns are:

storing sensitive information like API keys in code that's pushed to git can be insecure. Secrets may become public (i.e. over github), or they may leak between environments (i.e. a test environment accidentally using production credentials)
Using config files for each build environment is inflexible. You will inevitably end up with a requirement for another environment that needs different config values than test or production, or staging and so on.

We think this is mostly good advice, but in the context of this template, some things are to be considered to get some perspective:

Sensitive Information

Every value that you pass into your app and use it can be accessed by your users one way or another. For that reason, frontend apps (the kind this template is meant for) should generally do not contain sensitive information like API secrets or database keys. If however you do need those values in your build setup, for example, it is absolutely good practice to pass those in via environment variables - just don't don't pass them to your Vue app through the /variables files we discussed above.

Config files per environment are inflexible

We agree to that as well. But development, test and production are not so much build environments as they are different modes that the application can run in.

Since there are many situations where you want to run more than one of those modes in the same environment, but with different values for a given variable (i.e. running dev and test locally, as shown in an example further up), these different values have to be passed in with different environment variables. But these different variables have to be re-mapped to the same name if they ought to be accessible within the application:

While you might have these environment variables:

API_URL_DEV=dev.localhost:3000
API_URL_TEST=dev.localhost:3000

Your app only knows about process.env.API_URL, so we have to map the value of these environment variables to API_URL depending on the mode we are running in. That's what the files in /config/variables are for.

…tion.

LinusBorg · 2017-12-16T11:57:31Z

docs/commands.md

-> Starts a Node.js local development server. See [API Proxying During Development](proxy.md) for more details.
+Starts a Node.js local development server. See [API Proxying During Development](proxy.md) for more details.
+
+> Sets process.env.NODE_ENV to `'development'`


I added these notes to that people get the connection between the commands and the NODE_ENV the set. These define the "mode" the app is running in.

LinusBorg · 2017-12-16T12:00:31Z

template/build/webpack.prod.conf.js

    // see https://github.com/ampedandwired/html-webpack-plugin
    new HtmlWebpackPlugin({
-      filename: {{#if_or unit e2e}}process.env.NODE_ENV === 'testing'
+      filename: {{#if_or unit e2e}}process.env.VUE_TEST = 'e2e'


See /test/e2e/runner.js:

Here we set NODE_ENV to test, but for our e2e tests we run the webpack dev server with webpack.prod.conf.js, because the want to run e2e tests against a devserver that serves files like in production.

Therefore, we have to use a second env variable as a workaround to make the production config behave like NODE_ENV === 'production' for e2e tests.

LinusBorg · 2017-12-16T12:01:25Z

template/config/variables/development.js

+ * It should not contain any sensitive information. See README.
+ */
+module.exports = {
+  NODE_ENV: 'development'


Since we now aggregate variables and stingify their values in /build/utils, we don't have to stringify them here anymore - so no more '" "'double quotes!

LinusBorg · 2017-12-16T12:02:25Z

template/package.json

    {{/if_eq}}
    {{#if_eq runner "karma"}}
-    "unit": "cross-env BABEL_ENV=test karma start test/unit/karma.conf.js --single-run",
+    "unit": "karma start test/unit/karma.conf.js --single-run",


we set the NODE_ENV to test in the karma config, so BABEL_ENV will fall back to this.

LinusBorg · 2017-12-16T12:02:44Z

template/package.json

    "friendly-errors-webpack-plugin": "^1.6.1",
    "html-webpack-plugin": "^2.30.1",
-    "webpack-bundle-analyzer": "^2.9.0",
+    "minimist": "^1.2.0",


this can actually be removed, leftover from an experiment.

LinusBorg · 2017-12-16T12:04:38Z

template/build/vue-loader.conf.js

 const utils = require('./utils')
 const config = require('../config')
-const isProduction = process.env.NODE_ENV === 'production'
+const isProduction = process.env.NODE_ENV === 'production' || process.env.VUE_E2E === 'test'


See /test/e2e/runner.js:

Here we set NODE_ENV to test, but for our e2e tests we run the webpack dev server with webpack.prod.conf.js, because the want to run e2e tests against a devserver that serves files like in production.

Therefore, we have to use a second env variable as a workaround to make the production config behave like NODE_ENV === 'production' for e2e tests.

LinusBorg · 2017-12-16T12:17:16Z

Sidenote: The tests are failing because of memory issues on CircleCI. Maybe needs a workaround, but not relevant to this PR.

codeofsumit · 2017-12-16T13:03:38Z

Holy fuck. I know my comment doesn’t add anything useful but damn 👏👏. I’ll check this as soon as I can. Thanks for your work on this!!!

LinusBorg · 2017-12-16T13:31:39Z

Looking forward to it :)

codeofsumit · 2017-12-30T14:46:27Z

Wow again, I really appreciate you all are looking into this and advancing this template for our needs 👍 👏
While I probably can't provide the extensive in depth code review you're looking for (sorry about that), here are some thoughts:

The modes are the different ways your app can run:

npm run build ( = production mode, NODE_ENV='production')

I would recommend to rename "production mode" to "build mode" or "shipment mode". I think sentences like these causes confusion:

Running in production mode in a staging environment probably requires different environment variable values than in the production environment, etc.

Other than that I agree with the split in modes and environments.

A bigger topic

Strategies for different build environments.

Maybe I misunderstood but it looks like this setup again limits me to three different configs via .env vars.
API_URL_DEV, API_URL_TEST and API_URL_PROD. And they all go into the same .env file which (in my experience) is not how env vars are used.

If the application is built in any other environment than my local computer, the env vars should come from the server/environment and it's the developers/teams job to make them available. There is no .env file in the repo (in fact, it should be in .gitignore by default).

Which makes me question the usecase of API_URL_DEV. If I deploy to a development-environment or QA-environment, the mode I run the application in doesn't matter. The app gets it's config via process.env and the environment is providing those configs.

Mode and environment should not be coupled IMO.

If a developer choses to build the application always locally and then deploys it to the different stages, THEN the configs need to be available locally - but again, the mode should not matter.
Instead, a user should create multiple .env files which is a common practice afaik (we do it too).

For example:

.env for local development
.env.prod for a production environment
.env.int for an integration environment
.env.qa for an QA environment

Now let's say I want to deploy the app to QA and go through the two options: build locally vs build on the server.

Build locally

I run npm run build/dev --.env.qa which should pass the path of the specific .env file to the dotenv package. No matter which mode, it choses the correct .env file.

Build Remotely

I deploy my app and run npm run build/dev on the server. There are no .env files available. The env vars of .env.qa are in the node process and available via process.env - no matter which mode, the config is correct.

This solution eliminates any assumption about how many environments there are and there is no need to namespace anything.

Hope this helps in any way - let me know if I missed something and misunderstood the solution.
Thanks again for your work and get well into the new year!

yyx990803 · 2017-12-30T19:22:06Z

@codeofsumit FYI that's exactly how the env loading in vue-cli-next is handled.

You can have as many modes as you want, and each mode specifies its own env variables. So you do something like vue-cli-service build --mode=internal which loads env vars from .env.internal and .env.internal.local. And all env vars that start with VUE_APP_ will be available in your client side code via webpack.DefinePlugin.

LinusBorg · 2018-01-04T21:24:51Z

@codeofsumit thanks for the feedback, I need to find the time for a longer reply, hopefully before the weekend.

theoephraim · 2018-01-08T19:15:06Z

docs/env.md

 }
+```
+
+You will also find a file called `externals.js` in this folder. Use this file if to map any environment variables that you want to pass to your Vue app independent of the mode.


What is the reasoning behind the name "externals"?

Since this file is loaded regardless of "mode" it seems like the place to store the default values that will get overwritten with mode/env specific values - hence maybe "defaults" would make more sense than "externals"?

theoephraim · 2018-01-08T19:31:29Z

Just running into this in my app now.
I had been going down the route of using https://webpack.js.org/plugins/environment-plugin/ to set up env vars normally (ie using dotenv or whatever setup the user would want based on which mode they are in) and then just choosing a subset of keys to expose to the front-end. Your approach seems to be more flexible, although a little more verbose (manually mapping "build" keys to "exposed" ones). Perhaps another config option to just choose which keys to expose could make sense?

On a related note - I'm not sure why yet, but the define-plugin does work within vue components while the environment-plugin does not.

theoephraim · 2018-01-08T20:13:34Z

template/build/webpack.base.conf.js

+  },
+  plugins: [
+    new webpack.DefinePlugin({
+      'process.env': utils.stringifiedEnvVars


a little nitpicky, but this fn name makes me think that ALL env vars will be stringified and exposed.

theoephraim · 2018-01-08T20:26:02Z

docs/env.md

+You can define these values in the files with the name matching the mode.
+
+```javascript
+// config/variables/development.js


The naming of the variables dir here is also a little confusing. Aren't all your env vars and webpack config settings "variables"? Maybe config/exposed or config/public would be more clear that these are the variables that will be accessible to the build/browser.

Similarly, when I started using this template, I found the fact that config/index.js was mostly build-related configuration a bit confusing. Maybe rename to config/build.js or config/webpack? The more explicit import would likely make it more clear. It could also be worth a quick review of what config lives in there vs what lives in the build directory in the various webpack config files.

Perhaps config/index would be the file that loads various files and exposes the formats needed in different places throughout.

theoephraim · 2018-01-08T20:27:24Z

docs/env.md

+## Passing Environment Variables to your application

-Sometimes it is practical to have different config values according to the environment that the application is running in.
+You application can run in different modes, as documented in the chapter about [Commands](commands.md) - `development`, `test`, `production`. Each of these modes might require different default values for things like:


typo "Your"

theoephraim · 2018-01-08T20:32:01Z

docs/env.md

+
+We agree to that as well. But `development`, `test` and `production` are not so much build environments as they are different *modes* that the application can run in. 
+
+Since there are many situations where you want to run more than one of those modes in the same environment, but with different values for a given variable (i.e. running `dev` and `test` locally, as further up), these different values have to be passed in with different environment variables. But these different variables have to be re-mapped to the same name if they ought to be accessible within the application:


This flexibility is nice, but don't many people use multiple .env files (for different modes) to solve this problem?
Maybe worth adding a note about using separate .env files as the "switch" rather than remapping in the config/variables files.

BoyardIsBack · 2018-02-06T14:26:38Z

template/config/variables/production.js

+'use strict'
+/**
+ * This file should define variables that webpack should make available to 
+ * your application when running in development mode (NODE_ENV === 'production')


Typo : "when running in production mode"

BoyardIsBack · 2018-02-06T14:27:03Z

template/config/variables/test.js

+'use strict'
+/**
+ * This file should define variables that webpack should make available to 
+ * your application when running in development mode (NODE_ENV === 'test')


Typo : "when running in test mode"

LinusBorg · 2018-06-25T15:06:24Z

I won't finish this.

theoephraim · 2018-06-25T15:14:04Z

Will something else be taking its place instead?
If not I think it is important someone work on the config setup.
I'd be happy to take a crack at it.

LinusBorg · 2018-06-25T15:28:18Z

I won't put much energy into feature development for this template, since vue-cli 3 is already in RC now, and it doesn't make sense to start new projects with this template in my personal opinion. I would assume that this template won't be useful anymore compared to vue-cli 3 within weeks after vue-cli 3 has officially become stable.

And since there is no clean update path for this template from an old version to a new one, new features have very limited use for existing projects as well - you can switch those over to vue-cli 3 at least as easy, probably.

I'd be happy to merge changes to this template proviced they are tested solidly (which is not easy!) though, so go ahead if you want to improve this.

theoephraim · 2018-06-25T15:30:08Z

Ah - didn't realize vue-cli 3 will be replacing.
Thanks for the info!

LinusBorg added 19 commits December 14, 2017 13:23

Batman!

dc9828d

Batman!

00440d4

Some progress: a lot of fine tuning, and a first attempt at documenta…

1ae7180

…tion.

Further adjustments. re-wrote the README, changed bunch of file names.

a1b3279

bufixes

d3144d9

load .env before setting NODE_ENV

6ecc298

dito

782d226

remove unnecessary init code, since webpack config does that.

1ca1126

properly load env variables in jest setup

979b8da

adjust docs

0f615bf

bugfix

728a2ba

cater to stupid airbnb config

c458a7b

fix typo in jest setup

902a32e

fix paths in jest setup

823f6a5

add default .env file

4f9a324

slight docs ajustments

1ae8d51

remove the README, moved to docs

e7f285d

small docs fixes

431f208

further docs fixes

baa68dd

LinusBorg added the PR:Review required label Dec 16, 2017

LinusBorg commented Dec 16, 2017

View reviewed changes

make vue-loader behave as if in production during E2E tests

4f0f8eb

LinusBorg commented Dec 16, 2017

View reviewed changes

LinusBorg closed this Dec 16, 2017

LinusBorg reopened this Dec 16, 2017

LinusBorg added this to the 1.3.0 milestone Dec 16, 2017

LinusBorg added the help wanted label Dec 16, 2017

This was referenced Dec 17, 2017

Question: is circle ci need extra config (forked project build fail) ? #1182

Closed

env specific builds #903

Open

Import JSON does not work for E2E tests #835

Open

Have simple JS object for configuration #667

Open

LinusBorg mentioned this pull request Dec 27, 2017

using an .env file #1198

Closed

yyx990803 closed this Dec 30, 2017

yyx990803 reopened this Dec 30, 2017

theoephraim reviewed Jan 8, 2018

View reviewed changes

LinusBorg modified the milestones: 1.3.0, 1.4.0 Jan 13, 2018

BoyardIsBack reviewed Feb 6, 2018

View reviewed changes

BoyardIsBack mentioned this pull request Feb 6, 2018

Multiple builds #1163

Closed

LinusBorg closed this Jun 25, 2018


		We agree to that as well. But `development`, `test` and `production` are not so much build environments as they are different modes that the application can run in.

		Since there are many situations where you want to run more than one of those modes in the same environment, but with different values for a given variable (i.e. running `dev` and `test` locally, as further up), these different values have to be passed in with different environment variables. But these different variables have to be re-mapped to the same name if they ought to be accessible within the application:

[WIP] Improve support for .env files and different build environments (fix #1163 #903) #1178

[WIP] Improve support for .env files and different build environments (fix #1163 #903) #1178

Uh oh!

Conversation

LinusBorg commented Dec 16, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Preface

What this PR does

Reviewing

Documentation for this:

Passing Environment Variables to your application

How are these files processed?

How do I use these values in my application?

.env file support

Strategies for different build environments.

A word on 'seperation of config from code' according to "The 12 Factor App"

Sensitive Information

Config files per environment are inflexible

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LinusBorg Dec 16, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LinusBorg Dec 16, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LinusBorg commented Dec 16, 2017

Uh oh!

codeofsumit commented Dec 16, 2017

Uh oh!

LinusBorg commented Dec 16, 2017

Uh oh!

codeofsumit commented Dec 30, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

A bigger topic

Build locally

Build Remotely

Uh oh!

yyx990803 commented Dec 30, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LinusBorg commented Jan 4, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

theoephraim commented Jan 8, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LinusBorg commented Jun 25, 2018

Uh oh!

theoephraim commented Jun 25, 2018

Uh oh!

LinusBorg commented Jun 25, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

theoephraim commented Jun 25, 2018

Uh oh!

Reviewers

Assignees

Labels

LinusBorg commented Dec 16, 2017 •

edited

Loading

`.env` file support

LinusBorg Dec 16, 2017 •

edited

Loading

LinusBorg Dec 16, 2017 •

edited

Loading

codeofsumit commented Dec 30, 2017 •

edited

Loading

yyx990803 commented Dec 30, 2017 •

edited

Loading

LinusBorg commented Jun 25, 2018 •

edited

Loading