Shift from file-based configuration to environment variables #165

MattiSG · 2018-07-22T07:18:08Z

Main objective and features

This changeset shifts the configuration mechanism from file-based to environment variables.

This allows to create instances for new tax and benefit systems (countries) without committing a full config file to the shared repository, leaving it to each instance to define their configuration at runtime.

This also allows to run the Legislation Explorer on application-level cloud providers (PaaS) such as Heroku, which do not allow modification of the filesystem between checkout and execution.

In order to facilitate usage, support is also provided for a .env file that can list all environment variables, for these cases where the operator has control over the filesystem.

This changeset also:

Properly and exhaustively documents all options.
Simplifies configuration, removing unused options.
Updates all references to Piwik to Matomo following their name change.
Fixes cases where a configuration option would not be applied in a uniform way, or where default values would be inconsistent in different parts of the code.
Fixes tracking of first visited page (see https://github.com/joernroeder/piwik-react-router/releases).
Removes a dependency with no functional impact.

Clients impact

All automatic deployments will fail when this PR is merged. Since the configuration mechanism now relies on variables being injected, you will need to update your setup to define these variables.

For convenience, here are the previously defined configuration files as .env files, as well as suggested migration paths:

France

Define a .env file at the root of the current clone with the following data:

API_URL=https://fr.openfisca.org/api/v21
PATHNAME=/legislation
CHANGELOG_URL=https://github.com/openfisca/openfisca-france/blob/master/CHANGELOG.md
MATOMO_URL=https://stats.data.gouv.fr
MATOMO_SITE_ID=4
UI_STRINGS={"en":{"countryName":"France","search_placeholder":"smic, salaire net"},"fr":{"countryName":"France","search_placeholder":"smic, salaire net"}}

I recommend to start considering alternatives to deploy_prod.sh now, as it is very unlikely this France-specific file will survive in the long term.

Tunisia

Define a .env file at the root of the current clone with the following data:

API_URL=https://api.openfisca.tn
CHANGELOG_URL=https://github.com/openfisca/openfisca-tunisia/blob/master/CHANGELOG.md
UI_STRINGS={"en":{"countryName":"Tunisia"},"fr":{"countryName":"Tunisie"}}

I recommend to consider options for automatic deployment, as the app currently fails if the API is updated without the app being rebuilt.

Aotearoa

Define the following variables in Heroku:
Connected to Support loading arbitrary configuration files #147

API_URL=https://openfisca-aotearoa.herokuapp.com
CHANGELOG_URL=https://github.com/ServiceInnovationLab/openfisca-aotearoa/blob/master/CHANGELOG.md
UI_STRINGS={"en":{"countryName":"Aotearoa New Zealand","search_placeholder":"rates rebate, threshold"},"fr":{"countryName":"Nouvelle-Zélande","search_placeholder":"rates rebate, threshold"}}}

Activate automatic rebuilds on pushes to master.
Redefine the nz.openfisca.org DNS entry to point to the Heroku instance.

Br3nda · 2018-07-22T09:51:22Z

webpack.config.prod.babel.js

-        NODE_ENV: JSON.stringify('production'), // clean up some react stuff
-        BASENAME: JSON.stringify(basename),
-        COUNTRY_PRODUCTION_CONFIG: JSON.stringify(process.env.COUNTRY_PRODUCTION_CONFIG || 'france')
+        NODE_ENV: JSON.stringify(process.env.NODE_ENV),


✨ this is much better

Br3nda

Looks good to me.
Looks like some missing env for circle-ci.

fpagnoux

Thanks for the improvement and the great doc!

I'd like to understand more the rational behind the removal of the logger. I don't consider it blocking though.

fpagnoux · 2018-07-24T23:05:57Z

src/server/index.js

@@ -23,7 +21,7 @@ function startServer(state) {

  // Generic server errors (e.g. not caught by components)
  server.use((err, req, res, next) => {
-    winston.error(req.method + ' ' + req.url, {error: err})
+    console.error(req.method + ' ' + req.url, {error: err})


While dropping the logger altogether?

Seems like it add a purpose: #93

Do you think it's just to complex for what it brings?

It is only used in this single occurrence.
#93 was about moving away from email-based error notifications that were never set up properly and that no one ever read. I was asked by the implementer if I had suggestions on how to log and recommended winston.
Adding a logger-dedicated module only makes sense if we actually use it. Currently, there is no monitoring in place, and the only usage of the error-logging mechanism was this line of code. It will be trivial to revert when / if someone puts the effort to set up logging properly. On the other hand, having it right now means adding a (useless) dependency and a lot of config complexity in order to pass a config object.

fpagnoux · 2018-07-24T23:12:30Z

src/server/webpack-dev-server.js



-const WEBPACK_HOST = process.env.WEBPACK_HOST || 'localhost'
-const WEBPACK_PORT = parseInt(process.env.WEBPACK_PORT)
+const WEBPACK_HOST = process.env.WEBPACK_HOST || '0.0.0.0'


In this context, what is the difference between localhost and 0.0.0.0?

In any context: localhost is a hostname that is by default set (in a hosts file) to resolve to the loopback address 127.0.0.1.
0.0.0.0, on the other hand, is a non-routable meta-address, which means it won't resolve on its own (i.e. you can't access 0.0.0.0 on a client), but is a standard way to express, in a server, “all IPv4 addresses on the local machine”.

Concretely, if server S has 192.168.0.1 as its local address on a local network and client C has 192.168.0.2:

S can access S resources through 192.168.0.1 and 127.0.0.1.

C can access S resources through 192.168.0.1.

However, for the server software to listen to 192.168.0.1, S has to explicitly state that it handles incoming packets from 192.168.0.1. In many cases, you'll have a frontal server on S, so you don't need to set this up explicitly: if Nginx listens to 192.168.0.1 and redirects to some local port P, then the S server only needs to bind to 127.0.0.1:P.
Here, we don't have any frontal server. So the only way S can listen both to 192.168.0.1 and 127.0.0.1 (and other addresses from other interfaces) is to use the meta-address 0.0.0.0.

In this specific context, this means we're allowing the webpack dev server to be available from all clients, not only the ones on the same computer as the server, so I can do remote debugging from another device on the LAN for example.

Thanks a lot for the thorough explanation!

fpagnoux · 2018-07-25T00:08:47Z

README.md

@@ -20,48 +20,56 @@ npm install --production

 ## Configure your instance

-You will need to tell the Legislation Explorer server where your OpenFisca API instance can be reached, as well as where your repository resides for contributors to be directed to. This is done with a configuration file.
+You will need to tell the Legislation Explorer server where your OpenFisca API instance can be reached, as well as where your repository resides for contributors to be directed to. This is all done through environment variables, allowing you to use the same code for any instance by [injecting these elements at runtime](https://12factor.net/config).


https://12factor.net/build-release-run

That's a really interesting resource, thanks for sharing!

Anna-Livia

Thank you @MattiSG for this great PR ! I think it will make life easier for many people.

There are a few changes that I would require to accept the PR.
I tried to use the PATHNAME in the .env, and it did not work (the links in the legexp did not show the basename)

I tried to change the WEBPACK_HOST on the .env and it kept serving the asset on the default WEBPACK_HOST.

Anna-Livia · 2018-07-31T10:00:32Z

README.md

+
+This web app supports [Matomo](https://matomo.org) (ex-Piwik) analytics out of the box. To set it up, use the following environment variables:
+
+- `MATOMO_URL`: he base URL at which the Matomo instance can be contacted. For example: `MATOMO_URL=https://stats.data.gouv.fr MATOMO_SITE_ID=4 npm start`. Only taken into account if used in conjunction with `MATOMO_SITE_ID` and if `MATOMO_CONFIG` is not defined.


he base URL --> the base URL

Anna-Livia · 2018-07-31T10:02:18Z

README.md

+
+### Server and public URL
+
+- `PATHNAME` defines the path at which the Legislation Explorer is served. Defaults to `/`. For example: `PATHNAME=/legislation` to serve on `https://fr.openfisca.org/legislation`.


Why PATHNAME ? Most libraries seem to use BASENAME to desctibe the path the app is hosted on (e.g. piwik-react-router, react-router).

As stated in the commit message for 408c898:

Use Node API name for URL pathname

The Node API uses “pathname”. In the URI spec, this component is called the “path component”.

This authoritative answer is reinforced by the fact that “basename” is ambiguous with the result of a basename operation on a path (which drops its last part).

Anna-Livia · 2018-07-31T13:14:57Z

src/server/lang.js

@@ -1,8 +1,9 @@
 import config from '../config'

+import acceptLanguage from 'accept-language'


Why the change ?

I tend to document the rationale for changes in commit messages 🙂

For b97220b:

Unify import statements order
Group by distance: first group is for app imports, second group for dependencies, third group for standard library

Just for information, Python recommends the opposite order, and that's also the default choice in the ESlint plugin that deals with import order.

Yes, and that's also the one I usually use. It makes much more sense as it is safer for overrides, and reflects the priority of non-namespaced languages (i.e. CSS). I must have been quite confused when I changed the order to this 😕😞
I still feel that splitting in groups is an improvement, and this PR is lingering for too long.

Thanks a lot for the heads up though!!

Anna-Livia · 2018-07-31T13:44:00Z

src/config.js

+config.apiBaseUrl = process.env.API_URL || 'http://0.0.0.0:5000'
+config.changelogUrl = process.env.CHANGELOG_URL
+
+config.pathname = process.env.PATHNAME || '/'


The default here is '/' and is '' in render.jsx. is that voluntary ? If so, what is the benefit ?

This is not on purpose, but not changing this value was the easiest. I believe the usage of that value in render.jsx should be dropped altogether, but that would have made the PR even bigger.

Anna-Livia · 2018-07-31T14:36:16Z

package.json

-    "dev:app-server": "NODE_ENV=development PORT=2030 node index.js",
-    "dev:webpack-dev-server": "WEBPACK_PORT=2031 babel-node src/server/webpack-dev-server.js",
+    "dev:app-server": "NODE_ENV=development node --require dotenv/config index.js",
+    "dev:webpack-dev-server": "babel-node src/server/webpack-dev-server.js",


dotenv/config is not imported here.
However, process.env is called by /src/server/webpack-dev-server.js and /webpack.config.dev.js.

Anna-Livia · 2018-07-31T14:36:53Z

webpack.config.dev.js

@@ -11,8 +11,8 @@ import writeAssets from './src/server/write-assets'

 const assetsPath = path.resolve(__dirname, 'public')

-const WEBPACK_HOST = process.env.HOST || 'localhost'
-const WEBPACK_PORT = parseInt(process.env.PORT) + 1 || 2031
+const WEBPACK_HOST = process.env.WEBPACK_HOST || '0.0.0.0'


In my tests, the envar WEBPACK_HOST in .env was not taken into account.

It worked with the following code :

const my_env = require('dotenv').config().parsed const WEBPACK_HOST = my_env.WEBPACK_HOST || '0.0.0.0' const WEBPACK_PORT = parseInt(my_env.WEBPACK_PORT) || 2031

Thanks for this test!
I have not been able to understand the use case for changing WEBPACK_HOST and have thus probably been too lax on testing.

Can I ask how you tried to set this value? I feel it would probably be a consequence of #165 (comment).

I ended up removing the ability to set up WEBPACK_HOST and WEBPACK_PORT independently from HOST and PORT, it adds complexity and I can't see a proper use case for them. Please let me know if you had a use for it 😉

Anna-Livia · 2018-07-31T14:37:47Z

src/server/webpack-dev-server.js



-const WEBPACK_HOST = process.env.WEBPACK_HOST || 'localhost'
-const WEBPACK_PORT = parseInt(process.env.WEBPACK_PORT)
+const WEBPACK_HOST = process.env.WEBPACK_HOST || '0.0.0.0'


In my tests, the envar WEBPACK_HOST in .env was not taken into account.

It worked with the following code :

const my_env = require('dotenv').config().parsed const WEBPACK_HOST = my_env.WEBPACK_HOST || '0.0.0.0' const WEBPACK_PORT = parseInt(my_env.WEBPACK_PORT) || 2031

Anna-Livia · 2018-07-31T14:54:48Z

README.md

+
+> If it gets lengthy or you want to persist these values, you can also use a `.env` file.
+> Such a file is a plaintext file with name `.env` stored in the root directory of your legislation explorer instance (i.e. next to the `package.json` file). List all of your environment variables in the `$NAME=$VALUE` syntax, one per line. If you have trouble writing this file, read the [parsing rules](https://github.com/motdotla/dotenv#rules).
+> An example file is provided as `.env.example`, setting default values. You can copy it, but changing it won't have any impact: only a file named `.env` will be taken into account. You should not commit this file: it depends on each instance.


Why not have a commited .env that has all the default environment values ? That way there is one place where they are all explicited.
such as :

API_URL=https://fr.openfisca.org/api/v19 CHANGELOG_URL=https://github.com/openfisca/country-template/blob/master/CHANGELOG.md UI_STRINGS={"en":{"countryName":"the development environment"},"fr":{"countryName":"l’environnement de développement"}} HOST='0.0.0.0' PORT=2030 PATHNAME='/' WEBPACK_HOST='0.0.0.0' WEBPACK_PORT=2031 MATOMO_URL=https://stats.data.gouv.fr MATOMO_SITE_ID=4 NODE_ENV=development

I hoped this would have been clear enough with the reference to https://12factor.net/config and this:

You should not commit this file: it depends on each instance.

If we use the .env file to store default values, then all clones will have changes that will conflict whenever there is a config (default value, renaming…) change.

How could I improve the documentation here?

I'm sorry, I wasn't very precise in my comment.

I think the documentation is great and very clear. My request would be that the .env.example (and not the .env as I hastily wrote) file contain the available environment variable with their default value.

From previous experience as a user in similar situations, I liked having a .env.example containing all environment variables I could set, (possibly commented for the ones that were a little specific/uncommon).
It made my life easier while configuring. My 2 currency-cents ;)

Anna-Livia · 2018-07-31T15:09:34Z

package.json

@@ -4,13 +4,12 @@
  "description": "OpenFisca - Legislation Explorer",
  "scripts": {
    "build": "webpack --config ./webpack.config.prod.babel.js",


dotenv/config is not imported here.
However, process.env is called by webpack.
Is that intentional ?

No, you're right, it is important that PATHNAME can be taken from the .env file. Good catch, thanks!

MattiSG · 2018-08-01T19:39:26Z

Should close #167.

Anna-Livia

About the WEBPACK_HOST and WEBPACK_PORT envars :
(Github wound't let me comment on an outdated thread)
The only use I can see of the WEBPACK_PORT I can figure is if your port PORT + 1is already used.

Seem not that likely :)

Anna-Livia · 2018-08-02T09:30:37Z

README.md

+
+> If it gets lengthy or you want to persist these values, you can also use a `.env` file.
+> Such a file is a plaintext file with name `.env` stored in the root directory of your legislation explorer instance (i.e. next to the `package.json` file). List all of your environment variables in the `$NAME=$VALUE` syntax, one per line. If you have trouble writing this file, read the [parsing rules](https://github.com/motdotla/dotenv#rules).
+> An example file is provided as `.env.example`, setting default values. You can copy it, but changing it won't have any impact: only a file named `.env` will be taken into account. You should not commit this file: it depends on each instance.


I'm sorry, I wasn't very precise in my comment.

I think the documentation is great and very clear. My request would be that the .env.example (and not the .env as I hastily wrote) file contain the available environment variable with their default value.

Group by distance: first group is for app imports, second group for dependencies, third group for standard library

Remove winstonConfig config element

Remove redundant “config” suffix Piwik renamed to Matomo in 2018

Define default values and load actual ones in a single place

Document the options

Document default API URL value

This property was not used in any other place and was only used for this inference. Decrease vendor lock-in.

Does not variate for each instance, depends on the legislation explorer itself

Does not variate for each instance, was already not used anymore

The current loading mechanism loads the config file in the frontend too, which fails to compile dotenv into a frontend bundle.

Remove unnecessary complexity. When in dev mode, it is expected that user has full port definition freedom and can accomodate for finding two subsequent available ports.

Define config in a single place

Would trigger a failed build on production on merge. Will have to be updated to support the latest config system as documented in the PR.

MattiSG · 2018-08-09T05:25:06Z

Alright, thanks everybody for the thoughtful reviews. We're finally ready to merge! I just rebased, checked the diff, and will merge as soon as CI passes.

@openfisca/france-admin beware! As I did not get any reply in the last 18 days regarding deployment updates, I decided to take the safest direction and deactivated autodeploy (see 5da4e75). I will let you open another PR if you want to use CI for CD again.

@openfisca/aotearoa you should be ready to use with Heroku! Let me know if you need more help than what was explained earlier in #165 (comment) 🙂

MattiSG · 2018-08-09T05:49:29Z

We're stuck because of #169.

MattiSG · 2018-08-09T05:57:35Z

Fastest way out of this for @openfisca/aotearoa who would like to switch to Heroku is to merge this branch into the master of your fork.

Best solution is to open a separate PR to switch integration tests to the country template.

Necessary feature is to support the new slash syntax.

MattiSG requested review from Br3nda, verbman and sandcha July 22, 2018 07:18

MattiSG changed the title ~~Shifts from file-based configuration to environment variables~~ Shift from file-based configuration to environment variables Jul 22, 2018

Br3nda reviewed Jul 22, 2018

View reviewed changes

Br3nda approved these changes Jul 22, 2018

View reviewed changes

bonjourmauko added the flow:qualified-issues label Jul 23, 2018

fpagnoux reviewed Jul 24, 2018

View reviewed changes

fpagnoux reviewed Jul 25, 2018

View reviewed changes

Anna-Livia requested changes Jul 31, 2018

View reviewed changes

MattiSG mentioned this pull request Aug 1, 2018

Update piwik tracker #167

Merged

MattiSG requested a review from Anna-Livia August 1, 2018 21:03

Anna-Livia requested changes Aug 2, 2018

View reviewed changes

MattiSG requested a review from Anna-Livia August 7, 2018 03:42

Anna-Livia approved these changes Aug 7, 2018

View reviewed changes

MattiSG added 14 commits August 8, 2018 23:16

Unify import statements order

b58f1bf

Group by distance: first group is for app imports, second group for dependencies, third group for standard library

Remove almost unused winston dependency

2f4984d

Remove winstonConfig config element

Rename piwikConfig config key to matomo

1d42e6c

Remove redundant “config” suffix Piwik renamed to Matomo in 2018

Remove unused useCommitReferenceFromApi config key

452ff56

Load host and port through env variables only

86b7d39

Define default values and load actual ones in a single place

Unify webpack dev server host and port definition

eae5169

Document the options

Support setting API URL in a single place

09adf4d

Document default API URL value

Replace githubProject config key with changelogUrl

48dd346

This property was not used in any other place and was only used for this inference. Decrease vendor lock-in.

Fix whitespace

223c7b4

Improve env variable usage documentation

955a70b

Remove gitWebpageUrl config key

52e0f7a

Does not variate for each instance, depends on the legislation explorer itself

Remove unused websiteUrl config key

8e00b10

Does not variate for each instance, was already not used anymore

Support Matomo config through env variables

3b23a12

Support defining UI strings through env variable

920dab8

MattiSG added 15 commits August 8, 2018 23:16

Get dev assets base URL from webpack dev config

aa32fc3

Drop support for file-based configuration

0e3848b

Use Node API name for URL pathname

6c99b6a

Properly pass NODE_ENV to frontend

ee974a6

Support .env files

cbca978

Refactor .gitignore

fd44a22

Limit .env files support only through NPM scripts

fea8d83

The current loading mechanism loads the config file in the frontend too, which fails to compile dotenv into a frontend bundle.

Fix typo

a11fc1e

Derive webpack dev server host and port from app's

581b2bc

Remove unnecessary complexity. When in dev mode, it is expected that user has full port definition freedom and can accomodate for finding two subsequent available ports.

Improve error behaviour in webpack dev server boot

dd252bd

Support dotenv variables in webpack config

871ff98

Define config in a single place

Simplify documentation minor

fbf02e4

List all available env vars in .env.example

8891c19

Ensure config is always forwarded to frontend

8bdb23a

Clean up unused import

9d74fec

MattiSG force-pushed the config branch from d20a71d to 8891c19 Compare August 9, 2018 05:18

Remove France-specific autodeploy

5da4e75

Would trigger a failed build on production on merge. Will have to be updated to support the latest config system as documented in the PR.

fpagnoux merged commit 44585d9 into master Aug 9, 2018

fpagnoux deleted the config branch August 9, 2018 14:38

fpagnoux removed the flow:qualified-issues label Aug 9, 2018


		This web app supports [Matomo](https://matomo.org) (ex-Piwik) analytics out of the box. To set it up, use the following environment variables:

		- `MATOMO_URL`: he base URL at which the Matomo instance can be contacted. For example: `MATOMO_URL=https://stats.data.gouv.fr MATOMO_SITE_ID=4 npm start`. Only taken into account if used in conjunction with `MATOMO_SITE_ID` and if `MATOMO_CONFIG` is not defined.


		### Server and public URL

		- `PATHNAME` defines the path at which the Legislation Explorer is served. Defaults to `/`. For example: `PATHNAME=/legislation` to serve on `https://fr.openfisca.org/legislation`.

		@@ -1,8 +1,9 @@
		import config from '../config'

		import acceptLanguage from 'accept-language'

Shift from file-based configuration to environment variables #165

Shift from file-based configuration to environment variables #165

Conversation

MattiSG commented Jul 22, 2018 • edited by fpagnoux

Main objective and features

This changeset also:

Clients impact

France

Tunisia

Aotearoa

Choose a reason for hiding this comment

Br3nda left a comment • edited

Choose a reason for hiding this comment

fpagnoux left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Anna-Livia left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fpagnoux Aug 1, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Anna-Livia Aug 2, 2018 • edited by fpagnoux

Choose a reason for hiding this comment

fpagnoux Aug 2, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MattiSG commented Aug 1, 2018

Anna-Livia left a comment

Choose a reason for hiding this comment

Anna-Livia Aug 2, 2018 • edited by fpagnoux

Choose a reason for hiding this comment

MattiSG commented Aug 9, 2018

MattiSG commented Aug 9, 2018

MattiSG commented Aug 9, 2018

MattiSG commented Jul 22, 2018 •

edited by fpagnoux

Br3nda left a comment •

edited

fpagnoux left a comment •

edited

fpagnoux Aug 1, 2018 •

edited

Anna-Livia Aug 2, 2018 •

edited by fpagnoux

fpagnoux Aug 2, 2018 •

edited

Anna-Livia Aug 2, 2018 •

edited by fpagnoux