[#175119553] Centralize environment variables in single config module

[infantesimone]

This PR introduces the utils/config module which aims to be the unique interface for application configuration.

Every place in which the application used to directly access env variable has been refactored to use this module instead.

[pagopa-github-bot]

Affected stories

• ⚙️ #175119553: Implementare il meccanismo di configurazione "centralizzata" su tutte le functions

New dependencies added: dotenv.

dotenv

Author: Unknown

Description: Loads environment variables from .env file

Homepage: https://github.com/motdotla/dotenv#readme

Created	over 7 years ago
Last Updated	8 months ago
License	BSD-2-Clause
Maintainers	4
Releases	38
Direct Dependencies
Keywords	dotenv, env, .env, environment, variables, config and settings

README

dotenv

Dotenv is a zero-dependency module that loads environment variables from a .env file into process.env. Storing configuration in the environment separate from code is based on The Twelve-Factor App methodology.

Install

# with npm
npm install dotenv

# or with Yarn
yarn add dotenv

Usage

As early as possible in your application, require and configure dotenv.

require('dotenv').config()

Create a .env file in the root directory of your project. Add
environment-specific variables on new lines in the form of NAME=VALUE.
For example:

DB_HOST=localhost
DB_USER=root
DB_PASS=s1mpl3

process.env now has the keys and values you defined in your .env file.

const db = require('db')
db.connect({
  host: process.env.DB_HOST,
  username: process.env.DB_USER,
  password: process.env.DB_PASS
})

Preload

You can use the --require (-r) command line option to preload dotenv. By doing this, you do not need to require and load dotenv in your application code. This is the preferred approach when using import instead of require.

$ node -r dotenv/config your_script.js

The configuration options below are supported as command line arguments in the format dotenv_config_<option>=value

$ node -r dotenv/config your_script.js dotenv_config_path=/custom/path/to/your/env/vars

Additionally, you can use environment variables to set configuration options. Command line arguments will precede these.

$ DOTENV_CONFIG_<OPTION>=value node -r dotenv/config your_script.js

$ DOTENV_CONFIG_ENCODING=latin1 node -r dotenv/config your_script.js dotenv_config_path=/custom/path/to/.env

Config

config will read your .env file, parse the contents, assign it to
process.env,
and return an Object with a parsed key containing the loaded content or an error key if it failed.

const result = dotenv.config()

if (result.error) {
  throw result.error
}

console.log(result.parsed)

You can additionally, pass options to config.

Options

Path

Default: path.resolve(process.cwd(), '.env')

You may specify a custom path if your file containing environment variables is located elsewhere.

require('dotenv').config({ path: '/full/custom/path/to/your/env/vars' })

Encoding

Default: utf8

You may specify the encoding of your file containing environment variables.

require('dotenv').config({ encoding: 'latin1' })

Debug

Default: false

You may turn on logging to help debug why certain keys or values are not being set as you expect.

require('dotenv').config({ debug: process.env.DEBUG })

Parse

The engine which parses the contents of your file containing environment
variables is available to use. It accepts a String or Buffer and will return
an Object with the parsed keys and values.

const dotenv = require('dotenv')
const buf = Buffer.from('BASIC=basic')
const config = dotenv.parse(buf) // will return an object
console.log(typeof config, config) // object { BASIC : 'basic' }

Options

Debug

Default: false

You may turn on logging to help debug why certain keys or values are not being set as you expect.

const dotenv = require('dotenv')
const buf = Buffer.from('hello world')
const opt = { debug: true }
const config = dotenv.parse(buf, opt)
// expect a debug message because the buffer is not in KEY=VAL form

Rules

The parsing engine currently supports the following rules:

• BASIC=basic becomes {BASIC: 'basic'}
• empty lines are skipped
• lines beginning with # are treated as comments
• empty values become empty strings (EMPTY= becomes {EMPTY: ''})
• inner quotes are maintained (think JSON) (JSON={"foo": "bar"} becomes {JSON:"{\"foo\": \"bar\"}")
• whitespace is removed from both ends of unquoted values (see more on trim) (FOO= some value becomes {FOO: 'some value'})
• single and double quoted values are escaped (SINGLE_QUOTE='quoted' becomes {SINGLE_QUOTE: "quoted"})
• single and double quoted values maintain whitespace from both ends (FOO=" some value " becomes {FOO: ' some value '})
• double quoted values expand new lines (MULTILINE="new\nline" becomes

{MULTILINE: 'new
line'}

FAQ

Should I commit my .env file?

No. We strongly recommend against committing your .env file to version
control. It should only include environment-specific values such as database
passwords or API keys. Your production database should have a different
password than your development database.

Should I have multiple .env files?

No. We strongly recommend against having a "main" .env file and an "environment" .env file like .env.test. Your config should vary between deploys, and you should not be sharing values between environments.

In a twelve-factor app, env vars are granular controls, each fully orthogonal to other env vars. They are never grouped together as “environments”, but instead are independently managed for each deploy. This is a model that scales up smoothly as the app naturally expands into more deploys over its lifetime.

– The Twelve-Factor App

What happens to environment variables that were already set?

We will never modify any environment variables that have already been set. In particular, if there is a variable in your .env file which collides with one that already exists in your environment, then that variable will be skipped. This behavior allows you to override all .env configurations with a machine-specific environment, although it is not recommended.

If you want to override process.env you can do something like this:

const fs = require('fs')
const dotenv = require('dotenv')
const envConfig = dotenv.parse(fs.readFileSync('.env.override'))
for (const k in envConfig) {
  process.env[k] = envConfig[k]
}

Can I customize/write plugins for dotenv?

For dotenv@2.x.x: Yes. dotenv.config() now returns an object representing
the parsed .env file. This gives you everything you need to continue
setting values on process.env. For example:

const dotenv = require('dotenv')
const variableExpansion = require('dotenv-expand')
const myEnv = dotenv.config()
variableExpansion(myEnv)

What about variable expansion?

Try dotenv-expand

How do I use dotenv with import?

ES2015 and beyond offers modules that allow you to export any top-level function, class, var, let, or const.

When you run a module containing an import declaration, the modules it imports are loaded first, then each module body is executed in a depth-first traversal of the dependency graph, avoiding cycles by skipping anything already executed.

– ES6 In Depth: Modules

You must run dotenv.config() before referencing any environment variables. Here's an example of problematic code:

errorReporter.js:

import { Client } from 'best-error-reporting-service'

export const client = new Client(process.env.BEST_API_KEY)

index.js:

import dotenv from 'dotenv'
import errorReporter from './errorReporter'

dotenv.config()
errorReporter.client.report(new Error('faq example'))

client will not be configured correctly because it was constructed before dotenv.config() was executed. There are (at least) 3 ways to make this work.

1. Preload dotenv: node --require dotenv/config index.js (Note: you do not need to import dotenv with this approach)
2. Import dotenv/config instead of dotenv (Note: you do not need to call dotenv.config() and must pass options via the command line or environment variables with this approach)
3. Create a separate file that will execute config first as outlined in this comment on #133

Contributing Guide

See CONTRIBUTING.md

Change Log

See CHANGELOG.md

Who's using dotenv?

These npm modules depend on it.

Projects that expand it often use the keyword "dotenv" on npm.

Generated by 🚫 dangerJS

[codecov-commenter]

Codecov Report

Merging #110 into master will decrease coverage by 0.00%.
The diff coverage is 87.50%.

@@            Coverage Diff             @@
##           master     #110      +/-   ##
==========================================
- Coverage   84.43%   84.42%   -0.01%     
==========================================
  Files          24       25       +1     
  Lines         713      732      +19     
  Branches       76       77       +1     
==========================================
+ Hits          602      618      +16     
- Misses        106      109       +3     
  Partials        5        5              

Impacted Files	Coverage Δ
utils/config.ts	82.35% <82.35%> (ø)
StoreSpidLogs/index.ts	96.00% <100.00%> (+0.16%)	⬆️
utils/notification.ts	92.98% <100.00%> (+0.12%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 707c5f0...43c5e3e. Read the comment docs.

[AleDore]

lgtm. Moreover the title of the PR should be more descriptive ( something like "Centralize Env variables in single config module" )

[codecov-io]

Codecov Report

Merging #110 into master will increase coverage by 0.68%.
The diff coverage is 86.36%.

@@            Coverage Diff             @@
##           master     #110      +/-   ##
==========================================
+ Coverage   84.43%   85.11%   +0.68%     
==========================================
  Files          24       31       +7     
  Lines         713      766      +53     
  Branches       76       81       +5     
==========================================
+ Hits          602      652      +50     
- Misses        106      109       +3     
  Partials        5        5              

Impacted Files	Coverage Δ
utils/config.ts	82.35% <82.35%> (ø)
StoreSpidLogs/index.ts	95.83% <100.00%> (ø)
utils/notification.ts	92.72% <100.00%> (-0.13%)	⬇️
...nerated/notifications/DeleteInstallationMessage.ts	100.00% <0.00%> (ø)
generated/notifications/NotifyMessagePayload.ts	100.00% <0.00%> (ø)
generated/notifications/NotifyMessage.ts	100.00% <0.00%> (ø)
generated/backend/Platform.ts	100.00% <0.00%> (ø)
generated/notifications/InstallationId.ts	100.00% <0.00%> (ø)
...notifications/CreateOrUpdateInstallationMessage.ts	100.00% <0.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 707c5f0...fc47ef1. Read the comment docs.

[balanza]

lgtm besides of the use of env variables in unit tests. I see this is needed in some cases because test suites import the whole function's index.ts file, which contains the initialisation code (hence env config is expected) (see here).
Or in other cases, the handler directly imports a module which has a required env variable (see here).

I see those are legacy problems, maybe we can take advantage to solve them by refactor such functions. The scope is: to only test handler.ts modules and to not directly reference env var in them.

[gunzip]

I see those are legacy problems, maybe we can take advantage to solve them by refactor such functions. The scope is: to only test handler.ts modules and to not directly reference env var in them.

that's right, we can do it with another PR

[gunzip]

woops

[gunzip]

a question, does dotenv configured in jest.config.js read env.example by default ?
I know it reads .env by default but maybe env.example is included as well.

[balanza]

I think it's not, we might specify it explicitly

[gunzip]

done :)

[infantesimone]

Yes, the correct file name is .env.
There is a comment in the first line of this file that indicates to rename, the idea is to indicate that it is just a test file.

[balanza]

@infantesimone I see there are lint issues. No problem, you can fix them and push again. Try https://github.com/pagopa/git-hooks to prevent yourself from pushing bad code and save a roundtrip ;)

LGTM but fix lint errors before merging.

[gunzip]

lgtm, why still draft?