Variables in .env file no longer evaluated

[MathieuGilbert]

Something changed from 0.76.0 to 0.78.0. For an .env file like:

PIES=yummy
CAKES=$PIES

The loaded values are:

ENV['CAKES'] == "yummy" in 0.76.0
ENV['CAKES'] == "$PIES" in 0.78.0

Dotenv (1.0.2) seems to be doing its job properly, but Dotenv.load doesn't overwrite existing keys, which already have the un-evaluated variables from Foreman. I've only looked into it far enough to see at this point: https://github.com/ddollar/foreman/blob/master/lib/foreman/cli.rb#L88, engine.env is correct in 0.76.0, but incorrect in 0.78.0.

[seanconaty]

I've encountered this too in version 0.78.0.

[meson10]

+1

[YoranBrondsema]

+1

[sycobuny]

Dotenv was removed as a foreman dependency prior to version 0.78.0, and foreman now has its own .env processing, which is stripped down from Dotenv's (e.g., it doesn't have variable interpolation).

If you're using Dotenv programmatically, you can use the Dotenv.overload method instead, which clobbers any pre-existing variables that are redefined inside .env files you specify, instead of politely leaving them alone as Dotenv.load does.

Alternately, you can stop foreman from doing any of its own environment loading by having it read from an empty file (if you specify any -e/--env flags, it won't try any default locations):

[example]$ cat Procfile .env
# Procfile
showtreats: dotenv env | egrep 'CAKES|PIES'
# .env
PIES=yummy
CAKES=$PIES
[example]$ foreman start -e <()
14:18:33 showtreats.1 | started with pid 39607
14:18:33 showtreats.1 | CAKES=yummy
14:18:33 showtreats.1 | PIES=yummy
14:18:33 showtreats.1 | exited with code 0
14:18:33 system       | sending SIGTERM to all processes
[example]$

The -e/--env flags currently seem to be supported in the start, export, and run commands. And, if <() doesn't work (perhaps your shell doesn't support that syntax), /dev/null works just as well (foreman start -e /dev/null).

There was some discussion about vendoring Dotenv rather than removing it, but that didn't seem to make it in before Dotenv was completely excised.

For what it's worth, I'm in favor of having foreman's env loader do variable interpolation — either by vendoring Dotenv or simply mimicking the behavior — but the authors may consider this an edge case that does not need to be supported, so these workarounds may be necessary for the foreseeable future unless someone takes up that cause (and/or effort).

[mgrobelin]

I am not sure if this is right place, but sadly foreman 0.78 is not usable for me as it was before with 0.76.

Before upgrading I created files to define my ENV vars with a bash-like syntax. That's the way how I passed my ENV vars to a Rails application, but also to some bash-scripts (source- aka. .-command) and other application (parsing line by line).

This file (I called it development.env) may looked like this:

export BASE_DIR="/opt/foo"
export AGENT_DIR="/opt/foo/agent"
export HOSTNAME="localhost"
export USER="myuser"

export DISPLAY=":99" 
export PORT="8000" 

export DB_HOST="localhost"
export DB_USER="myuser"
export DB_PASS="XXX"

Which enabled me to pass that ENV vars like this:

foreman run rake myapp:setup --env config/environments/development.env # or
foreman run rails console --env config/environments/development.env

With foremand 0.78 this procedure seems to have changed, because when running things like above I only receive..

invalid option: --env

Did I misused foreman or is this a bug or does that feature (passing env files to foreman) do not even exist anymore?

[YoranBrondsema]

@mgrobelin I had the same problem after upgrading to 0.78. You can fix it by adding the --env parameter before your command, so like

foreman run --env config/environments/development.env rake myapp:setup # or
foreman run --env config/environments/development.env rails console

[mgrobelin]

@YoranBrondsema thanks for pointing this out - it makes sense to place arguments directly after run-command. Sadly this just solves the invalid option message, but do not pick up my ENV vars :(

0.76:

 # GOOD, prints my current ENV plus the configured variables from envfile
foreman run --env=config/environments/development.env rails r "puts ENV.inspect"

 # OK, this prints my current ENV plus the configured variables from envfile. BUT this may should not work, as --env may be handled as an option to my rails [...] command ?!?!
foreman run rails r "puts ENV.inspect" --env=config/environments/development.env

0.78:

# BAD, prints my current ENV, but does not uses the variables configured in my envfile
foreman run --env config/environments/development.env rails r "puts ENV.inspect"

# OK, can't boot up Rails due it lacks my configured db-connection ENV vars so this don't work at all
foreman run rails r "puts ENV.inspect" --env=config/environments/development.env

After spending some time I guess, that envfile of 0.78 follows different conventions than it was true for the versions before:

1. you can't use export-statements in envfile anymore
2. you can't use sharp (#) for comments at end of line anymore

Example:

export MY_EXPORTED_VAR="foo"
MY_OTHER_VAR="bar" # this is a comment

--- don't work in 0.78 anymore, needs to be ---

MY_EXPORTED_VAR="foo"
# this is a comment
MY_OTHER_VAR="bar"

Is it possible to make those two expectations true again? It's necessary for my use case to have an envfile that is bash-(alike)-conform.

[koenpunt]

easy fix I use for run is:

env `cat .env` && foreman run rails console

[n-rodriguez]

If you want to restore the old behavior :

Create a foreman file In your Rails bin directory with this content :

#!/usr/bin/env ruby

require 'foreman/cli'
require_relative '../lib/deploy_it/core_ext/foreman/engine_patch'

Foreman::CLI.start

And in lib/deploy_it/core_ext/foreman/engine_patch : https://github.com/jbox-web/deploy-it/blob/master/bin/foreman

[adamlwgriffiths]

Still no news on this?!

[ddollar]

If anyone that is being affected by this would like to submit a PR I would be grateful. Thank you! My time is currently quite limited so I will likely not be addressing non-critical issues for quite some time.

[ddollar]

Foreman is currently only reading static env vars out of a .env file. Any change to this behavior could be proposed as a new issue or PR but for now I will err on the side of simplicity.

[donnoman]

This problem bit us inadvertently. We use Dotenv in our Rails installation and we had a .env with

ENCRYPTED_ATTRIBUTES_ALGORITHM="aes-256-gcm" # run rake encryption:rotate when this changes

When we ran in development no issues. But when we introduced foreman, really strange things began to happen.

until we found this finally during debugging:

Donovan Bray [5:34]  
this is kinda wierd

(byebug) Encryptor.default_options[:algorithm]
"\"aes-256-gcm\" # run rake encryption:rotate when this changes"

the comment is IN the value

If you are going to play with .env files, you need to treat them like Dotenv treats them.

[mullen3]

@ddollar I would be happy to submit a PR which re-introduced Dotenv as a dependency, but it sounds like that's not what you're looking for based on the conversation in #505. Can you elaborate on the "dependency hell" issues you had with Dotenv?

[VinceG3]

The main problem I see is that the current version of Foreman makes it impossible to also use Dotenv, and Foreman does not let you turn off environment processing easily. The easiest workaround seems to be the -e /dev/null switch.

What I would suggest is adding a --no-env switch that turns off the environment processing and putting a section in the README explaining the change. I can probably come up with a PR sometime next week if wanted.

[bogdan8]

my servic in systemctl

PartOf=businessbook-web.target

[Service]
User=bobo
WorkingDirectory=/home/bobo/apps/businessbook/releases/20170321095358
Environment=PORT=%i
Environment="DATABASE_USERNAME=businessbook" "DATABASE_PASSWORD=123456" "SECRET_KEY_BASE=a" ExecStart=/bin/bash -lc 'bundle exec puma -C config/puma.rb'
Restart=always
StandardInput=null
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=%n
KillMode=mixed
TimeoutStopSec=5

How do I make it so?

PartOf=businessbook-web.target

[Service]
User=bobo
WorkingDirectory=/home/bobo/apps/businessbook/current
Environment=PORT=%i
Environment="DATABASE_USERNAME=businessbook" 
Environment="DATABASE_PASSWORD=a" 
Environment="SECRET_KEY_BASE=aa" 
ExecStart=/bin/bash -lc 'bundle exec puma -C config/puma.rb'
Restart=always
StandardInput=null
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=%n
KillMode=mixed
TimeoutStopSec=5

Error if Environment one line

'Mar 21 13:00:31 bobo systemd[1]: [/etc/systemd/system/businessbook-web.target.wants/../businessbook-web@.service:8] Invalid environment assignment, ignoring: "DATABASE_USERNAME=businessbook" "DATABASE_PASSWORD=a" "SECRET_KEY_BASE=aa" ExecStart=/bin/bash -lc 'bundle exec puma -C config/puma.rb'

[edslocomb]

Variable interpolation and comments in the .env file are not "nice-to-haves," they're requirements. I can get by without them in toy projects or prototypes, but not in a mature codebase with dozens of environment variables that need to be understood and kept in sync.

Fortunately, the solution is straightforward:

gem install foreman -v 0.76

[danieltdt]

Dotenv is a zero-dependency library. Isn't time to consider adding it as a dependency again or, at least, update the vendorized version of it?