Cannot use environment variables defined in .env in `pipenv run` and scripts in Pipfile

[hmasdev]

Issue description

I want to use environment variables defined in .env in pipenv run and scripts in Pipfile.
For example, when FOO=SPAM is in .env and show-foo = "echo $FOO" is in Pipfile as a script, I want to get SPAM by running pipenv run echo '$FOO' or pipenv run show-foo.

Expected result

Running pipenv run echo '$FOO' or pipenv run show-foo returns SPAM with the following Pipfile and .env:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]

[dev-packages]

[requires]
python_version = "3.10"

[scripts]
show-foo = "echo $FOO"

FOO=SPAM

Actual result

Running pipenv run echo '$FOO' or pipenv run show-foo behave as follows:

$ pipenv run echo '$FOO'
Loading .env environment variables...
$FOO
$ pipenv run show-foo
Loading .env environment variables...
$FOO

$FOO is not evaluated even if FOO=SPAM in .env.

Steps to replicate

1. create the following Pipfile

   [[source]]
   url = "https://pypi.org/simple"
   verify_ssl = true
   name = "pypi"
   
   [packages]
   
   [dev-packages]
   
   [requires]
   python_version = "3.10"
   
   [scripts]
   show-foo = "echo $FOO"
   

2. create .env

   FOO=SPAM

3. run pipenv run echo '$FOO' or pipenv run show-foo

   $ pipenv run echo '$FOO'
   Loading .env environment variables...
   $FOO
   $ pipenv run show-foo
   Loading .env environment variables...
   $FOO

---

$ pipenv --support

Pipenv version: '2022.1.8'

Pipenv location: '/home/hmasu/.pyenv/versions/3.10.2/lib/python3.10/site-packages/pipenv'

Python location: '/home/hmasu/.pyenv/versions/3.10.2/bin/python3.10'

Python installations found:

• 3.10.2: /home/hmasu/.pyenv/versions/3.10.2/bin/python3
• 3.10.2: /home/hmasu/.pyenv/versions/3.10.2/bin/python
• 3.10.2: /home/hmasu/.pyenv/versions/3.10.2/bin/python3
• 3.9.8: /home/hmasu/.pyenv/versions/3.9.8/bin/python3.9
• 3.8.5: /usr/bin/python3.8
• 3.8.5: /usr/bin/python3
• 3.8.5: /bin/python3.8
• 3.8.5: /bin/python3
• 3.7.12: /home/hmasu/.pyenv/versions/3.7.12/bin/python3.7
• 2.7.18: /usr/bin/python2
• 2.7.18: /usr/bin/python2.7
• 2.7.18: /bin/python2
• 2.7.18: /bin/python2.7

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.10.2',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '5.4.72-microsoft-standard-WSL2',
 'platform_system': 'Linux',
 'platform_version': '#1 SMP Wed Oct 28 23:40:43 UTC 2020',
 'python_full_version': '3.10.2',
 'python_version': '3.10',
 'sys_platform': 'linux'}

System environment variables:

• SHELL
• PYENV_HOOK_PATH
• PYENV_SHELL
• WSL_DISTRO_NAME
• PYENV_VERSION
• NAME
• PWD
• LOGNAME
• HOME
• LANG
• WSL_INTEROP
• PYENV_DIR
• TERM
• USER
• DISPLAY
• SHLVL
• PYENV_ROOT
• WSLENV
• XDG_DATA_DIRS
• PATH
• HOSTTYPE
• PIP_SHIMS_BASE_MODULE
• PIP_DISABLE_PIP_VERSION_CHECK
• PYTHONDONTWRITEBYTECODE
• PIP_PYTHON_PATH
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /home/hmasu/.pyenv/versions/3.10.2/bin:/home/hmasu/.pyenv/libexec:/home/hmasu/.pyenv/plugins/python-build/bin:/home/hmasu/.pyenv/shims:/home/hmasu/.pyenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/Program Files/nodejs:/mnt/c/Program Files/WindowsApps/CanonicalGroupLimited.Ubuntu20.04onWindows_2004.2022.8.0_x64__79rhkp1fndgsc:/mnt/c/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.1/bin:/mnt/c/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.1/libnvvp:/mnt/c/windows/system32:/mnt/c/windows:/mnt/c/windows/System32/Wbem:/mnt/c/windows/System32/WindowsPowerShell/v1.0:/mnt/c/windows/System32/OpenSSH:/mnt/c/Program Files (x86)/NVIDIA Corporation/PhysX/Common:/mnt/c/Program Files/NVIDIA Corporation/NVIDIA NvDLISR:/mnt/c/Program Files/dotnet:/mnt/c/Program Files/NVIDIA Corporation/Nsight Compute 2019.4.0:/mnt/c/ProgramData/chocolatey/bin:/mnt/c/Program Files/nodejs:/mnt/c/Program Files/Microsoft SQL Server/130/Tools/Binn:/mnt/c/Program Files/Microsoft SQL Server/Client SDK/ODBC/170/Tools/Binn:/mnt/c/Program Files/Git/cmd:/mnt/c/Program Files (x86)/Microsoft SQL Server/150/DTS/Binn:/mnt/c/Program Files/Azure Data Studio/bin:/mnt/c/WINDOWS/system32:/mnt/c/WINDOWS:/mnt/c/WINDOWS/System32/Wbem:/mnt/c/WINDOWS/System32/WindowsPowerShell/v1.0:/mnt/c/WINDOWS/System32/OpenSSH:/mnt/c/Program Files/Docker/Docker/resources/bin:/mnt/c/ProgramData/DockerDesktop/version-bin:/mnt/c/Users/hmasu/.windows-build-tools/python27:/mnt/c/Users/hmasu/AppData/Local/Microsoft/WindowsApps:/mnt/c/Users/hmasu/bat:/mnt/c/tools/cuda/bin:/mnt/c/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.0/bin:/mnt/c/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.0/extras/CUPTI/libx64:/mnt/c/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.0/include:/mnt/c/Users/hmasu/AppData/Roaming/npm:/mnt/c/Users/hmasu/AppData/Local/Programs/Microsoft VS Code/bin:/mnt/c/Program Files/LLVM/bin:/mnt/c/Users/hmasu/AppData/Local/Microsoft/WindowsApps:/mnt/c/Users/hmasu/AppData/Local/Programs/Julia-1.6.0/bin:/snap/bin
• SHELL: /bin/bash
• LANG: C.UTF-8
• PWD: /mnt/d/workspace/pipenv-.env-exp

---

Contents of Pipfile ('/mnt/d/workspace/pipenv-.env-exp/Pipfile'):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]

[dev-packages]

[requires]
python_version = "3.10"

[scripts]
show-foo = "echo $FOO"

Contents of Pipfile.lock ('/mnt/d/workspace/pipenv-.env-exp/Pipfile.lock'):

{
  "_meta": {
    "hash": {
      "sha256": "fedbd2ab7afd84cf16f128af0619749267b62277b4cb6989ef16d4bef6e4eef2"
    },
    "pipfile-spec": 6,
    "requires": {
      "python_version": "3.10"
    },
    "sources": [
      {
        "name": "pypi",
        "url": "https://pypi.org/simple",
        "verify_ssl": true
      }
    ]
  },
  "default": {},
  "develop": {}
}

[matteius]

@hmasdev It works if you do: pipenv run echo $FOO

Loading .env environment variables...
SPAM

Not sure yet how to make the script section treat $FOO as a variable though, its clearly being treated as the string $FOO.

[matteius]

Related issue: #3901

[hmasdev]

@matteius
thanks for solving my issue :)

[hmasdev]

@hmasdev It works if you do: pipenv run echo $FOO

Loading .env environment variables...
SPAM

Not sure yet how to make the script section treat $FOO as a variable though, its clearly being treated as the string $FOO.

As @matteius mentioned, I have tried pipenv run echo $FOO. But the result is different from Matteius's. My result is empty but Matteius's result is SPAM.
I think that the reason is that $FOO in pipenv run echo $FOO is not evaluated in the virtual environment created by pipenv but evaluated in the current environment, which is outside of the virtual environment created by pipenv.

Here I have two questions:

1. Where is $FOO in pipenv run echo $FOO evaluated?
2. What makes my result different from Matteius's?

$ cat Pipfile
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]

[dev-packages]

[requires]
python_version = "3.10"

[scripts]
show-foo = "echo $FOO"
$ cat .env
FOO=SPAM
$ pipenv run echo $FOO
Loading .env environment variables...

$ 

NOTE: The setting is same as #4975 (comment) .

[matteius]

@hmasdev Can you test with my proposed fixed in branch: issue-4975-env-variables-not-expanded ?

[hmasdev]

@matteius
I have found that my expected results are realized on issue-4975-env-variables-not-expanded 🎉 Thank you 🎉

$ pipenv run echo '$FOO'
Loading .env environment variables...
SPAM
$ pipenv run show-foo
Loading .env environment variables...
SPAM

By the way, is the following result what you intended on issue-4975-env-variables-not-expanded?

 $ pipenv run echo $FOO
Loading .env environment variables...

pipenv run echo $FOO outputs the empty string.

[matteius]

By the way, is the following result what you intended on issue-4975-env-variables-not-expanded?

Not really, I think the master branch worked without the quotes for the run example ... that is kind of odd really. I'll have to think about it more.
EDIT: Actually @hmasdev funny story, I just tried it on main and it also required the quotes, it is on my other branch that upgrades some other vendor'd dependencies that it seems to not require the quotes. So I think this is fine as is and the other issue you mentioned should be fixed when we eventually release: #4969
Double EDIT: Actually my branch it also needs quotes, and git shell on windows needs quotes ... I am not sure what I did at 3am.

[matteius]

2022.3.23 has been released!