JSON Values are improperly stored

[mslinn]

python-dotenv does not store valid JSON. The JSON standard requires double quotes and will not accept single quotes, nor will the parser.

JSON values are escaped to death by python-dotenv because double quotes are used to delimit values. For example, instead of storing JSON values nicely, like this:

MY_JSON='{"a": "b"}'

They are stored as:

MY_JSON="{\"a\": \"b\"}"

... which is not JSON. Yes, it could be transformed into JSON.

python-dotenv reads values surrounded by single quotes without any problem. Is there a technical reason why values are not stored by default surrounded by single quotes?

[Flimm]

Hi @mslinn . Thanks for creating a new issue.

Let's take the example .env that you included:

MY_JSON="{\"a\": \"b\"}"

Now let's parse this file using dotenv in Python:

import os
from dotenv import load_dotenv
import json
import pprint

load_dotenv()
parsed_json = json.loads(os.getenv("MY_JSON"))
pprint.pprint(parsed_json)

If you run this Python file, you will see that the output is:

{'a': 'b'}

In other words, it parsed the .env file correctly, and so we could load the JSON correctly. Double quotes were correctly escaped in a string literal surrounded by double quotes. So I would not agree that JSON values are improperly stored.

Now, you may still have a preference for using single quotes instead of double quotes. May I ask, how are you creating the .env files? Are you creating them manually, or are you using the python-dotenv CLI?

[mslinn]

The motivation for this issue is for .env files to be portable between dotenv implementations, and to be usable without any preprocessing. Simply changing to single quotes would accomplish that.

[Flimm]

That seems like a good goal, to achieve comptability with different .env parsers. Do you have an example of a parser that would parse the string quoted with single quotes correctly, but would fail with double quotes?

Older versions of python-dotenv could not handle string literals surrounded with single quotes:

$ pip install -U 'python-dotenv==0.6.3'
$ cat .env
MY_JSON='{"a": "b"}'
$ python3 settings.py
  File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

This is because older versions of python-dotenv would treat single quotes as part of the value, instead of just as quotes surrounding the value.

Using source .env in the Bash command-line works for both single quotes and double quotes.

If you can't find any examples of parsers that break with double quotes, I think it would be preferable to keep the existing behaviour of python-dotenv, in order to preserve compatibility with older versions of python-dotenv.

[mslinn]

I am writing docs and blogging, using Jekyll plugins. I would have to write a special filter just so that JSON contents would display properly when including portions of .env files if I used python-dotenv.

[bbc2]

Just to clarify, you are using the CLI (dotenv set MY_JSON '{...}') to edit the .env file, right? If you are using that, then yes, it won't use single quotes at the moment.

If all you want is show portions of .env files, then you should be able to use single quotes.

The dotenv CLI is quite limited at the moment. Supporting both types of quotes might be good since they have different semantics in python-dotenv (e.g. variables are not expanded in single-quoted values).

[mslinn]

Yes, bash expansion is also inhibited by single quotes as well.

[Flimm]

I'm not very familiar with Jekyll. Does it parse .env files? I had a look at the links that you posted, but I couldn't find an answer to this question. I'm sorry if I'm being obtuse.

You could use the environment variables in .env in Jekyll like this in a shell like Bash:

$ set -a
$ source .env
$ jekyll ....

That way, Jekyll would have access to all the settings in .env as environment variables. source in Bash will interpret the double-quoted string literal as intended.

[mslinn]

Jekyll does not know about .env files. I wrote Jekyll plugins to help me document code. Some of those plugins read and filter data from files. I would like to be able to include portions of .env files in the docs. If the files do not store proper JSON my task becomes much harder.

[bbc2]

Unless I missed something you didn't answer my question. Do you really need to use the CLI? Otherwise, just write the single-quoted JSON value in the file and python-dotenv will be able to read it the way you want.

Regardless of that problem, I'll consider switching to single-quotes in the implementation of dotenv set so that there's no surprise with variable expansion.

[bbc2]

This should be fixed by #330 when it's merged. In case you'd like to test it, I'd be happy to know what you think about it.