Expressing the absence of a key in a configuration

[eksortso]

Some issues caused by not having null are discussed in this thread: https://twitter.com/mitsuhiko/status/1346455555435593729

Originally posted by @jab in #802 (comment)

---

The thread on Twitter mentioned two use cases for a potential "null" in TOML. One of them discusses "inheritance," which TOML expressly does not have in and of itself, but the matter of combining multiple TOML configurations is a thing that's come up a lot in the past. It's in that context that I bring up the issue here.

The thread author expressed a need to be able to take out a config value when one was previously supplied by the original TOML document. Presumably, no sentinel logic can be used in the consuming app to do this, which I find suspect.

What would be the best way to address an issue like this? Is there an elegant way to do that without introducing a null value and all that such a value would imply?

[marzer]

The best way to address this is to not address it at all, imo. The situation the tweet author describes seems somewhat contrived and suggests poor application design. "A poor tradesman blames his tools."

[jab]

Tagging @mitsuhiko (who started the Twitter thread linked above) in case you'd like to join this discussion.

Also tagging @rochabruno, author of Dynaconf, who may have thought about this, since Dynaconf popularizes the use of TOML with inheritance (or as the Dynaconf docs call it, layered environments). If you click the "settings.toml" tab there, you'll see this example:

[default]
name = ""
[development]
name = "developer"
[production]
name = "admin"

This layered configuration approach very nicely addresses a very common need for application configuration in different environments. But TOML's lack of a null value makes it impossible to use TOML in this way when a lower layer needs to override a non-null value it inherits from a higher layer and set it to null. At this point, Dynaconf users would need to switch from using the Dynaconf default of settings.toml to one of the other formats that support null (e.g. YAML).

@marzer and any others who feel this is poor design, it'd be helpful if you could give an explanation of why this is poor design that takes us through example use cases that this design addresses and what you would consider a better design that addresses those use cases.

[jab]

One other important point here is that you can't easily just change the design of libraries you're using, and libraries often accept null as a valid value for various settings. In this case, TOML's lack of null makes it impossible to explicitly set such a setting to null.

Point being, whatever you think of this design, a lot of software in the real world is designed this way, so it's worth acknowledging that TOML requires such software to change its design if you want to use it.

Maybe worth creating a separate issue for this, but as long as TOML does not support null, the docs should offer an explanation and a good, real-world example of how the TOML maintainers suggest that users work around this.

[marzer]

@jab As @eksortso noted there's no reason a sentinel value can't be used to represent an explicit null, like an empty string. In the example you gave it seems highly unlikely an empty string would be a valid value for a name field, so an empty string would serve as a stand-in for null just fine.

If that feels contrived or clunky, that's because it is, but so are all configuration inheritance schemes that allow this sort of hierarchy inversion. Overriding a parent value is one thing, but a need to "reset" a parent value to the null state in a child just means the child is in the wrong place in the tree and likely even belongs on a different branch entirely (hence the poor design).

(Worth noting that my first use of TOML was writing an application that used it in an inheritance context; I encountered the issue in the tweet a few times, and each time it served as an accurate hint that I needed to re-think the structure of my data.)

Of course there's always going to be exceptions to the above and you can't design out everything or perfectly anticipate the growth needs of an application, but there's nothing else I can add that hasn't already been covered by the discussion in #146.

[eksortso]

Point being, whatever you think of this design, a lot of software in the real world is designed this way, so it's worth acknowledging that TOML requires such software to change its design if you want to use it.

A while back, we considered if TOML ought to appeal to apps that used legacy INI formats. Ultimately, it was considered better to appeal to new apps. We didn't want to deal with all that legacy stuff.

The situation here involves software that already uses TOML. The legacy to take into account is much less shallow. These are newer programs, newer systems. They can change. So I'd say the proper approach towards these newer apps is, yes, to advise them to correct their designs.

I spent a little time looking at Dynaconf. They're inspired by the 12-factor application guide, which advocates for keeping each and every config setting orthogonal from the others, even between environments. That implies that a design that requires a setting be completely unset on a different layer is not taking such advocacy seriously, and needs to be modified.

Maybe worth creating a separate issue for this, but as long as TOML does not support null, the docs should offer an explanation and a good, real-world example of how the TOML maintainers suggest that users work around this.

That's what we're here for. It's why I split out this issue from the previous one. It seemed interesting so I thought it was worth hashing out.

I would take @marzer's advice to heart, and work with your app developers to adjust the way their configuration logic handles these layers.

[pradyunsg]

The best way to address this is to not address it at all, imo.

And I'm in agreement. I'm not sure this is a problem that needs solving in TOML. 🤷