Re-thinking config-rs

[matthiasbeyer]

Project: Re-thinking config-rs

---

(Please have a look at https://github.com/matthiasbeyer/config-rs-ng/ for the current efforts)

---

I want to rethink how this crate works, what this crate provides in means of features and what it does purposely not provide.

This issue is a tracking issue for the whole project.

Current features (as of 0.13.x)

• Deserialization from various formats
  • Toml
  • Json
  • INI
  • Yaml
  • Ron
  • Json5
• "Layering" of configuration
• Different kinds of config sources
  • sync/async
  • Environment
• Setting defaults
• Setting overwrites
• Accessing fields using JSONPath-style selectors

Accepted future features

As soon as a feature suggestion is discussed and agreed upon with the config-rs community, it will be moved to the "Accepted" Panel in the project board and listed here.

Timeline

There is no timeline when or whether this project is done. We will implement things when we think it is time.

Help wanted

Help is very much welcome. If you're a user of this crate, please submit your ideas!

Please open one issue per idea for features you want to see in the config-rs implementation of the future.
No idea is too small for an issue.

---

This issue will be constantly updated!

[crumplecup]

I first ran across this crate reading Luca Palmieri's Zero to Production in Rust, and this allows me to use configuration files in ways that I had wanted to, but did not know how to do until the Config crate made it dead simple for me.

Being featured in a popular Rust book seems like a good place to be, as a starting point. I ended up here on the docs page because the Palmieri book is locked to the 0.12 version, and I ran into deprecation warnings when I upgraded to the latest crate version. It took me a trivial amount of effort to refactor my code using the builder pattern. I cannot offer much vision for future direction, but for my two cents, I think the builder pattern is an improvement over the previous variety of Config methods, and feels idiomatic, as it is so widely used in other popular crates.

[jchimene]

Is this solely a runtime configuration tool? Should it interoperate with Cargo and other
Software Bill of Materials tools as a configuration management tool that's made available for devops? I use this for runtime configuration in a Rust/PHP environment.

[matthiasbeyer]

The scope of this library is loading (from different sources) configuration and making it available to the running application conveniently, for rust application code.
So yes, this is for runtime configuration loading and parsing.

[conradludgate]

If it helps, I'll document how config is currently used in a couple of my projects:

At work, where we deploy web applications in kubernetes deployments:

Our main configuration is built into a yaml file.
We also load some secrets from vault into some more yaml files. These are all injected into the container.
Lastly, there are some extra secrets that we encode in environment variables.

For https://github.com/ellie/atuin (cli app and server):

For client, the intent is that the user can modify their config.toml file in order to change how atuin runs. For development purposes of quick testing, we also have the environment parsing enabled.

For the server, there's only a small set of configuration, but similar idea where we have both a server.toml and environment parsing. Depending on how a user wants to host their server, one might be easier to configure than the other.

---

So far, I've never made use of the async (read remote http) configuration. For my work production apps, we don't allow defaults (mis-configuration is a bug). But for atuin, almost everything has a default.

[szarykott]

I'd like to add something to discussion, it is not directly connected to this crate, hence I am not opening a new issue.

I've always percieved this crate as a library that contains core logic related to loading, layering and exposing configuration in applications. Those few pull requests I submitted added extensibility points to the mechanism (for example Format trait). Those provide certain degree of freedom to users and release this crate from responsibility of supporting every configuration format or source possible.

Think about #328 - even though clap is de facto standard Rust library, it has many alternatives and some people might ask why config does not support those alternatives as well.

This is why I want to propose that, following general Rust trend visible in, for instance, tokio, some specific configuration sources or formats (k8s, clap, http, consul, etc.) are implemented as separate crates that depend on this one - thanks to Source and Format trait it is possible.

An obvious alternative is to have feature flags per dependency. I'd like to hear your thought about it.

[matthiasbeyer]

It is always hard to decide whether to implement something behind a feature flag or as a seperate crate. If we think about supported formats (think toml, json,...etc), I guess feature flags would be the right way. For support of something like clap, we might end up putting it into a seperate crate, yes.

[Dessix]

My suggestion would be to prefer separate crates, as it avoids some of the issues I've seen in the current iteration- such as expression path semantics being unavoidable (and inaccessible) for anything outside of the crate.

As one result of the in-crate paradigm, Source::collect_to is essentially impossible to properly implement via any means that is not in terms of an existing source, besides leaving the default. Due to Rust lacking a way to call the "base" version of a trait default implementation, this is non-extensible.

I suspect a flexible core with source types defined in separate crates would allow for a model that encourages extensibility, and ensure that such roadblocks won't be hit by downstream developers.

[matthiasbeyer]

I just published a very very basic mockup of my idea of the very baseline for config-rs in this branch.

For now, I removed the old codebase in this branch. This branch WILL be rebased and modified, do NOT base work on this! If you want to send patches for this branch, please make sure you publish them ASAP after you wrote them, because you might have to redo all the things if I rebase this!

---

That said, here's some explanation what my ideas in this branch are for now:

• Config loading is abstracted via a trait
  • This means that the loading mechanism must provide context, where the configuration came from, etc.
  • Loading is done to an intermediate format (ConfigElement here).
  • Loaded ConfigElements are stored in ConfigObject objects, that also contain context information (source, for example. But more is possible)
• Layering is not collapsed. This means that we safe all layers always
• Accessing values (ConfigElements) means grabbing "through" these layers to find the relevant ConfigElement object
  • This means that accessing values is a bit more costly than before, but
  • Context information is preserved and can be used easily ("get value at foo.bar and also tell me where this came from")
  • Default values are nothing more than "the lowest layer" and do not need special handling
  • Override values are nothing more than "the highest layer" and do not need special handling

Not yet there, but definitively relevant for this very first mockup:

• Async loading (very important!)
• JSON-Path style accessing (prepared in this PR, not even mocked yet though)
• Deserializing to T (this might be more involved)
• Self-describing config types (see re-thinking: Self-describing configuration #339) - this is important because if we want something like this, it will influence the design of the core of the crate, I guess. So we would need to decide early.

---

After we decided on some of the basic ideas and so on, we must work out a way how the examples and the integration tests that I simply removed can be added back and how we can implement our public API in a way that the examples/tests must only be adapted in a very very minimal way.

[dbofmmbt]

When I was searching for config-related crates, config and figment were the two that seemed more robust. As we're having this "re-thinking" process here, it may be worth it to take its approach in consideration.

[dbofmmbt]

@Dessix no, I just wanted to highlight that this one and figment were the ones I considered more interesting.

[Dessix]

@matthiasbeyer I realized that one of the limitations I'm currently running into is the separation between my actual configuration (something I generally want to statically type and check against) and the config representation of it.

Right now, config stores a few core primitive types in a hierarchy, but it is nearly unaware of its contents- with the only typed information conveyed by serde. I'm considering a system with a similar paradigm that stores typed information, but exposing that type information to the config handler real-time with something like bevy_reflect so arbitrary types can be checked against as boundaries when applying config sources, instead of just when unpacking the result.

Treating configurations in a pseudo-typed manner might even fix the 500-mile-email problem that is currently a vulnerability in config-rs: Right now, arbitrary key names will be accepted, even if they aren't supported options- and a rename of an optional parameter can lead to the sort of back-versioning issue described in the story.

In a pseudo-typed world, config could enforce that some type in the expected configuration tree provides that path or aliases to it. Rather than a runtime-typed tree, the backing registry could be a series of "partials" which may only be hydrated once they are sufficiently fulfilled. As a nice benefit, indirection could be performed on such types through something like your mentioned ConfigElement, which could both allow runtime hot-reloading and futures-stream-subscription, and provide a means of accessing these "typed" child-values prior to full hydration.

[matthiasbeyer]

FWIW, I extracted my work into a new repository, so we can more easily collaborate (and I hopefully get up my buttocks to get something going here 😆 ).

If someone's interested: https://github.com/matthiasbeyer/config-rs-ng

[matthiasbeyer]

Hello everyone!

After basic derive macro support is merged now, I would like to ask everyone interested in trying out config-rs-ng!

Here is a call-for-participation blog article!

If you have the time, I'd appreciate if you had a look!

[pksunkara]

@matthiasbeyer I think #431 and #435 are both needed to be thought of when developing the next gen because this is something that needs support from early on instead of being added later.

[matthiasbeyer]

@matthiasbeyer I think #431 and #435 are both needed to be thought of when developing the next gen because this is something that needs support from early on instead of being added later.

Agreed and added as issue in matthiasbeyer/config-rs-ng#76!

[kate-shine]

If you have the time, I'd appreciate if you had a look!

From the article:

we also should think hard about environment-variable support, although I am a bit hesitant because in config-rs that is a feature that is half-working and half-broken and I think is also rather hard to get right)

Configuration of services by env variables is a really common usecase with containers, and also provides an easy way to do overrides of config when testing something. Its absence would be a bit of dealbreaker for me.

I didn't notice any issue with it while using the current config-rs, what's broken with it?

[matthiasbeyer]

Its absence would be a bit of dealbreaker for me.

Yes, to me too!

I didn't notice any issue with it while using the current config-rs, what's broken with it?

It's mostly the issue that setting deeply nested fields of a configuration structure. But there's a crate for that now: https://crates.io/crates/envious