Split toml file

[benoitmasson]

I'm using a TOML file for my site configuration with Hugo.

My configuration is pretty long and I would like to split it into several files. However, Hugo requires that the configuration is found in a single file named config.toml, preventing me to split it into several sub-files (since I can refer to only one of them).

Not talking about my particular case anymore, wouldn't it be interesting and useful to have a kind of include "config_local.toml" keyword, to be able to add extra key-value pairs in another file? Or is there another simple way to do something similar?

This would allow global vs. local configuration, easier sharing and versioning, and probably other things that I didn't think of...

[BurntSushi]

I think this is more a feature request for Hugo than for TOML. I don't see any reason why the configuration format needs to standardize on file inclusion.

Failing all of that, I feel like it'd be pretty straight-forward to cobble a shell script together that merged your TOML files into the one required by Hugo.

[benoitmasson]

Of course, I agree with both comments, I can make a specific request and/or script.

I was thinking that maybe I was not the only one to have such an usage of TOML configuration files, and that maybe splitting could be interesting in other contexts. However, I'm not an expert in file standardization so I'll let you decide in the end whether it's meaningful or not ;-)

[TheElectronWill]

I don't think file inclusion corresponds to "a minimal language". I agree that it may be useful sometimes, but I prefer that the software using TOML manages it.

[eternaleye]

Another possible way of framing the problem is TOML documents that push some of their structure out to the file system - I'm partially inspired by how Rust handles modules.

Given a directory structure like this:

• foo.toml.d
  • bar.toml
  • baz.toml.d
    • qux.toml
    • corge.toml
  • grault.toml

then loading foo.toml.d would result in a data structure with the three keys "bar", "baz", and "grault" - the values of "bar" and "grault" are simply the data structures described by those TOML files, and "baz" would similarly have two keys, "qux" and "corge".

Note the lack of any "top-level" textual TOML file - this ensures that each key has one canonical definition, with no ambiguity.

[mojombo]

I think file inclusion violates the minimalism principle TOML is striving for. I'd expect that if a host application thinks that config will become complex enough to warrant multiple files, it will implement a way to accommodate that. I also really don't want to open the security can-of-worms that comes with TOML files slurping in other user specified files. Thanks for the excellent discussion on this matter.

[structurechart]

I have never used TOML, but I just came across this post.

I think inclusion is useful in some real cases.

For instance, let's say a company licenses out an application framework to run on its private Linux server, and the 3rd party (licensed the framework) will run on that private Linux server with its own user account (so others besides the admin of that company can see that file).

The application framework would like to include a TOML file configured by the 3rd party. However, the 3rd party doesn't want to expose the secret parameters in its own TOML. Then, the solution to get around it is to change the code in the framework to expect two separates TOML files and then when a lookup is required, it looks up in both TOML files. Or another solution is that the company needs to give the 3rd party a copy of its TOML file, and the 3rd party manually (or through script) to copy and paste into its own TOML file.

[Nemecsek]

I am positive about adding include to simplify repeating configurations.

In Telegraf it happens to read a number of devices, all with the same configuration. Only IP address/modbus address change.

It is quite boring and error-prone to copy/paste for each device all the lines (in my case the same 50+ lines are repeated 13 times). If I could define a device_registers.toml with the common settings to all 13 devices, and include it in the final config.toml, that would ease it greatly.
Instead of a jumbo 700+ single TOML file, I would have 2 files, of 100 + 50 lines.

[eksortso]

@Nemecsek Can you add logic to your application to check for the presence of special keys in your configuration? Here's a way to do an include.

1. Parse your main TOML file and create a configuration table.
2. Scan through the tables in your configuration for keys with a special name. Let's say, "%include".
3. For each such table:

• Read the value of "%include", which will be the location of the TOML file that you wish to include.
• Load that file and parse it. It will be a configuration table.
• Go back to your original configuration to the table with the include, and merge into that particular table the newly loaded table. (You could cache that included table and refer back to it instead of rereading the file if you know that the files are static.)

4. When you're done, you'll have a complete configuration with all the files included that you wanted to include originally.

That's one way to do it. Another is to just load up all files that you want in your complete configuration, then read from the ones that are needed. If you know how all these files are related to each other, then you can reliably code a composite configuration without merging tables.

[Nemecsek]

@eksortso, Yes of course this is an option and it is what I personally do. That alas means I cannot use it directly for Telegraf, for example, where I would need this feature the most, but need a "precompilation" phase first to join the to-be-included TOML files.

The library is starred 14k+ times, and that means it is widely used in the wild.
If this feature was added natively it would benefit indirectly A LOT of other projects using it.

As an example, here is a problem at hand.

I need to configure a remote server for Telegraf to read two different kind of devices, each with its own list of registers (same names but different modbus addresses).

20 units are DEVICE_A (50 registers are read from this kind of unit) ; 15 units are DEVICE_B (20 registers).

A single TOML file will contain at least 20x50 + 15x20 = 1300 lines, plus the boilerplate = 1700+ lines.

I transfer the monster TOML to the server. After 6 months somebody needs to add a new register to read on DEVICE_B and the "precompiler" is not available (you cannot imagine work conditions by customers, it is sometimes a lot to have a power source for the laptop, leave alone a Net connection).

So this unlucky somebody needs to edit the monster TOML with vi, search/replace hoping not to mingle the register names that are common between the devices.

What is the server's Telegraf service too could accept the include? We would only add ONE line in the description of DEVICE_B and voilà! it's done...

[marzer]

That alas means I cannot use it directly for Telegraf, for example, where I would need this feature the most, but need a "precompilation" phase first to join the to-be-included TOML files.

@Nemecsek If you're trying to solve a design problem in Telegraf by complaining about something in TOML I'd say you're barking up the wrong tree. File a feature request with those developers and have them implement the 'precompilation' phase you need, or some other solution to help your specific issue.

The library is starred 14k+ times, and that means it is widely used in the wild. If this feature was added natively it would benefit indirectly A LOT of other projects using it.

1. This isn't the repo for a library, this is the config language specification (it is separately implemented in many languages on many platforms in other repos elsewhere)
2. That it would benefit "A LOT" of projects is spurious; at best we can say it would benefit specifically you, and maybe some others who want this feature (e.g. 1-2 other people in this thread). A quick search of the TOML issues suggest there have been very few people that need this more generally.

[Nemecsek]

@marzer, It looks like you are taking it quite personally.
I am not "complaining" but "suggesting why it could be interesting". That is why I am discussing about the language specifications, not about specific implementations.

I have been a programmer for 30+ years and know quite well this "purity" against "real life problems" fight.
I only want to underline that if something is a problem for one person only, it doesn't mean it is not a problem.
No need to discuss furthermore about this topic. Bye

[marzer]

It looks like you are taking it quite personally

Huh? Not at all. I was just illustrating that this is likely the wrong level at which your problem should be solved. Besides, there's already plenty of good reasons for not adding this to the language outlined by others above.

[eksortso]

Would it be possible, @mojombo or @pradyunsg, to open up a discussions tab on this project? I don't believe that this topic is ever going to go away (I'm ambivalent towards it, but allowing includes may still contribute to better consumer software designs), and I'd rather have a permanent place to talk about it than under a closed issue from five years ago.

[dgutson]

What finally happened with this, why this was closed? Likewise #36
Including repeated information, avoiding DRY, is a good thing.

[pradyunsg]

See the closing note: #397 (comment)

[eksortso]

Well, the issue was closed six years ago, and @mojombo gave his reasons for that. Those reasons still hold up.

The DRY principle applies to programming languages for a reason. If one function can do a job, then you only need to call/document/test/maintain that one function. But TOML is not a programming language. It's a configuration language, and it will remain simple. That's a big promise for us to keep.

If your application requires multiple configuration files, then you need to call/document/test/maintain the routines that bring in all those configurations and assign them context and meanings in your application. That is appropriately your domain.

[gl-yziquel]

The problem, IMO, is not about having TOML support includes out of the box. It's about having a convention that people agree upon to implement the preprocessing phase in some external library.

That need for include is real to my mind. If TOML doesn't support it, a preprocessing library should and it should be brain dead simple to replace toml with that library.

Higher up in that thread, there was a suggestion for %include. I'd suggest #%include "subfile.toml" instead to ensure this is a comment and thus doesn't break toml parsing. I'd also suggest ensuring that all file paths in that include are considered relative to the main toml file. I'd also suggest, once such a preprocessing library works out of the box, to have toml emit a warning of sorts when it encounters #% include.