New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a /config dir #5422

Closed
bep opened this Issue Nov 8, 2018 · 23 comments

Comments

Projects
None yet
3 participants
@bep
Copy link
Member

bep commented Nov 8, 2018

This is issue is vaguely described other places but is long over due. This description should be concrete enough to implement:

  • Add a configDir setting, default /config.
  • Add a --environment flag (and setting), which defaults to production for hugo and development for hugo server
  • Configuration keys in /config will override keys in config.toml

A sample tree could look like this:

config
├── _default
│   ├── config.toml
│   ├── languages.toml
│   └── mediaTypes.toml
└── production
    ├── config.toml
    ├── mediaTypes.toml
    └── related.toml
  • config.toml allows top level keys (baseURL) etc. to be set per environment.
  • The other filenames maps to a configuration key, so languages.toml could look like this:
  [en]
    contentDir = "content/en"
    languageName = "English"
    weight = 1
  [zh]
    contentDir = "content/zh"
    languageName = "中文"
    weight = 2

There are no clever merging inside /config -- we pick the most specific file (for production and mediaTypes in the example above, we pick the file below /production).

This will (in its first iteration) be a project only thing.

/cc @regisphilibert @budparr @kaushalmodi

@bep bep added the Proposal label Nov 8, 2018

@regisphilibert

This comment has been minimized.

Copy link

regisphilibert commented Nov 8, 2018

Was just thinking about it yesterday having to manage many menu items from the config.yaml. "Boy I wish there was a native way to isolate those menu items in their own file..."

I assume we can add as many files as we want in order to create a config objects. This is not limited to the one you mentioned?

There are no clever merging inside

What do you mean by clever ? I just wish the config/production/config.yaml could overwrite some values of config/_default/config.yaml such as the baseURL or publishDir while using its many "shared" parameters.

@bep

This comment has been minimized.

Copy link
Member

bep commented Nov 8, 2018

This is not limited to the one you mentioned?

No, the scope is endless.

What do you mean by clever ?

I should be more clear (and less clever):

  • In /config we pick the most specific file (config.toml, mediaTypes.toml etc.)
  • mediaTypes.toml wlil be a complete configuration that will replace any in config.toml (there are some special rules for themes here ... but that is little bit on the side)
  • Keys inside config.toml below /config will override the root config, so you can override single values there (baseURL etc).
@budparr

This comment has been minimized.

Copy link

budparr commented Nov 8, 2018

Love this. Slightly uncomfortable with environment variable defaults, but I suppose that if one went to the trouble of using them at all, they'd be aware of the defaults.

Question (hopefully not too off topic). Would it be possible to do a build with no config at all? Right now I believe Hugo looks for a config file, no matter how minimal (even empty). Would be cool to be zero config with nothing required other than a minimal layouts/index.html file. Then Hugo could be "The world’s fastest ZERO CONFIG framework for building websites" :)

@bep bep added Enhancement and removed Proposal labels Nov 8, 2018

@bep bep added this to the v0.52 milestone Nov 8, 2018

@bep

This comment has been minimized.

Copy link
Member

bep commented Nov 8, 2018

Slightly uncomfortable with environment variable defaults, but I suppose that if one went to the trouble of using them at all, they'd be aware of the defaults.

You mean the development/production logic?

Yes, it's strictly a little outside the scope of this issue, but doing it at the same time is considerably cheaper. And the number of overrides should be very small (my example is bad) -- and I think this makes it more explicit than the if production we have today.

@regisphilibert

This comment has been minimized.

Copy link

regisphilibert commented Nov 8, 2018

Yes I suspect environment only key/value pair will be limited to baseURL, googleAnalytics and a few params. So most of the config settings will happen in _default.

This will be an awesome improvements on the way Hugo currently deals with config! 🤩

@budparr

This comment has been minimized.

Copy link

budparr commented Nov 8, 2018

Yes, I meant that I explicitly set environment variables and that control is predictable to me. Not a big deal though.

@bep

This comment has been minimized.

Copy link
Member

bep commented Nov 8, 2018

Yes, I meant that I explicitly set environment variables and that control is predictable to me

Note that if we add environment as a new config key, any value in HUGO_ENVIRONMENT will still work.

@bep

This comment has been minimized.

Copy link
Member

bep commented Nov 15, 2018

An important addendum to the above, having thought about this a little. One of the big benefactors of this feature is sites with many languages (and with that really a long languages section with menu definitions etc.).

So, for the languages config I think we need to allow "one dot" in the filename, e.g.:

config
├── _default
│   ├── config.toml
│   ├── languages.en.toml
│   ├── languages.toml
│   └── mediaTypes.toml
└── production
    ├── config.toml
    ├── mediaTypes.toml
    └── related.toml

So in the above, languages.en.toml represents the English configuration and will replace any languages.en key in languages.toml.

/cc @regisphilibert @budparr

@regisphilibert

This comment has been minimized.

Copy link

regisphilibert commented Nov 15, 2018

But could we still have "standalone file" config for each languages (menus.en.toml, menus.fr.toml) ?

@bep

This comment has been minimized.

Copy link
Member

bep commented Nov 15, 2018

But could we still have "standalone file" config for each languages (menus.en.toml, menus.fr.toml) ?

menus isn't a configuration key in Hugo, I think. This issue is getting scope creeped as it is, so we're not adding config options that isn't already there.

I think it helps to think about this issue as "very technical".

@bep bep modified the milestones: v0.52, v0.53 Nov 28, 2018

bep added a commit to bep/hugo that referenced this issue Dec 5, 2018

@bep

This comment has been minimized.

Copy link
Member

bep commented Dec 5, 2018

@regisphilibert I've dived into this issue now, and it does not look too hard. There is one issue that you can help me with. I will think out loud a little. It is related to your last comment; I think we need to be a little more clever when it comes to language handling.

Taken from the Hugo docs site:

languages.zh.menu.main  => array
languages.zh.params => map

(I have made a note to create menus (plural) as an alias -- to get it in line with the others)

I have put the above to wrap my head around and make sure we can represent this in all the config formats. So, only JSON allows arrays without a root, so we need to avoid (or work around) that.

With that, I was thinking that the 2 above examples could look like:

config/menus.zh.toml:

[[main]]
name = "函数"
weight = 30
[[main]]
name = "变量"
weight = 35
[[sidebar]]
name = "变量"
weight = 35

This means that if you have several menus, you need to put them in the same file for that language.

config/params.zh.toml:

description = "A description in that language"

The above makes sense to me ...?

bep added a commit to bep/hugo that referenced this issue Dec 5, 2018

@regisphilibert

This comment has been minimized.

Copy link

regisphilibert commented Dec 5, 2018

This means that if you have several menus, you need to put them in the same file for that language.

Of course. The important part here is that multilingual projects will not loose this "file isolation" feature by having to dump all their config objets into one "language config" file.

(I have made a note to create menus (plural) as an alias -- to get it in line with the others)

Agree! Current key is counter-intuitive (I always forget it's supposed to be singular).

There is one issue that you can help me with.

Not sure how I can help beside supporting this logic (config/menus.zh.toml) as it is exactly what I would wish as a user (#5422 (comment)).

@bep

This comment has been minimized.

Copy link
Member

bep commented Dec 5, 2018

Not sure how I can help beside supporting this logic

That's the help I needed.

bep added a commit to bep/hugo that referenced this issue Dec 6, 2018

bep added a commit to bep/hugoDocs that referenced this issue Dec 7, 2018

bep added a commit to bep/hugoDocs that referenced this issue Dec 7, 2018

bep added a commit to bep/hugoDocs that referenced this issue Dec 8, 2018

@bep

This comment has been minimized.

Copy link
Member

bep commented Dec 8, 2018

@budparr and @regisphilibert -- given the "config tree" below:

config
├── _default
│   ├── config.toml
│   ├── languages.toml
│   ├── menus
│   │   ├── menu.en.toml
│   │   └── menu.zh.toml
│   ├── params.1.toml
│   └── params.2.toml
└── development
    └── params.2.toml

Do you understand the meaning of params.1.toml etc or is it too clever?

The thing is, the main rule has to be that if languages etc. is set for development, we don't use the one in _default. I cannot know if the user wants to merge without some additional information. There are, however, situations where you have a lot of common params, but where only some of them is different for a given environment, hence the numbering.

Also note that the menus sub folder is only a way to organize your files, it can be named anything, and I will fix "menu vs menus".

@regisphilibert

This comment has been minimized.

Copy link

regisphilibert commented Dec 8, 2018

Do you understand the meaning of params.1.toml etc or is it too clever?

I'm guessing this is a way of defining a group of params to be merged vs another group to be left as is.

Let me share a personal yet very common (I think) use case to make sure I'm following.

Using the CLI command hugo --config config.yaml,config.prod.yaml, I have a config.yaml with a lot of settings common to both environement, and a config.prod.yaml with a distinct baseURL and params.api_url. Those will override the one present in config.yaml.
If a key present in config.yaml needs to be absent in production I suppose I could set it to false in config.prod.yaml.

By what I understand this is how I would replicate this behaviour using the new config directory feature:

config
├── _default
│   ├── config.toml (baseURL set here)
│   ├── params.1.toml (params common to both env. set here)
│   └── params.2.toml (api_url set here)
└── production
│   ├── config.toml (baseURL overriden here)
    └── params.2.toml (api_url overriden here)

Am I following?

@bep

This comment has been minimized.

Copy link
Member

bep commented Dec 8, 2018

Am I following?

Yes.

The main motivation here is that if you have 120 generic params + 1 environment specific param, you don't want to repeat the 121 params in every environment.

Also note that this is a general pattern; you could also do config.5.toml-- the number is a key element, but it also controls the order (params.2.toml is added after 1).

@regisphilibert

This comment has been minimized.

Copy link

regisphilibert commented Dec 9, 2018

So still with the previous use case I could not do the following:

config
├── _default
│   ├── config.toml (baseURL set here)
│   └── params.toml (params common to both env. set here + default api_url)
└── production
│   ├── config.toml (baseURL overriden here)
    └── params.toml (default api_url overriden here with production api_url)

And expect the params common to both env to be set in the production env. as well.

If this is a technical decision then I'm happy to trade this very light complexification to be able to manage configuration with directories and filenames.

If this is a usability decision on the other hand, I'd like to understand the logic behind it as it presently escapes me.

@bep

This comment has been minimized.

Copy link
Member

bep commented Dec 9, 2018

Your last example will not work. This is merged on "file level": We pick the most specific file, e.g. production/params.toml.

Note that it would be technically possible to implement it so your particular example would work. But that would break other examples/use cases, examples being that I would want a different set of languages, output formats, whatever, in the different environments, so we cannot merge blindly.

@regisphilibert

This comment has been minimized.

Copy link

regisphilibert commented Dec 9, 2018

But that would break other examples/use cases

But I think this could be solved differently by having Hugo merge blindly while having the user exercise a much expected discipline of never putting in _default something not shared by every environment, unless it can be overridden by a false value.
I would not add a "production only" Output Format in _default, and I don't expect any one would.

Again my suggestion may not be sustainable, but as I cannot think of a use case for which this basic discipline would not suffice, I must insist on asking you or other users if this "blind merging" could not be OK, providing it does not complicate your task of course.

This would avoid introducing and documenting the file numbering system and make the new config management all the more intuitive by matching the behaviour of hugo --config's blind merging logic even though I suspect it is currently underused.

@regisphilibert

This comment has been minimized.

Copy link

regisphilibert commented Dec 9, 2018

Here's the use case I can think of to defend the numbering file implementation:

I have 5 environments, I need a certain output format in all of them but one. This means with blind merging, I cannot put it in _default and therefore have to copy and paste this output format settings in the 4 environments needing it.

I'm suspecting this use case will be rare and I'm afraid we may complexify a feature for the vast majority of use cases only to benefit a very few.

@bep

This comment has been minimized.

Copy link
Member

bep commented Dec 9, 2018

I will sleep on it.

@bep

This comment has been minimized.

Copy link
Member

bep commented Dec 10, 2018

@regisphilibert I have slept, and you are right. I was thinking too hard about this. If we for some really good reason need "my variant" we could add some naming standard for that case later. Thanks for the input, always appreciated.

@regisphilibert

This comment has been minimized.

Copy link

regisphilibert commented Dec 10, 2018

Anytime, glad I could help.

bep added a commit to bep/hugo that referenced this issue Dec 10, 2018

bep added a commit to bep/hugo that referenced this issue Dec 10, 2018

bep added a commit to bep/hugo that referenced this issue Dec 10, 2018

bep added a commit to bep/hugo that referenced this issue Dec 10, 2018

bep added a commit to bep/hugo that referenced this issue Dec 10, 2018

bep added a commit to bep/hugo that referenced this issue Dec 10, 2018

bep added a commit to bep/hugo that referenced this issue Dec 10, 2018

bep added a commit to bep/hugoDocs that referenced this issue Dec 11, 2018

bep added a commit to bep/hugo that referenced this issue Dec 11, 2018

bep added a commit to bep/hugo that referenced this issue Dec 11, 2018

Add /config dir support
This commit adds support for a configuration directory (default `config`). The different pieces in this puzzle are:

* A new `--environment` (or `-e`) flag. This can also be set with the `HUGO_ENVIRONMENT` OS environment variable. The value for `environment` defaults to `production` when running `hugo` and `development` when running `hugo server`. You can set it to any value you want (e.g. `hugo server -e "Sensible Environment"`), but as it is used to load configuration from the file system, the letter case may be important. You can get this value in your templates with `{{ hugo.Environment }}`.
* A new `--configDir` flag (defaults to `config` below your project). This can also be set with `HUGO_CONFIGDIR` OS environment variable.

If the `configDir` exists, the configuration files will be read and merged on top of each other from left to right; the right-most value will win on duplicates.

Given the example tree below:

If `environment` is `production`, the left-most `config.toml` would be the one directly below the project (this can now be omitted if you want), and then `_default/config.toml` and finally `production/config.toml`. And since these will be merged, you can just provide the environment specific configuration setting in you production config, e.g. `enableGitInfo = true`. The order within the directories will be lexical (`config.toml` and then `params.toml`).

```bash
config
├── _default
│   ├── config.toml
│   ├── languages.toml
│   ├── menus
│   │   ├── menus.en.toml
│   │   └── menus.zh.toml
│   └── params.toml
├── development
│   └── params.toml
└── production
    ├── config.toml
    └── params.toml
```

Some configuration maps support the language code in the filename (e.g. `menus.en.toml`): `menus` (`menu` also works) and `params`.

Also note that the only folders with "a meaning" in the above listing is the top level directories below `config`. The `menus` sub folder is just added for better organization.

We use `TOML` in the example above, but Hugo also supports `JSON` and `YAML` as configuration formats. These can be mixed.

Fixes gohugoio#5422

@bep bep closed this in 7829474 Dec 11, 2018

bep added a commit to bep/hugoDocs that referenced this issue Dec 24, 2018

bep added a commit to bep/hugoDocs that referenced this issue Dec 24, 2018

regisphilibert added a commit to regisphilibert/rp_hugo that referenced this issue Dec 31, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment