Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch config to YAML #2475

Open
ralsina opened this Issue Aug 26, 2016 · 40 comments

Comments

Projects
7 participants
@ralsina
Copy link
Member

ralsina commented Aug 26, 2016

Rationale:

  • We currently can't introduce new options in the user's config
  • We can't migrate configs when we change options
  • We could do those things if the config file was not python

Plan:

  • Write a conf.yaml that matches current conf.py (including comments)
  • Make nikola init generate a conf.yaml and no conf.py
  • If there is a conf.py, read it.
  • If there is no conf.py read the conf.yaml

We can even group settings by thematic sections, and just smush them smartly together on reading so we don't need to reflect the sectioning in our code.

NOTE for this to be worthwhile, we need to have comments that survive roundtrips. While pyyaml doesn't support that, https://pypi.python.org/pypi/ruamel.yaml/0.6 does

NOTE Previously this issue suggested TOML. No, TOML sucks because of it's table syntax.

@ralsina ralsina added this to the Whenever milestone Aug 26, 2016

@ralsina ralsina closed this Aug 26, 2016

@ralsina ralsina reopened this May 21, 2017

@ralsina ralsina changed the title Switch config to TOML Switch config to YAML May 21, 2017

@felixfontein

This comment has been minimized.

Copy link
Contributor

felixfontein commented May 21, 2017

If I understand it correctly, currently the only way to define your own filter is to define it in your config as a Python function. That won't work with YAML.

So if I didn't understood this wrongly, we need some way to provide filters by plugins.

@ralsina

This comment has been minimized.

Copy link
Member Author

ralsina commented May 21, 2017

IIRC we already have a way to specify filters in metadata which works. It's not perfect, of course.

Also, a way to tweak things via conf.py for advanced usage is still doable.

@felixfontein

This comment has been minimized.

Copy link
Contributor

felixfontein commented May 22, 2017

In metadata of what? Posts? But I want to specify a custom filter for every HTML file, for example, or for JS files (which are not geneated by posts). I don't see how I can do that curently without writing code in the config file.

@Kwpolska

This comment has been minimized.

Copy link
Member

Kwpolska commented May 22, 2017

Huge 👎 for YAML. Vim and VSCode both have issues with editing YAML and indenting it right. Also, YAML is plagued by ambiguity. For example, values are strings, unless they aren’t, and sometimes they need quotes. Multi-line strings have two syntaxes. Single characters can drastically change the meaning of values (| vs >, - vs ?).

YAML may look easy, but to me it’s more complicated than the existing .py file.

@ralsina

This comment has been minimized.

Copy link
Member Author

ralsina commented May 22, 2017

@felixfontein which is why I am working on it. I need to see if there is a way to do this without losing interesting functionality. As it is now, there is absolutely no functionality loss. Have some patience :-)

@Kwpolska at least we can change a value in YAML from Nikola and write it back to disk, which we can't in python. Having that would make our whole deprecation strategy much cleaner.

I could do the same thing using configobj ... it's the only other config format I know where comments survive roundtrips.

@felixfontein

This comment has been minimized.

Copy link
Contributor

felixfontein commented May 22, 2017

@ralsina: I'm just trying to bring up things which could be problematic as early as possible :)

@richieadler

This comment has been minimized.

Copy link

richieadler commented May 22, 2017

@ralsina Besides my +1 for YAML, I assume you're thinking of using ruamel.yaml to preserve the comments on roundtrips (the original pyYAML parser doesn't AFAIK).

@ralsina

This comment has been minimized.

Copy link
Member Author

ralsina commented May 22, 2017

@richieadler exactly

@felixfontein

This comment has been minimized.

Copy link
Contributor

felixfontein commented Jun 3, 2017

@ralsina: I've been thinking a bit about filters. What do you think of the following?

  1. Add a function to register filters to the Nikola site object; this allows plugins to register them in their set_site call.
  2. Move the # Configure filters block (here) in Nikola.__init__ after initialization of at least one plugin type (for example after this line) and prepend a loop which looks for strings, and looks up the strings in the registered filters dict to resolve the actual filters (with fallback "keep the string as-is" if it doesn't show up in the registry).

We should then add a little plugin which registers the default filters, so we can start replacing the FILTERS = { ... } setting with Python callables with strings. We should prefix the names with something, though, to avoid confusions with direct command lines. Maybe nikola.optipng or #optipng or something like that for the optipng filter (as an example)?

@Kwpolska

This comment has been minimized.

Copy link
Member

Kwpolska commented Jun 3, 2017

👍, but this still isn’t enough for everything. There could be some optional .py file for config that requires callables that would complement the yaml config file (eg. filters can be specified in YAML for Nikola built-ins or plugins and in .py for user-defined extras; global context = YAML for strings + .py for custom functions)

@felixfontein

This comment has been minimized.

Copy link
Contributor

felixfontein commented Jun 3, 2017

@Kwpolska: you should be able to insert that function into global context via a little ConfigPlugin. I'll also have to move my filter definitions (which are currently in the config file) into a little plugin.

@Kwpolska

This comment has been minimized.

Copy link
Member

Kwpolska commented Jun 3, 2017

Writing a plugin may be a bit scary to newcomers.

@felixfontein

This comment has been minimized.

Copy link
Contributor

felixfontein commented Jun 3, 2017

That's true. Having something like:

# Will take all exported functions from the Python file and register them as filters
FILTERS_PYTHON_FILE = 'my-filters.py'

# Provides a Python file with definitions for the global context, and a dictionary
# specifying which identifiers from the Python file end up as which objects in the
# global context.
GLOBAL_CONTEXT_PYTHON_FILE = "global-context.py"
GLOBAL_CONTEXT_PYTHON_MAP = {
  "post_lead_format": "my_post_lead_format_python_function_name_as_defined_in_python_file",
}

as you mentioned would be way better for that. I'd personally like to see both. I probably won't use the Python file approach (as I'm comfortable with plugins and already have a bunch of them), but that shouldn't stop others from using them :)

@Kwpolska

This comment has been minimized.

Copy link
Member

Kwpolska commented Jun 3, 2017

I was thinking of something like this.

conf.yaml

FILTERS = {".html": ["#add_headerlinks"]}
GLOBAL_CONTEXT = {"foo": "value"}

extraconf.py

def filter_html(x):
    return x

def bar(x):
    return x

FILTERS = {".html": filter_html}
GLOBAL_CONTEXT = {"bar": bar}

Nikola (pseudocode)

conf = yaml.safe_load('conf.yaml')
if os.path.exists('extraconf.py'):
    extraconf = vars(__import__('extraconf'))
    for k, v in extraconf.items():
        if k in conf and isinstance(conf[k], dict):
            conf[k].update(v)  # special FILTERS handling?
        else:
            conf[k] = v
@felixfontein

This comment has been minimized.

Copy link
Contributor

felixfontein commented Jun 3, 2017

That would essentially destroy the most important advantage of a YAML config, namely the ability to modify it, as you can now define everything in the Python config as well and a program parsing the YAML file would have no idea which parts are overridden by the Python config. (Of course, it could load/interpret the Python config, but that's kind of messy and requires a lot of extra logic to make sure nothing goes wrong when updating the YAML file.)

@Kwpolska

This comment has been minimized.

Copy link
Member

Kwpolska commented Jun 3, 2017

We could limit the extraconf.py file to a few variables and ignore the rest.

@felixfontein

This comment has been minimized.

Copy link
Contributor

felixfontein commented Jun 3, 2017

That's possible. We still then need to define a merging order. This is particularly important for filters: if a filter for .html files is defined in both config files, which filters are executed first? Depending on the filters, the execution order is important after all.

@felixfontein felixfontein referenced this issue Jun 4, 2017

Merged

Allowing to register filters. #2819

3 of 3 tasks complete
@felixfontein

This comment has been minimized.

Copy link
Contributor

felixfontein commented Jun 4, 2017

I created a PR for registering filters.

@ralsina

This comment has been minimized.

Copy link
Member Author

ralsina commented Jun 4, 2017

For filters I agree writing a plugin is scary. BUT we could just have a filters/ where the user throws foo.py which always has to have a def filter() in it.

Then we import it and rename it to foo().

I think it matches with what we do with templates/ where it's a lightweight alternative to a full theme.

@felixfontein

This comment has been minimized.

Copy link
Contributor

felixfontein commented Jun 4, 2017

That sounds good as well. Then you could write filters/foo or filters.foo in the FILTERS config to use them.

But how about GLOBAL_CONTEXT entries? For that, we need something else.

@Kwpolska

This comment has been minimized.

Copy link
Member

Kwpolska commented Jun 4, 2017

write filters.foo

That kinda conflicts with the just-merged code. Perhaps we could make it nikola.filters for the built-in filters?


Now, there’s one more thing left to discuss. Perhaps an entirely new config format is worthy of v8? We have quite a lot of cruft to remove in that release, and if we’re looking for reasons, this is the one. v7.0.0 was released in May 2014, or 3793 commits ago. (Of course, if anyone has ideas for other refactorings/changes/revolutions, speak up now!)

@felixfontein

This comment has been minimized.

Copy link
Contributor

felixfontein commented Jun 4, 2017

@filters: It only conflicts if you use the same names ;-) It has been filters.xxx before in post metadata, so even if we adjust the names (to say nikola.filters.xxx), we have to support the old filters.xxx names as well. Or maybe support both names, and let filters in filters/ override default ones?

@v8: how about allowing to mix both configs in v7 (with YAML config overriding Python config if both are there), and then remove Python config in v8?

@Kwpolska

This comment has been minimized.

Copy link
Member

Kwpolska commented Jun 4, 2017

Okay, let’s keep it as-is. But when would said v8 release happen if not now?

@felixfontein

This comment has been minimized.

Copy link
Contributor

felixfontein commented Jun 4, 2017

I'm fine with starting to work on v8. Maybe we should release something before really starting on that, especially to fix #2816.

@Kwpolska

This comment has been minimized.

Copy link
Member

Kwpolska commented Jun 4, 2017

That’d be v7.8.7 which I’m going to release tomorrow or on Tuesday. And there’s nothing to stop us from other bugfix v7 releases. @ralsina, opinions?

@ralsina

This comment has been minimized.

Copy link
Member Author

ralsina commented Jun 4, 2017

@Kwpolska @felixfontein: How about we do 7.8.7 then we have a couple of months without stable releases to break stuff.

@felixfontein

This comment has been minimized.

Copy link
Contributor

felixfontein commented Jun 4, 2017

Sounds good to me!

@Kwpolska Kwpolska referenced this issue Jun 12, 2017

Closed

Version 8 #1718

26 of 26 tasks complete

@Kwpolska Kwpolska added this to To Do in Version 8 Jun 12, 2017

@ralsina

This comment has been minimized.

Copy link
Member Author

ralsina commented Apr 24, 2018

So. I have tried. I have failed.

I think the right approach for this is to write a framework for configuration that supports defining the options, adding help, and generating / modifying config files via a API.

I sort of have the idea of how to do it, but trying to do it inside Nikola makes no sense. I should just do that separately and then migrate nikola to using it.

So, I propose we take this out of v8, and if we have everything else (or enough?) we just release it.

Master is so much nicer than 7.x it deserves a release.

Opinions @Kwpolska @felixfontein ?

@felixfontein

This comment has been minimized.

Copy link
Contributor

felixfontein commented Apr 24, 2018

From the TODOs listed in #1718, I'd still like to see #2761 implemented before 8.0 is released. Since @h4ckninja doesn't seem to be working on this anymore, I can work on this if no one else does. I should be able to produce at least a first version this weekend.

@felixfontein

This comment has been minimized.

Copy link
Contributor

felixfontein commented Apr 24, 2018

Besides that, yes: I also would prefer master to be released not too far in the future :)

@ralsina

This comment has been minimized.

Copy link
Member Author

ralsina commented Apr 24, 2018

@felixfontein sure, looks simple enough, go ahead

@Kwpolska

This comment has been minimized.

Copy link
Member

Kwpolska commented Apr 24, 2018

Here’s a plan:

  1. Recon — look for issues to handle soon (like #2761@felixfontein, go ahead) and do them
  2. Release v8 beta to PyPI and announce it (blog + nikola-discuss + python-announce list), preferably this week (I can do it if there are no objections)
  3. After ~two weeks, if there are no large complaints, release v8 proper to the world. (Migration guide: be wary of crumbs.tmpl and bootstrap3, update your config — write a neat blog post for it)
  4. Perhaps there is some good config management framework that we could use?
  5. If there isn’t, start a new project for the configuration management framework (preferably under the @getnikola org for better visibility), inspired by Django migrations, that can be integrated into anything and is very flexible.
  6. When it’s ready, release v8.1.0 (or so) with YAML config. // extra task: metadata.sample.yml → metadata.yml in demo gallery

Opinions?

# sample migration set -- basic idea
from fancyconfigframework import MigrationType
MIGRATIONS = [
    (MigrationType.RENAME, 'DISABLE_INDEXES_PLUGIN_TOO_MANY_WORDS', 'DISABLE_INDEXES'),

    # handle hide_sourcelink → show_sourcelink
    (MigrationType.RENAME, 'HIDE_SOURCELINK', 'SHOW_SOURCELINK'),
    (MigrationType.NEGATE_BOOL, 'SHOW_SOURCELINK'),

    (MigrationType.RUN_PYTHON, some_callable_here)
]
@ralsina

This comment has been minimized.

Copy link
Member Author

ralsina commented Apr 25, 2018

+1 looks like a great idea to me

@felixfontein

This comment has been minimized.

Copy link
Contributor

felixfontein commented Apr 26, 2018

+1 I like this too :)

@Kwpolska Kwpolska moved this from To Do to Postponed in Version 8 May 3, 2018

@polyzen

This comment has been minimized.

Copy link
Contributor

polyzen commented Aug 24, 2018

@gwax

This comment has been minimized.

Copy link
Contributor

gwax commented Sep 7, 2018

Personally, my approach to upgrades / migrations is to create a new, empty nikola install and diff the fresh conf.py file against my current one. This gives me a very good view of new/changed/removed attributes.

It does, to some degree, require that I keep my configuration in roughly the same structure as the stock configuration but it makes upgrades incredibly easy. There might be room for automating / simplifying this process.

@gwax

This comment has been minimized.

Copy link
Contributor

gwax commented Sep 7, 2018

I'm a little worried about the idea of using YAML configs and automatically migrating configurations on updates. Specific to YAML, I am worried about:

  • PyYAML discards comments on load and cannot dump comments
  • PyYAML does not maintain dictionary order on load/dump

The ordering and comments in my configuration file are absolutely critical to my ability to understand how I have things configured. When I upgrade nikola, reading through the comments in new sections helps me understand what each new configuration directive means.

Losing comments and reordering configurations will discard important context and potentially make diffs very hard to read.

@ralsina

This comment has been minimized.

Copy link
Member Author

ralsina commented Sep 8, 2018

@kayhayen

This comment has been minimized.

Copy link
Contributor

kayhayen commented Oct 1, 2018

@Kwpolska I just checked elektra for yaml, but it shares the same issue regarding loss of comments: https://www.libelektra.org/plugins/yamlcpp

As somebody who is going to upgrade Nikola soon going to 8.0.1, I used to do what @gwax suggested, even for small releases, observing differences in generated output, indicating the need for it, until it became to laborious and then I stopped updating Nikola. Or wait, not true, in another instance, I auto update one instance, but that wasn't really a good idea I think.

I will now follow the suggested route going to the last release 7 release with that method, then 8 again with @gwax method. And then stop again.

To me YAML is a step back from Python as config, but I am coder, and few people are, and that is your audience. I like code as config, and wish for factories to generate config. It was nice to e.g. check stuff right there, e.g. having the copyright date being automatically adjusted, filter functions defined right there in conf.py, and I will miss that. No big deal though.

@Kwpolska

This comment has been minimized.

Copy link
Member

Kwpolska commented Oct 1, 2018

@kayhayen We’ve already found something that works (it’s in the first post in this thread): https://pypi.org/project/ruamel.yaml/0.6/

round trip mode that includes comments (block mode, key ordering kept)

As for the v8 migration, the route you intend to take is a fairly good one — don’t forget to read the blog post though: https://getnikola.com/blog/upgrading-to-nikola-v8.html .

We attempt to mention all config changes in CHANGES.txt, but we might forget about something along the way.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.