config: add support for formatted keys

[alexfikl]

This adds support for entries like these in the configuration file (see discussion in #662):

[settings]
add-file-name = {doc[year]}-{doc[author]}-{doc[title]}
ref-format.jinja2 = {{ doc.author_list|slice(3)|join("", attribute="family") }}{{ doc.year }}

where the first setting uses the default formatter (whatever formatter is set to) and the second setting overwrites the default and forces the jinja2 formatter. In general, key[.formatter] is how this looks like.

For this, it adds some functionality:

• papis.config.getformattedstring: this looks thorough the config file for any version of key.[formatter] and returns the last one it finds.
• papis.strings.FormattedString: a class that's just a (formatter, string) tuple and gets passed to papis.format.format to allow picking the formatter for each individual string. This is mostly inspired by prompt_toolkit.FormattedText.
• papis.cli.FormattedStringParamType: a click.ParamType that automatically converts command-line parameters to FormattedString. It also shows a nice text --file-name FORMATTED-TEXT in the command line to mark entries that can be format strings.
• Update the rest of the code to use the above where appropriate!

TODO:

• Need to check what other keys should use formatted strings.
• Not sure the folder name construction in papis.commands.add::run works correctly with formatted strings. Need to add some tests for that.
• Probably needs some more testing to make sure nothing is broken. (Currently using this daily, so hopefully will catch some things)

Fixes #662.

[hermlon]

I just tried your fork because I need this feature, but I somehow can't start the ref-format with a lowercase letter. I'm using this Jinja format string:

ref-format.jinja2 = {{ doc.author_list[0]['family']|lower }}{{ doc.year }}

But I get the ref: Sanderson12

I guess there is some code upper-casing the ref-format again, since the Jinja2 documentation clearly states that the lower filter lowers all the characters.

[alexfikl]

I just tried your fork because I need this feature, but I somehow can't start the ref-format with a lowercase letter. I'm using this Jinja format string:

ref-format.jinja2 = {{ doc.author_list[0]['family']|lower }}{{ doc.year }}

@hermlon Can confirm that this doesn't work as expected. I'll look into it!
Thanks a lot for testing ❤️ this touches a lot of parts, so needs quite a bit more testing besides the unit tests..

[alexfikl]

@hermlon Turns out my case was just a config error. I added a test for the case you mentioned (I think), i.e. setting a ref-format.jinja2, and it seems to work as expected.

Can you give more details about your setup? Do other things work, e.g. does setting ref-format.jinja2 = {{ doc.year }} give the expected change in the ref? Can you also set ref-format in the config to see if it's picking that one by mistake?

EDIT: Gah, nevermind, that was another issue with my setup. Yes, your format didn't work because we do apparently capitalize things by default here:

papis/papis/bibtex.py

Lines 411 to 416 in c1a9d91

	return string.capwords(str(slugify.slugify(
	ref,
	lowercase=False,
	word_boundary=False,
	separator=" ",
	regex_pattern=allowed_characters))).replace(" ", "")

That doesn't sound like a good idea, so we can make a PR to revert it. Thanks for pointing that out and sorry for the confusion!

[OopsYao]

I just tried your fork because I need this feature, but I somehow can't start the ref-format with a lowercase letter. I'm using this Jinja format string:

ref-format.jinja2 = {{ doc.author_list[0]['family']|lower }}{{ doc.year }}

But I get the ref: Sanderson12

I guess there is some code upper-casing the ref-format again, since the Jinja2 documentation clearly states that the lower filter lowers all the characters.

I tried the commit 126825c; it works well, no abnormalities found yet. 👏 One thing is that something like papis update --set ref.jinja2 [ref-format] is not supported currently. Don't know if this needs extra heavy work.

[alexfikl]

I tried the commit 126825c; it works well, no abnormalities found yet. 👏

Awesome! Thank for you testing!

One thing is that something like papis update --set ref.jinja2 [ref-format] is not supported currently. Don't know if this needs extra heavy work.

Hm, it shouldn't be too hard to implement. I'll look into it! There are a few more places that take values that should be formatted from the command line, so we need some nice solution for them.

EDIT: I've looked a bit at this and there's two issues here:

• For the --set key value option that a few of the commands take, we can just use key[.formatter] and select the formatter that way. This shouldn't be hard to implement.
• Other arguments, e.g. papis add --file-name [FORMAT] ..., don't really have a way to specify the formatter. I'm not sure what a good way to do that would be. We can always recommend using

papis --set add-file-name.jinja2 [FORMAT] add ...

instead, but not sure that's very nice..

[alexfikl]

@OopsYao The latest version should also support papis update --set ref.jinja2 [format]. Let me know if you try it and something doesn't work (I just very briefly tested it)!

[kiike]

I have been trying this feature for a couple of weeks and I really like it!

I think the Jinja2 formatting is a great addition, and the formatted keys give a lot of flexibility.

While the config is very compact, I find it unusual. Maybe I would've expected:

add-folder.format = {{doc.author}}
add-folder.formatter = jinja2

Thinking about it, this all adds complexity, while settling for Jinja2 as default and only formatter would simplify the code. The default Python formatter is simpler, but if we have a well-known syntax, Jinja2, which is more powerful, could that be the only option?

[alexfikl]

I have been trying this feature for a couple of weeks and I really like it!

@kiike Thank you for trying this out! ❤️

While the config is very compact, I find it unusual. Maybe I would've expected:

add-folder.format = {{doc.author}}
add-folder.formatter = jinja2

Why is this better? I find it a bit repetitive for multiple options, e.g.

header-format.format = ...
header-format.formatter = jinja2
add-file-name.format = ...
add-file-name.formatter = jinja2
add-folder.format = ...
add-folder.formatter = jinja2

and annoyingly allows these to be set apart, e.g.

header-format.format = ...
picktool = ...
opentool = ...
add-confirm = ...
header-format.formatter = jinja2

Thinking about it, this all adds complexity, while settling for Jinja2 as default and only formatter would simplify the code. The default Python formatter is simpler, but if we have a well-known syntax, Jinja2, which is more powerful, could that be the only option?

I'm not sure I agree with this.

First, the jinja2 formatter has been available for a long time and you can already switch to it if it feels more natural. The main issue with that was just that all the default values in Papis are defined for the Python formatter, so it's a bit of a drag to redefine all of them.

I also don't think removing the plugins infrastructure for the formatters and having the jinja2 one as the only implementation is likely to happen. At least I have some custom formatters used to hook into formatting references and file names that would be quite annoying to do otherwise..

[kiike]

I have been trying this feature for a couple of weeks and I really like it!

@kiike Thank you for trying this out! ❤️

While the config is very compact, I find it unusual. Maybe I would've expected:

add-folder.format = {{doc.author}}
add-folder.formatter = jinja2

Why is this better? I find it a bit repetitive for multiple options, e.g.

header-format.format = ...
header-format.formatter = jinja2
add-file-name.format = ...
add-file-name.formatter = jinja2
add-folder.format = ...
add-folder.formatter = jinja2

You're right. It gets tedious easily.

I'm not sure I agree with this.

First, the jinja2 formatter has been available for a long time and you can already switch to it if it feels more natural. The main issue with that was just that all the default values in Papis are defined for the Python formatter, so it's a bit of a drag to redefine all of them.

Yes. Having to define manually all of them from scratch is quite a drag. Plus it's not straightforward to see the changes without being put off by all the broken strings, or exceptions, and errors and warnings.

I also don't think removing the plugins infrastructure for the formatters and having the jinja2 one as the only implementation is likely to happen. At least I have some custom formatters used to hook into formatting references and file names that would be quite annoying to do otherwise..

Makes sense. Especially in your case, with the custom formatters.

Thank you for explaining this!

[alexfikl]

Yes. Having to define manually all of them from scratch is quite a drag. Plus it's not straightforward to see the changes without being put off by all the broken strings, or exceptions, and errors and warnings.

Yeah, not quite sure how to solve that in a nice way. That was mostly what this is trying to work around: instead of rewriting all the config settings using e.g. jinja2, you can just rewrite the ones where you need a bit more power.

There may be something to be said about switching to jinja2 as the default though. Especially if this goes in, we can probably do that in a backwards compatible way (or at least will less pain).