nixos/dovecot: Add proper escaping and support for more value types to plugin settings

[ntninja]

Description of changes

The added escaping functions cover the entire range of Dovecot supported config value constructs other than string-includes and variable references (neither of which has an equivalent in Nix) and are intended to be reusable for further structed configuration enhancements. Support for string-includes and variable references can be re-added (compared to the status quo where they happen to work due to lack of escaping) when they are needed.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[ntninja]

Sorry for bumping this, but it would be really great if this was merged before 24.05 (since it’d be a backward-compatibility hazard afterwards)

@2xsaiko (name in maintainers appears to be dated, btw)

[2xsaiko]

Very cool, thanks!

(I didn't see this until now. Maybe I have to additionally add myself to the maintainers file to get reliably notified about stuff I maintain.)

(name in maintainers appears to be dated, btw)

Hm? It's not.

[2xsaiko]

Suggested change

	abort "mkDovecotValue: value not supported: ${toPretty {} value}"
	throw "mkDovecotValue: value not supported: ${toPretty {} value}"

[2xsaiko]

Actually, abort works too, I suppose. I just see throw usually, so use whichever you want.

[ntninja]

Think I found abort in the docs somewhere, seen throw elsewhere too, I’ll change it.

[2xsaiko]

Suggested change

	in if match "[[:space:]]*<.*|.*[[:space:]#\"\\].*|" string != null
	in if match ''[[:space:]]*<.*|.*[[:space:]#"\].*|'' string != null

so you can copy it verbatim

[2xsaiko]

With this, you'll have to pass the list as-is to pluginSettings for sieve.plugins, sieve.extensions and sieve.globalExtensions, otherwise it will quote the values:

plugin {
  [...]
  sieve_extensions = "+notify +imapflags +vnd.dovecot.filter"
  sieve_global_extensions = "+vnd.dovecot.environment +vnd.dovecot.pipe"
  sieve_pipe_bin_dir = /nix/store/9g0l06qac224hd2m0ixj0mhf25mh0662-sieve-pipe-bins
  sieve_plugins = "sieve_imapsieve sieve_extprograms"
} 

[ntninja]

Is this about having to write

services.dovecot = {
    pluginSettings = {
        sieve_extensions = ["+notify" "+imapflags" "+vnd.dovecot.filter"];
    };
};

vs

services.dovecot = {
    pluginSettings = {
        sieve_extensions = "+notify +imapflags +vnd.dovecot.filter";
    };
};

?

If so, that is intended behaviour since the former denotes a list of strings (which this is), while the latter denotes a since string containing spaces. (Having it declared as a list also means others can later use mkBefore/mkAfter in their own configs to easily extend it without conflicts or mkForce.)

Dovecot isn’t very strict about these type differences in practice, so both of the above actually work. On the hand, a configuration value of sieve_extensions = "+notify" "+imapflags" "+vnd.dovecot.filter" doesn’t work, even though the documentation says it should. As far as I can tell, all of those “should work but doesn’t” cases boil down to being space-separated words which the regex mentioned in the previous comment gracefully handles by making sure the quoting is only applied when it is actually needed due to the contents containing “special characters” (spaces, quotation marks, backslashes, file redirects, comment signs, empty strings). I actually have an extended version of what I’m proposing here (uses the same code also for global assignments, mailboxes, services and service listeners) in use and as long as one follows the only-quote-when-needed rule, even complex configurations are successfully generated and parsed by Dovecot with this.

[2xsaiko]

No, this is about the settings that I mentioned setting your second variant instead of the first. Even if it works (tbh I didn't assume it would) I think it would be good to change it.

[2xsaiko]

I guess with this at least sieve.plugins could be removed since it just passes the value through directly.
If you do that though then I'd make pluginSettings a submodule with freeformType like here so the description and example can be kept.

[ntninja]

Updated to replace sieve.plugins, sieve.extensions & sieve.globalExtensions with submodule declarations. I removed the transitional aids mentioned below again after realizing that those options never had been in NixOS stable.

    (mkRenamedOptionModule [ "services" "dovecot2" "sieve" "plugins" ] [ "services" "dovecot2" "pluginSettings" "sieve_plugins" ])
    (mkChangedOptionModule [ "services" "dovecot2" "sieve" "extensions" ] [ "services" "dovecot2" "pluginsSettings" "sieve_extensions" ]
      (config: map (el: "+${el}") config.services.dovecot2.sieve.extensions))
    (mkChangedOptionModule [ "services" "dovecot2" "sieve" "globalExtensions" ] [ "services" "dovecot2" "pluginsSettings" "sieve_global_extensions" ]
      (config: map (el: "+${el}") config.services.dovecot2.sieve.globalExtensions))

    sieve = {
      extensions = mkOption {
        default = [];
        description = "Deprecated in favour of {config}`services.dovecot2.pluginSettings.sieve_extensions` – Extra Sieve extensions to enable for user scripts";
        example = [ "notify" "imapflags" "vnd.dovecot.filter" ];
        type = types.listOf types.str;
      };

      globalExtensions = mkOption {
        default = [];
        example = [ "vnd.dovecot.environment" ];
        description = "Deprecated in favour of {config}`services.dovecot2.pluginSettings.sieve_global_extensions` – Extra Sieve extensions to enable for global scripts";
        type = types.listOf types.str;
      };
    };

[2xsaiko]

Suggested change

	default = sievePipeBinScriptDirectory;
	default = null;

This should be set as part of config if sieve.pipeBins is non-empty so it doesn't get silently overwritten.

[2xsaiko]

Suggested change

	pluginSettings = (lib.mapAttrs (n: lib.mkDefault) (
	sieveScriptSettings // imapSieveMailboxSettings
	)) // (lib.mapAttrs (n: lib.mkBefore) (filterAttrs (n: v: v != []) {
	sieve_global_extensions = (
	optional (cfg.sieve.pipeBins != []) "+vnd.dovecot.pipe"
	);
	sieve_plugins = (
	optional (cfg.imapsieve.mailbox != []) "sieve_imapsieve"
	++ optional (cfg.sieve.pipeBins != []) "sieve_extprograms"
	);
	}));
	pluginSettings = mkMerge [
	sieveScriptSettings
	imapSieveMailboxSettings
	(mkIf (cfg.sieve.pipeBins != []) {
	sieve_plugins = [ "sieve_extprograms" ];
	sieve_global_extensions = [ "+vnd.dovecot.pipe" ];
	})
	(mkIf (cfg.imapsieve.mailbox != []) {
	sieve_plugins = [ "sieve_imapsieve" ];
	})
	];

This is getting a bit messy. Also that mkDefault shouldn't be needed anymore since both sieveScriptSettings and imapSieveMailboxSettings will be empty until the user sets the corresponding options (and conflicts should error for these).

[ntninja]

This is a cool trick! 👍

[ntninja]

Haven’t had time to test it yet sadly but I think I applied all the changes you requested. I also omitted the default = null; from the declared options again since I don’t think that was needed in the first place – I’d keep the facility to explicitly specify options as null to omit them from the generated Dovecot configuration, since it allows users to explicitly unset options that would otherwise be set by some NixOS module.

[ntninja]

Tested it now and appears to work! Turns out the default = null; is still required for making evaluation happy. Also added some more docs on pluginSettings, otherwise no changes compared to yesterday.

[2xsaiko]

Suggested change

	in if match ''[[:space:]]*<.*|.*[[:space:]#"\].*|'' string != null
	in if match ''[[:space:]]*<.*|.*[[:space:]].*|'' string != null

Actually, this regexp is invalid, I didn't check that before. Is this maybe what it should be?

[ntninja]

No, the regexp works just fine.

The following Nix configuration:

{
    # …
    services.dovecot2.pluginSettings = {
         test_backslash = "\\";                    # → "\\\\"
         test_comment = "a#b";                     # → "a#b"
         test_empty = "";                          # → ""
         test_multiline = " a a \n  b b \n c c ";  # → " a a  \\\n  b b  \\\n c c "
         test_multiline_trailing = "a\nb\nc\n\n";  # → "a \\\nb \\\nc"
         test_quote = "\"";                        # → "\\""
         test_space = " ";                         # → " "
         test_unquoted = "abc";                    # → abc
    };
}

Results, using the submitted regexp, in the following Dovecot configuration:

plugin {
    test_backslash = "\\"
    test_comment = "a#b"
    test_empty = ""
    test_multiline = " a a  \
      b b  \
     c c "
    test_multiline_trailing = "a \
    b \
    c"
    test_quote = "\""
    test_space = " "
    test_unquoted = abc
}

The condition if match ''[[:space:]]*<.*|.*[[:space:]#"\\].*|^$'' string != null also appears to work.

[2xsaiko]

Oh, I guess this hits NixOS/nix#1537. It fails to evaluate on my Mac.

error: invalid regular expression '[[:space:]]*<.*|.*[[:space:]#"\].*|'

The | needs to be escaped for it to work everywhere:

Suggested change

	in if match ''[[:space:]]*<.*|.*[[:space:]#"\].*|'' string != null
	in if match ''[[:space:]]*<.*\|.*[[:space:]#"\].*\|'' string != null

edit: Oh wait, no, those are alternatives as in (a|b). Didn't know that worked without the parentheses.

[ntninja]

I checked the linked issue but the mentioned problem does not appear to be present inside a Darling (macOS on Linux emulator) container, so I’m unable to reproduce this macOS-only behaviour locally. The pattern is supposed to say “starts with < (ignoring whitespace) OR contains at least one whitespace, #, " or \ character OR is empty” in any case.

I’ve updated the regexp to something that should hopefully work on macOS too – if it’s just the empty variant it dislikes. Please also check if the test vector I posted produces the same results: #286184 (comment) (there actually was a bug that single-lined strings containing only whitespace would also be trimmed).

[2xsaiko]

Great, looks like that works! The test cases give the same output here.

I thought maybe you could get the same behavior by overriding nix to use clangStdenv but apparently not. Well, doesn't matter now anyway.

[ntninja]

@RaitoBezarius @2xsaiko Sorry for bumping this, but can this be merged for 24.05 since it’d be a break change afterwards?

[2xsaiko]

I'm sorry, I can't do anything here since I can't merge PRs. Others who might be interested in reviewing this: @GaetanLepage @nlewo (since you recently touched dovecot in simple-nixos-mailserver, but feel free to ignore)

Otherwise, try posting in https://discourse.nixos.org/t/prs-ready-for-review/3032.

[RaitoBezarius]

Since the last widespread breakage, I'm unfortunately a bit risk averse on large scale changes on Dovecot2, I'd appreciate more testing in this PR, maybe, including things that try to reproduce what SNM is trying to do.

[nlewo]

I submitted a NixOS Mailserver MR to test this MR:

• MR: https://gitlab.com/simple-nixos-mailserver/nixos-mailserver/-/merge_requests/324
• Hydra jobset: https://hydra.nix-community.org/jobset/simple-nixos-mailserver/324#tabs-evaluations

Tests are currently failing because of:

error: The option `nodes.server.services.dovecot2.sieve.extensions' does not exist.

@ntninja Could you provide the migration path?

[2xsaiko]

It should be services.dovecot2.sieve.extensions = ["foo" "bar" "baz"]; -> services.dovecot2.pluginSettings.sieve_extensions = ["+foo" "+bar" "+baz"];.

[ntninja]

@nlewo There used to be a migration path for these options initially, but I removed it after realizing that those options never have been in NixOS stable. (cf #286184 (comment))

You are attempting what RaitoBezarius asked for there, right? (I hadn’t heard of SNM before, these changes – and some – are for Modoboa, which I’ve privately packaged.)