add builtins.toTOML

[chessai]

Request

We already have builtins.fromTOML, it would be useful to have builtins.toTOML as well.

Ideal solution

Implement builtins.toTOML.

Alternatives I've considered

This snippet by @nprindle, which I'm not sure is 100% correct but seems to work:

  toTOML =
    let
      # Escape a TOML key; if it is a string that's a valid identifier, we don't
      # need to add quotes
      tomlEscapeKey = val:
        # Identifier regex taken from https://toml.io/en/v1.0.0-rc.1#keyvalue-pair
        if builtins.isString val && builtins.match "[A-Za-z0-9_-]+" val != null
          then val
          else toJSON val;

      # Escape a TOML value
      tomlEscapeValue = toJSON;

      # Render a TOML value that appears on the right hand side of an equals
      tomlValue = v:
        if builtins.isList v
          then "[${string.concatMapSep ", " tomlValue v}]"
        else if builtins.isAttrs v
          then "{${string.concatMapSep ", " ({ name, value }: tomlKV name value) (set.toList v)}}"
        else tomlEscapeValue v;

      # Render an inline TOML "key = value" pair
      tomlKV = k: v: "${tomlEscapeKey k} = ${tomlValue v}";

      # Turn a prefix like [ "foo" "bar" ] into an escaped header value like
      # "foo.bar"
      dots = string.concatMapSep "." tomlEscapeKey;

      # Render a TOML table with a header
      tomlTable = oldPrefix: k: v:
        let
          prefix = oldPrefix ++ [k];
          rest = go prefix v;
        in "[${dots prefix}]" + string.optional (rest != "") "\n${rest}";

      # Render a TOML array of attrsets using [[]] notation. 'subtables' should
      # be a list of attrsets.
      tomlTableArray = oldPrefix: k: subtables:
        let prefix = oldPrefix ++ [k];
        in string.concatMapSep "\n\n" (v:
          let rest = go prefix v;
          in "[[${dots prefix}]]" + string.optional (rest != "") "\n${rest}") subtables;

      # Wrap a string in a list, yielding the empty list if the string is empty
      optionalNonempty = str: list.optional (str != "") str;

      # Render an attrset into TOML; when nested, 'prefix' will be a list of the
      # keys we're currently in
      go = prefix: attrs:
        let
          attrList = set.toList attrs;

          # Render values that are objects using tables
          tableSplit = list.partition ({ value, ... }: builtins.isAttrs value) attrList;
          tablesToml = string.concatMapSep "\n\n"
            ({ name, value }: tomlTable prefix name value)
            tableSplit._0;

          # Use [[]] syntax only on arrays of attrsets
          tableArraySplit = list.partition
            ({ value, ... }: builtins.isList value && value != [] && list.all builtins.isAttrs value)
            tableSplit._1;
          tableArraysToml = string.concatMapSep "\n\n"
            ({ name, value }: tomlTableArray prefix name value)
            tableArraySplit._0;

          # Everything else becomes bare "key = value" pairs
          pairsToml = string.concatMapSep "\n" ({ name, value }: tomlKV name value) tableArraySplit._1;
        in string.concatSep "\n\n" (list.concatMap optionalNonempty [
          pairsToml
          tablesToml
          tableArraysToml
        ]);
    in go [];

[stale]

I marked this as stale due to inactivity. → More info

[expipiplus1]

@chessai Where are all those string/set/list functions such as string.concatSep coming from? they don't seem to exist in nixpkgs's lib.

[chessai]

@chessai Where are all those string/set/list functions such as string.concatSep coming from? they don't seem to exist in nixpkgs's lib.

From https://github.com/chessai/nix-std

[stale]

I marked this as stale due to inactivity. → More info

[ghost]

I wish we had this.

buildRustPackage reads in a Cargo.toml. I find myself needing to modify that file (in nix) and write the modified Cargo.toml back out again.

[dzmitry-lahoda]

there are 2 solutions to toml

1. render json and yq it into toml, minus - you cannot read markup, and then generate markup and then read it again, in strict mode of evaliaiton
2. render via nixpkgs.lib.toToml when function provided here. it makes sure scenario from 1 works.

so there is no need to be build int.

why fromJson and fromToml built in?

because it allows for strict evaliation. so it can allow to interpret any local file of config (json, yaml, toml) to be just nix.

so making it nixpkgs.lib.toml.toToml seems more right.

so may be move issue to nixpkgs repo, not nix

[jkachmar]

+1 to this; Nix already depends on toml11 to provide builtins.fromTOML, and that library appears to support encoding to TOML as well.

with this dependency in place, it seems like a clear win to make this a builtin rather than a library function.

[Aleksanaa]

Any work on this?

[fricklerhandwerk]

Triaged in the Nix team meeting:

This must be guarded by an experimental feature to be merged. We need more time to make sure the round-trip works, as we'll have to commit to the output format forever. We have to consider if we will keep it, because it's a liability for reproducibility.

@roberth will take over review.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-06-02-nix-team-meeting-minutes-59/28666/1

[roberth]

We've decided not to hardcode a toTOML into Nix, because it would be a disproportionate liability when it comes to long term reproducibility of evaluation. By relying on a library, we are subjected to all changes in its output format, which unlike the parser has many degrees of freedom. Such changes to the toml library would be acceptable to normal library consumers who only care about the meaning of the generated file, but for Nix this is not acceptable, because the output of evaluation must be byte for byte reproducible, as it affects derivation hashes.

I'm sorry to disappoint y'all for now, but there's a way forward.

• pkgs.formats.toml may already cover your needs, if you're using the TOML file in a built configuration file (ie one that's ok to have in the store)
• otherwise, I would recommend to write a pure-nix implementation of the toTOML function in the Nixpkgs library; in case you need it to be pure nix.

[chessai]

For anyone's information: There is already a toTOML function in https://github.com/chessai/nix-std (which is why I opened this issue)