pkgs.formats.toml: fix TOML semantics by upgrading tomlkit

[luizribeiro]

Description of changes

This fixes #237521 by upgrading tomlkit to 0.11.8.

It also introduces tests to make sure this behavior is kept, as some of our configurations (such as telegraf) depend on it.

Tested these changes by running nix build .\#tests.pkgs-lib.formats.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 23.11 Release Notes (or backporting 23.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

[infinisil]

Also, does this need to be backported?

[infinisil]

Downgrading packages should be exceptional: Is the new behavior a bug or a feature?

• If it's a feature we should either
  • Introduce a new tomlkit_11_4 attribute and use that, or
  • Define the older version internally to the pkgs.formats code (this way we don't have a new attribute to maintain)
• If it's a bug we should add a comment here saying that we're waiting for the bug to get fixed before upgrading.

[luizribeiro]

You're absolutely right :)

It turns out that tomlkit 0.11.8 (the latest version) doesn't have this bug either. So doing an upgrade and introducing the tests seems way more appropriate.

The challenge is that tomlkit is now built with poetry and.... poetry also depends on tomlkit.

So I think until we bootstrap poetry itself, we need to keep 0.11.6 as tomlkit_11_6, similar to your suggestion.

I'll make those changes. Hopefully I don't mess it up :-)

Thanks for the quick review, btw!

[luizribeiro]

Not sure if this is the cleanest way to go about this, so I'm open to suggestions.

Also, does this need to be backported?

We need to have it merged into the 23.05 release, if that's what you are asking (sorry, I'm not yet very familiar with the nixpkgs release process / terminology).

[luizribeiro]

Alright I'm calling it a day on this for now. Thanks for the help and sorry for all the churn :)

I've added a commit to my branch which has option 3 implemented that I mentioned above (a patch which makes tomlkit be built with setuptools instead of poetry).

I'll squash / drop those commits depending on the solution we go with before we merge this in.

[luizribeiro]

After discussing this upstream on python-poetry/tomlkit#305, it turns out that poetry-core should not depend on tomlkit. poetry-core has no dependencies on other libraries and should be used to build tomlkit instead of poetry.

I've updated this PR accordingly.

[infinisil]

Nice! Could you split this into separate commits like this?

1. Remove poetry dependency on tomlkit and set `format = "pyproject" on tomlkit
2. Update tomlkit to 0.11.8
3. Introduce the test case

[infinisil]

Awesome thanks!

[mweinelt]

f689956 broke pylint and by extension all nixos tests.

[vcunat]

Note that this PR was tagged by OfBorg as very large rebuild and thus it should not have been merged to master.

[infinisil]

Ah sorry I forgot to pay attention the label. Here's a Hydra failure for reference, also fails on x86_64-linux. After some quick error searching it looks like this commit in tomlkit introduced that error, and it's only included in 0.11.8, not 0.11.7. I can confirm that only upgrading to 0.11.7 fixes the pylint build.

[infinisil]

Looks like pylint is aware of this problem: pylint-dev/pylint#8633

[mweinelt]

Picked into #244135.