python3Packages.watchdog: fix build

[oxij]

The newly disabled test fails on my machine.

The other change is to make syntax highlight happy.

[oxij]

Hydra thinks it's fine https://hydra.nixos.org/build/292199074 so switched this to be against staging.

I have no idea why that test works on Hydra but fails locally, but it does.

[FliegendeWurst]

Maybe some invocation of ulimit could increase the max number of open files?

[oxij]

My build machine runs with default `security.pam.loginLimits` and similar, so it is limited by whatever `nixbld` user is allowed to have by default (which, for me, `sudo --user nixbld1 /bin/sh -c 'ulimit -n'` says, is 1024). That test tries to open a file 2048 times in a row, so it will always fail with the default `ulimit`s on my machine. I guess it could be set to a higher value on Hydra or something. Or maybe newer kernels set the default `ulimit -n` higher?

[FliegendeWurst]

Try preCheck = "ulimit -n $(ulimit -Hn)". This will raise the max. number of open files to the "hard limit", which should be higher than 1024.

[mweinelt]

Are you building in single user mode by chance? Hydra uses the nix-daemon and inherits its increased ulimit.

[oxij]

@mweinelt I do, yes.

I pushed a new commit raising ulimit -n to 2x of what that test needs, similar to what @FliegendeWurst suggested, and it now passes.

[FliegendeWurst]

Do the other tests open at most 2048 files?

[oxij]

Well, it's actually a bit more than 2048, since pytest itself opens files too, but yes, it's the only test that opens that many files, AFAICS.

[oxij]

ping?

[mweinelt]

I'm not super convinced we should manually manage ulimits like that in a derivation.

This is absolutely your environment putting constraints on the build sandbox. Can't you simply lift the fd limits or use NIX_REMOTE=daemon instead?

[oxij]

I'm not super convinced we should manually manage ulimits like that in a derivation.

Without this commit applied: ``` $ grep -r 'ulimit -n' pkgs | wc -l 16 ``` Most of those are also in `preCheck`s, so fixes like this is quite common. In fact, this is the most common usage of `ulimit` calls in Nixpkgs overall.

This is absolutely your environment putting constraints on the build sandbox. Can't you simply lift the fd limits or use `NIX_REMOTE=daemon` instead?

I could, but I don't think I should be lifting limits I did not add. My environment is the default environment of all non-NixOS systems and when using Nix from under root on NixOS. Requiring such changes would imply that, on any similar system (which is probably most of them), when this packages is not available from Hydra, or when using Nix with substituters disabled, a huge chunk of Nixpkgs will become unbuildable without adding special sauce lines to `limits.conf` and rebooting first. I would not expect such to be the default behaviour.

[emilazy]

This isn’t the appropriate place to work around this issue. The existing instances of the pattern in Nixpkgs are basically all there to work around an issue with unreasonably low macOS limits and Hydra not using the daemon that was resolved at the NixOS infra level; see #351783 for what led to that investigation. (I should rebase that PR to drop the existing instances at some point.)

Nix already raises the limit for the daemon in its shipped systemd service and launchd daemon definitions. This issue with single‐user mode should be addressed in Nix by having it raise the limit to match the one it ships for the daemon, to increase uniformity of the build environment. Failing that, I would recommend just adjusting your system configuration. If we do work around the Nix bug, it should be in stdenv, not individual packages.