redisTestHook,memcachedTestHook: init; prevent hanging Darwin build after test failures

[ofalvai]

Looking at Hydra's "timed out jobs" list, I noticed many Darwin timeouts where the last log lines were all about Redis. It turns out the background Redis job is not terminated correctly on macOS when checkPhase fails (thus postCheck never runs).

I reviewed all Redis test usage in pkgs and added the same workaround, as well as connection retry where it was missing.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 25.05 Release Notes (or backporting 24.11 and 25.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[afh]

It seems this PR adds almost the same code to the preCheck phase of various packages. Would it be possible and feasible to refactor that code into a script that can be re-used by these packages?

[ofalvai]

I was thinking the same, but I'm not familiar enough with nixpkgs to know how to approach this. Do you have a specific idea in mind? Maybe a hook?

[toonn]

I asked in the NixOS dev room and emily suggested a hook indeed. And in addition to mark the failing packages as broken or badPlatforms.

[mweinelt]

Create a hook to integrate redis into the check phase.

Check out pkgs/by-name/po/postgresqlTestHook which has a similar use case.

[toonn]

Emily also mentioned the following:

I'm also not sure that MAX_RETRIES stuff is a good idea since Hydra builders are often under heavy load and test timeouts usually just break things unnecessarily

I sort of suspect fixing the underlying issue is just a matter of putting a nohup in front of the server call

I'm not sure nohup would prevent the process being assigned to launchd but worth looking into I suppose.

[ofalvai]

Thank you for the review @mweinelt. The easier ones are fixed now. Next, I'll figure out the unix domain socket thing and then resolve the merge conflict.

[ofalvai]

@mweinelt Redis socket support is now ready. However, Memcached works with either a network connection or a socket, it doesn't accept connections on both: https://docs.memcached.org/serverguide/configuring/#unix-sockets

Do you think it would still be useful to add a socket-only mode to memcachedTestHook?

[mweinelt]

We can probably revisit UDS later.

[mweinelt]

Looks good. Just some minor things left.

[mweinelt]

Suggested change

	cli = "${redis}/bin/redis-cli";
	server = "${redis}/bin/redis-server";
	cli = lib.getExe' redis "redis-cli";
	server = lib.getExe' redis "redis-server";

[mweinelt]

Suggested change

	memcached = "${lib.getExe memcached}";
	nc = "${lib.getExe netcat}";
	memcached = lib.getExe memcached;
	nc = lib.getExe netcat;

[mweinelt]

redis-cli seems to be a bit noisy printing connection errors, while waiting for redis-server to start

django-cacheops> starting redis
django-cacheops> waiting for redis to be ready
django-cacheops> Could not connect to Redis at /build/run/redis.sock: No such file or directory

[ofalvai]

Do you have a better idea than 2>/dev/null? I'm afraid it would make eventual debugging a nightmare. Maybe we should sleep 1 first once?

[mweinelt]

We could check whether the socket exists (test -s "$REDIS_SOCKET") or the listener has bound to the port (nc -z localhost "$redisTestPort") before interrogating it with redis-cli. 🤔

[mweinelt]

And please check if you have removed unused redis-server arguments from these packages. Found one in rq for example.

[mweinelt]

Eval failure in python3.pkgs.rq.

[mweinelt]

Thank you!