WIP: Add garbage collector to docker registry #1

ironpinguin · 2018-04-06T13:17:00Z

Motivation for this change

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

Ma27 · 2018-04-06T13:52:13Z

nixos/modules/services/misc/docker-registry.nix

+
+      script = ''
+        ${pkgs.docker-distribution}/bin/registry garbage-collect ${configFile}
+        ${lib.optionalString enableRedisCache '${pkgs.systemd}/bin/systemctl restart docker-registry'}


single quotes won't work as single quotes can be part of variable names.
Use '' for multiline strings or " for single line strings and you're safe %)

- 1. delete uploaded manifest - 2. run gc to remove all depending blobs from filesystem (and restart registry) - 3. check if blobs are deleted - 4. push again the image - 5. check if blobs exists in filesystem (means that the registry was correct restarted)

Ma27

Thanks a lot for your change! I just found some minor aspects to dscuss, can test it tonight, when I'm at home :)

Ma27 · 2018-04-06T16:44:53Z

nixos/modules/services/misc/docker-registry.nix

+          <manvolnum>7</manvolnum></citerefentry>) of the time at
+          which the garbage collect will occur.
+        ''
+      };


the indent and the additional closing bracket are obviously broken %)

Ma27 · 2018-04-06T16:47:08Z

nixos/modules/services/misc/docker-registry.nix

@@ -95,16 +97,32 @@ in {
      default = {};
      type = types.attrsOf types.str;
    };
+
+    enableGarbageCollect = mkOption {


you might want to have a look at mkEnableOption

Ma27 · 2018-04-06T16:54:36Z

nixos/tests/docker-registry.nix

@@ -43,7 +43,15 @@ import ./make-test.nix ({ pkgs, ...} : {
    $client2->succeed("docker images | grep scratch");

    $client2->succeed(
-      'curl -fsS -X DELETE registry:8080/v2/scratch/manifests/$(curl registry:8080/v2/scratch/manifests/latest | jq ".fsLayers[0].blobSum" | sed -e \'s/"//g\')'
+      'curl -fsS -X DELETE registry:8080/v2/scratch/manifests/$(curl -fsS -I -H"Accept: application/vnd.docker.distribution.manifest.v2+json" registry:8080/v2/scratch/manifests/latest | grep Docker-Content | sed -e\'Docker-Content-Digest: //\' | tr -d \'\r\')'


I don't remember exactly how the JSON looked like, however I think that a solution with jq might be way easier to read

The json looks now different with the Header Accept: application/vnd.docker.distribution.manifest.v2+json

The correct digest is not part of the json but of the HTTP Header. And also it is only the correct one with the given Header :-(

This is a example to see it better:
curl -fsS -i -H'Accept: application/vnd.docker.distribution.manifest.v2+json' localhost:5000/v2/scratch/manifests/latest

HTTP/1.1 200 OK Content-Length: 522 Content-Type: application/vnd.docker.distribution.manifest.v2+json Docker-Content-Digest: sha256:0eec87df34024c2c53db0faae043ab2175902c5877a488eb9053413e03f32661 Docker-Distribution-Api-Version: registry/2.0 Etag: "sha256:0eec87df34024c2c53db0faae043ab2175902c5877a488eb9053413e03f32661" X-Content-Type-Options: nosniff Date: Sat, 07 Apr 2018 09:36:57 GMT { "schemaVersion": 2, "mediaType": "application/vnd.docker.distribution.manifest.v2+json", "config": { "mediaType": "application/vnd.docker.container.image.v1+json", "size": 887, "digest": "sha256:b8edfce180eb3b092c8248c81f8c208b6f835308a2041d11cd2389d6741035ad" }, "layers": [ { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 49, "digest": "sha256:3c2cba919283a210665e480bcbf943eaaf4ed87a83f02e81bb286b8bdead0e75" } ] }

All 3 digests are also blobs:

sha256:0eec87df34024c2c53db0faae043ab2175902c5877a488eb9053413e03f32661 contains the images meta information

sha256:b8edfce180eb3b092c8248c81f8c208b6f835308a2041d11cd2389d6741035ad
contains the container config as json

sha256:3c2cba919283a210665e480bcbf943eaaf4ed87a83f02e81bb286b8bdead0e75 is the first root fs layer referenced by both other blobs json's

- Also change enableRedisCache to mkEnableOption

ironpinguin

Done

ironpinguin · 2018-04-07T07:49:50Z

nixos/tests/docker-registry.nix

@@ -43,7 +43,15 @@ import ./make-test.nix ({ pkgs, ...} : {
    $client2->succeed("docker images | grep scratch");

    $client2->succeed(
-      'curl -fsS -X DELETE registry:8080/v2/scratch/manifests/$(curl registry:8080/v2/scratch/manifests/latest | jq ".fsLayers[0].blobSum" | sed -e \'s/"//g\')'
+      'curl -fsS -X DELETE registry:8080/v2/scratch/manifests/$(curl -fsS -I -H"Accept: application/vnd.docker.distribution.manifest.v2+json" registry:8080/v2/scratch/manifests/latest | grep Docker-Content | sed -e\'Docker-Content-Digest: //\' | tr -d \'\r\')'


The json looks now different with the Header Accept: application/vnd.docker.distribution.manifest.v2+json

The correct digest is not part of the json but of the HTTP Header. And also it is only the correct one with the given Header :-(

ironpinguin · 2018-04-07T09:34:42Z

nixos/tests/docker-registry.nix

@@ -43,7 +43,15 @@ import ./make-test.nix ({ pkgs, ...} : {
    $client2->succeed("docker images | grep scratch");

    $client2->succeed(
-      'curl -fsS -X DELETE registry:8080/v2/scratch/manifests/$(curl registry:8080/v2/scratch/manifests/latest | jq ".fsLayers[0].blobSum" | sed -e \'s/"//g\')'
+      'curl -fsS -X DELETE registry:8080/v2/scratch/manifests/$(curl -fsS -I -H"Accept: application/vnd.docker.distribution.manifest.v2+json" registry:8080/v2/scratch/manifests/latest | grep Docker-Content | sed -e\'Docker-Content-Digest: //\' | tr -d \'\r\')'


This is a example to see it better:
curl -fsS -i -H'Accept: application/vnd.docker.distribution.manifest.v2+json' localhost:5000/v2/scratch/manifests/latest

HTTP/1.1 200 OK Content-Length: 522 Content-Type: application/vnd.docker.distribution.manifest.v2+json Docker-Content-Digest: sha256:0eec87df34024c2c53db0faae043ab2175902c5877a488eb9053413e03f32661 Docker-Distribution-Api-Version: registry/2.0 Etag: "sha256:0eec87df34024c2c53db0faae043ab2175902c5877a488eb9053413e03f32661" X-Content-Type-Options: nosniff Date: Sat, 07 Apr 2018 09:36:57 GMT { "schemaVersion": 2, "mediaType": "application/vnd.docker.distribution.manifest.v2+json", "config": { "mediaType": "application/vnd.docker.container.image.v1+json", "size": 887, "digest": "sha256:b8edfce180eb3b092c8248c81f8c208b6f835308a2041d11cd2389d6741035ad" }, "layers": [ { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 49, "digest": "sha256:3c2cba919283a210665e480bcbf943eaaf4ed87a83f02e81bb286b8bdead0e75" } ] }

All 3 digests are also blobs:

sha256:0eec87df34024c2c53db0faae043ab2175902c5877a488eb9053413e03f32661 contains the images meta information

sha256:b8edfce180eb3b092c8248c81f8c208b6f835308a2041d11cd2389d6741035ad
contains the container config as json

sha256:3c2cba919283a210665e480bcbf943eaaf4ed87a83f02e81bb286b8bdead0e75 is the first root fs layer referenced by both other blobs json's

Ma27 · 2018-04-07T19:48:59Z

thanks for elaborating!
However to the tests actually pass for you?
I currently get the following output (after applying several minor fixes):

client2: must succeed: curl -fsS -X DELETE registry:8080/v2/scratch/manifests/$(curl -fsS -I -H"Accept: application/vnd.docker.distribution.manifest.v2+json" registry:8080/v2/scratch/manifests/latest | grep docker-content-digest | sed -e 's/docker-content-digest: //' | tr -d '\r')
registry# [   25.046352] docker-registry-start[645]: time="2018-04-07T19:45:30Z" level=info msg="response completed" go.version=go1.9.4 http.request.host="registry:8080" http.request.id=82197b6d-d7eb-4313-830e-362618417c07 http.request.method=HEAD http.request.remoteaddr="192.168.1.2:33248" http.request.uri="/v2/scratch/manifests/latest" http.request.useragent="curl/7.58.0" http.response.contenttype="application/vnd.docker.distribution.manifest.v2+json" http.response.duration=3.558273ms http.response.status=200 http.response.written=522 instance.id=aabbe96b-9745-4294-8fb5-8cad0fb0e1b6 version="v2.6.2+unknown"
registry# [   25.053704] docker-registry-start[645]: 192.168.1.2 - - [07/Apr/2018:19:45:30 +0000] "HEAD /v2/scratch/manifests/latest HTTP/1.1" 200 522 "" "curl/7.58.0"
registry# [   25.088081] docker-registry-start[645]: 192.168.1.2 - - [07/Apr/2018:19:45:30 +0000] "DELETE /v2/scratch/manifests/ HTTP/1.1" 404 19 "" "curl/7.58.0"
client2# curl: (22) The requested URL returned error: 404 Not Found
client2: exit status 22
client2: output:
error: command `curl -fsS -X DELETE registry:8080/v2/scratch/manifests/$(curl -fsS -I -H"Accept: application/vnd.docker.distribution.manifest.v2+json" registry:8080/v2/scratch/manifests/latest | grep docker-content-digest | sed -e 's/docker-content-digest: //' | tr -d '\r')' did not succeed (exit code 22)
command `curl -fsS -X DELETE registry:8080/v2/scratch/manifests/$(curl -fsS -I -H"Accept: application/vnd.docker.distribution.manifest.v2+json" registry:8080/v2/scratch/manifests/latest | grep docker-content-digest | sed -e 's/docker-content-digest: //' | tr -d '\r')' did not succeed (exit code 22)
cleaning up
killing client2 (pid 634)
killing client1 (pid 593)
killing registry (pid 612)
vde_switch: EOF on stdin, cleaning up and exiting
vde_switch: Could not remove ctl dir '/build/vde1.ctl': Directory not empty
builder for '/nix/store/hghlmf01mhzadda7mbrr5cj7jhvc8s5y-vm-test-run-docker-registry.drv' failed with exit code 255
error: build of '/nix/store/hghlmf01mhzadda7mbrr5cj7jhvc8s5y-vm-test-run-docker-registry.drv' failed

Will start investigating for now, let's see how far we get %)

Ma27 · 2018-04-08T17:16:32Z

merged as dd7d4080209da5b261ed9bc9bb91e369c8eb0def into docker-registry-enhancements.

This fix is required for the raspherry pi 3 with glibc 2.27, otherwise the kernel panics in initrd with: ``` <<< NixOS Stage 1 >>> loading module dm_mod... running udev... kbd_mode: KDSKBMODE: Inappropriate ioctl for device Gstarting device mapper and LVM... [ 1.969164] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b [ 1.969164] [ 1.978476] CPU: 0 PID: 1 Comm: init Not tainted 4.16.8 #1-NixOS [ 1.984580] Hardware name: Raspberry Pi 3 Model B (DT) [ 1.989801] Call trace: [ 1.992301] dump_backtrace+0x0/0x1c8 [ 1.996025] show_stack+0x24/0x30 [ 1.999396] dump_stack+0x9c/0xc0 [ 2.002766] panic+0x124/0x294 [ 2.005872] complete_and_exit+0x0/0x30 [ 2.009771] do_group_exit+0x40/0xa8 [ 2.013406] get_signal+0x280/0x5b0 [ 2.016954] do_signal+0x88/0x240 [ 2.020325] do_notify_resume+0xd8/0x130 [ 2.024311] work_pending+0x8/0x10 [ 2.027774] SMP: stopping secondary CPUs [ 2.031763] Kernel Offset: disabled [ 2.035308] CPU features: 0x0802004 [ 2.038850] Memory Limit: none [ 2.041963] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b [ 2.041963] [ 2.865264] random: crng init done ``` Suse has done the same to circumvent crashes with hostname resolving in glibc 2.27 on aarch64.

Sync with upstream

This adds some initial values for .dir-locals.el. Mainly this is useful for using bug-reference-mode. So if you have bug-reference-mode enabled - > M-x bug-reference-mode You will see as clickable text like this: Fixes NixOS#15 (NixOS#12) Closed NixOS#1252 issue #1

Get Master

…g with it Change introduced in NixOS#50672. The patch does not apply on older protobuf like protobuf3_1 ``` $ nix-build -E 'with import ./. { }; python3Packages.protobuf.override { protobuf = protobuf3_1; }' unpacking sources unpacking source archive /nix/store/1zdyl0cxaa8ha2v1zp75zzdjd6j99d0m-source source root is source setting SOURCE_DATE_EPOCH to timestamp 315619200 of file source/util/python/BUILD patching sources applying patch /nix/store/yagx7hvylnnjq7lxbcia0y5lq1r736w3-0a59054c30e4f0ba10f10acfc1d7f3814c63e1a7.patch patching file google/protobuf/pyext/descriptor.cc Hunk #1 succeeded at 55 (offset -1 lines). patching file google/protobuf/pyext/descriptor_containers.cc patching file google/protobuf/pyext/descriptor_pool.cc Hunk #1 succeeded at 47 (offset -1 lines). patching file google/protobuf/pyext/extension_dict.cc Hunk #1 FAILED at 53. 1 out of 1 hunk FAILED -- saving rejects to file google/protobuf/pyext/extension_dict.cc.rej patching file google/protobuf/pyext/message.cc Hunk #1 succeeded at 82 (offset 3 lines). Hunk #2 succeeded at 1425 (offset -104 lines). ``` Since the patch isn't necessary on python versions older than 3.7, let's only apply it for version 3.7. This means that most things using older protobuf implementation will now be able to build when using an older pythonPackage set (as is most probably the case anyway). This still leaves protobuf 3.1 using packages hanging, but the errors will be localized to those that would be breaking anyway with the upgrade to 3.7 as default python.

The sed expression actually caused an error on hydra: ``` sed: -e expression #1, char 18: Invalid collation character ``` It seems to build fine without it however.

Fix pius update

roundcube: tests - add space to db password, check setup script worked

openvswitch improvements

Fix example for xkbOptions

$ diffoscope '...-hdf5-1.12.1' '...-hdf5-1.12.1.check' --- ...-hdf5-1.12.1/lib/libhdf5.settings +++ ...-hdf5-1.12.1.check/lib/libhdf5.settings -1,17 +1,17 @@ HDF5 Version: 1.12.1 - Configured on: Thu Oct 28 17:42:30 UTC 2021 + Configured on: Sat Nov 6 19:02:02 UTC 2021 Configured by: nixbld@ Host system: x86_64-unknown-linux-gnu - Uname information: Linux localhost 5.10.76 #1-NixOS SMP Wed Oct 27 07:56:57 UTC 2021 x86_64 GNU/Linux + Uname information: Linux localhost 5.14.15 #1-NixOS SMP Wed Oct 27 07:59:56 UTC 2021 x86_64 GNU/Linux The patch removes `Configured on`, `Configured by` and `Uname information` fields.

Without the change build fails as: patch -p0 < patches/zkc-3.4.5-gcc-8.patch 2>&1 patching file zkc-3.4.5/c/src/zookeeper.c Hunk #1 FAILED at 3418. 1 out of 1 hunk FAILED -- saving rejects to file zkc-3.4.5/c/src/zookeeper.c.rej

This effectively fixes the majority of all VM tests which were broken because `/dev/vda` (or any other block device) wasn't mountable: machine # mounting /dev/vda on /... machine # mount: mounting /dev/vda on /mnt-root/ failed: No such device[ 2.820976] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100 machine # [ 2.821757] CPU: 0 PID: 1 Comm: init Not tainted 5.10.72 #1-NixOS machine # [ 2.821757] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014 machine # [ 2.821757] Call Trace: machine # [ 2.821757] dump_stack+0x6b/0x83 machine # [ 2.821757] panic+0x101/0x2c8 machine # [ 2.821757] do_exit.cold+0x14/0xb3 machine # [ 2.821757] do_group_exit+0x33/0xa0 machine # [ 2.821757] __x64_sys_exit_group+0x14/0x20 machine # [ 2.821757] do_syscall_64+0x33/0x40 machine # [ 2.821757] entry_SYSCALL_64_after_hwframe+0x44/0xa9 machine # [ 2.821757] RIP: 0033:0x7f67ec2800f6 machine # [ 2.821757] Code: 00 4c 8b 0d 2c 5d 11 00 eb 19 66 2e 0f 1f 84 00 00 00 00 00 89 d7 89 f0 0f 05 48 3d 00 f0 ff ff 77 22 f4 89 d7 44 89 c0 0f 05 <48> 3d 00 f0 ff ff 76 e2 f7 d8 64 41 89 01 eb da 66 2e 0f 1f 84 00 machine # [ 2.821757] RSP: 002b:00007fff8f5a71d8 EFLAGS: 00000202 ORIG_RAX: 00000000000000e7 machine # [ 2.821757] RAX: ffffffffffffffda RBX: 0000000000699704 RCX: 00007f67ec2800f6 machine # [ 2.821757] RDX: 0000000000000001 RSI: 000000000000003c RDI: 0000000000000001 machine # [ 2.821757] RBP: 0000000000000004 R08: 00000000000000e7 R09: ffffffffffffff80 machine # [ 2.821757] R10: 00007f67ec33f3e0 R11: 0000000000000202 R12: 000000000000000b machine # [ 2.821757] R13: 00007fff8f5a75a8 R14: 0000000000000000 R15: 00000000004fc198 machine # [ 2.821757] Kernel Offset: 0x31e00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) machine # [ 2.821757] Rebooting in 1 seconds.. This happened because the kernel failed to load modules such as `ext4` from `boot.initrd.availableKernelModules`[1] on e.g. a `mount(2)` syscall. The problem is that `kmod` isn't linked against `libpthread.so.0` anymore because it got merged into `libc.so.6` (however, the .so still exists), but still needs it: machine # newfstatat(AT_FDCWD, "/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-glibc-2.34-36/lib/x86_64", 0x7ffd951114c0, 0) = -1 ENOENT (No such file or directory) machine # openat(AT_FDCWD, "/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-glibc-2.34-36/lib/x86_64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) machine # newfstatat(AT_FDCWD, "/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-glibc-2.34-36/lib/x86_64", 0x7ffd951114c0, 0) = -1 ENOENT (No such file or directory) machine # openat(AT_FDCWD, "/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-glibc-2.34-36/lib/libpthread.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) machine # newfstatat(AT_FDCWD, "/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-glibc-2.34-36/lib", 0x7ffd951114c0, 0) = -1 ENOENT (No such file or directory) machine # openat(AT_FDCWD, "/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-glibc-2.34-36/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) machine # writev(2, [{iov_base="/nix/store/kdc9n48ksdc1a8y8w512w"..., iov_len=69}, {iov_base=": ", iov_len=2}, {iov_base="error while loading shared libra"..., iov_len=36}, {iov_base=": ", iov_len=2}, {iov_base="libpthread.so.0", iov_len=15}, {iov_base=": ", iov_len=2}, {iov_base="cy machine # ) = 184 machine # exit_group(127) = ? machine # +++ exited with 127 +++ machine # mount: mounting /dev/vda on /mnt-root/ failed: No such device machine # [ 19.167180] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100 machine # [ 19.167711] CPU: 0 PID: 1 Comm: init Not tainted 5.10.72 #1-NixOS This is not a problem * inside stage-1 because `LD_LIBRARY_PATH` points to `$out/lib` of extra-utils where `libpthread.so.6` also exists. * on a running system because `${pkgs.glibc}/lib` is part of kmod's rpath. However this is a problem inside the kernel which calls `modprobe` (in our case `kmod`) to load modules and doesn't know about `LD_LIBRARY_PATH`. Also, the rpath-reference was nuked. To work around this, the kernel's `modprobe` (i.e. `/proc/sys/kernel/modprobe`) now points to a wrapper which explicitly declares `LD_LIBRARY_PATH`. We can't use `makeWrapper` here because `modprobe` itself must not be renamed. Otherwise, `kmod` (which is the link-target of `modprobe`) won't work because it expects `argv[0] == "modprobe"` to perform modprobe's tasks. [1] https://nixos.org/manual/nixos/stable/options.html#opt-boot.initrd.availableKernelModules

SQLAlchemy-Utils v0.36.6 package override build is failing. This is due to a patch in the original SQLAlchemy-Utils package which broke the build of this package override: ```bash > applying patch /nix/store/pd6anhwbf0in3r3jhi3sbn5v2fjs0mf2-skip-database-tests.patch > patching file conftest.py > Hunk #1 FAILED at 61. > Hunk #2 succeeded at 98 (offset -10 lines). ``` These SQLAlchemy package overrides were originaly added to fix incompatibilities with Flask-Admin. See commit 05ae01f However with Flask-Admin >= v1.5.6, several SQLAlchemy compatibility patches were added: * https://flask-admin.readthedocs.io/en/latest/changelog/ We can now safely remove these package overrides to make bukuserver work again.

This reverts commit 246216e. 3.9.1 does not compile due to patch collision: > applying patch /nix/store/sickncxw0s730j6gfrnlsi5ndgysi6la-libxml2-cmake-find-package.patch > patching file CMakeLists.txt > Hunk #1 FAILED at 42. Fixing it is not trivial as upstream started bundling libxml2: NixOS#182941 (comment) Let's revert the update for now.

Without this change it segfaults when trying to play any media: $ jellyfinmediaplayer Warning: Ignoring XDG_SESSION_TYPE=wayland on Gnome. Use QT_QPA_PLATFORM=wayland to run on Wayland anyway. libpng warning: iCCP: known incorrect sRGB profile Logging to /home/bf/.local/share/jellyfinmediaplayer/logs/jellyfinmediaplayer.log Cannot load libcuda.so.1 Segmentation fault (core dumped) The backtrace shows pipewire being at fault: $ coredumpctl debug [...] Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f711428c9bb in core_event_demarshal_remove_id () from /nix/store/nhffrd7f15dhfbkwzgayq7hhzmdvdy19-pipewire-0.3.63-lib/lib/pipewire-0.3/libpipewire-module-protocol-native.so [Current thread is 1 (Thread 0x7f6ffdc87640 (LWP 1360949))] (gdb) bt #0 0x00007f711428c9bb in core_event_demarshal_remove_id () from /nix/store/nhffrd7f15dhfbkwzgayq7hhzmdvdy19-pipewire-0.3.63-lib/lib/pipewire-0.3/libpipewire-module-protocol-native.so #1 0x00007f711428886c in process_remote () from /nix/store/nhffrd7f15dhfbkwzgayq7hhzmdvdy19-pipewire-0.3.63-lib/lib/pipewire-0.3/libpipewire-module-protocol-native.so #2 0x00007f7114288e68 in on_remote_data () from /nix/store/nhffrd7f15dhfbkwzgayq7hhzmdvdy19-pipewire-0.3.63-lib/lib/pipewire-0.3/libpipewire-module-protocol-native.so #3 0x00007f7114310efe in loop_iterate () from /nix/store/nhffrd7f15dhfbkwzgayq7hhzmdvdy19-pipewire-0.3.63-lib/lib/spa-0.2/support/libspa-support.so #4 0x00007f71266fe7f2 in do_loop () from /nix/store/nhffrd7f15dhfbkwzgayq7hhzmdvdy19-pipewire-0.3.63-lib/lib/libpipewire-0.3.so.0 #5 0x00007f7128b08e86 in start_thread () from /nix/store/ayfr5l52xkqqjn3n4h9jfacgnchz1z7s-glibc-2.35-224/lib/libc.so.6 #6 0x00007f7128b8fce0 in clone3 () from /nix/store/ayfr5l52xkqqjn3n4h9jfacgnchz1z7s-glibc-2.35-224/lib/libc.so.6 (gdb) Standalone mpv doesn't segfault (when directly playing the underlying media files). I don't know why. Fixes: b97cda7 ("mpv-unwrapped: 0.34.1 -> 0.35.0") Fixes NixOS#205141 Ref jellyfin/jellyfin-media-player#341

Without this change it segfaults when trying to play any media: $ jellyfinmediaplayer Warning: Ignoring XDG_SESSION_TYPE=wayland on Gnome. Use QT_QPA_PLATFORM=wayland to run on Wayland anyway. libpng warning: iCCP: known incorrect sRGB profile Logging to /home/bf/.local/share/jellyfinmediaplayer/logs/jellyfinmediaplayer.log Cannot load libcuda.so.1 Segmentation fault (core dumped) The backtrace shows pipewire being at fault: $ coredumpctl debug [...] Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f711428c9bb in core_event_demarshal_remove_id () from /nix/store/nhffrd7f15dhfbkwzgayq7hhzmdvdy19-pipewire-0.3.63-lib/lib/pipewire-0.3/libpipewire-module-protocol-native.so [Current thread is 1 (Thread 0x7f6ffdc87640 (LWP 1360949))] (gdb) bt #0 0x00007f711428c9bb in core_event_demarshal_remove_id () from /nix/store/nhffrd7f15dhfbkwzgayq7hhzmdvdy19-pipewire-0.3.63-lib/lib/pipewire-0.3/libpipewire-module-protocol-native.so #1 0x00007f711428886c in process_remote () from /nix/store/nhffrd7f15dhfbkwzgayq7hhzmdvdy19-pipewire-0.3.63-lib/lib/pipewire-0.3/libpipewire-module-protocol-native.so #2 0x00007f7114288e68 in on_remote_data () from /nix/store/nhffrd7f15dhfbkwzgayq7hhzmdvdy19-pipewire-0.3.63-lib/lib/pipewire-0.3/libpipewire-module-protocol-native.so #3 0x00007f7114310efe in loop_iterate () from /nix/store/nhffrd7f15dhfbkwzgayq7hhzmdvdy19-pipewire-0.3.63-lib/lib/spa-0.2/support/libspa-support.so #4 0x00007f71266fe7f2 in do_loop () from /nix/store/nhffrd7f15dhfbkwzgayq7hhzmdvdy19-pipewire-0.3.63-lib/lib/libpipewire-0.3.so.0 #5 0x00007f7128b08e86 in start_thread () from /nix/store/ayfr5l52xkqqjn3n4h9jfacgnchz1z7s-glibc-2.35-224/lib/libc.so.6 #6 0x00007f7128b8fce0 in clone3 () from /nix/store/ayfr5l52xkqqjn3n4h9jfacgnchz1z7s-glibc-2.35-224/lib/libc.so.6 (gdb) Standalone mpv doesn't segfault (when directly playing the underlying media files). I don't know why. Fixes: b97cda7 ("mpv-unwrapped: 0.34.1 -> 0.35.0") Fixes NixOS#205141 Ref jellyfin/jellyfin-media-player#341 (cherry picked from commit 3c528bc)

Previously, hashcat was unable to use CUDA at runtime, and would warn: > Failed to initialize the NVIDIA main driver CUDA runtime library. > Failed to initialize NVIDIA RTC library. > * Device #1: CUDA SDK Toolkit not installed or incorrectly installed. > CUDA SDK Toolkit required for proper device support and utilization. > Falling back to OpenCL runtime. This remedies that, at least on NixOS.

had to use clangStdenv because /nix/store/1qa12idg59nlxjjdrw4hfyxdkhfd3nrq-clang-wrapper-11.1.0/bin/ar: CMakeFiles/ananicy_cpp_bpf_c.dir/src/bpf_program_utils.c.o: plugin needed to handle lto object ananicy-cpp> /nix/store/1qa12idg59nlxjjdrw4hfyxdkhfd3nrq-clang-wrapper-11.1.0/bin/ranlib: libananicy_cpp_bpf_c.a(bpf_program_utils.c.o): plugin needed to handle lto object ananicy-cpp> [100%] Linking CXX executable ananicy-cpp ananicy-cpp> /nix/store/zsvpmlddl2i5mpzm031a99xfpn410m5b-binutils-2.40/bin/ld: /build/cchtSHrR.ltrans5.ltrans.o: in function `ProcessQueue::stop()': ananicy-cpp> <artificial>:(.text+0x2274): undefined reference to `destroy_bpf_program' ananicy-cpp> /nix/store/zsvpmlddl2i5mpzm031a99xfpn410m5b-binutils-2.40/bin/ld: /build/cchtSHrR.ltrans5.ltrans.o: in function `ProcessQueue::init()': ananicy-cpp> <artificial>:(.text+0x8460): undefined reference to `initialize_bpf_program' ananicy-cpp> /nix/store/zsvpmlddl2i5mpzm031a99xfpn410m5b-binutils-2.40/bin/ld: <artificial>:(.text+0x846c): undefined reference to `bpf_program_init_events' ananicy-cpp> /nix/store/zsvpmlddl2i5mpzm031a99xfpn410m5b-binutils-2.40/bin/ld: <artificial>:(.text+0x8486): undefined reference to `destroy_bpf_program' ananicy-cpp> /nix/store/zsvpmlddl2i5mpzm031a99xfpn410m5b-binutils-2.40/bin/ld: /build/cchtSHrR.ltrans5.ltrans.o: in function `std::thread::_State_impl<std::thread::_Invoker<std::tuple<ProcessQueue::start()::{lambda(std::stop_token const&)#1}, std::stop_token> > >::_M_run() [clone .lto_priv.0]': ananicy-cpp> <artificial>:(.text+0x861d): undefined reference to `destroy_bpf_program' ananicy-cpp> collect2: error: ld returned 1 exit status ananicy-cpp> make[2]: *** [CMakeFiles/ananicy-cpp.dir/build.make:325: ananicy-cpp] Error 1 ananicy-cpp> make[1]: *** [CMakeFiles/Makefile2:100: CMakeFiles/ananicy-cpp.dir/all] Error 2 ananicy-cpp> make: *** [Makefile:136: all] Error 2

Upstream supports php 8.0/8.1 for the 3.3.0 release. The upgrade to 8.2 caused a type mismatch in carbon. > PHP message: Exception: Code: 0, Message: Carbon\Carbon::setLastErrors(): Argument #1 ($lastErrors) must be of type array, bool given, called in /nix/store/2prnw9qya9kaks2rwvd6fkrz0c7l5ygd-engelsystem-3.3.0/share/engelsystem/vendor/nesbot/carbon/src/Carbon/Traits/Creator.php on line 98, File: vendor/nesbot/carbon/src/Carbon/Traits/Creator.php:928

1.2.1: Bug fix release: Single bug fix (#1) that fixes regression in `perf` tool caused by libbpf resetting its custom catch-all `SEC()` handler on explicit `bpf_program__set_type()` call. Given setting custom `SEC()` handlers is rarely used and pretty esoteric feature of libbpf, most users should not be affected. 1.2.2: One more fix: - Fix (#2) possible double-free in USDT-related libbpf code, which happens when libbpf runs out of space in `__bpf_usdt_specs` map due to having too many unique USDT specs. Running out of space can be mitigated by bumping up `BPF_USDT_MAX_SPEC_CNT` define before including `bpf/usdt.bpf.h` header in BPF-side code. This will prevent the double-free as a side effect (and will make it possible to successfully attach all requested USDTs), which is a recommended work-around for libbpf versions prior to v1.2.2. Link: libbpf/libbpf@e4d3827 #1 Link: libbpf/libbpf@f117080 #2

Pull in _FORTIFY_SOURCE=3 stack smashing fix. Without the change on current `master` `rtorrent` crashes at start as: *** buffer overflow detected ***: terminated __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44 44 pthread_kill.c: No such file or directory. (gdb) bt #0 __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44 #1 0x00007ffff7880af3 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at pthread_kill.c:78 #2 0x00007ffff7831c86 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26 #3 0x00007ffff781b8ba in __GI_abort () at abort.c:79 #4 0x00007ffff781c5f5 in __libc_message (fmt=fmt@entry=0x7ffff7992540 "*** %s ***: terminated\n") at ../sysdeps/posix/libc_fatal.c:150 #5 0x00007ffff7910679 in __GI___fortify_fail (msg=msg@entry=0x7ffff79924e6 "buffer overflow detected") at fortify_fail.c:24 #6 0x00007ffff790eea4 in __GI___chk_fail () at chk_fail.c:28 NixOS#7 0x00007ffff790ea85 in ___snprintf_chk (s=<optimized out>, maxlen=<optimized out>, flag=<optimized out>, slen=<optimized out>, format=<optimized out>) at snprintf_chk.c:29 NixOS#8 0x0000000000472acf in utils::Lockfile::try_lock() () NixOS#9 0x000000000044b524 in core::DownloadStore::enable(bool) () NixOS#10 0x00000000004b1f7b in Control::initialize() () NixOS#11 0x000000000043000b in main ()

duckdb 0.9.0 without git deepcopy and hash version.

Without the change `amdgpu-pro` tried to build `libffi-3.3` with patches for `libffi-3.4` and failed as: patching file include/ffi_common.h Hunk #1 succeeded at 103 with fuzz 2 (offset -25 lines). can't find file to patch at input line 36 THe change uses `libffi_3_3` instead. Closes: NixOS#279955

Since ba83271 the build fails with applying patch /nix/store/46rxbbvl2l3mrxb50y9rzy7ahgx0lraj-d741901dddd731895346636c0d3556c6fa51fbe6.patch patching file tests/hazmat/primitives/test_aead.py Hunk #1 FAILED at 56. Hunk #2 FAILED at 197. Hunk #3 FAILED at 378. Hunk #4 FAILED at 525. Hunk #5 FAILED at 700. Hunk #6 FAILED at 844. 6 out of 6 hunks FAILED -- saving rejects to file tests/hazmat/primitives/test_aead.py.rej

Without the change `unnethack` startup crashes as: (gdb) bt #0 __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44 #1 0x00007f734250c0e3 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at pthread_kill.c:78 #2 0x00007f73424bce06 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26 #3 0x00007f73424a58f5 in __GI_abort () at abort.c:79 #4 0x00007f73424a67a1 in __libc_message (fmt=fmt@entry=0x7f734261e2f8 "*** %s ***: terminated\n") at ../sysdeps/posix/libc_fatal.c:150 #5 0x00007f734259b1d9 in __GI___fortify_fail (msg=msg@entry=0x7f734261e2df "buffer overflow detected") at fortify_fail.c:24 #6 0x00007f734259ab94 in __GI___chk_fail () at chk_fail.c:28 NixOS#7 0x00000000005b2ac5 in strcpy (__src=0x7ffe68838b00 "Shall I pick a character's race, role, gender and alignment for you? [YNTQ] (y)", __dest=0x7ffe68838990 "\001") at /nix/store/B0S2LKF593R3585038WS4JD3LYLF2WDX-glibc-2.38-44-dev/include/bits/string_fortified.h:79 NixOS#8 curses_break_str (str=str@entry=0x7ffe68838b00 "Shall I pick a character's race, role, gender and alignment for you? [YNTQ] (y)", width=width@entry=163, line_num=line_num@entry=1) at ../win/curses/cursmisc.c:275 NixOS#9 0x00000000005b3f51 in curses_character_input_dialog (prompt=prompt@entry=0x7ffe68838cf0 "Shall I pick a character's race, role, gender and alignment for you?", choices=choices@entry=0x7ffe68838d70 "YNTQ", def=def@entry=121) at ../win/curses/cursdial.c:211 NixOS#10 0x00000000005b9ca0 in curses_choose_character () at ../win/curses/cursinit.c:556 NixOS#11 0x0000000000404eb1 in main (argc=<optimized out>, argv=<optimized out>) at ./../sys/unix/unixmain.c:309 which corresponds to `gcc` warning: ../win/curses/cursmisc.c: In function 'curses_break_str': ../win/curses/cursmisc.c:275:5: warning: '__builtin___strcpy_chk' writing one too many bytes into a region of a size that depends on 'strlen' [-Wstringop-overflow=] 275 | strcpy(substr, str); | ^ I did not find a single small upstream change that fixes it. Let's disable `fortify3` until next release. Closes: NixOS#292113

This adds some extremely helpful and popular encoders in by default: * openjpeg * celt * libwebp * libaom On the `master` branch, closure size for ffmpeg-headless went up 18.5 MiB. ``` $ nix store diff-closures nixpkgs#ffmpeg-headless^bin .#ffmpeg-headless^bin celt: ∅ → 0.11.3, +168.4 KiB ffmpeg-headless: +70.0 KiB giflib: ∅ → 5.2.2, +398.7 KiB lcms2: ∅ → 2.16, +466.2 KiB lerc: ∅ → 4.0.0, +840.2 KiB libaom: ∅ → 3.9.0, +8047.8 KiB libdeflate: ∅ → 1.20, +427.0 KiB libtiff: ∅ → 4.6.0, +655.9 KiB libvmaf: ∅ → 3.0.0, +2665.0 KiB libwebp: ∅ → 1.4.0, +2559.7 KiB openjpeg: ∅ → 2.5.2, +1525.1 KiB zstd: ∅ → 1.5.6, +1158.0 KiB $ nvd diff $(nix build nixpkgs#ffmpeg-headless^bin --print-out-paths --no-link) $(nix build .#ffmpeg-headless^bin --print-out-paths --no-link) <<< /nix/store/4n60lnj3zkjpasd4c56bzhpx2m8lc1sx-ffmpeg-headless-6.1.1-bin >>> /nix/store/884f487w5hac6rs94jq6hq5zqkxdv666-ffmpeg-headless-6.1.1-bin Added packages: [A.] #1 celt 0.11.3 [A.] #2 giflib 5.2.2 [A.] #3 lcms2 2.16 [A.] #4 lerc 4.0.0 [A.] #5 libaom 3.9.0 [A.] #6 libdeflate 1.20 [A.] NixOS#7 libtiff 4.6.0 [A.] NixOS#8 libvmaf 3.0.0 [A.] NixOS#9 libwebp 1.4.0 x2 [A.] NixOS#10 openjpeg 2.5.2 [A.] NixOS#11 zstd 1.5.6 Closure size: 66 -> 78 (15 paths added, 3 paths removed, delta +12, disk usage +18.5MiB). ```

Strongly inspired by the forgejo counterpart[1], for the following reasons: * The feature is broken with the current module and crashes on authentication with the following stacktrace (with a PAM service `gitea` added): server # Stack trace of thread 1008: server # #0 0x00007f3116917dfb __nptl_setxid (libc.so.6 + 0x8ddfb) server # #1 0x00007f3116980ae6 setuid (libc.so.6 + 0xf6ae6) server # #2 0x00007f30cc80f420 _unix_run_helper_binary (pam_unix.so + 0x5420) server # #3 0x00007f30cc8108c9 _unix_verify_password (pam_unix.so + 0x68c9) server # #4 0x00007f30cc80e1b5 pam_sm_authenticate (pam_unix.so + 0x41b5) server # #5 0x00007f3116a84e5b _pam_dispatch (libpam.so.0 + 0x3e5b) server # #6 0x00007f3116a846a3 pam_authenticate (libpam.so.0 + 0x36a3) server # NixOS#7 0x00000000029b1e7a n/a (.gitea-wrapped + 0x25b1e7a) server # NixOS#8 0x000000000047c7e4 n/a (.gitea-wrapped + 0x7c7e4) server # ELF object binary architecture: AMD x86-64 server # server # [ 42.420827] gitea[897]: pam_unix(gitea:auth): unix_chkpwd abnormal exit: 159 server # [ 42.423142] gitea[897]: pam_unix(gitea:auth): authentication failure; logname= uid=998 euid=998 tty= ruser= rhost= user=snenskek It only worked after turning off multiple sandbox settings and adding `shadow` as supplementary group to `gitea.service`. I'm not willing to maintain additional multiple sandbox settings for different features, especially given that it was probably not used for quite a long time: * There was no PR or bugreport about sandboxing issues related to PAM. * Ever since the module exists, it used the user `gitea`, i.e. it had never read-access to `/etc/shadow`. * Upstream has it disabled by default[2]. If somebody really needs it, it can still be brought back by an overlay updating `tags` accordingly and modifying the systemd service config. [1] 07641a9 [2] https://docs.gitea.com/usage/authentication#pam-pluggable-authentication-module

Add garbage collector to docker registry

ca7d236

Ma27 reviewed Apr 6, 2018

View reviewed changes

Extend Test with run gc

179ef9a

- 1. delete uploaded manifest - 2. run gc to remove all depending blobs from filesystem (and restart registry) - 3. check if blobs are deleted - 4. push again the image - 5. check if blobs exists in filesystem (means that the registry was correct restarted)

Ma27 reviewed Apr 6, 2018

View reviewed changes

ironpinguin and others added 3 commits April 7, 2018 09:50

change to mkEnableOption and fix typos

17d348a

- Also change enableRedisCache to mkEnableOption

Fix copy & paste string

db9f58b

Fix a other typo

ad71bb4

ironpinguin commented Apr 7, 2018

View reviewed changes

Ma27 and others added 6 commits April 7, 2018 21:49

fix evaluation + applying minor fixes

6a8194f

start fixing tests

7c31c36

stupid me

fa65288

review comment

ef2fd10

just restart in the end

8d05940

Add wait for open docker-registry port

384749a

Ma27 closed this Apr 8, 2018

Ma27 deleted the docker-registry-enhancements-with-gc branch April 10, 2018 12:07

Ma27 pushed a commit that referenced this pull request Jun 21, 2018

Merge pull request #1 from NixOS/master

cfbef4a

Sync with upstream

Ma27 pushed a commit that referenced this pull request Aug 27, 2018

Merge pull request #1 from NixOS/master

17f152a

Get Master

Ma27 pushed a commit that referenced this pull request Mar 7, 2019

sewer: Fix build

19fc5cb

The sed expression actually caused an error on hydra: ``` sed: -e expression #1, char 18: Invalid collation character ``` It seems to build fine without it however.

Ma27 pushed a commit that referenced this pull request Mar 9, 2019

sewer: Fix build

82a3ab0

The sed expression actually caused an error on hydra: ``` sed: -e expression #1, char 18: Invalid collation character ``` It seems to build fine without it however.

Ma27 pushed a commit that referenced this pull request Jul 20, 2019

Merge pull request #1 from kierdavis/pius-fix

ef8b90e

Fix pius update

Ma27 pushed a commit that referenced this pull request Nov 22, 2019

Merge pull request #1 from DinCahill/fix/roundcube

d7982b7

roundcube: tests - add space to db password, check setup script worked

Ma27 pushed a commit that referenced this pull request Nov 24, 2019

Merge pull request #1 from kmcopper/openvswitch-improvements

0f5366d

openvswitch improvements

Ma27 pushed a commit that referenced this pull request Jan 6, 2020

Merge pull request #1 from juliancoffee/juliancoffee-patch-1

5851c9c

Fix example for xkbOptions

Ma27 pushed a commit that referenced this pull request May 14, 2022

goku 0.5.1 -> 0.5.2 (#1)

65c8b91

Ma27 pushed a commit that referenced this pull request Oct 4, 2023

Merge pull request #1 from robert-manchester/duckdb-0.9_nohash

875eb2d

duckdb 0.9.0 without git deepcopy and hash version.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Add garbage collector to docker registry #1

WIP: Add garbage collector to docker registry #1

ironpinguin commented Apr 6, 2018

Ma27 Apr 6, 2018

Ma27 left a comment

Ma27 Apr 6, 2018

Ma27 Apr 6, 2018

Ma27 Apr 6, 2018

ironpinguin Apr 7, 2018

ironpinguin Apr 7, 2018

ironpinguin left a comment

ironpinguin Apr 7, 2018

ironpinguin Apr 7, 2018

Ma27 commented Apr 7, 2018

Ma27 commented Apr 8, 2018

WIP: Add garbage collector to docker registry #1

WIP: Add garbage collector to docker registry #1

Conversation

ironpinguin commented Apr 6, 2018

Motivation for this change

Things done

Choose a reason for hiding this comment

Ma27 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ironpinguin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ma27 commented Apr 7, 2018

Ma27 commented Apr 8, 2018