Unstable channel hasn't updated in 2 weeks

[peti]

https://github.com/nixos/nixpkgs-channels is currently at a commit that's 13 days old. The channel appears to be stuck mostly because of failing Chromium and kde4 tests at https://hydra.nixos.org/job/nixos/trunk-combined/tested#tabs-constituents.

[vcunat]

The problem with telepathy_qt not building was fixed some time ago and Hydra even has binaries for it already.

[Mathnerd314]

Past issues: #7083 #7589 #8057 #10353 #11097.

I think this is more a problem with Hydra than with nixpkgs...

[vcunat]

I'm getting loads of timeouts just on evaluation, consistently during the last months, most of staging attempts seem to timeout...

[domenkozar]

There are issues with tests itself, like https://hydra.nixos.org/build/29218960/nixlog/1/raw

[domenkozar]

cc @aszlig I have a feeling that tests are indeed way slower now without msize. We should definitely look into moving 9p into fuse.

[ttuegel]

Evaluation is now failing with

hydra-eval-jobs returned signal 6:
warning: SQLite database is busy
Too many heap sections: Increase MAXHINCR or MAX_HEAP_SECTS

[domenkozar]

@ttuegel I've forced an eval, let's see.

Current culprit are tests, they timeout now. I think a5bc11f made things worse by making 9p really slow (that was the intention of bigger msize).

[domenkozar]

It worked, but all tests still fail: http://hydra.nixos.org/eval/1235190

[copumpkin]

https://github.com/NixOS/nixpkgs-channels/tree/nixpkgs-unstable is even farther behind, having last updated a month and a half ago 😦

[domenkozar]

That's due to darwin: http://hydra.nixos.org/build/29453741/nixlog/1/raw

[copumpkin]

@domenkozar that should be fixed now by 527f357, thanks!

[dezgeg]

I don't think the 9p tweaks are causing the test breakage, since they're quite broken on 15.09 as well: http://hydra.nixos.org/job/nixos/release-15.09/tested#tabs-constituents, and those changes weren't cherry-picked there.

[domenkozar]

@dezgeg Eelco fixed that today (it was a Nix bug). I've fixed Chromium (hopefully, it did build locally).

What's left is:

• lvm test fails due to missing /mnt mount http://hydra.nixos.org/build/29615754/nixlog/2/raw
• firewall test fails to resolve DNS http://hydra.nixos.org/build/29614963/log/raw

[domenkozar]

Actually tested job is totally broken..

[vcunat]

On 15.09 it is also rather broken ATM, but there it's waiting to evaluate a forced rebuild, so maybe it's not really that bad.

[vcunat]

Now we're getting thousands of abortions due to unexpected end of file.

[domenkozar]

Yeah, hopefully @edolstra will have time to look at it soon :)

[exi]

Wow, if i run the current master firefox.nix test, it fails immediately.

If I just put a "sleep 15" in nixpkgs/nixos/modules/virtualisation/qemu-vm.nix:385 to force a random rebuild, it works sometimes.

This looks like a weird timing bug + something i cannot identify => why does it work if i force a local rebuilt?

Running the firefox test from current master without any modifications just yield this every time:

machine# mke2fs 1.42.13 (17-May-2015)
machine# Creating filesystem with 131072 4k blocks and 32768 inodes
machine# Filesystem UUID: d33dcb2f-c3a3-4bc9-8148-5ea90bb65da6
machine# Superblock backups stored on blocks: 
machine#    32768, 98304
machine# 
machine# Allocating group tables: done                            
machine# Writing inode tables: done                            
machine# Creating journal (4096 blocks): [    0.730133] usb 1-1: new full-speed USB device number 2 using uhci_hcd
machine# done
machine# Writing superblocks and filesystem accounting information: done
machine# 
machine# chmod: /mnt-root/tmp: No such file or directory

This error disappears even when i just add a "echo test" into nixpkgs/nixos/modules/virtualisation/qemu-vm.nix:385

[obadz]

Any idea what causes this error?

http://hydra.nixos.org/build/29873328/log#line-1262

cleaning up
killing backend1 (pid 65)
killing proxy (pid 76)
killing client (pid 87)
killing backend2 (pid 98)
vde_switch: EOF on stdin, cleaning up and exiting
vde_switch: Could not remove ctl dir '/tmp/nix-build-vm-test-run-proxy.drv-0/vde1.ctl': Directory not empty
builder for ‘/nix/store/xfwy21mpamrrlmh8956hfjfv5ryxhr9p-vm-test-run-proxy.drv’/...-vm-test-run-proxy.drv’ failed with exit code 255

[domenkozar]

I think it was out of HDD on hydra (which should be fixed now).

On Mon, Jan 4, 2016 at 3:54 PM, obadz notifications@github.com wrote:

Any idea what causes this error?

http://hydra.nixos.org/build/29873328/log#line-1262

cleaning up
killing backend1 (pid 65)
killing proxy (pid 76)
killing client (pid 87)
killing backend2 (pid 98)
vde_switch: EOF on stdin, cleaning up and exiting
vde_switch: Could not remove ctl dir '/tmp/nix-build-vm-test-run-proxy.drv-0/vde1.ctl': Directory not empty
builder for ‘/nix/store/xfwy21mpamrrlmh8956hfjfv5ryxhr9p-vm-test-run-proxy.drv’/...-vm-test-run-proxy.drv’ failed with exit code 255

—
Reply to this email directly or view it on GitHub
#11865 (comment).

[domenkozar]

28 days.

[domenkozar]

swraid test is failing with out of memory exception, reverting 00aac09 actually fixes it but then following error occurs:

machine: must succeed: nixos-rebuild switch >&2
machine# > > > > > > > > > > > > > > > > > > > building Nix...
machine# [    9.366267] dhcpcd[692]: eth0: probing for an IPv4LL address
machine# [    9.495965] systemd[1]: Started Nix Daemon.
machine# [    9.502463] nix-daemon[716]: accepted connection from pid 715, user root (trusted)
machine# [    9.619185] nix-daemon[716]: accepted connection from pid 719, user root (trusted)
machine# download-from-binary-cache.pl: could not download ‘https://cache.nixos.org/37lk94lzfgmf3x3si23bi7xwgrhyy177.narinfo’ (Curl error 6)
machine# download-from-binary-cache.pl: could not download ‘https://cache.nixos.org/6flnja8068w1lwnhjw88f9lsgf77lg89.narinfo’ (Curl error 6)
machine# download-from-binary-cache.pl: could not download ‘https://cache.nixos.org/fl641x87p42i4r0nf74brm6lwjvapa4g.narinfo’ (Curl error 6)
machine# download-from-binary-cache.pl: could not download ‘https://cache.nixos.org/aymhkd65g8fy03ar4c3yssp180dnh4pv.narinfo’ (Curl error 6)
machine# download-from-binary-cache.pl: could not download ‘https://cache.nixos.org/4l3nkwjv4z2lvkhxr1i94f4lqw5isswn.narinfo’ (Curl error 6)
machine# download-from-binary-cache.pl: could not download ‘https://cache.nixos.org/w3y63q62a4y99m9j51zx05acspw1wxkq.narinfo’ (Curl error 6)
machine# download-from-binary-cache.pl: could not download ‘https://cache.nixos.org/z4z6la8bkkhc227n68vrq7zqbfdmrw4b.narinfo’ (Curl error 6)
machine# download-from-binary-cache.pl: could not download ‘https://cache.nixos.org/k8i9i1vmlipaxajj95bgn2zjkdmz36v7.narinfo’ (Curl error 6)
machine# download-from-binary-cache.pl: could not download ‘https://cache.nixos.org/k0vqprjmxybr7clvfljk13zsdjwklcch.narinfo’ (Curl error 6)
machine# download-from-binary-cache.pl: could not download ‘https://cache.nixos.org/nyw8s7kqbwapr0hlx3xxw6j3zqa1dlsx.narinfo’ (Curl error 6)
machine# download-from-binary-cache.pl: could not download ‘https://cache.nixos.org/ldj18wicxhjf0739kr08p0rbnsby7zr9.narinfo’ (Curl error 6)
machine# these derivations will be built:
machine#   /nix/store/4z0jdl9baq0sv7353n5xv738bxinx08x-bootstrap-tools.tar.xz.drv
machine#   /nix/store/8pvv9kjda7hs0mrrn9km3f5wmhp3fygs-mirrors-list.drv
machine#   /nix/store/6brhkj9bxyg56jhj62m2gvnjn5nm38dy-nix-1.10.tar.xz.drv
machine#   /nix/store/pi866p41dsp26rlpq5r79h2a35gx2vrn-busybox.drv
machine#   /nix/store/rlbfc41y1hkdhympy67lbfg8r6nc93ab-bootstrap-tools.drv
machine#   /nix/store/x40as51v8pj51s33l8hmziqrx8dcs8vc-stdenv-linux-boot.drv
machine#   /nix/store/zrlcc9g3vw579c3vskvgb387ad7x6gjy-mirrors-list.drv
machine#   /nix/store/l3ibd0rcvaqq0n8p5j780b0gw9zyykpc-pkg-config-0.29.tar.gz.drv
machine#   /nix/store/q1rl94kqphsx90hd765vwxc2vk8l0d48-pkg-config-0.29.drv
machine#   /nix/store/mq209nls1l67nrhv6qxqak7c60wxipz5-nix-1.10.drv
machine# download-from-binary-cache.pl: could not download ‘https://cache.nixos.org/37lk94lzfgmf3x3si23bi7xwgrhyy177.narinfo’ (Curl error 6)
machine# download-from-binary-cache.pl: could not download ‘https://cache.nixos.org/6flnja8068w1lwnhjw88f9lsgf77lg89.narinfo’ (Curl error 6)
machine# download-from-binary-cache.pl: could not download ‘https://cache.nixos.org/aymhkd65g8fy03ar4c3yssp180dnh4pv.narinfo’ (Curl error 6)
machine# download-from-binary-cache.pl: could not download ‘https://cache.nixos.org/fl641x87p42i4r0nf74brm6lwjvapa4g.narinfo’ (Curl error 6)
machine# download-from-binary-cache.pl: could not download ‘https://cache.nixos.org/4l3nkwjv4z2lvkhxr1i94f4lqw5isswn.narinfo’ (Curl error 6)
machine# download-from-binary-cache.pl: could not download ‘https://cache.nixos.org/w3y63q62a4y99m9j51zx05acspw1wxkq.narinfo’ (Curl error 6)
machine# download-from-binary-cache.pl: could not download ‘https://cache.nixos.org/k0vqprjmxybr7clvfljk13zsdjwklcch.narinfo’ (Curl error 6)
machine# download-from-binary-cache.pl: could not download ‘https://cache.nixos.org/k8i9i1vmlipaxajj95bgn2zjkdmz36v7.narinfo’ (Curl error 6)
machine# download-from-binary-cache.pl: could not download ‘https://cache.nixos.org/z4z6la8bkkhc227n68vrq7zqbfdmrw4b.narinfo’ (Curl error 6)
machine# building path(s) ‘/nix/store/4l3nkwjv4z2lvkhxr1i94f4lqw5isswn-mirrors-list’
machine# download-from-binary-cache.pl: could not download ‘https://cache.nixos.org/nyw8s7kqbwapr0hlx3xxw6j3zqa1dlsx.narinfo’ (Curl error 6)
machine# download-from-binary-cache.pl: could not download ‘https://cache.nixos.org/ldj18wicxhjf0739kr08p0rbnsby7zr9.narinfo’ (Curl error 6)
machine# building path(s) ‘/nix/store/nyw8s7kqbwapr0hlx3xxw6j3zqa1dlsx-bootstrap-tools.tar.xz’
machine# downloading ‘http://tarballs.nixos.org/stdenv-linux/x86_64/8d66a51a872af1ab58edc68a2ebddcc79958b563/bootstrap-tools.tar.xz’...
machine# error: unable to download ‘http://tarballs.nixos.org/stdenv-linux/x86_64/8d66a51a872af1ab58edc68a2ebddcc79958b563/bootstrap-tools.tar.xz’: Couldn't resolve host name (6)
machine# builder for ‘/nix/store/4z0jdl9baq0sv7353n5xv738bxinx08x-bootstrap-tools.tar.xz.drv’ failed with exit code 1
machine# cannot build derivation ‘/nix/store/rlbfc41y1hkdhympy67lbfg8r6nc93ab-bootstrap-tools.drv’: 1 dependencies couldn't be built
machine# building path(s) ‘/nix/store/aymhkd65g8fy03ar4c3yssp180dnh4pv-nix-1.10.tar.xz’
machine# cannot build derivation ‘/nix/store/l3ibd0rcvaqq0n8p5j780b0gw9zyykpc-pkg-config-0.29.tar.gz.drv’: 1 dependencies couldn't be built
machine# cannot build derivation ‘/nix/store/q1rl94kqphsx90hd765vwxc2vk8l0d48-pkg-config-0.29.drv’: 1 dependencies couldn't be built
machine# cannot build derivation ‘/nix/store/mq209nls1l67nrhv6qxqak7c60wxipz5-nix-1.10.drv’: 1 dependencies couldn't be built
machine# error: build of ‘/nix/store/mq209nls1l67nrhv6qxqak7c60wxipz5-nix-1.10.drv’ failed
machine: exit status 1
machine: output: 
error: command `nixos-rebuild switch >&2' did not succeed (exit code 1)

[copumpkin]

@domenkozar that looks like my ec2-config test failing, but I see no indication of my test being in there. Can you disable ec2-config for now and see if that fixes it?

Edit: or at least, it's exactly the same failure, caused by calling nixos-rebuild inside a VM test inside a sandbox, so it has no internet connection and can't retrieve the files it needs.

[domenkozar]

Probably most of the tests fail this way. I'll bisect. @ttuegel any idea why would above revert fix the issue?

[ttuegel]

@domenkozar No idea, looks totally unrelated.

[domenkozar]

@rickynils the above test fails to build because of ca0c460, you can test it with nix-build -A tests.installer.swraid.x86_64-linux nixos/release.nix. You'll probably have to revert 00aac09 first to get rid of message out of memory

[ttuegel]

@domenkozar I realized why 00aac09 causes the swraid test to fail: the swraid test uses nix-env -i instead of nix-env -iA, so nix-env is searching through the entire package set. That commit adds many new packages.

[obadz]

@ttuegel nice catch!

[domenkozar]

@ttuegel fixed in 313fa2e.

@rickynils I've reverted ca0c460. since it's blocking a channel update. Sorry I couldn't find time to fix the issue, please open a PR and make sure one of the installer tests is passing.

[rickynils]

@domenkozar no problems, I'll look into it.

[lucabrunox]

I'm seeing new errors in nixos tests (http://hydra.nixos.org/build/30325559/log/raw):

machine# qemu-system-x86_64: -drive index=0,id=drive1,file=/tmp/nix-build-vm-test-run-installer-btrfsSimple.drv-0/vm-state-machine/machine.qcow2,if=virtio,cache=none,werror=report: file system may not support O_DIRECT
machine# qemu-system-x86_64: -drive index=0,id=drive1,file=/tmp/nix-build-vm-test-run-installer-btrfsSimple.drv-0/vm-state-machine/machine.qcow2,if=virtio,cache=none,werror=report: Could not open '/tmp/nix-build-vm-test-run-installer-btrfsSimple.drv-0/vm-state-machine/machine.qcow2': Invalid argument
vde_switch: EOF data port: No such file or directory
error: the VM quit before connecting

Is that an out of disk space?

[dezgeg]

The file system may not support O_DIRECT makes me think it could be related to the cache=none change in 6353f58

[lucabrunox]

@dezgeg that's not supposed to be a problem, it's only a notice

[dezgeg]

To me that sounds like the /tmp filesystem on that machine is incapable of being used for cache=none disk access for some reason, which makes QEMU then error out.

[domenkozar]

That's unfortunate, it seems to not work with fuse and journaled FS: http://www.mygnu.de/index.php/2012/02/qemu-kvm-with-cachenone-fails-on-ext4-filesystem-with-journal_data-option/

I'll revert.

[jgeerds]

This looks much better: https://hydra.nixos.org/job/nixos/trunk-combined/tested#tabs-constituents

[domenkozar]

Yes, it was a long fight but it appears we'll get a channel update tomorrow.

[obadz]

Thanks for pushing this through @domenkozar! 👏

[NeQuissimus]

👏

[vcunat]

Great work! The tested job succeeded twice in a row now. The big channel hasn't updated yet, but that's probably only waiting for non-critical builds to finish.

[lucabrunox]

Yep. Let's close this?

[domenkozar]

updated.