bitwarden_rs: remove failing integration test #118072

Munksgaard · 2021-03-30T14:10:10Z

This test is hitting a timeout because it's trying to build
firefox-unwrapped. The idea is good, test that the web service works with
different database backends, but building firefox is way too big a task for a
small service like this.

Here's the logs from one of the failing tests:

https://hydra.nixos.org/build/140338120

From the looks of it, the only other test to use firefox-unwrapped and selenium
in the same way is nginx-etag, which also times out:

https://hydra.nixos.org/build/140337879

If there is no alternative way of automating the integration tests, I therefore
propose that we delete them. Right now, the only purpose they serve is to block
newer releases on unstable.

If you want me to, I can also create a PR to remove the failing nginx-etag test.

Motivation for this change

Things done

This test is hitting a timeout because it's trying to build firefox-unwrapped. The idea is good, test that the web service works with different database backends, but building firefox is way too big a task for a small service like this. Here's the logs from one of the failing tests: https://hydra.nixos.org/build/140338120 From the looks of it, the only other test to use firefox-unwrapped and selenium in the same way is nginx-etag, which also times out: https://hydra.nixos.org/build/140337879 If there is no alternative way of automating the integration tests, I therefore propose that we delete them. Right now, the only purpose they serve is to block newer releases on unstable.

Munksgaard · 2021-03-30T14:11:12Z

@SuperSandro2000 I think I figured out why the bitwarden_rs tests were failing.

SuperSandro2000 · 2021-03-30T14:52:31Z

Maybe firefox-unwrapped was not cached?

mweinelt · 2021-03-30T14:56:35Z

I feel very strongly that this is the wrong way to tackle this issue. The test isn't even part of the tested set, so I don't see how it could block anything.

Firefox should mostly come from the cache: https://hydra.nixos.org/job/nixos/trunk-combined/nixpkgs.firefox-unwrapped.x86_64-linux/all

The bitwarden tests fail regardless.

https://hydra.nixos.org/job/nixos/trunk-combined/nixos.tests.bitwarden.sqlite.x86_64-linux/all
https://hydra.nixos.org/job/nixos/trunk-combined/nixos.tests.bitwarden.postgresql.x86_64-linux/all
https://hydra.nixos.org/job/nixos/trunk-combined/nixos.tests.bitwarden.mysql.x86_64-linux/all

Munksgaard · 2021-03-30T15:37:40Z

Firefox should mostly come from the cache: https://hydra.nixos.org/job/nixos/trunk-combined/nixpkgs.firefox-unwrapped.x86_64-linux/all

Ah, it actually seems like the x86_64 tests are now passing!

However, aarch64 is still failing.

Maybe the problem is that firefox is not cached for aarch64? https://hydra.nixos.org/job/nixos/trunk-combined/nixpkgs.firefox-unwrapped.aarch64-linux/all

Munksgaard · 2021-03-30T15:38:49Z

The test isn't even part of the tested set, so I don't see how it could block anything.

I'm not sure what you mean by this?

mweinelt · 2021-03-30T16:48:17Z

The test isn't even part of the tested set, so I don't see how it could block anything.

I'm not sure what you mean by this?

The list of tests that decide whether a channel bump can occur.

nixpkgs/nixos/release-combined.nix

Lines 36 to 158 in 2446625

    
           tested = 
        
             let 
        
               onFullSupported = x: map (system: "${x}.${system}") supportedSystems; 
        
               onAllSupported = x: map (system: "${x}.${system}") (supportedSystems ++ limitedSupportedSystems); 
        
               onSystems = systems: x: map (system: "${x}.${system}") 
        
                 (pkgs.lib.intersectLists systems (supportedSystems ++ limitedSupportedSystems)); 
        
             in pkgs.releaseTools.aggregate { 
        
               name = "nixos-${nixos.channel.version}"; 
        
               meta = { 
        
                 description = "Release-critical builds for the NixOS channel"; 
        
                 maintainers = with pkgs.lib.maintainers; [ eelco fpletz ]; 
        
               }; 
        
               constituents = pkgs.lib.concatLists [ 
        
                 [ "nixos.channel" ] 
        
                 (onFullSupported "nixos.dummy") 
        
                 (onAllSupported "nixos.iso_minimal") 
        
                 (onSystems ["x86_64-linux" "aarch64-linux"] "nixos.amazonImage") 
        
                 (onSystems ["x86_64-linux"] "nixos.iso_plasma5") 
        
                 (onSystems ["x86_64-linux"] "nixos.iso_gnome") 
        
                 (onFullSupported "nixos.manual") 
        
                 (onSystems ["x86_64-linux"] "nixos.ova") 
        
                 (onSystems ["aarch64-linux"] "nixos.sd_image") 
        
                 (onSystems ["x86_64-linux"] "nixos.tests.boot.biosCdrom") 
        
                 (onSystems ["x86_64-linux"] "nixos.tests.boot.biosUsb") 
        
                 (onFullSupported "nixos.tests.boot-stage1") 
        
                 (onSystems ["x86_64-linux"] "nixos.tests.boot.uefiCdrom") 
        
                 (onSystems ["x86_64-linux"] "nixos.tests.boot.uefiUsb") 
        
                 (onSystems ["x86_64-linux"] "nixos.tests.chromium") 
        
                 (onFullSupported "nixos.tests.containers-imperative") 
        
                 (onFullSupported "nixos.tests.containers-ip") 
        
                 (onSystems ["x86_64-linux"] "nixos.tests.docker") 
        
                 (onFullSupported "nixos.tests.ecryptfs") 
        
                 (onFullSupported "nixos.tests.env") 
        
                 (onFullSupported "nixos.tests.firefox-esr") 
        
                 (onFullSupported "nixos.tests.firefox") 
        
                 (onFullSupported "nixos.tests.firewall") 
        
                 (onFullSupported "nixos.tests.fontconfig-default-fonts") 
        
                 (onFullSupported "nixos.tests.gnome3") 
        
                 (onFullSupported "nixos.tests.gnome3-xorg") 
        
                 (onSystems ["x86_64-linux"] "nixos.tests.hibernate") 
        
                 (onFullSupported "nixos.tests.i3wm") 
        
                 (onSystems ["x86_64-linux"] "nixos.tests.installer.btrfsSimple") 
        
                 (onSystems ["x86_64-linux"] "nixos.tests.installer.btrfsSubvolDefault") 
        
                 (onSystems ["x86_64-linux"] "nixos.tests.installer.btrfsSubvols") 
        
                 (onSystems ["x86_64-linux"] "nixos.tests.installer.luksroot") 
        
                 (onSystems ["x86_64-linux"] "nixos.tests.installer.lvm") 
        
                 (onSystems ["x86_64-linux"] "nixos.tests.installer.separateBootFat") 
        
                 (onSystems ["x86_64-linux"] "nixos.tests.installer.separateBoot") 
        
                 (onSystems ["x86_64-linux"] "nixos.tests.installer.simpleLabels") 
        
                 (onSystems ["x86_64-linux"] "nixos.tests.installer.simpleProvided") 
        
                 (onSystems ["x86_64-linux"] "nixos.tests.installer.simpleUefiSystemdBoot") 
        
                 (onSystems ["x86_64-linux"] "nixos.tests.installer.simple") 
        
                 (onSystems ["x86_64-linux"] "nixos.tests.installer.swraid") 
        
                 (onFullSupported "nixos.tests.ipv6") 
        
                 (onFullSupported "nixos.tests.keymap.azerty") 
        
                 (onFullSupported "nixos.tests.keymap.colemak") 
        
                 (onFullSupported "nixos.tests.keymap.dvorak") 
        
                 (onFullSupported "nixos.tests.keymap.dvp") 
        
                 (onFullSupported "nixos.tests.keymap.neo") 
        
                 (onFullSupported "nixos.tests.keymap.qwertz") 
        
                 (onFullSupported "nixos.tests.latestKernel.login") 
        
                 (onFullSupported "nixos.tests.lightdm") 
        
                 (onFullSupported "nixos.tests.login") 
        
                 (onFullSupported "nixos.tests.misc") 
        
                 (onFullSupported "nixos.tests.mutableUsers") 
        
                 (onFullSupported "nixos.tests.nano") 
        
                 (onFullSupported "nixos.tests.nat.firewall-conntrack") 
        
                 (onFullSupported "nixos.tests.nat.firewall") 
        
                 (onFullSupported "nixos.tests.nat.standalone") 
        
                 (onFullSupported "nixos.tests.networking.scripted.bond") 
        
                 (onFullSupported "nixos.tests.networking.scripted.bridge") 
        
                 (onFullSupported "nixos.tests.networking.scripted.dhcpOneIf") 
        
                 (onFullSupported "nixos.tests.networking.scripted.dhcpSimple") 
        
                 (onFullSupported "nixos.tests.networking.scripted.link") 
        
                 (onFullSupported "nixos.tests.networking.scripted.loopback") 
        
                 (onFullSupported "nixos.tests.networking.scripted.macvlan") 
        
                 (onFullSupported "nixos.tests.networking.scripted.privacy") 
        
                 (onFullSupported "nixos.tests.networking.scripted.routes") 
        
                 (onFullSupported "nixos.tests.networking.scripted.sit") 
        
                 (onFullSupported "nixos.tests.networking.scripted.static") 
        
                 (onFullSupported "nixos.tests.networking.scripted.virtual") 
        
                 (onFullSupported "nixos.tests.networking.scripted.vlan") 
        
                 (onFullSupported "nixos.tests.networking.networkd.bond") 
        
                 (onFullSupported "nixos.tests.networking.networkd.bridge") 
        
                 (onFullSupported "nixos.tests.networking.networkd.dhcpOneIf") 
        
                 (onFullSupported "nixos.tests.networking.networkd.dhcpSimple") 
        
                 (onFullSupported "nixos.tests.networking.networkd.link") 
        
                 (onFullSupported "nixos.tests.networking.networkd.loopback") 
        
                 # Fails nondeterministically (https://github.com/NixOS/nixpkgs/issues/96709) 
        
                 #(onFullSupported "nixos.tests.networking.networkd.macvlan") 
        
                 (onFullSupported "nixos.tests.networking.networkd.privacy") 
        
                 (onFullSupported "nixos.tests.networking.networkd.routes") 
        
                 (onFullSupported "nixos.tests.networking.networkd.sit") 
        
                 (onFullSupported "nixos.tests.networking.networkd.static") 
        
                 (onFullSupported "nixos.tests.networking.networkd.virtual") 
        
                 (onFullSupported "nixos.tests.networking.networkd.vlan") 
        
                 (onFullSupported "nixos.tests.systemd-networkd-ipv6-prefix-delegation") 
        
                 (onFullSupported "nixos.tests.nfs3.simple") 
        
                 (onFullSupported "nixos.tests.nfs4.simple") 
        
                 (onFullSupported "nixos.tests.openssh") 
        
                 (onFullSupported "nixos.tests.pantheon") 
        
                 (onFullSupported "nixos.tests.php.fpm") 
        
                 (onFullSupported "nixos.tests.php.httpd") 
        
                 (onFullSupported "nixos.tests.php.pcre") 
        
                 (onFullSupported "nixos.tests.plasma5") 
        
                 (onFullSupported "nixos.tests.predictable-interface-names.predictableNetworkd") 
        
                 (onFullSupported "nixos.tests.predictable-interface-names.predictable") 
        
                 (onFullSupported "nixos.tests.predictable-interface-names.unpredictableNetworkd") 
        
                 (onFullSupported "nixos.tests.predictable-interface-names.unpredictable") 
        
                 (onFullSupported "nixos.tests.printing") 
        
                 (onFullSupported "nixos.tests.proxy") 
        
                 (onFullSupported "nixos.tests.sddm.default") 
        
                 (onFullSupported "nixos.tests.shadow") 
        
                 (onFullSupported "nixos.tests.simple") 
        
                 (onFullSupported "nixos.tests.switchTest") 
        
                 (onFullSupported "nixos.tests.udisks2") 
        
                 (onFullSupported "nixos.tests.xfce") 
        
                 (onSystems ["i686-linux"] "nixos.tests.zfs.installer") 
        
                 (onFullSupported "nixpkgs.emacs") 
        
                 (onFullSupported "nixpkgs.jdk") 
        
                 ["nixpkgs.tarball"] 
        
               ]; 
        
             };

Munksgaard · 2021-03-30T17:03:27Z

Interesting, I didn't know about that.

Then why is bitwarden_rs still in version v1.17.0 on unstable?

https://search.nixos.org/packages?channel=unstable&show=bitwarden_rs&from=0&size=50&sort=relevance&query=bitwarden

mweinelt · 2021-03-30T17:16:15Z

Search must be outdated.

https://hydra.nixos.org/job/nixos/trunk-combined/nixpkgs.bitwarden_rs.x86_64-linux

mweinelt · 2021-03-30T17:32:42Z

See NixOS/nixos-search#295

JJJollyjim · 2021-03-30T18:33:55Z

Thanks mweinelt, I agree there is no issue here. The test is not "building firefox", it depends on firefox, like it depends on linux and systemd.

Munksgaard · 2021-03-30T19:12:21Z

Thank you all for helping out ❤️

github-actions bot added the 6.topic: nixos label Mar 30, 2021

ofborg bot added 10.rebuild-darwin: 0 10.rebuild-linux: 0 labels Mar 30, 2021

mweinelt requested a review from JJJollyjim March 30, 2021 14:56

mweinelt added the invalid label Mar 30, 2021

JJJollyjim closed this Mar 30, 2021

Munksgaard deleted the remove-bitwarden-test branch March 30, 2021 19:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bitwarden_rs: remove failing integration test #118072

bitwarden_rs: remove failing integration test #118072

Munksgaard commented Mar 30, 2021 •

edited

Munksgaard commented Mar 30, 2021

SuperSandro2000 commented Mar 30, 2021

mweinelt commented Mar 30, 2021 •

edited

Munksgaard commented Mar 30, 2021 •

edited

Munksgaard commented Mar 30, 2021

mweinelt commented Mar 30, 2021

Munksgaard commented Mar 30, 2021

mweinelt commented Mar 30, 2021

mweinelt commented Mar 30, 2021

JJJollyjim commented Mar 30, 2021

Munksgaard commented Mar 30, 2021

bitwarden_rs: remove failing integration test #118072

bitwarden_rs: remove failing integration test #118072

Conversation

Munksgaard commented Mar 30, 2021 • edited

Motivation for this change

Things done

Munksgaard commented Mar 30, 2021

SuperSandro2000 commented Mar 30, 2021

mweinelt commented Mar 30, 2021 • edited

Munksgaard commented Mar 30, 2021 • edited

Munksgaard commented Mar 30, 2021

mweinelt commented Mar 30, 2021

Munksgaard commented Mar 30, 2021

mweinelt commented Mar 30, 2021

mweinelt commented Mar 30, 2021

JJJollyjim commented Mar 30, 2021

Munksgaard commented Mar 30, 2021

Munksgaard commented Mar 30, 2021 •

edited

mweinelt commented Mar 30, 2021 •

edited

Munksgaard commented Mar 30, 2021 •

edited