Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nixosTests: Reraise exceptions in subtests so they stop the test #79328

Merged
merged 2 commits into from Apr 30, 2020

Conversation

@tfc
Copy link
Contributor

@tfc tfc commented Feb 6, 2020

Motivation for this change

#72828

While porting the printing test i realized that if an assert triggers within a subtest, then this is logged but does not lead to test abort. This was bad because without successful assertions the rest of the test didn't make any sense.

I changed this passage of the test driver so that it gives extra-info in which subtest the problem occured and then leave it to the global exception handling to fail the whole test.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.
@tfc tfc force-pushed the tfc:python-test-driver-reraise branch from 140a416 to 555b982 Feb 6, 2020
@tfc tfc changed the title Reraise exceptions in subtests so they stop the test nixosTests: Reraise exceptions in subtests so they stop the test Feb 6, 2020
@tfc tfc requested a review from flokli Feb 6, 2020
@ofborg ofborg bot added the 6.topic: nixos label Feb 6, 2020
@flokli
Copy link
Contributor

@flokli flokli commented Feb 6, 2020

This basically depends on whether we want to use subtests to describe individual independent things / testcases on machines that don't necessarily depend on each other (A), or just use them as a way to group / give a heading to a bunch of statements (B).

The current behaviour (both in python and perl) is to add the an error message to failed_tests, but keep going with the next one if I'm not mistaken (cc @globin #74089), which assumes (A)

However, I think we mostly do B in many tests, even though we probably shouldn't. WDYT?

I don't think we should change semantics here with the perl driver still around.

@tfc
Copy link
Contributor Author

@tfc tfc commented Feb 6, 2020

This basically depends on whether we want to use subtests to describe individual independent things / testcases on machines that don't necessarily depend on each other (A), or just use them as a way to group / give a heading to a bunch of statements (B).

It generally makes sense to enable the user to decide which things should make tests fail and which should not.

The current behaviour (both in python and perl) is to add the an error message to failed_tests, but keep going with the next one if I'm not mistaken (cc @globin #74089), which assumes (A)

True, but i observed this behavior to do no good in the many many tests that i have ported.

However, I think we mostly do B in many tests, even though we probably shouldn't. WDYT?

The vast majority of tests that i have seen assumes behavior B. Behavior B is also a good default setting in my opinion.

We could longterm give users some wrapper that is similar to subtest but that is explicit in isolating exceptions from the code outside so users can kind of add an error summary.
Such summary as "the following subtests failed" is not helpful in many tests, because the failing subtests were collected, but then the whole test times out after some steps did not work that would have been prerequisites to following steps.

@flokli
Copy link
Contributor

@flokli flokli commented Feb 6, 2020

I agree that a lot of tests currenly assume B.

I'm just hesitant introducing semantic changes with both python and perl driver around, so would really feel better if we pick this up when the perl driver isn't around anymore.

@tfc
Copy link
Contributor Author

@tfc tfc commented Feb 6, 2020

I'm just hesitant introducing semantic changes with both python and perl driver around, so would really feel better if we pick this up when the perl driver isn't around anymore.

I agree that it is not cool to have different semantics in different implementations that share the same names.

But we have a lot of things in subtests that are necessary prerequisites to their next steps, but won't blow up visibly, leading to tests that time out (e.g. something must be done, but that fails, and then a test loops on wait_until_x which then can never happen) and are hard to debug.

Regarding the minority of still existing perl tests that remain untouched and working until they are ported (and people will see that they get more fails instead of less then, which is always better than the other way around), i suggest putting this in.

@worldofpeace
Copy link
Member

@worldofpeace worldofpeace commented Feb 6, 2020

Just want to mention if there's any decision on this please merge after branch off.

@flokli
Copy link
Contributor

@flokli flokli commented Feb 6, 2020

@flokli flokli mentioned this pull request Feb 7, 2020
0 of 10 tasks complete
@flokli
Copy link
Contributor

@flokli flokli commented Apr 30, 2020

I stumbled more than once into python tests failing in a previous subtests, while I was staring at the later (still running) output.

As this doesn't change semantics of passing tests, and as tests run isolated, I don't have any strong objections against this.

tfc and others added 2 commits Feb 6, 2020
With the tests now bailing out early on a failing subtest, we don't need
to keep a list of failed tests, or the number of total tests
@flokli flokli force-pushed the tfc:python-test-driver-reraise branch from 555b982 to 3cdd558 Apr 30, 2020
@flokli
Copy link
Contributor

@flokli flokli commented Apr 30, 2020

I pushed some cleanups, PTAL.

@tfc
Copy link
Contributor Author

@tfc tfc commented Apr 30, 2020

@flokli looks good!

@flokli flokli merged commit ae70c38 into NixOS:master Apr 30, 2020
14 checks passed
14 checks passed
Evaluation Performance Report Evaluator Performance Report
Details
grahamcofborg-eval ^.^!
Details
grahamcofborg-eval-check-maintainers matching changed paths to changed attrs...
Details
grahamcofborg-eval-check-meta config.nix: checkMeta = true
Details
grahamcofborg-eval-darwin nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="3cdd558"; rev="3cdd558a6f785fb3a934c7c1820b3576c9e8b5b6"; } ./pkgs/t
Details
grahamcofborg-eval-lib-tests nix-build --arg pkgs import ./. {} ./lib/tests/release.nix
Details
grahamcofborg-eval-nixos nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="3cdd558"; rev="3cdd558a6f785fb3a934c7c1820b3576c9e8b5b6"; } ./nixos/
Details
grahamcofborg-eval-nixos-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="3cdd558"; rev="3cdd558a6f785fb3a934c7c1820b3576c9e8b5b6"; } ./nixos/
Details
grahamcofborg-eval-nixos-options nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="3cdd558"; rev="3cdd558a6f785fb3a934c7c1820b3576c9e8b5b6"; } ./nixos/
Details
grahamcofborg-eval-nixpkgs-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="3cdd558"; rev="3cdd558a6f785fb3a934c7c1820b3576c9e8b5b6"; } ./pkgs/t
Details
grahamcofborg-eval-nixpkgs-tarball nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="3cdd558"; rev="3cdd558a6f785fb3a934c7c1820b3576c9e8b5b6"; } ./pkgs/t
Details
grahamcofborg-eval-nixpkgs-unstable-jobset nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="3cdd558"; rev="3cdd558a6f785fb3a934c7c1820b3576c9e8b5b6"; } ./pkgs/t
Details
grahamcofborg-eval-package-list nix-env -qa --json --file .
Details
grahamcofborg-eval-package-list-no-aliases nix-env -qa --json --file . --arg config { allowAliases = false; }
Details
flokli added a commit to flokli/nixpkgs that referenced this pull request May 1, 2020
5150378 fixed the long-broken
nixosTests.networking.virtual.

With all tests failures fixed, and NixOS#79328 making debugging much easier,
let's re-add it to the tested jobset.
@flokli flokli mentioned this pull request May 1, 2020
3 of 10 tasks complete
flokli added a commit to flokli/nixpkgs that referenced this pull request May 9, 2020
5150378 fixed the long-broken
nixosTests.networking.virtual.

With all tests failures fixed, and NixOS#79328 making debugging much easier,
let's re-add it to the tested jobset.
flokli added a commit to flokli/nixpkgs that referenced this pull request May 9, 2020
5150378 fixed the long-broken
nixosTests.networking.virtual.

With all tests failures fixed, and NixOS#79328 making debugging much easier,
let's re-add it to the tested jobset.
flokli added a commit to flokli/nixpkgs that referenced this pull request Aug 27, 2020
5150378 fixed the long-broken
nixosTests.networking.virtual.

With all tests failures fixed, and NixOS#79328 making debugging much easier,
let's re-add it to the tested jobset.
peti added a commit that referenced this pull request Aug 28, 2020
5150378 fixed the long-broken
nixosTests.networking.virtual.

With all tests failures fixed, and #79328 making debugging much easier,
let's re-add it to the tested jobset.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants
You can’t perform that action at this time.