Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nixos/testing: add deprecation notice for Perl VM tests #79335

Merged
merged 3 commits into from Feb 10, 2020

Conversation

@flokli
Copy link
Contributor

@flokli flokli commented Feb 6, 2020

Most VM tests have been migrated to use the python test driver
(introduced in #71684), the migration is tracked in #72828 (which also
thankfully uncovered and fixed many currently broken tests)

While increasing the acceptance and adoption of NixOS integration tests
by using a more popular language, there was also nobody willing to do
larger refactors in the currently very convoluted test infrastructure.

We plan to remove the perl infrastructure between the 20.03 and 20.09
release, to be able to do these refactorings.

Some people might be using Perl tests in their internal CI, so print a
warning for 20.03, and give users time to move to the python testing
infrastructure.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.
@flokli flokli added this to the 20.03 milestone Feb 6, 2020
@flokli flokli requested review from grahamc, Mic92, fpletz, lheckemann and globin Feb 6, 2020
Copy link
Contributor

@jonringer jonringer left a comment

I agree with the intent of this PR.

Just concerned that the warning message may get lost as nixos tests are particularly noisy. Using nix build it might be useful, but nix-build will definitely blanket the warning with noise.

@ofborg ofborg bot added the 6.topic: nixos label Feb 6, 2020
@flokli
Copy link
Contributor Author

@flokli flokli commented Feb 6, 2020

@worldofpeace
Copy link
Member

@worldofpeace worldofpeace commented Feb 6, 2020

@worldofpeace I assume you're adding a general explanation to the release notes as well?

There's one thing that's being forgot here aside from the deprecation notice and it's nixosTest uses nixos/lib/testing.nix.

@worldofpeace
Copy link
Member

@worldofpeace worldofpeace commented Feb 6, 2020

@flokli I believe I will be adding that as a highlight.

@fpletz
Copy link
Member

@fpletz fpletz commented Feb 7, 2020

While in principle being in favour of this, I have a question about your refactoring plans: Will the current Python driver API (both Nix and Python) stay compatible? If we advocate to move away from the Perl test infra, we should have something ready that will still be compatible for at least the next release in half a year.

If there a big changes planned, keeping the Perl stuff around could be worthwhile and a manageable technical debt because both test drivers could be cleanly separated with the Perl driver in maintenance mode.

If the Python API is considered stable, awesome! 👍

@flokli
Copy link
Contributor Author

@flokli flokli commented Feb 7, 2020

@flokli flokli force-pushed the flokli:deprecate-perl-vm-tests branch from 503dfbd to 101dd1f Feb 8, 2020
@flokli flokli force-pushed the flokli:deprecate-perl-vm-tests branch from c12ef95 to d43a353 Feb 9, 2020
worldofpeace and others added 3 commits Feb 9, 2020
This still referred to the legacy make-test.nix (Perl)
Most VM tests have been migrated to use the python test driver
(introduced in #71684), the migration is tracked in #72828 (which also
thankfully uncovered and fixed many currently broken tests)

While increasing the acceptance and adoption of NixOS integration tests
by using a more popular language, there was also nobody willing to do
larger refactors in the currently very convoluted test infrastructure.

We plan to remove the perl infrastructure between the 20.03 and 20.09
release, to be able to do these refactorings.

Some people might be using Perl tests in their internal CI, so print a
warning for 20.03, and give users time to move to the python testing
infrastructure.
@flokli flokli force-pushed the flokli:deprecate-perl-vm-tests branch from d43a353 to 0945178 Feb 9, 2020
@worldofpeace worldofpeace requested review from aszlig and adisbladis Feb 10, 2020
@worldofpeace
Copy link
Member

@worldofpeace worldofpeace commented Feb 10, 2020

This PR does update nixosTest to use the python driver, and you can't use the nixpkgs function to access the perl one anymore. @flokli noted in IRC that he didn't think that would be problematic.

@flokli
Copy link
Contributor Author

@flokli flokli commented Feb 10, 2020

Well, nixosTest seems to be quite newer the import ./make-test.nix pattern, which can still be used to execute perl tests.

I don't think it makes any sense to introduce a nixosTestPerl at least 😆

@worldofpeace
Copy link
Member

@worldofpeace worldofpeace commented Feb 10, 2020

Well, nixosTest seems to be quite newer the import ./make-test.nix pattern, which can still be used to execute perl tests.

I don't think it makes any sense to introduce a nixosTestPerl at least laughing

I think similarly that having anything like a nixosTestPerl would be pointless if it's just going to be removed in 20.09.

@worldofpeace worldofpeace merged commit aa4ba50 into NixOS:master Feb 10, 2020
13 checks passed
13 checks passed
Evaluation Performance Report Evaluator Performance Report
Details
grahamcofborg-eval ^.^!
Details
grahamcofborg-eval-check-maintainers matching changed paths to changed attrs...
Details
grahamcofborg-eval-check-meta config.nix: checkMeta = true
Details
grahamcofborg-eval-darwin nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A darwin-tested
Details
grahamcofborg-eval-nixos nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release-combined.nix -A tested
Details
grahamcofborg-eval-nixos-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release.nix -A manual
Details
grahamcofborg-eval-nixos-options nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release.nix -A options
Details
grahamcofborg-eval-nixpkgs-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A manual
Details
grahamcofborg-eval-nixpkgs-tarball nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A tarball
Details
grahamcofborg-eval-nixpkgs-unstable-jobset nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A unstable
Details
grahamcofborg-eval-package-list nix-env -qa --json --file .
Details
grahamcofborg-eval-package-list-no-aliases nix-env -qa --json --file . --arg config { allowAliases = false; }
Details
@worldofpeace worldofpeace mentioned this pull request Feb 23, 2020
1 of 11 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants
You can’t perform that action at this time.