Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runInLinuxVM, test-driver: pass host's cpu type to guest vm #59855

Merged
merged 1 commit into from Feb 2, 2020

Conversation

@volth
Copy link
Contributor

@volth volth commented Apr 18, 2019

kvm64 is the most generic CPU, which does not support SSE4.2, AVX and other ISA extentions.

There are already some packages which use extended opcodes. They cannot be run in VM.

With the acceptance of #59225 and #59148 even glibc would have AVX instructions in memcpy() code (if platform.gcc.arch is at least sandybridge) so almost all the packages from host's nix store shared with the VM via 9p won't run in the VM if the the VM does not have the same CPU as the host.

@volth
Copy link
Contributor Author

@volth volth commented Apr 18, 2019

@GrahamcOfBorg test xrdp

Copy link
Contributor

@tomfitzhenry tomfitzhenry left a comment

Is this a breaking change? -cpu kvm64 makes it easier to perform live migrations (since -cpu kvm64 is a lowest-common-denominator), and thus users who are currently performing live migrations (?) of guests between hosts may no longer be able to perform live migrations.

Perhaps this should be mentioned in the release notes.

@volth
Copy link
Contributor Author

@volth volth commented Nov 17, 2019

'kvm64' is the most generic CPU, which does not support SSE4.2, AVX and other ISA extentions.
@volth volth force-pushed the volth:qemu-cpu-passthru branch from 1421e5e to 2bd296a Jan 15, 2020
@volth
Copy link
Contributor Author

@volth volth commented Jan 15, 2020

rebased

@Ericson2314
Copy link
Member

@Ericson2314 Ericson2314 commented Jan 20, 2020

Can we make this deterministic?

@volth
Copy link
Contributor Author

@volth volth commented Jan 20, 2020

It is as deterministic as running code without virtualization.

@Ericson2314
Copy link
Member

@Ericson2314 Ericson2314 commented Jan 20, 2020

@volth Well, I don't really like that impurity either :). But haven't you done tremendous work adding stuff to *Platform.parsed so we can both be deterministic and not exclusively support some ancient version?

@volth
Copy link
Contributor Author

@volth volth commented Jan 20, 2020

Besides that platform.gcc.arch, there are tons of program which do runtime cpu detection (mainly math, graphics and multimedia) and now we have non-determinism because of difference between non-virtualized run (which exposes all features in /proc/cpuinfo) and test-driver run on "deterministic cpu".

Even putting my work aside, the question is still here: what is determinism here: run tests on "deterministic cpu"? or getting same results as on the bare hardware?

@Ericson2314
Copy link
Member

@Ericson2314 Ericson2314 commented Jan 20, 2020

I really do prefer the former. I would like Nix to someday set the CPU mode or something to improve the bare metal case too. I don't like feature detection be it configure scripts or run time fat binary tricks.

@volth
Copy link
Contributor Author

@volth volth commented Jan 20, 2020

I really do prefer the former.

I tend to like the latter :) Just to avoid green tests and failures after deployment.

I don't like feature detection be it configure scripts or run time fat binary tricks.

Fat binary tricks have sense even combined with per-closure platform.gcc.arch: I am using mostly platform.gcc.arch = "sandybridge" as "sandybridge" is the oldest platform here. But many servers are newer, and runtime detection could handle it.

Not to say that everyone but us (and maybe 2-3 people more) do not set platform.gcc.arch at all and runtime detection is the only way to take advantage on AVX

@Ericson2314
Copy link
Member

@Ericson2314 Ericson2314 commented Feb 2, 2020

Well if @matthewbauer thinks this is fine I'll relent for now, but let's create an issue.

@Ericson2314 Ericson2314 merged commit 14fbd41 into NixOS:master Feb 2, 2020
12 checks passed
12 checks passed
Evaluation Performance Report Evaluator Performance Report
Details
grahamcofborg-eval ^.^!
Details
grahamcofborg-eval-check-meta config.nix: checkMeta = true
Details
grahamcofborg-eval-darwin nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A darwin-tested
Details
grahamcofborg-eval-nixos nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release-combined.nix -A tested
Details
grahamcofborg-eval-nixos-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release.nix -A manual
Details
grahamcofborg-eval-nixos-options nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release.nix -A options
Details
grahamcofborg-eval-nixpkgs-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A manual
Details
grahamcofborg-eval-nixpkgs-tarball nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A tarball
Details
grahamcofborg-eval-nixpkgs-unstable-jobset nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A unstable
Details
grahamcofborg-eval-package-list nix-env -qa --json --file .
Details
grahamcofborg-eval-package-list-no-aliases nix-env -qa --json --file . --arg config { allowAliases = false; }
Details
@@ -17,9 +17,9 @@ in
else throw "Unknown QEMU serial device for system '${pkgs.stdenv.hostPlatform.system}'";

qemuBinary = qemuPkg: {
x86_64-linux = "${qemuPkg}/bin/qemu-kvm -cpu kvm64";

This comment has been minimized.

@jslight90

jslight90 Feb 27, 2020
Contributor

This change is causing all of my NixOS tests to fail with error: [Errno 32] Broken pipe when calling machine.shell.send. I am trying to run the tests on a NixOS VM running in Hyper-V. I'm not sure if the nested virtualization is the cause of this.

This comment has been minimized.

@conferno

conferno Feb 27, 2020
Contributor

Does qemu-kvm -cpu host work or fail with some error?
"Broken pipe" might indicate the qemu process exit.
It might print an useful message before exit

This comment has been minimized.

@jslight90

jslight90 Feb 27, 2020
Contributor

It fails with the following error:

qemu-system-x86_64: error: failed to set MSR 0x48b to 0x358ae00000000
qemu-system-x86_64: /build/qemu-4.2.0/target/i386/kvm.c:2947: kvm_put_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.
Aborted
@nixos-discourse

This comment has been minimized.

Copy link

@nixos-discourse nixos-discourse commented on 2bd296a Apr 16, 2020

This commit has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/new-constraint-between-release-19-09-and-20-03-in-package-qemu/6297/2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

9 participants
You can’t perform that action at this time.