fuzz: different results for `f64sqrt`

[abrown]

Test Case

(module
  (type (;0;) (func (param f64) (result f64)))
  (func (;0;) (type 0) (param f64) (result f64)
    local.get 0
    f64.sqrt
  )
  (export "test" (func 0))
)

See crash log, crash input, and the WAT/Wasm file: fuzz-bug.zip.

Steps to Reproduce

$ RUST_LOG=wasmtime_fuzzing=debug cargo +nightly fuzz run differential_meta fuzz/artifacts/differential_meta/crash-4a7edd5b645711ba09c0356fceb14040e130e948

Expected Results

Wasmtime and the spec interpreter should have the same results for f64.sqrt.

Actual Results

Wasmtime and the spec interpreter have different results:

[2022-08-10T18:27:26Z DEBUG wasmtime_fuzzing::oracles] Evaluating: test([F64(18442240474082181119)])
[2022-08-10T18:27:26Z DEBUG wasmtime_fuzzing::oracles]  -> results on spec: Ok([F64(9221120237041090560)])
[2022-08-10T18:27:26Z DEBUG wasmtime_fuzzing::oracles]  -> results on wasmtime: Ok([F64(18444492273895866368)])

Versions and Environment

Wasmtime version or commit: meta-diff branch

Operating system: Fedora 35

Architecture: x86-64

Extra Info

@conrad-watt, any thoughts on this one?

[alexcrichton]

This looks like a case where the debug printing could probably be improved along the lines of what wasmtime-wast does. The F64(N) is actually a float where the bit pattern of the float is represented by N which means the values in question are:

    -1.7976931348623157e308
    NaN
    NaN

so both the spec interpreter and Wasmtime are producing NaN, they're just producing different signs for the nan. I think spec-wise this is allowed and indicates that we'll need to perform nan normalization of results perhaps.

This is only an issue for the single-instruction-module generator since the wasm-smith generator already has nan canonicalization enabled at the wasm bytecode level to paper over differences between engines.

[abrown]

Thanks; I've updated my branch to take this into account (somewhat).