Fix flaky behavior of test_xeb_fidelity

[pavoljuhas]

Increase tolerance for fidelities calculated with and without state
vector amplitudes. Avoid flaky, random-seed dependent test failures
on a slightly excessive difference.

Example: https://github.com/quantumlib/Cirq/actions/runs/6726576064/job/18283075054#step:6:598

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (2f7d732) 97.83% compared to head (a53a76c) 97.83%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6337      +/-   ##
==========================================
- Coverage   97.83%   97.83%   -0.01%     
==========================================
  Files        1108     1108              
  Lines       96352    96352              
==========================================
- Hits        94270    94269       -1     
- Misses       2082     2083       +1     

Files	Coverage Δ
...-core/cirq/experiments/fidelity_estimation_test.py	100.00% <100.00%> (ø)

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[NoureldinYosri]

lets use pytest.approx instead

assert f2 == pytest.approx(f, rel=1e-6)

This will assert that $f2$ equals $f$ up to relative error of 1e-6. This should be more robust unless $f$ is almost zero in which case we can make it

assert f2 == pytest.approx(f, rel=1e-6, abs=1e-6)

so that it works for all cases

[pavoljuhas]

@NoureldinYosri - thank you for the suggestion, done in 9ddcbaa.
Good to know about pytest.approx it is however not used in cirq. numpy.allclose does the same thing here so I opted for that one.

[NoureldinYosri]

relative tolerance works only for large numbers. for small number absolute tolerance is needed so

np.allclose(f, f2, atol=1e-6, rtol=1e-6)

[pavoljuhas]

In this case the initial absolute value comparison was better as the fidelity is supposed to be on the [0, 1] interval.
For some reason the round-off errors on Mac got it over the threshold.