VCF lossless conversion tests are failing for Zarr 2.11.0

[tomwhite]

From https://github.com/pystatgen/sgkit/runs/5401218690?check_suite_focus=true (which is truncated due to lots of output):

2022-03-03T15:45:05.4820016Z =========================== short test summary info ============================
2022-03-03T15:45:05.4820284Z FAILED sgkit/tests/io/vcf/test_vcf_lossless_conversion.py::test_lossless_conversion[CEUTrio.20.21.gatk3.4.g.vcf.bgz]
2022-03-03T15:45:05.4820484Z FAILED sgkit/tests/io/vcf/test_vcf_lossless_conversion.py::test_lossless_conversion[all_fields.vcf]
2022-03-03T15:45:05.4820638Z = 2 failed, 813 passed, 32 skipped, 2 xfailed, 1 xpassed in 259.62s (0:04:19) ==

I haven't been able to reproduce this, but it looks like some fields are nan (e.g. QUAL) which is odd:

2022-03-03T15:45:05.4783854Z #CHROM     POS     ID      REF     ALT     QUAL    FILTER  INFO    FORMAT  s1      s2
2022-03-03T15:45:05.4783952Z 1  1       .       G       A,C     nan     PASS    IB0     .       .       .

[tomwhite]

I looked into this a bit more and found that it's due to changes in Zarr 2.11.0. In that release, chunks with no data are not written to disk. However, we use two types of NaN value to encode missing vs fill (see https://github.com/pystatgen/vcf-zarr-spec/blob/main/vcf_zarr_spec.md#missing-and-fill-values) both of which are different to regular NaN. So the QUAL field, which is all missing in this example, is not written to disk and then is read as a regular NaN when read back.

It's possible to use the old Zarr behaviour, by setting write_empty_chunks=True when creating the Zarr file. However, Xarray creates the Zarr file for us, and I couldn't see a way of passing this through to Xarray at first glance.

For the moment we should probably pin to zarr<2.11.0 while we fix this. It needs a bit more investigation but may need changes in both Zarr and Xarray.

[jeromekelleher]

We've hit this issue with tsinfer as well @tomwhite . For reference:

• Problems with zarr 2.11.0 tskit-dev/tsinfer#628
• Incorrect default fill value causes byte arrays to become numeric when write_empty_chunks=False zarr-developers/zarr-python#965

[tomwhite]

Thanks for the references @jeromekelleher!

[tomwhite]

Opened pydata/xarray#6347

[tomwhite]

Opened zarr-developers/zarr-specs#194

[tomwhite]

Fixed in zarr-developers/zarr-python#834