RuntimeError with NetCDF4FileHandler

[gerritholl]

Describe the bug

When trying to use the NetCDF4FileHandler to access the contents of my data file, it fails with a RuntimeError (see below for details).

To Reproduce

# Your code here
from satpy.readers.netcdf_utils import NetCDF4FileHandler
from satpy.readers.fci_l1c_fdhsi import FCIFDHSIFileHandler
from satpy.utils import debug_on
debug_on()
import datetime

ncfh = NetCDF4FileHandler(
        '/media/nas/x21308/2019_05_Testdata/MTG/FCI/FDHSI/uncompressed/20170410_RC70/W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--CHK-BODY--L2P-NC4E_C_EUMT_20170410113551_GTT_DEV_20170410113042_20170410113051_N__C_0070_0006.nc',
        {'pflag': 'W', 'location_indicator': 'XX-EUMETSAT-Darmstadt', 'data_designator': 'IMG+SAT', 'spacecraft_id': 1, 'data_source': 'FCI', 'processing_level': '1C', 'type': 'RRAD', 'subtype': 'FDHSI', 'coverage': 'FD', 'subsetting': '', 'component1': 'CHK', 'component2': 'BODY', 'component3': '', 'purpose': 'L2P', 'format': 'NC4E', 'oflag': 'C', 'originator': 'EUMT', 'processing_time': datetime.datetime(2017, 4, 10, 11, 35, 51), 'facility': 'GTT', 'environment': 'DEV', 'start_time': datetime.datetime(2017, 4, 10, 11, 30, 42), 'end_time': datetime.datetime(2017, 4, 10, 11, 30, 51), 'processing_mode': 'N', 'special_compression': '', 'disposition_mode': 'C', 'repeat_cycle_in_day': 70, 'count_in_repeat_cycle': 6},
        {'file_reader': FCIFDHSIFileHandler, 'file_patterns': ['{pflag}_{location_indicator},{data_designator},MTI{spacecraft_id:1d}+{data_source}-{processing_level}-{type}-{subtype}-{coverage}-{subsetting}-{component1}-{component2}-{component3}-{purpose}-{format}_{oflag}_{originator}_{processing_time:%Y%m%d%H%M%S}_{facility}_{environment}_{start_time:%Y%m%d%H%M%S}_{end_time:%Y%m%d%H%M%S}_{processing_mode}_{special_compression}_{disposition_mode}_{repeat_cycle_in_day:>04d}_{count_in_repeat_cycle:>04d}.nc'], 'file_type': 'fci_l1c_fdhsi'})

print(ncfh.file_content)

(file available to core PyTroll developers)

Expected behavior

I expect the contents of the file to be shown.

Actual results

/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/site-packages/pyspectral/config.py:75: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  config = recursive_dict_update(config, yaml.load(fp_))
Traceback (most recent call last):
  File "/tmp/mwe8.py", line 12, in <module>
    print(ncfh.file_content)
  File "netCDF4/_netCDF4.pyx", line 3901, in netCDF4._netCDF4.Variable.__repr__
  File "netCDF4/_netCDF4.pyx", line 3910, in netCDF4._netCDF4.Variable.__unicode__
  File "netCDF4/_netCDF4.pyx", line 4037, in netCDF4._netCDF4.Variable.dimensions.__get__
  File "netCDF4/_netCDF4.pyx", line 3969, in netCDF4._netCDF4.Variable._getdims
  File "netCDF4/_netCDF4.pyx", line 1857, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: Not a valid ID

Environment Info:

• OS: openSUSE 15.0
• Satpy Version: v0.14.2-64-ga2aae4a4

[gerritholl]

Reading data variables that do exist often results in KeyError.

I can read the file fine directly with netCDF4, maybe it's a problem in xarray or in some combination of libraries, or maybe there's something seriously wrong with the file.

[gerritholl]

Here no RuntimeError, but a KeyError where I expected the data to be found:

from satpy.readers.netcdf_utils import NetCDF4FileHandler
from satpy.readers.fci_l1c_fdhsi import FCIFDHSIFileHandler
from satpy.utils import debug_on
debug_on()
import datetime
import netCDF4

fn = '/media/nas/x21308/2019_05_Testdata/MTG/FCI/FDHSI/uncompressed/20170410_RC70/W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--CHK-BODY--L2P-NC4E_C_EUMT_20170410113551_GTT_DEV_20170410113042_20170410113051_N__C_0070_0006.nc'
ncfh = NetCDF4FileHandler(
        fn,
        {'pflag': 'W', 'location_indicator': 'XX-EUMETSAT-Darmstadt', 'data_designator': 'IMG+SAT', 'spacecraft_id': 1, 'data_source': 'FCI', 'processing_level': '1C', 'type': 'RRAD', 'subtype': 'FDHSI', 'coverage': 'FD', 'subsetting': '', 'component1': 'CHK', 'component2': 'BODY', 'component3': '', 'purpose': 'L2P', 'format': 'NC4E', 'oflag': 'C', 'originator': 'EUMT', 'processing_time': datetime.datetime(2017, 4, 10, 11, 35, 51), 'facility': 'GTT', 'environment': 'DEV', 'start_time': datetime.datetime(2017, 4, 10, 11, 30, 42), 'end_time': datetime.datetime(2017, 4, 10, 11, 30, 51), 'processing_mode': 'N', 'special_compression': '', 'disposition_mode': 'C', 'repeat_cycle_in_day': 70, 'count_in_repeat_cycle': 6},
        {'file_reader': FCIFDHSIFileHandler, 'file_patterns': ['{pflag}_{location_indicator},{data_designator},MTI{spacecraft_id:1d}+{data_source}-{processing_level}-{type}-{subtype}-{coverage}-{subsetting}-{component1}-{component2}-{component3}-{purpose}-{format}_{oflag}_{originator}_{processing_time:%Y%m%d%H%M%S}_{facility}_{environment}_{start_time:%Y%m%d%H%M%S}_{end_time:%Y%m%d%H%M%S}_{processing_mode}_{special_compression}_{disposition_mode}_{repeat_cycle_in_day:>04d}_{count_in_repeat_cycle:>04d}.nc'], 'file_type': 'fci_l1c_fdhsi'})

try:
    ncfh["data/vis_05/measured"]
except KeyError:
    print("satpy can't find my data")
else:
    print("satpy finds it")

with netCDF4.Dataset(fn, "r") as ds:
    try:
        ds["data"]["vis_05"]["measured"]
    except KeyError:
        print("neither can netCDF4")
    else:
        print("but netCDF4 can!")

Resulting in:

satpy can't find my data
but netCDF4 can!

[djhoese]

What do you get if you do ncfh.file_content.keys()? I wonder if the utility class can't handle nested groups. Looking at the code it should handle it, but I'm not sure I've ever needed it.

What do you get if you do:

nc_group = xr.open_dataset(self.filename, group='data/vis_05')
var = nc_group['measured']

[gerritholl]

Replacing the bit after constructing the NetCDF4FileHandler and before the context manager with

key = "data/vis_05/measured"
keys = ncfh.file_content.keys()
print("found", len(keys), "keys")
vis05_meas_keys = [k for k in keys if k.startswith(key)]
print("including", len(vis05_meas_keys), "starting with", key)
print(key, "in keys", key in keys)
try:
    ncfh[key]
except KeyError:
    print("but satpy can't find my data")
else:
    print("and satpy finds it")

the output becomes

found 3913 keys
including 106 starting with data/vis_05/measured
data/vis_05/measured in keys False
but satpy can't find my data
but netCDF4 can!

[gerritholl]

If I try key="data/vis_05/measured/effective_radiance", then the data are found, so maybe this is all a misunderstanding on my side (perhaps a need for documentation improvement) on whether this key-indexing is supposed to work for Groups at all? Or is it only supposed to work for Variables?

[gerritholl]

It doesn't read groups, not variables and attributes.