Issue loading MODIS Aqua data #626

andyprata · 2019-02-20T19:17:58Z

Describe the bug
I have no issue loading Terra data, but when I load MODIS data from Aqua into a Scene object I get the following error.

To Reproduce

from satpy.scene import Scene
from satpy import find_files_and_readers

path_data = '/home/aprata/satellite/etna/modis_aqua/data/20190511225/'
files = find_files_and_readers(base_dir=path_data,
                               sensor='modis',
                               reader='hdfeos_l1b')
print("Reading files...")
scn = Scene(filenames=files)

Expected behavior
Data should load into the Scene object.

Actual results

[DEBUG: 2019-02-20 20:07:05 : satpy.readers] Reading ['/home/aprata/anaconda3/envs/satpy-env/lib/python3.6/site-packages/satpy/etc/readers/modis_l1b.yaml']
[DEBUG: 2019-02-20 20:07:05 : satpy.scene] Setting 'PPP_CONFIG_DIR' to '/home/aprata/anaconda3/envs/satpy-env/lib/python3.6/site-packages/satpy/etc'
[DEBUG: 2019-02-20 20:07:05 : satpy.readers] Reading ['/home/aprata/anaconda3/envs/satpy-env/lib/python3.6/site-packages/satpy/etc/readers/modis_l1b.yaml']
Reading files...
[DEBUG: 2019-02-20 20:07:05 : satpy.readers.yaml_reader] Assigning to modis_l1b: ['/home/aprata/satellite/etna/modis_aqua/data/20190511225/MYD02QKM.A2019051.1225.061.NRT.hdf', '/home/aprata/satellite/etna/modis_aqua/data/20190511225/MYD02HKM.A2019051.1225.061.NRT.hdf', '/home/aprata/satellite/etna/modis_aqua/data/20190511225/MYD021KM.A2019051.1225.061.NRT.hdf', '/home/aprata/satellite/etna/modis_aqua/data/20190511225/MYD03.A2019051.1225.061.NRT.hdf']
Traceback (most recent call last):
  File "/home/aprata/anaconda3/envs/satpy-env/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2683, in safe_execfile
    self.compile if shell_futures else None)
  File "/home/aprata/anaconda3/envs/satpy-env/lib/python3.6/site-packages/IPython/utils/py3compat.py", line 188, in execfile
    exec(compiler(f.read(), fname, 'exec'), glob, loc)
  File "/home/aprata/PycharmProjects/etna/src/plot_modis_btd.py", line 25, in <module>
    scn = Scene(filenames=files)
  File "/home/aprata/anaconda3/envs/satpy-env/lib/python3.6/site-packages/satpy/scene.py", line 154, in __init__
    reader_kwargs=reader_kwargs)
  File "/home/aprata/anaconda3/envs/satpy-env/lib/python3.6/site-packages/satpy/scene.py", line 199, in create_reader_instances
    ppp_config_dir=self.ppp_config_dir)
  File "/home/aprata/anaconda3/envs/satpy-env/lib/python3.6/site-packages/satpy/readers/__init__.py", line 657, in load_readers
    reader_instance.create_filehandlers(loadables)
  File "/home/aprata/anaconda3/envs/satpy-env/lib/python3.6/site-packages/satpy/readers/yaml_reader.py", line 500, in create_filehandlers
    filename_set)
  File "/home/aprata/anaconda3/envs/satpy-env/lib/python3.6/site-packages/satpy/readers/yaml_reader.py", line 487, in new_filehandlers_for_filetype
    return list(filtered_iter)
  File "/home/aprata/anaconda3/envs/satpy-env/lib/python3.6/site-packages/satpy/readers/yaml_reader.py", line 460, in filter_fh_by_metadata
    for filehandler in filehandlers:
  File "/home/aprata/anaconda3/envs/satpy-env/lib/python3.6/site-packages/satpy/readers/yaml_reader.py", line 403, in new_filehandler_instances
    yield filetype_cls(filename, filename_info, filetype_info, *req_fh)
  File "/home/aprata/anaconda3/envs/satpy-env/lib/python3.6/site-packages/satpy/readers/modis_l1b.py", line 122, in __init__
    HDFEOSFileReader.__init__(self, filename, filename_info, filetype_info)
  File "/home/aprata/anaconda3/envs/satpy-env/lib/python3.6/site-packages/satpy/readers/modis_l1b.py", line 64, in __init__
    self.metadata = self.read_mda(self.sd.attributes()['CoreMetadata.0'])
  File "/home/aprata/anaconda3/envs/satpy-env/lib/python3.6/site-packages/satpy/readers/modis_l1b.py", line 96, in read_mda
    val = eval(val)
  File "<string>", line 1
    ("MYD01.61.2019-051T12:25:00.000000Z.NA.29878844.500100_1.hdf", "MYD03LUT.coeff_V6.1.4", "PM1EPHND_NRT.A2019051.1220.061.2019051125628", "PM1EPHND_NRT.A2019051.1225.061.2019051125628", "PM1EPHND_NRT.A2019051.1230.061.2019051125628", "
                                                                                                                                                                                                                                             ^
SyntaxError: EOL while scanning string literal

Environment Info:

OS: Linux
SatPy Version: 0.11.2
PyResample Version: 1.10.3

Additional context
You can download the data here (note you'll need an EarthData login):
https://nrt4.modaps.eosdis.nasa.gov/archive/allData/61/MYD021KM/2019/051/MYD021KM.A2019051.1225.061.NRT.hdf
https://nrt4.modaps.eosdis.nasa.gov/archive/allData/61/MYD02QKM/2019/051/MYD02QKM.A2019051.1225.061.NRT.hdf
https://nrt4.modaps.eosdis.nasa.gov/archive/allData/61/MYD02HKM/2019/051/MYD02HKM.A2019051.1225.061.NRT.hdf
https://nrt4.modaps.eosdis.nasa.gov/archive/allData/61/MYD03/2019/051/MYD03.A2019051.1225.061.NRT.hdf

The text was updated successfully, but these errors were encountered:

djhoese · 2019-02-20T19:20:43Z

Please read the error message you were given. have you edited any satpy source code? This is a syntax error, seems like a missing or extra quote. I'm guessing you typed a filename incorrectly. You should receive a SyntaxError here regardless of Terra versus Aqua.

andyprata · 2019-02-20T19:51:19Z

I haven't edited any satpy code. I followed the Traceback and I think it's something to do with the way a string is split using the '\n' character. See line 83 of modis_l1b.py

I haven't typed in any file names either... I used the find_files_and_readers utility and it returned:

{'modis_l1b': ['/home/aprata/satellite/etna/modis_aqua/data/20190511225/MYD02QKM.A2019051.1225.061.NRT.hdf',
  '/home/aprata/satellite/etna/modis_aqua/data/20190511225/MYD02HKM.A2019051.1225.061.NRT.hdf',
  '/home/aprata/satellite/etna/modis_aqua/data/20190511225/MYD021KM.A2019051.1225.061.NRT.hdf',
  '/home/aprata/satellite/etna/modis_aqua/data/20190511225/MYD03.A2019051.1225.061.NRT.hdf']}

Which looks fine. So I think it's an issue in modis_l1b.py

djhoese · 2019-02-20T19:59:02Z

So after a quick glance at the code, it seems that the reader is trying to run the metadata string from the file as a python "literal". So trying to convert a list of filenames or whatever in to a python list of filenames. From the way that the parsing is going it is ending up with an extra quotation mark on one line that shouldn't be there. Meaning the software that wrote this file put a newline (I'm guessing) after a quotation mark.

@mraspaud @adybbroe did one of you write the metadata parsing code? We may have to be smarter about this, but iirc one of the recent PRs for MODIS (#611 ) refactored a lot of this (FYI @LTMeyer).

mraspaud · 2019-02-20T21:11:32Z

I did write this a long time ago, and indeed interpreting the metadata string could lead to problems if the metadata is not properly formatted.

LTMeyer · 2019-02-21T10:36:35Z

I got similar errors trying to read MODIS 35 L2 products, without any modification of the HDF file nor of the reader itself. I believe the errors is raised from the following lines (https://github.com/pytroll/satpy/blob/master/satpy/readers/modis_l1b.py#L92-L96) probably because of the metadata format in MODIS HDF L2 products.
To avoid this, I've changed the metadata parser logic to use regular expression (see 5cd228e). It solved this particular issue. However, I don't know if it is optimal or consistent with other readers.

mraspaud · 2019-03-12T10:03:48Z

@andyprata Looks like the data links aren't valid anymore. I'd like to test this, would you mind telling me were I can find the data ?

andyprata · 2019-03-12T11:10:46Z

@mraspaud They must've moved out of the near-real time directory. You can download the data here now:

https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/61/MYD021KM/2019/051/MYD021KM.A2019051.1225.061.2019052150226.hdf
https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/61/MYD02QKM/2019/051/MYD02QKM.A2019051.1225.061.2019052150226.hdf
https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/61/MYD02HKM/2019/051/MYD02HKM.A2019051.1225.061.2019052150226.hdf
https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/61/MYD03/2019/051/MYD03.A2019051.1225.061.2019052145615.hdf

mraspaud · 2019-03-12T11:51:14Z

Thanks, I got the data now... and it works for me, ie I can't reproduce the problem with the last satpy/master, v0.11.2, v.0.12.0 in python 3 in a conda environment.

Just guessing, but could it be something with the pyhdf library used ? which one are you using ?

Here is the metadata I get:

{'INVENTORYMETADATA': {'ADDITIONALATTRIBUTES': {'ADDITIONALATTRIBUTESCONTAINER': {'ADDITIONALATTRIBUTENAME': {'VALUE': 'identifier_product_doi_authority'},
                                                                                  'INFORMATIONCONTENT': {'PARAMETERVALUE': {'VALUE': 'http://dx.doi.org'}}}},
                       'ASSOCIATEDPLATFORMINSTRUMENTSENSOR': {'ASSOCIATEDPLATFORMINSTRUMENTSENSORCONTAINER': {'ASSOCIATEDINSTRUMENTSHORTNAME': {'VALUE': 'MODIS'},
                                                                                                              'ASSOCIATEDPLATFORMSHORTNAME': {'VALUE': 'Aqua'},
                                                                                                              'ASSOCIATEDSENSORSHORTNAME': {'VALUE': 'MODIS'}}},
                       'COLLECTIONDESCRIPTIONCLASS': {'SHORTNAME': {'VALUE': 'MYD03'},
                                                      'VERSIONID': {'VALUE': 61}},
                       'ECSDATAGRANULE': {'DAYNIGHTFLAG': {'VALUE': 'Day'},
                                          'LOCALGRANULEID': {'VALUE': 'MYD03.A2019051.1225.061.2019052145615.hdf'},
                                          'LOCALVERSIONID': {'VALUE': '6.0.4'},
                                          'PRODUCTIONDATETIME': {'VALUE': '2019-02-21T14:56:15.000Z'},
                                          'REPROCESSINGACTUAL': {'VALUE': 'processed '
                                                                          'once'},
                                          'REPROCESSINGPLANNED': {'VALUE': 'further '
                                                                           'update '
                                                                           'is '
                                                                           'anticipated'}},
                       'GROUPTYPE': 'MASTERGROUP',
                       'INPUTGRANULE': {'INPUTPOINTER': {'VALUE': ('MYD01.61.2019-051T12:25:00.000000Z.NA.159272202.500100_1.hdf',
                                                                   'MYD03LUT.coeff_V6.1.4',
                                                                   'PM1EPHND.P2019051.1200.003.2019052143129',
                                                                   'PM1ATTNR.P2019051.1200.003.2019052143530')}},
                       'MEASUREDPARAMETER': {'MEASUREDPARAMETERCONTAINER': {'PARAMETERNAME': {'VALUE': 'Geolocation'},
                                                                            'QAFLAGS': {'AUTOMATICQUALITYFLAG': {'VALUE': 'Passed'},
                                                                                        'AUTOMATICQUALITYFLAGEXPLANATION': {'VALUE': 'Set '
                                                                                                                                     'to '
                                                                                                                                     "'Failed' "
                                                                                                                                     'if '
                                                                                                                                     'processing '
                                                                                                                                     'error '
                                                                                                                                     'occurred, '
                                                                                                                                     'set '
                                                                                                                                     'to '
                                                                                                                                     "'Passed' "
                                                                                                                                     'otherwise'},
                                                                                        'SCIENCEQUALITYFLAG': {'VALUE': 'Not '
                                                                                                                        'Investigated'}},
                                                                            'QASTATS': {'QAPERCENTMISSINGDATA': {'VALUE': 0},
                                                                                        'QAPERCENTOUTOFBOUNDSDATA': {'VALUE': 0}}}},
                       'ORBITCALCULATEDSPATIALDOMAIN': {'ORBITCALCULATEDSPATIALDOMAINCONTAINER': {'EQUATORCROSSINGDATE': {'VALUE': '2019-02-20'},
                                                                                                  'EQUATORCROSSINGLONGITUDE': {'VALUE': -172.683316485128},
                                                                                                  'EQUATORCROSSINGTIME': {'VALUE': '13:06:32.740093'},
                                                                                                  'ORBITNUMBER': {'VALUE': 89360}}},
                       'PGEVERSIONCLASS': {'PGEVERSION': {'VALUE': '6.1.4'}},
                       'RANGEDATETIME': {'RANGEBEGINNINGDATE': {'VALUE': '2019-02-20'},
                                         'RANGEBEGINNINGTIME': {'VALUE': '12:25:00.000000'},
                                         'RANGEENDINGDATE': {'VALUE': '2019-02-20'},
                                         'RANGEENDINGTIME': {'VALUE': '12:30:00.000000'}},
                       'SPATIALDOMAINCONTAINER': {'HORIZONTALSPATIALDOMAINCONTAINER': {'GPOLYGON': {'GPOLYGONCONTAINER': {'GRING': {'EXCLUSIONGRINGFLAG': {'VALUE': 'N'}},
                                                                                                                          'GRINGPOINT': {'GRINGPOINTLATITUDE': {'VALUE': (29.5171344332795,
                                                                                                                                                                          26.1481221584725,
                                                                                                                                                                          43.2445763295087,
                                                                                                                                                                          47.7960695907611)},
                                                                                                                                         'GRINGPOINTLONGITUDE': {'VALUE': (25.3839215625266,
                                                                                                                                                                           1.80398746929754,
                                                                                                                                                                           -6.50867170993159,
                                                                                                                                                                           23.0259943482517)},
                                                                                                                                         'GRINGPOINTSEQUENCENO': {'VALUE': (1,
                                                                                                                                                                            2,
                                                                                                                                                                            3,
                                                                                                                                                                            4)}}}}}}}}

andyprata · 2019-03-12T13:22:48Z

Ah ok. Yes I don't get any errors either with the nominal data. I tried again on the near-real time (NRT) data and I can reproduce the SyntaxError. So it's probably because the NRT data has a different metadata format. I would definitely find a NRT MODIS reader useful.

I've uploaded the original NRT data to my Google Drive if you would like to test on it:
https://drive.google.com/file/d/1u4nvc2aSpo9efeZjwGz0_XjAfQCA9n58/view?usp=sharing
https://drive.google.com/file/d/1InG2_hYvgskGfAMK72In5-9Kx4GPhgXV/view?usp=sharing
https://drive.google.com/file/d/1P5Koe5WTC-nBDBd9ob6-NARagcv3xelf/view?usp=sharing
https://drive.google.com/file/d/1QQOnULvUE9F-7NXRVFOYQzj__t4xtLXL/view?usp=sharing

mraspaud · 2019-03-12T13:35:21Z

Yep, now I can reproduce the problem :) Thanks!

mraspaud · 2019-03-12T14:37:23Z

@andyprata I believe it's fixed in #648, you're very welcome to test it.

mraspaud added bug component:readers labels Mar 12, 2019

mraspaud self-assigned this Mar 12, 2019

mraspaud mentioned this issue Mar 12, 2019

Add wrapped line support for metadata in modis_l1b reader #648

Merged

5 tasks

mraspaud closed this as completed in #648 Mar 13, 2019

LTMeyer mentioned this issue Mar 13, 2019

Add MODIS L2 reader #611

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issue loading MODIS Aqua data #626

Issue loading MODIS Aqua data #626

andyprata commented Feb 20, 2019 •

edited by mraspaud

djhoese commented Feb 20, 2019

andyprata commented Feb 20, 2019 •

edited by mraspaud

djhoese commented Feb 20, 2019

mraspaud commented Feb 20, 2019

LTMeyer commented Feb 21, 2019

mraspaud commented Mar 12, 2019

andyprata commented Mar 12, 2019

mraspaud commented Mar 12, 2019 •

edited

andyprata commented Mar 12, 2019

mraspaud commented Mar 12, 2019

mraspaud commented Mar 12, 2019

Issue loading MODIS Aqua data #626

Issue loading MODIS Aqua data #626

Comments

andyprata commented Feb 20, 2019 • edited by mraspaud

djhoese commented Feb 20, 2019

andyprata commented Feb 20, 2019 • edited by mraspaud

djhoese commented Feb 20, 2019

mraspaud commented Feb 20, 2019

LTMeyer commented Feb 21, 2019

mraspaud commented Mar 12, 2019

andyprata commented Mar 12, 2019

mraspaud commented Mar 12, 2019 • edited

andyprata commented Mar 12, 2019

mraspaud commented Mar 12, 2019

mraspaud commented Mar 12, 2019

andyprata commented Feb 20, 2019 •

edited by mraspaud

andyprata commented Feb 20, 2019 •

edited by mraspaud

mraspaud commented Mar 12, 2019 •

edited