'NoneType' object has no attribute 'items' when indexing dataset.

[jankovicgd]

Hey all I get an AttributeError when trying to index via some custom code. I have a minimum code sample below. Am I missing anything? I tried to set sources={} in the dcm.Dataset but it is checked and fails before below error. datacube dataset add works properly.

Expected behaviour

Indexed data into the database

Actual behaviour

Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/nikola/.vscode/extensions/ms-python.python-2020.6.91350/pythonFiles/lib/python/debugpy/__main__.py", line 45, in <module>
    cli.main()
  File "/home/nikola/.vscode/extensions/ms-python.python-2020.6.91350/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 430, in main
    run()
  File "/home/nikola/.vscode/extensions/ms-python.python-2020.6.91350/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 267, in run_file
    runpy.run_path(options.target, run_name=compat.force_str("__main__"))
  File "/usr/lib/python3.8/runpy.py", line 263, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/usr/lib/python3.8/runpy.py", line 96, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/lib/python3.8/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/nikola/Projects/dcArchitect/other/test.py", line 28, in <module>
    idxer.index_datasets()
  File "/home/nikola/Projects/dcArchitect/src/dcarchitect/indexer.py", line 73, in index_datasets
    dc.index.datasets.add(dataset)
  File "/home/nikola/Projects/dcArchitect/.venv/lib/python3.8/site-packages/datacube-1.8.1-py3.8.egg/datacube/index/_datasets.py", line 190, in add
    ds_by_uuid = flatten_datasets(dataset)
  File "/home/nikola/Projects/dcArchitect/.venv/lib/python3.8/site-packages/datacube-1.8.1-py3.8.egg/datacube/model/utils.py", line 324, in flatten_datasets
    traverse_datasets(ds, proc, id_map=id_map, depth_map=depth_map)
  File "/home/nikola/Projects/dcArchitect/.venv/lib/python3.8/site-packages/datacube-1.8.1-py3.8.egg/datacube/model/utils.py", line 290, in traverse_datasets
    proc(ds, cbk)
  File "/home/nikola/Projects/dcArchitect/.venv/lib/python3.8/site-packages/datacube-1.8.1-py3.8.egg/datacube/model/utils.py", line 279, in visit_post_order
    for k, v in sorted_items(ds.sources):
  File "/home/nikola/Projects/dcArchitect/.venv/lib/python3.8/site-packages/datacube-1.8.1-py3.8.egg/datacube/utils/py.py", line 72, in sorted_items
    return sorted(d.items(), key=key, reverse=reverse)
AttributeError: 'NoneType' object has no attribute 'items'

Steps to reproduce the behaviour

import datacube
import datacube.model as dcm

# md_mapping, dataset_type, ds -> loaded from file or somehow constructed

md = dcm.MetadataType(md_mapping)
dataset_type = dcm.DatasetType(md, dataset_type)
dataset = dc.Dataset(dataset_type, ds, [ds_path])

with datacube.Datacube(app="test") as dc:
    dc.index.datasets.add(dataset)

Environment information

• Which datacube --version are you using? 1.8.1
• What datacube deployment/enviornment are you running against? Local venv with python3.8.2

[Kirill888]

It's because of the confusing metadata relocation logic. What appears as .sources on dataset object, has to be stored here instead in dataset definition document:

lineage:
  source_datasets: {}

I don't know what was the original reason for this...

[jankovicgd]

I still get the same error. Example Dataset.metadata_doc:

{'extent': {'center_dt': '2019-04-24T05:26:21.500000',
            'coord': {'ll': {'lat': 10.428304773879523,
                             'lon': 46.551778699301416},
                      'lr': {'lat': 11.712077471815588,
                             'lon': 46.70859051248336},
                      'ul': {'lat': 10.192431517262827,
                             'lon': 47.43268391106645},
                      'ur': {'lat': 11.497321921234711,
                             'lon': 47.59280260389223}},
            'from_dt': '2019-04-24T05:26:09',
            'to_dt': '2019-04-24T05:26:34'},
 'format': {'name': 'GeoTIFF'},
 'grid_spatial': {'projection': {'geo_ref_points': {'ll': {'x': 4800000.0,
                                                           'y': 1500000.0},
                                                    'lr': {'x': 4900000.0,
                                                           'y': 1500000.0},
                                                    'ul': {'x': 4800000.0,
                                                           'y': 1600000.0},
                                                    'ur': {'x': 4900000.0,
                                                           'y': 1600000.0}},
                                 'spatial_reference': 'PROJCS["Azimuthal_Equidistant",GEOGCS["WGS '
                                                      '84",DATUM["WGS_1984",SPHEROID["WGS '
                                                      '84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0],UNIT["degree",0.0174532925199433],AUTHORITY["EPSG","4326"]],PROJECTION["Azimuthal_Equidistant"],PARAMETER["latitude_of_center",53],PARAMETER["longitude_of_center",24],PARAMETER["false_easting",5837287.81977],PARAMETER["false_northing",2121415.69617],UNIT["metre",1,AUTHORITY["EPSG","9001"]]]'}},
 'id': 'dd1ea43b-eec9-44b8-b143-5fdf648db705',
 'image': {'bands': {'VH': {'layer': 1,
                            'path': '/home/nikola/Data/Projects/ACUBE/S1_TUW/sig0/SIG0-----_SGRTA01_S1B_IWGRDH1VHD_20190424_052609--_EU010M_E048N015T1.tif'},
                     'VV': {'layer': 1,
                            'path': '/home/nikola/Data/Projects/ACUBE/S1_TUW/sig0/SIG0-----_SGRTA01_S1B_IWGRDH1VVD_20190424_052609--_EU010M_E048N015T1.tif'}}},
 'instrument': {'name': 'SAR'},
 'lineage': {'source_datasets': {}},
 'name': 'TUW_SIG0_S1_20190424052609_20190424052634',
 'platform': {'code': 'Sentinel-1'},
 'product_type': 'TUW_SIG0_S1'}

[Kirill888]

How does your metadata and product definition documents look like?

[Kirill888]

Try adding with_lineage=False when calling .datasets.add(..), this should work around your issue. Also consider using this class for making Dataset objects for indexing:

datacube-core/datacube/index/hl.py

Lines 199 to 232 in dcab0e0

	class Doc2Dataset:
	"""Used for constructing `Dataset` objects from plain metadata documents.
	This requires a database connection to perform the automatic matching against
	available products.
	There are options for including and excluding the products to match against,
	as well as how to deal with source lineage.
	Once constructed, call with a dictionary object and location URI, eg::
	resolver = Doc2Dataset(index)
	dataset = resolver(dataset_dictionary, 'file:///tmp/test-dataset.json')
	index.dataset.add(dataset)
	:param index: an open Database connection
	:param list products: List of product names against which to match datasets
	(including lineage datasets). If not supplied we will
	consider all products.
	:param list exclude_products: List of products to exclude from matching
	:param fail_on_missing_lineage: If True fail resolve if any lineage
	datasets are missing from the DB
	:param verify_lineage: If True check that lineage datasets in the
	supplied document are identical to DB versions
	:param skip_lineage: If True ignore lineage sub-tree in the supplied
	document and construct dataset without lineage datasets
	:param eo3: 'auto'/True/False by default auto-detect EO3 datasets and pre-process them
	"""

[jankovicgd]

Thanks for the heads up. Works but with a db constraint error. In hindsight in my code I should have probably extended DatasetType and Dataset because I have similar concepts that extract and construct yml files based on arbitrary tiffs, netcdfs and minimal configuration.