TypeError: unhashable type: 'list' when making fused wofs and fc cache

[alexgleith]

Using odc-stats = 0.2.1.dev2619 and odc-dscache = 0.2.1.dev2286

Stack trace is

odc-stats save-tasks --frequency annual --grid africa-30 --year 2018 fc_ls+wofs_ls
Processing 61,778 datasets
Training compression dictionary
Traceback (most recent call last):
  File "/env/bin/odc-stats", line 8, in <module>
    sys.exit(main())
  File "/env/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/env/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/env/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/env/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/env/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/env/lib/python3.8/site-packages/odc/stats/_cli_save_tasks.py", line 162, in save_tasks
    ok = tasks.save(
  File "/env/lib/python3.8/site-packages/odc/stats/tasks.py", line 270, in save
    dss_slice = list(islice(dss, 0, 100))
  File "/env/lib/python3.8/site-packages/odc/stats/_cli_save_tasks.py", line 230, in filter
    for _, ds_group in groups:
  File "/env/lib/python3.8/site-packages/odc/index/_index.py", line 243, in ordered_dss
    dss = dc.find_datasets(**q.search_terms)
  File "/env/lib/python3.8/site-packages/datacube/api/core.py", line 448, in find_datasets
    return list(self.find_datasets_lazy(**search_terms))
  File "/env/lib/python3.8/site-packages/datacube/api/core.py", line 463, in find_datasets_lazy
    query = Query(self.index, **kwargs)
  File "/env/lib/python3.8/site-packages/datacube/api/query.py", line 102, in __init__
    datacube_product = index.products.get_by_name(product)
  File "/env/lib/python3.8/site-packages/datacube/index/_products.py", line 278, in get_by_name
    return self.get_by_name_unsafe(name)
  File "/env/lib/python3.8/site-packages/cachetools/func.py", line 61, in wrapper
    v = cache[k]
  File "/env/lib/python3.8/site-packages/cachetools/lru.py", line 14, in __getitem__
    value = cache_getitem(self, key)
  File "/env/lib/python3.8/site-packages/cachetools/cache.py", line 44, in __getitem__
    return self.__data[key]
  File "/env/lib/python3.8/site-packages/cachetools/keys.py", line 19, in __hash__
    self.__hashvalue = hashvalue = hash(self)
TypeError: unhashable type: 'list'

[alexgleith]

Products are accessible in the DE Africa Sandbox

[Kirill888]

@alexgleith it's a "bug" in datacube-core, introduced in this commit:

opendatacube/datacube-core@00ddcbb

specifically this line:

datacube_product = index.products.get_by_name(product)

https://github.com/opendatacube/datacube-core/blob/88e4c8c4984cec4b19c1987571d094712d4332bc/datacube/api/query.py#L102

basically added code assumes that product= parameter is a single product name and not a a list of products. Hard to blame @petewa for adding this, support for product being a list of strings is not documented nor tested in datacube-core.

The "proper" way to handle this is to "determine what product was meant from the query", which is actually quite involved process of guessing which products might be matching the query given search terms, realistically I'm not even sure if one can do dc.load without specifying which product (or several) you want, but find_datasets is surely supposed to work with that, except nothing in datacube-core tests for this use-case. @jeremyh might provide more info on the topic of proper way to "resolve product(s) from query".

[jeremyh]

@jeremyh might provide more info on the topic of proper way to "resolve product(s) from query".

This is what index.products.search() was supposed to do. It "searches" for products that support+match the given search query args.

It's used by index.datasets.search() to decide which products to search, and takes the same set of args. (... at least, last time I looked)

[alexgleith]

Made a change over here: opendatacube/datacube-core#1201

Not sure how to test. Does anyone have a suggestion?

[alexgleith]

Fixed in opendatacube/datacube-core#1201