Declaring tuples in catalog.yml

[ghost]

Hi guys,

I am trying specify the filters argument (as one of the load_args) while loading a dask.ParquetDataSet. According to this document, the filters need to be declared as a List[Tuple]] or a List[List[Tuple]]. But tuples are not natively supported in yaml files. So, I wonder if there is a kedro way to specify the filters argument. If not, is there a simple work-around?

Thank you very much for your help.

[datajoely]

Good question @lamhm - two things I'd suggest:

• Can you try using a YAML list instead of a tuple? Skimming the code it doesn't look like they do any isinstance checks and may just rely on the filters being in an iterable.
• If that doesn't work I would subclass dask.ParquetDataSet and add a simple mechanism for defining filters as YAML list structures and covert them to tuples explicitly. I've hacked together something quickly that should give you an idea of what I mean:

This would expose a new top level load_filters argument in your YAML definition if you want to use it.

[ghost]

Thanks a lot, @datajoely. Following your suggestion, I have just tried passing a List[List] (instead of a List[Tuple]) into the filters argument. The dataset was loaded and filtered correctly.

[filpano]

For what it's worth, using a plain List[List] using the documented YAML syntax (see "Block sequences"):

  load_args:
    filters:
      - 
        - 'itemId'
        - '='
        - 'fb895e07-86ed-490f-952a-7f1b96819d15'

unfortunately does not work when using pyarrow, since dask.dataframe.io.parquet.arrow#_construct_collection_plan flattens the filters:

        if filters is not None:
            for filter in flatten(filters, container=list):
                col, op, val = filter

which causes something like the above to fail with ValueError: too many values to unpack (expected 3).

So, a custom DataSet seems to be the way to go.