Snakemake expand equivalent #3308

netphantom · 2023-11-14T16:19:37Z

netphantom
Nov 14, 2023

Hi all,
I am new to Kedro and I have some experience using Snakemake.

I need to run multiple pipelines with different inputs, so I have configured in my parameters.yml something like:

neural_network_heads: [100, 200, 300]

I would like that Kedro takes into account one at the time each value, and run 3 pipelines.
Using Snakemake, I put the expand rule and it took care of it. Is it possible to do the same in Kedro?

Thanks

astrojuanlu · 2023-11-14T16:48:27Z

astrojuanlu
Nov 14, 2023
Maintainer

Hello @netphantom! The way to do this in Kedro at the moment is by using namespaces. You can see an example in the tutorial: https://docs.kedro.org/en/stable/tutorial/add_another_pipeline.html#modular-pipelines

Essentially you have to:

Tweak your parameters.yml to put each value in a different namespace
Create three different pipelines in your create_pipeline function
Optionally create different outputs in catalog.yml, adding a namespace as a prefix

There is an evolution of this approach to make things more automatic, described in this blog post https://getindata.com/blog/kedro-dynamic-pipelines/

Could you give it a try and let us know how it went?

1 reply

astrojuanlu Nov 14, 2023
Maintainer

I also pasted this use case in #1606, where we are collecting evidence to support the case for making this use case simpler in Kedro.

netphantom · 2023-11-14T16:59:24Z

netphantom
Nov 14, 2023
Author

Hi @astrojuanlu !
Thank you very much for your quick reply. So far I had very few problems in setting up the project and the pipelines. I thought that I was missing something from the official documentation, because I didn't see anything related to that option.
Thank you again for the blog post, I came across that, but wasn't sure that it was the right path to follow.

Tomorrow I will try to implement everything and let you know how it goes.

I personally believe that this kind of feature would be very useful for creating fully test matrix and make exploratory analysis complete

1 reply

astrojuanlu Nov 14, 2023
Maintainer

I personally believe that this kind of feature would be very useful for creating fully test matrix and make exploratory analysis complete

I completely agree :)

netphantom · 2023-11-17T12:26:08Z

netphantom
Nov 17, 2023
Author

I think I am wrongly setting the parameters.yaml.

For the sake of simplicity I report only some of the params here.

I have some basic dataset and NN options in a shape like:

base_options:
   window:
       size: 100
       prediction_size: 10
   training:
     epochs: 1000
    learning_rate: 0.01
   heads:
   -

At some point I have specified a new namespace:

nn_heads:
  base_options: ${..base_options}

  nn_100:
    _overrides:
      heads: "100"
    base_options: ${merge:${..base_options},${._overrides}}


  nn_200:
    _overrides:
      heads: "200"
    base_options: ${merge:${..base_options},${._overrides}}

In settings.py I have:

DYNAMIC_PIPELINES_MAPPING = {
    "nn_heads": ["nn_100", "nn_200"]
}

In pipeline.py

    for namespace, variants in settings.DYNAMIC_PIPELINES_MAPPING.items():
        for variant in variants:
            pipes.append(
                pipeline(
                    pipeline_instance,
                    inputs={"dataset": f"{namespace}.dataset"},
                    namespace=f"{namespace}.{variant}",
                    tags=[variant, namespace],
                )
            )

But if I run with kedro run, I have:

InterpolationKeyError: Interpolation key 'base_options' not found
    full_key: nn_heads.nn_100.base_options
    object_type=dict

Am I missing something?

7 replies

netphantom Nov 21, 2023
Author

@marrrcin Yes, it is an adaptation from it :).
If I change the ${base_options}, I have a similar error: UnsupportedInterpolationType: Unsupported interpolation type merge

If I debug until the error, I can see that the inter_args are correctly populated with the values I give. It seems that the resolver is None, but I don't know why. Here I paste the stacktrace of the error.

Traceback (most recent call last):
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/omegaconf/basecontainer.py", line 242, in get_node_value
    node = node._dereference_node()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/omegaconf/base.py", line 246, in _dereference_node
    node = self._dereference_node_impl(throw_on_resolution_failure=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/omegaconf/base.py", line 277, in _dereference_node_impl
    return parent._resolve_interpolation_from_parse_tree(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/omegaconf/base.py", line 584, in _resolve_interpolation_from_parse_tree
    resolved = self.resolve_parse_tree(
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/omegaconf/base.py", line 764, in resolve_parse_tree
    return visitor.visit(parse_tree)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/antlr4/tree/Tree.py", line 34, in visit
    return tree.accept(self)
           ^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/omegaconf/grammar/gen/OmegaConfGrammarParser.py", line 206, in accept
    return visitor.visitConfigValue(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/omegaconf/grammar_visitor.py", line 101, in visitConfigValue
    return self.visit(ctx.getChild(0))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/antlr4/tree/Tree.py", line 34, in visit
    return tree.accept(self)
           ^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/omegaconf/grammar/gen/OmegaConfGrammarParser.py", line 342, in accept
    return visitor.visitText(self)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/omegaconf/grammar_visitor.py", line 298, in visitText
    return self.visitInterpolation(c)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/omegaconf/grammar_visitor.py", line 125, in visitInterpolation
    return self.visit(ctx.getChild(0))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/antlr4/tree/Tree.py", line 34, in visit
    return tree.accept(self)
           ^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/omegaconf/grammar/gen/OmegaConfGrammarParser.py", line 1041, in accept
    return visitor.visitInterpolationResolver(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/omegaconf/grammar_visitor.py", line 179, in visitInterpolationResolver
    return self.resolver_interpolation_callback(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/omegaconf/base.py", line 750, in resolver_interpolation_callback
    return self._evaluate_custom_resolver(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/omegaconf/base.py", line 702, in _evaluate_custom_resolver
    raise UnsupportedInterpolationType(
omegaconf.errors.UnsupportedInterpolationType: Unsupported interpolation type merge

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/kedro/framework/cli/cli.py", line 139, in main
    super().main(
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/kedro/framework/cli/project.py", line 453, in run
    session.run(
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/kedro/framework/session/session.py", line 418, in run
    catalog = context._get_catalog(  # noqa: protected-access
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/kedro/framework/context/context.py", line 284, in _get_catalog
    feed_dict = self._get_feed_dict()
                ^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/kedro/framework/context/context.py", line 299, in _get_feed_dict
    params = self.params
             ^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/kedro/framework/context/context.py", line 237, in params
    params = self.config_loader["parameters"]
             ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/kedro/config/omegaconf_config.py", line 209, in __getitem__
    raise exc
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/kedro/config/omegaconf_config.py", line 200, in __getitem__
    base_config = self.load_and_merge_dir_config(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/kedro/config/omegaconf_config.py", line 334, in load_and_merge_dir_config
    return OmegaConf.to_container(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/omegaconf/omegaconf.py", line 573, in to_container
    return BaseContainer._to_content(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/omegaconf/basecontainer.py", line 292, in _to_content
    value = get_node_value(key)
            ^^^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/omegaconf/basecontainer.py", line 247, in get_node_value
    value = BaseContainer._to_content(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/omegaconf/basecontainer.py", line 292, in _to_content
    value = get_node_value(key)
            ^^^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/omegaconf/basecontainer.py", line 247, in get_node_value
    value = BaseContainer._to_content(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/omegaconf/basecontainer.py", line 292, in _to_content
    value = get_node_value(key)
            ^^^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/omegaconf/basecontainer.py", line 244, in get_node_value
    conf._format_and_raise(key=key, value=None, cause=e)
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/omegaconf/base.py", line 231, in _format_and_raise
    format_and_raise(
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/omegaconf/_utils.py", line 899, in format_and_raise
    _raise(ex, cause)
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/omegaconf/_utils.py", line 797, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/omegaconf/basecontainer.py", line 242, in get_node_value
    node = node._dereference_node()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/omegaconf/base.py", line 246, in _dereference_node
    node = self._dereference_node_impl(throw_on_resolution_failure=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/omegaconf/base.py", line 277, in _dereference_node_impl
    return parent._resolve_interpolation_from_parse_tree(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/omegaconf/base.py", line 584, in _resolve_interpolation_from_parse_tree
    resolved = self.resolve_parse_tree(
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/omegaconf/base.py", line 764, in resolve_parse_tree
    return visitor.visit(parse_tree)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/antlr4/tree/Tree.py", line 34, in visit
    return tree.accept(self)
           ^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/omegaconf/grammar/gen/OmegaConfGrammarParser.py", line 206, in accept
    return visitor.visitConfigValue(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/omegaconf/grammar_visitor.py", line 101, in visitConfigValue
    return self.visit(ctx.getChild(0))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/antlr4/tree/Tree.py", line 34, in visit
    return tree.accept(self)
           ^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/omegaconf/grammar/gen/OmegaConfGrammarParser.py", line 342, in accept
    return visitor.visitText(self)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/omegaconf/grammar_visitor.py", line 298, in visitText
    return self.visitInterpolation(c)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/omegaconf/grammar_visitor.py", line 125, in visitInterpolation
    return self.visit(ctx.getChild(0))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/antlr4/tree/Tree.py", line 34, in visit
    return tree.accept(self)
           ^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/omegaconf/grammar/gen/OmegaConfGrammarParser.py", line 1041, in accept
    return visitor.visitInterpolationResolver(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/omegaconf/grammar_visitor.py", line 179, in visitInterpolationResolver
    return self.resolver_interpolation_callback(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/omegaconf/base.py", line 750, in resolver_interpolation_callback
    return self._evaluate_custom_resolver(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/PycharmProjects/data_analysis/Kedro/venv/lib/python3.11/site-packages/omegaconf/base.py", line 702, in _evaluate_custom_resolver
    raise UnsupportedInterpolationType(
omegaconf.errors.UnsupportedInterpolationType: Unsupported interpolation type merge
    full_key: nn_heads.nn_100.base_options
    object_type=dict
python-BaseException

marrrcin Nov 21, 2023
Collaborator

Have you registered your custom resolver in the settings.py? merge resolver is my own:

from copy import deepcopy

def merge_dicts(dict1, dict2):
    """
    Recursively merge two dictionaries.

    Args:
        dict1 (dict): The first dictionary to merge.
        dict2 (dict): The second dictionary to merge.

    Returns:
        dict: The merged dictionary.
    """
    result = deepcopy(dict1)
    for key, value in dict2.items():
        if key in result and isinstance(result[key], dict) and isinstance(value, dict):
            result[key] = merge_dicts(result[key], value)
        else:
            result[key] = value
    return result


CONFIG_LOADER_ARGS = {
    "custom_resolvers": {
        "merge": merge_dicts,
     }
}

netphantom Nov 22, 2023
Author

You're correct. I added the function to the spaceship settings.py, thus it was not finding it.

I think now things are almost fixed, I am just having a little troubles in mapping the parameters.
In the pipeline.py reported in the first post, I have the first node is set like this:

node(
    func=extract_data,
    inputs=["full_dataset", "nn_heads", "base_options:dataset"],
    outputs=["dataframe", "description", "error"],
    name="extract_data_node",

It returns an error where it seems that is not able to find the parameters in base_options:

ValueError: Pipeline input(s) {'params:nn_heads.nn_100.training', 'params:nn_heads.nn_100.class_info', 'nn_heads.full_dataset', 'params:nn_heads.nn_100.dataset', 'nn_heads.nn_100.error_code', 
'params:nn_heads.nn_100.network_parameters', 'params:nn_heads.nn_100.window'} not found in the DataCatalog

In the data catalog.yml I have set:

  "{namespace}.{variant}.full_dataset":
        type: pandas.CSVDataSet
        filepath: data/05_model_input/scaled_dataset.csv
        load_args:
            index_col: "Timestamp"
            parse_dates: True

The output of the node are partials, so I didn't added them to the catalog.

marrrcin Nov 23, 2023
Collaborator

What is the namespace of this node? Namespacing should match between parameters and pipeline() constructors in the code.

netphantom Nov 24, 2023
Author

Ok I think I got it right and now everything works. Thank you very much @marrrcin .

The "problem" was, in the fact, that in my namespace I had to specify the top level definition of my parameters list. I short, I wrote

    for namespace, variants in settings.DYNAMIC_PIPELINES_MAPPING.items():
        for variant in variants:
            pipes.append(
                pipeline(
                    pipeline_instance,
                    inputs={"dataset": f"{namespace}.dataset"},
                    namespace=f"{namespace}.{variant}",
                    tags=[variant, namespace],
                )
            )

While I had to specify in my namespace the base_options. It was not clear to me. Too many new things in few time. Thus, changing the code into:

    for namespace, variants in settings.DYNAMIC_PIPELINES_MAPPING.items():
        for variant in variants:
            pipes.append(
                pipeline(
                    pipeline_instance,
                    inputs={"dataset": f"{namespace}.dataset"},
                    namespace=f"{namespace}.{variant}.base_options",
                    tags=[variant, namespace],
                )
            )

I was able to recall into the nodes each inner variable using the params:... prefix.

I definitely like Kedro for data science, it is very specific and it gives a solid project structure to work with. I'm going to develop more with it, and I would really appreciate if you can find an "easier" way to make dynamic pipelines.

Thank you again

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Snakemake expand equivalent #3308

{{title}}

Replies: 3 comments 9 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Snakemake expand equivalent #3308

netphantom Nov 14, 2023

Replies: 3 comments · 9 replies

astrojuanlu Nov 14, 2023 Maintainer

astrojuanlu Nov 14, 2023 Maintainer

netphantom Nov 14, 2023 Author

astrojuanlu Nov 14, 2023 Maintainer

netphantom Nov 17, 2023 Author

netphantom Nov 21, 2023 Author

marrrcin Nov 21, 2023 Collaborator

netphantom Nov 22, 2023 Author

marrrcin Nov 23, 2023 Collaborator

netphantom Nov 24, 2023 Author

netphantom
Nov 14, 2023

Replies: 3 comments 9 replies

astrojuanlu
Nov 14, 2023
Maintainer

astrojuanlu Nov 14, 2023
Maintainer

netphantom
Nov 14, 2023
Author

astrojuanlu Nov 14, 2023
Maintainer

netphantom
Nov 17, 2023
Author

netphantom Nov 21, 2023
Author

marrrcin Nov 21, 2023
Collaborator

netphantom Nov 22, 2023
Author

marrrcin Nov 23, 2023
Collaborator

netphantom Nov 24, 2023
Author