Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[sdk] Error occurs for component_input_parameter in v2 compiler #5854

Closed
TrsNium opened this issue Jun 14, 2021 · 2 comments
Closed

[sdk] Error occurs for component_input_parameter in v2 compiler #5854

TrsNium opened this issue Jun 14, 2021 · 2 comments

Comments

@TrsNium
Copy link
Contributor

TrsNium commented Jun 14, 2021

Environment

  • KFP SDK version:
    1.6.3

Steps to reproduce

Define and execute a pipeline with the following code

slack_notification_op = comp.load_component_from_text(
    read_file("components/slack-notification/component.yaml")
)

def get_before_days(date: str, fmt: str, days: int) -> str:
    """
    Subtract the number of days from the date from the date and the number of days specified in the argument.

    Parameters
    ----------
    date: str
        Original Date
    fmt: str
        Date formate with a (strftime, strptime) format
    days: int
        The number of days to subtract.
    """
    from datetime import datetime, timedelta

    dt = datetime.strptime(date, fmt)
    new_dt = dt - timedelta(days=days)
    return new_dt.strftime(fmt)

@dsl.pipeline(
    name="test",
    description="",
)
def feature_extract_pipeline(
    vertex_pipelines_project_id: str,
    dataset_id: str,
    pipelines_url: str,
    slack_webhook: str,
    target_date: str,
):
    with dsl.ExitHandler(
        exit_op=slack_notification_op(
            job_name="{{workflow.name}}",
            webhook_url=slack_webhook,
            status="{{workflow.status}}",
            pipelines_url=f"{pipelines_url}/pipeline/#/runs/details/{{{{workflow.uid}}}}",
        )
    ):
        get_before_a_days_op = comp.create_component_from_func(
            func=get_before_days, base_image="python:alpine"
        )
        before_a_day = get_before_a_days_op(
            target_date, "%Y%m%d", 1
        )

from kfp.v2 import compiler
import tempfile
with tempfile.TemporaryDirectory() as tempdir:
    package_path = os.path.join(tempdir, "pipeline.json")
    # NOTE: The type_check argument is probably not working.
    compiler.Compiler().compile(pipeline_func=feature_extract_pipeline, package_path=package_path)

following is error messsage

ata/.pyenv/versions/3.7.1rc1/lib/python3.7/site-packages/kfp/v2/compiler/compiler.py", line 687, in _group_to_dag_spec
    input_artifacts_in_current_dag,
  File "/Users/takuya.hirata/.pyenv/versions/3.7.1rc1/lib/python3.7/site-packages/kfp/dsl/component_spec.py", line 338, in update_task_inputs_spec
    component_input_parameter, parent_component_inputs)
AssertionError: component_input_parameter: pipelineparam--pipelineparam--target_date not found. All inputs:

I got same error occurred when using from kfp.v2.dsl import component.
If you use a python function with component, you will get an error like the one above

Expected result

No errors occur during compilation.


Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

@chensun chensun self-assigned this Jun 16, 2021
@zijianjoy zijianjoy added this to Needs triage in KFP SDK Triage via automation Jun 18, 2021
@TrsNium
Copy link
Contributor Author

TrsNium commented Jun 18, 2021

I noticed that I can compile without using the exit handler.
I just did a print debugging.
The target_date is recognized as a input_parameter of the subgroup component, but there is no such parameter in the parent, which may be causing the above error.

**********************group.name***************
group.name: exit-handler-1, rootgroup_name: 8a1b7dd5281c40c3aee5c15a24814903
********************root spec******************
input_definitions {
  parameters {
    key: "category_models"
    value {
      type: STRING
    }
  }
  parameters {
    key: "dataset_id"
    value {
      type: STRING
    }
  }
  parameters {
    key: "metric_bucket_url"
    value {
      type: STRING
    }
  }
  parameters {
    key: "metric_image"
    value {
      type: STRING
    }
  }
  parameters {
    key: "output_gcs_path"
    value {
      type: STRING
    }
  }
  parameters {
    key: "parallel_num"
    value {
      type: INT
    }
  }
  parameters {
    key: "pipelines_url"
    value {
      type: STRING
    }
  }
  parameters {
    key: "service_project_id"
    value {
      type: STRING
    }
  }
  parameters {
    key: "slack_webhook"
    value {
      type: STRING
    }
  }
  parameters {
    key: "snap_item_limit"
    value {
      type: STRING
    }
  }
  parameters {
    key: "target_date"
    value {
      type: STRING
    }
  }
  parameters {
    key: "vertex_pipelines_project_id"
    value {
      type: STRING
    }
  }
}
dag {
  tasks {
    key: "exit-handler-1"
    value {
      task_info {
        name: "exit-handler-1"
      }
      component_ref {
        name: "comp-exit-handler-1"
      }
    }
  }
}

********************group component spec******************
{'comp-exit-handler-1': }

********************subgroup params*****************
[{{pipelineparam:op=;name=target_date}}]

if group.name == rootgroup_name:
group_component_spec = pipeline_spec.root
else:
group_component_spec = pipeline_spec.components[group_component_name]

subgroup_params = [param for param, _ in subgroup_inputs]

exit-handler group does not have parameters, So I figured the error would occur.

@TrsNium
Copy link
Contributor Author

TrsNium commented Jun 22, 2021

@chensun hello.
Could you review this PR? #5899

KFP SDK Triage automation moved this from Needs triage to Closed Jun 22, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Development

No branches or pull requests

2 participants