Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Task result not passed to nested loop #1350

Closed
typeness opened this issue Sep 27, 2023 · 1 comment · Fixed by #1351
Closed

Task result not passed to nested loop #1350

typeness opened this issue Sep 27, 2023 · 1 comment · Fixed by #1351
Assignees
Labels

Comments

@typeness
Copy link

/kind bug

What steps did you take and what happened:
[A clear and concise description of what the bug is.]

The following DSL generates Tekton that does not pass task results to the inner loop pipeline-120ca-for-loop-4

from kfp import dsl, components
from kfp.components import load_component_from_text
from kfp_tekton.tekton import Loop

op1_yaml = '''\
name: 'my-in-coop1'
inputs:
- {name: item, type: Integer}
- {name: param, type: Integer}
implementation:
    container:
        image: library/bash:4.4.23
        command: ['sh', '-c']
        args:
        - |
          set -e
          echo op1 "$0" "$1"
        - {inputValue: item}
        - {inputValue: param}
'''




@dsl.pipeline(name='pipeline')
def pipeline():

    cel_run_bash_script = load_component_from_text(r"""
        name: cel-run-bash-script
        inputs: []
        outputs:
        - {name: env-variables}
        implementation:
          container:
            image: aipipeline/cel-eval:latest
            command: [cel]
            args: [--apiVersion, custom.tekton.dev/v1alpha1, --kind, VariableStore, --name,
              1234567-var-store, --lit_0, 'a', --taskSpec,
              '{}']
            fileOutputs: {}
    """)()
    with Loop(cel_run_bash_script.output) as item:
        with Loop(cel_run_bash_script.output) as item2:
            op1_template = components.load_component_from_text(op1_yaml)
            op1 = op1_template(1, 10)


if __name__ == '__main__':
  from kfp_tekton.compiler import TektonCompiler as Compiler
  from kfp_tekton.compiler.pipeline_utils import TektonPipelineConf
  tekton_pipeline_conf = TektonPipelineConf()
  tekton_pipeline_conf.set_tekton_inline_spec(True)
  tekton_pipeline_conf.set_resource_in_separate_yaml(True)
  Compiler().compile(pipeline, __file__.replace('.py', '.yaml'), tekton_pipeline_conf=tekton_pipeline_conf)
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  name: pipeline
  annotations:
    tekton.dev/output_artifacts: '{}'
    tekton.dev/input_artifacts: '{}'
    tekton.dev/artifact_bucket: mlpipeline
    tekton.dev/artifact_endpoint: minio-service.kubeflow:9000
    tekton.dev/artifact_endpoint_scheme: http://
    tekton.dev/artifact_items: '{"my-in-coop1": []}'
    sidecar.istio.io/inject: "false"
    tekton.dev/template: ''
    pipelines.kubeflow.org/big_data_passing_format: $(workspaces.$TASK_NAME.path)/artifacts/$ORIG_PR_NAME/$TASKRUN_NAME/$TASK_PARAM_NAME
    pipelines.kubeflow.org/pipeline_spec: '{"name": "pipeline"}'
  labels:
    pipelines.kubeflow.org/pipelinename: ''
    pipelines.kubeflow.org/generation: ''
spec:
  pipelineSpec:
    tasks:
    - name: cel-run-bash-script
      params:
      - name: lit_0
        value: a
      taskSpec:
        apiVersion: custom.tekton.dev/v1alpha1
        kind: VariableStore
        spec: {}
        metadata:
          labels:
            pipelines.kubeflow.org/cache_enabled: "true"
          annotations:
            pipelines.kubeflow.org/component_spec_digest: '{"name": "cel-run-bash-script",
              "outputs": [{"name": "env-variables"}], "version": "cel-run-bash-script@sha256=2cb7a267ff559aa219e1f0554002d093cbd9efb18997692171ee99335950d35b"}'
    - name: pipeline-ed566-for-loop-2
      params:
      - name: loop-item-param-1
        value: $(tasks.cel-run-bash-script.results.env-variables)
      taskSpec:
        apiVersion: custom.tekton.dev/v1alpha1
        kind: PipelineLoop
        spec:
          pipelineSpec:
            params:
            - name: loop-item-param-1
              type: string
            tasks:
            - name: pipeline-ed566-for-loop-4
              params: []
              taskSpec:
                apiVersion: custom.tekton.dev/v1alpha1
                kind: PipelineLoop
                spec:
                  pipelineSpec:
                    params:
                    - name: loop-item-param-3
                      type: string
                    tasks:
                    - name: my-in-coop1
                      taskSpec:
                        steps:
                        - name: main
                          args:
                          - |
                            set -e
                            echo op1 "$0" "$1"
                          - '1'
                          - '10'
                          command:
                          - sh
                          - -c
                          image: library/bash:4.4.23
                        metadata:
                          labels:
                            pipelines.kubeflow.org/cache_enabled: "true"
                          annotations:
                            pipelines.kubeflow.org/component_spec_digest: '{"name":
                              "my-in-coop1", "outputs": [], "version": "my-in-coop1@sha256=8ccab3a28a39a406554d964865f2ccb0aed854a43b6de827f613eff2bccd6f8f"}'
                  iterateParam: loop-item-param-3
                metadata:
                  labels:
                    pipelines.kubeflow.org/cache_enabled: "true"
          iterateParam: loop-item-param-1
        metadata:
          labels:
            pipelines.kubeflow.org/cache_enabled: "true"

What did you expect to happen:

I expect pipeline-ed566-for-loop-4 task to have non-empty params

params:
- name: cel-run-bash-script-env-variables-loop-item
  value: $(params.cel-run-bash-script-env-variables-loop-item)

Additional information:
[Miscellaneous information that will assist in solving the issue.]

Environment:

  • Python Version (use python --version):
  • SDK Version:
  • Tekton Version (use tkn version):
  • Kubernetes Version (use kubectl version):
  • OS (e.g. from /etc/os-release):
@Tomcli
Copy link
Member

Tomcli commented Sep 27, 2023

/assign

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants