[sdk] How to apply PVCs to pipeline tasks when using KFP SDK v2

[yshimizu37]

Environment

• KFP version:
  I could not find out my KFP version.
  Not sure this is a valuable information, but I am using Kubeflow v1.6(manually installed).

• KFP SDK version:
  2.0.0b8

• All dependencies version:
  kfp 2.0.0b8
  kfp-pipeline-spec 0.1.16
  kfp-server-api 2.0.0a6

Steps to reproduce

Here is my source code.

@dsl.component
def data_preprocessing():
  # do some data tasks

@dsl.pipeline(name="AI Training Run")
def ai_training_run(
  epochs: int=1,
  batch_size: int=64,
  learning_rate: float=0.001,
  dataset_pvc_name: str="dataset",
) -> Model:
  dataset_dir = "/mnt/dataset"

  # proprocessing dataset
  preprocess_data_task = data_preprocessing()
  preprocess_data_task.apply(
    onprem.mount_pvc(dataset_pvc_name, 'datavol', dataset_dir)
  ).set_display_name('Data preprocessing')

It could be compiled successfully when I was using KFP 1.8.17.

$ python3 ai-training-run.py
/opt/deepops/env/lib/python3.8/site-packages/kfp/compiler/compiler.py:79: UserWarning: V2_COMPATIBLE execution mode is at Beta quality. Some pipeline features may not work as expected.
  warnings.warn('V2_COMPATIBLE execution mode is at Beta quality.'

But in KFP 2.0.0b8, It doesn't work.

$ python3 ai-training-run.py 
/opt/deepops/env/lib/python3.8/site-packages/kfp/deprecated/dsl/type_utils.py:28: FutureWarning: Module kfp.dsl.type_utils is deprecated and will be removed in KFP v2.0. Please use from kfp.components.types.type_utils instead.
  warnings.warn(
Traceback (most recent call last):
  File "ai-training-run.py", line 58, in <module>
    def ai_training_run(
  File "/opt/deepops/env/lib/python3.8/site-packages/kfp/components/pipeline_context.py", line 67, in pipeline
    return component_factory.create_graph_component_from_func(func)
  File "/opt/deepops/env/lib/python3.8/site-packages/kfp/components/component_factory.py", line 501, in create_graph_component_from_func
    return graph_component.GraphComponent(
  File "/opt/deepops/env/lib/python3.8/site-packages/kfp/components/graph_component.py", line 56, in __init__
    pipeline_outputs = pipeline_func(*args_list)
  File "ai-training-run.py", line 89, in ai_training_run
    preprocess_data_task.apply(
AttributeError: 'PipelineTask' object has no attribute 'apply'

I know 'PipelineTask' class does not have 'apply' method anymore, but am curious that are there other ways to mount PVCs to pipeline tasks?

Expected result

finding out how to mount PVCs to pipeline tasks.

Materials and Reference

---

Impacted by this bug? Give it a 👍.

[chensun]

@yshimizu37 PVC is not supported yet in SDK v2. It's on our roadmap to be supported once platform-specific runtime support is ready. (Join kubeflow-discuss to gain access to the doc).

/cc @connor-mccarthy

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[connor-mccarthy]

@yshimizu37, please take a look at the kfp-kubernetes reference docs.