[bug] urllib3 breaks lightweight components in Vertex AI and kubeflow pipelines with 2.0.x

[Davidnet]

Environment

kfp[all]==2.0.0b15 # Although the version seems to impact all kfp versions see issue 930

• How do you deploy Kubeflow Pipelines (KFP)?
  Vertex AI Pipelines

• KFP version:
  Vertex AI Pipelines

• KFP SDK version:
  kfp[all]==2.0.0b15

Steps to reproduce

I have created a repo that with the only dependency being: kfp[all]==2.0.0b15 and run this component:

# %%
from kfp import dsl


@dsl.component
def print_component(str_exmpl: str) -> None:
    print(str_exmpl)


@dsl.pipeline
def pipeline():
    print_task = print_component(str_exmpl="Hello World!")


if __name__ == "__main__":
    import kfp

    print(kfp.__version__)
    kfp.compiler.Compiler().compile(pipeline, "pipeline.yaml")

This fails:

Traceback (most recent call last):
  File "/home/davidnet/.pyenv/versions/issue-kubeflow/lib/python3.11/site-packages/requests_toolbelt/_compat.py", line 48, in <module>
    from requests.packages.urllib3.contrib import appengine as gaecontrib
ImportError: cannot import name 'appengine' from 'requests.packages.urllib3.contrib' (/home/davidnet/.pyenv/versions/issue-kubeflow/lib/python3.11/site-packages/urllib3/contrib/__init__.py)

This is tracked in the repo: branch issue/compile
Now, pinning the urllib dependency:

$ cat requirements.txt
kfp[all]==2.0.0b15
urllib3==1.26.15

Makes the pipeline compile. but then when running on vertex it fails :

"message": "ImportError: cannot import name 'appengine' from 'urllib3.contrib' (/usr/local/lib/python3.7/site-packages/urllib3/contrib/__init__.py)",

With the same component.

Now technically all my pipelines are failing and the only solution we think with our team is to pin urllib in each lightweight component.

This is tracked in this repo (both logs, and the compiled yaml can be found) in here branch issue/running

Expected result

Pipeline run succesfully

Materials and reference

Resources: https://github.com/Davidnet/kubeflow_request_issue

Labels

/area backend
/area sdk
/area components

---

Impacted by this bug? Give it a 👍.

[Davidnet]

Fixed in: #9323

[connor-mccarthy]

This issue is resolved in the latest v1 and v2 releases: kfp==1.8.21 and kfp==2.0.0b16.

Preferred solution
Pip install and recompile your pipelines with kfp==2.0.0-beta.16 or kfp==1.8.21 to observe the fix. Recompilation is necessary because the KFP SDK compiler injects the KFP SDK version used for compilation into the pipeline YAML/JSON. This is the same version that will be installed and used at component runtime.

Alternative solutions
Alternatively, if you don't have the Python available for recompilation, you can search and replace

kfp==2.0.0-beta.14 (or whatever your current version is)

with

kfp==2.0.0-beta.16 in your compiled component/pipeline YAML/JSON.

If for some reason you cannot use kfp==2.0.0-beta.16 or kfp==1.8.21, you can manually pip install urllib3<2.0 in your local environment (the one used for authoring/compilation) and in your component at runtime:

@dsl.component(packages_to_install=['urllib3<2.0'])
def identity(string: str) -> str:
    return string

[ChaseLyall]

Root Cause
Python Software Foundation requests released an arguably breaking change in a minor version: https://github.com/psf/requests/releases/tag/v2.30.0.

Impact
Lightweight Python Components compiled by KFP SDK versions earlier than kfp==1.8.21 and kfp==2.0.0b16 will fail with error:

ImportError: cannot import name 'appengine' from 'requests.packages.urllib3.contrib'

This is because requests is pip installed at runtime by several KFP dependencies: requirements.in.

Other component types (Containerized Python Components and Container Components) are unaffected. We recommend that production workloads use these component types (namely pre-built container images).

Long Term Improvements
We are assessing how we can reduce this risk in the future.