error uploading pipeline from github URL

[amygdala]

I'm using whatever kfp version is Build commit: ca58b22.

When I try to upload a pipeline version, I get the following error. I double checked that I could 'wget' the URL and unarchive the result. (Am I misinterpreting what "Package Url means?)

Related: why doesn't this UI support browsing to upload a file, rather than requiring a URL? That makes things much less convenient IMO.

[amygdala]

Update: it looks like github URLs are problematic somehow. With a url like this:
https://raw.githubusercontent.com/amygdala/pipelines/automl_client_update/components/gcp/automl/tables_eval_pipeline.py.tar.gz , which downloads okay via curl or wget, I see the error above.
If I try with this URL, which points to the same archive file:
https://storage.googleapis.com/aju-btest2/arghh/tables_eval_pipeline.py.tar.gz
things work okay.

IMO this issue with github urls is particularly problematic since we apparently can not upload an archive file from the local machine via the version UI, unlike the case with a (non-version) Pipeline upload.

cc @jingzhang36

[jingzhang36]

We now have another api method UploadPipelineVersion that works with local files. Examples are at
https://www.kubeflow.org/docs/pipelines/tutorials/sdk-examples/#creating-a-pipeline-and-a-pipeline-version-using-the-sdk

Also, if using CreatePipelineVersion with url, there is some known issue regarding which types of urls are supported. We recommend using https://storage.googleapis.com/ <gcs bucket name>/ <file name>

[amygdala]

/reopen

I'm sorry, I don't view this as fixed-- it's pretty important that GitHub URLs are supported IMO.

[k8s-ci-robot]

@amygdala: Reopened this issue.

In response to this:

/reopen

I'm sorry, I don't view this as fixed-- it's pretty important that GitHub URLs are supported IMO.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[amygdala]

Just pinging on this... it continues to annoy me each time I would like to load from a github URL and it breaks (since that's really the only remote URL source I normally use for pipelines). I expect that if it annoys me so often, it is surely annoying other users too.

I think there just needs to be some frontend support for handling redirects. Does the js package being used support that?

[Bobgy]

Thanks, this sounds useful. I've put it to triage track, but if any contributor is willing to jump in. You are welcome to do so.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[amygdala]

let's keep this open, we really should fix it.

[amygdala]

/lifecycle frozen

[zijianjoy]

Hello @amygdala , thank you for providing the sample for reproducing the issue! I went through the steps as below:

Environment

I am using v1.3.0 for backend image: gcr.io/ml-pipeline/api-server:1.3.0. And I am using the latest change of frontend code on GitHub.

Observation

I have no problem uploading a revision of pipeline using raw github content URL as you provided (https://raw.githubusercontent.com/amygdala/pipelines/automl_client_update/components/gcp/automl/tables_eval_pipeline.py.tar.gz ). And below is a screenshot of a successful upload UI and the network requests:

From this screenshot, you can see that a revision of pipeline request is sent to {domain}/apis/v1beta1/pipeline_versions endpoint, with query parameter of the githubusercontent URL. And on the right hand side is a graph of pipeline after successful response.

Action

This issue might have been fixed already. You can try redeploying your KFP with new version. If the issue persists or resolves, please reply back here and I am happy to dig in further with you. Reference: KFP release note

[zijianjoy]

Received response from user that the issue has been fixed, and new UI allows you to upload pipeline revision file from local.