Pipelines End-to-end on GCP: fix example & update screenshots #1611

amygdala · 2020-02-07T15:11:07Z

https://www.kubeflow.org/docs/gke/pipelines/pipelines-tutorial/

got a reminder from someone (doing a hosted pipelines-related bug bash) that the screenshots here need to be updated. And maybe more tweaks? Let's check it during the docs sprint. I'll try to do it.

issue-label-bot · 2020-02-07T15:11:14Z

Issue-Label Bot is automatically applying the labels:

Label	Probability
kind/feature	0.53

Please mark this comment with 👍 or 👎 to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

amygdala · 2020-02-07T15:11:40Z

/kind bug

sarahmaddox · 2020-02-07T17:20:54Z

/assign @amygdala

Thanks @amygdala!

sarahmaddox · 2020-02-08T16:33:03Z

@amygdala I've moved this issue to In progress in the doc sprint Kanban board, because the issue is assigned to you. Happy sprinting!

amygdala · 2020-02-10T20:20:30Z

This example no longer runs properly either; will look into it. Getting some kind of memory reference error related to ksonnet (!).
Also, the python pipeline spec. could use some syntax updates.

amygdala · 2020-02-10T20:47:51Z

hi @jinchihe , not sure if you're the right one to ping on this: when I tried running the example, I got the following ksonnet-related segfault error in the serve step.
I don't have visibility as to what's going on in the docker image that this step is using (gcr.io/ml-pipeline/ml-pipeline-kubeflow-deployer:7775692adf28d6f79098e76e839986c9ee55dd61)

Are you the right person to look into this? (I'm thinking it might be best to find a non-ksonnet implementation of this step.)

Switched to context "kubeflow".
+ cd /src/github.com/kubeflow/kubeflow
+ git checkout v0.2.5
HEAD is now at a4486ccd... Fix bug with updating an existing deployment. (#1475)
+ cd /opt
+ echo 'Initializing KSonnet app...'
+ ks init tf-serving-app
Initializing KSonnet app...
level=info msg="Using context \"kubeflow\" from kubeconfig file \"/root/.kube/config\""
level=info msg="Creating environment \"default\" with namespace \"default\", pointing to \"version:v1.14.8\" cluster at address \"https://kubernetes.default\""
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x456452]

goroutine 1 [running]:
github.com/ksonnet/ksonnet/vendor/github.com/ksonnet/ksonnet-lib/ksonnet-gen/ksonnet.parsePaths(0xc4204c2300, 0x6, 0x1, 0xe)
	/Users/oren.shomron/go/src/github.com/ksonnet/ksonnet/vendor/github.com/ksonnet/ksonnet-lib/ksonnet-gen/ksonnet/paths.go:33 +0x1f1
github.com/ksonnet/ksonnet/vendor/github.com/ksonnet/ksonnet-lib/ksonnet-gen/ksonnet.NewCatalog(0xc4204c2300, 0xc420a7acd0, 0x1, 0x1, 0x40, 0x0, 0x0)
	/Users/oren.shomron/go/src/github.com/ksonnet/ksonnet/vendor/github.com/ksonnet/ksonnet-lib/ksonnet-gen/ksonnet/catalog.go:76 +0x1de
github.com/ksonnet/ksonnet/vendor/github.com/ksonnet/ksonnet-lib/ksonnet-gen/ksonnet.GenerateLib(0xc420ece030, 0xe, 0x0, 0x0, 0x0)
	/Users/oren.shomron/go/src/github.com/ksonnet/ksonnet/vendor/github.com/ksonnet/ksonnet-lib/ksonnet-gen/ksonnet/ksonnet.go:25 +0xce
github.com/ksonnet/ksonnet/pkg/util/kslib.astBuilder(0xc420ece030, 0xe, 0xc421600000, 0x4b73df, 0x7ffe00, 0x0, 0x0, 0x0)
	/Users/oren.shomron/go/src/github.com/ksonnet/ksonnet/pkg/util/kslib/ksonnet.go:87 +0x40
github.com/ksonnet/ksonnet/pkg/util/kslib.Ksonnet(0xc421600000, 0x4b73df, 0x7ffe00, 0x0, 0x0, 0x0)
	/Users/oren.shomron/go/src/github.com/ksonnet/ksonnet/pkg/util/kslib/ksonnet.go:83 +0x3fd
github.com/ksonnet/ksonnet/pkg/lib.(*defaultKsLibGenerator).Generate(0xc420282eb0, 0x4eeb1b, 0xc420465050, 0xf)
	/Users/oren.shomron/go/src/github.com/ksonnet/ksonnet/pkg/lib/lib.go:61 +0x8e
github.com/ksonnet/ksonnet/pkg/lib.(*Manager).GenerateLibData(0xc4200e0e40, 0xf, 0x1f19de0)
	/Users/oren.shomron/go/src/github.com/ksonnet/ksonnet/pkg/lib/lib.go:105 +0x4c
github.com/ksonnet/ksonnet/pkg/app.updateLibData(0x1f19de0, 0x2d46a70, 0xc420465050, 0xf, 0xc42077a5a0, 0x17, 0xc4203a6ca8, 0xc4209d3230, 0x48364c, 0xc4203a6ca8)
	/Users/oren.shomron/go/src/github.com/ksonnet/ksonnet/pkg/app/app.go:149 +0x89
github.com/ksonnet/ksonnet/pkg/app.(*App010).AddEnvironment(0xc420184980, 0xc42093b980, 0xc420465050, 0xf, 0x2a24000, 0x3, 0x3)
	/Users/oren.shomron/go/src/github.com/ksonnet/ksonnet/pkg/app/app010.go:81 +0x423
github.com/ksonnet/ksonnet/pkg/env.(*creator).Create(0xc4201727e0, 0xc4201727e0, 0x100)
	/Users/oren.shomron/go/src/github.com/ksonnet/ksonnet/pkg/env/create.go:117 +0x9c3
github.com/ksonnet/ksonnet/pkg/env.Create(0x1f21920, 0xc420184980, 0xc4209886c0, 0x1a, 0x19aa68a, 0x7, 0x19aa68a, 0x7, 0xc420465050, 0xf, ...)
	/Users/oren.shomron/go/src/github.com/ksonnet/ksonnet/pkg/env/create.go:42 +0x157
github.com/ksonnet/ksonnet/pkg/appinit.(*initApp).Run(0xc420a7b900, 0x7f239ac84d90, 0x0)
	/Users/oren.shomron/go/src/github.com/ksonnet/ksonnet/pkg/appinit/init.go:88 +0x452
github.com/ksonnet/ksonnet/pkg/appinit.Init(0x1f19de0, 0x2d46a70, 0x7ffe9ccc3200, 0xe, 0xc420988220, 0x13, 0x0, 0x0, 0xc420465050, 0xf, ...)
	/Users/oren.shomron/go/src/github.com/ksonnet/ksonnet/pkg/appinit/init.go:33 +0x14a
github.com/ksonnet/ksonnet/pkg/actions.(*Init).Run(0xc420172510, 0xc420172510, 0x0)
	/Users/oren.shomron/go/src/github.com/ksonnet/ksonnet/pkg/actions/init.go:106 +0x121
github.com/ksonnet/ksonnet/pkg/actions.RunInit(0xc42017cb10, 0xc42017dcb0, 0xe)
	/Users/oren.shomron/go/src/github.com/ksonnet/ksonnet/pkg/actions/init.go:37 +0x5c
github.com/ksonnet/ksonnet/pkg/clicmd.runAction(0xe, 0xc42017cb10, 0x19c51e1, 0x17)
	/Users/oren.shomron/go/src/github.com/ksonnet/ksonnet/pkg/clicmd/actions.go:115 +0x66
github.com/ksonnet/ksonnet/pkg/clicmd.newInitCmd.func1(0xc420992f00, 0xc4204c5a80, 0x1, 0x1, 0x0, 0x0)
	/Users/oren.shomron/go/src/github.com/ksonnet/ksonnet/pkg/clicmd/init.go:153 +0x752
github.com/ksonnet/ksonnet/vendor/github.com/spf13/cobra.(*Command).execute(0xc420992f00, 0xc4204c5a50, 0x1, 0x1, 0xc420992f00, 0xc4204c5a50)
	/Users/oren.shomron/go/src/github.com/ksonnet/ksonnet/vendor/github.com/spf13/cobra/command.go:753 +0x468
github.com/ksonnet/ksonnet/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc420968000, 0x7, 0x19a69fe, 0x4)
	/Users/oren.shomron/go/src/github.com/ksonnet/ksonnet/vendor/github.com/spf13/cobra/command.go:843 +0x30a
github.com/ksonnet/ksonnet/vendor/github.com/spf13/cobra.(*Command).Execute(0xc420968000, 0x2d46a70, 0xc420044004)
	/Users/oren.shomron/go/src/github.com/ksonnet/ksonnet/vendor/github.com/spf13/cobra/command.go:791 +0x2b
main.main()
	/Users/oren.shomron/go/src/github.com/ksonnet/ksonnet/cmd/ks/main.go:48 +0x148

amygdala · 2020-02-10T20:52:19Z

Update: I'll try this more recent component definition: https://github.com/kubeflow/pipelines/blob/master/components/kubeflow/deployer/component.yaml to see if that helps.

Update 2: using a newer image fixed the issue (kinda), so please ignore my earlier comment, @jinchihe. There are still some problems with this op, but I think the KFP team is the right group to ask.
(There's a problem with the serving step as well, but I think Dan S is the one to ping on that).

update 3: okay, this example seems completely broken ...

amygdala · 2020-02-11T02:02:37Z

related: kubeflow/pipelines#3037

amygdala · 2020-02-11T03:00:51Z

in addition to the probs with the serving step, the webapp step of the pipeline is getting a ksonnet-related seg fault.

joeliedtke · 2020-02-11T16:44:22Z

Chatted with @amygdala and @sarahmaddox: since this tutorial does not currently work, I will remove this tutorial from the site while it is being revised.

jtfogarty · 2020-04-07T02:00:04Z

/priority p2

renat-abbyazov · 2020-04-10T19:32:26Z

Had no experience with segfault error but got 'No Major.Minor.Patch elements found', fixed by adding --api-spec=version:v1.15.0 to all ks commands

ks init --api-spec=version:v1.15.0 tf-serving-app

after that had webserving app deployed successfully.

aadityak33 · 2020-05-20T17:22:22Z

Hi, Attached is the pipeline with the latest docker images for train, serve, and web-ui. Getting the following error in the web-ui.

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x456452]
goroutine 1 [running]:
github.com/ksonnet/ksonnet/vendor/github.com/ksonnet/ksonnet-lib/ksonnet-gen/ksonnet.parsePaths(0xc42017c900, 0x7, 0x1, 0xe)
/Users/oren.shomron/go/src/github.com/ksonnet/ksonnet/vendor/github.com/ksonnet/ksonnet-lib/ksonnet-gen/ksonnet/paths.go:33 +0x1f1
github.com/ksonnet/ksonnet/vendor/github.com/ksonnet/ksonnet-lib/ksonnet-gen/ksonnet.NewCatalog(0xc42017c900, 0xc420ab8cd0, 0x1, 0x1, 0x40, 0x0, 0x0)
/Users/oren.shomron/go/src/github.com/ksonnet/ksonnet/vendor/github.com/ksonnet/ksonnet-lib/ksonnet-gen/ksonnet/catalog.go:76 +0x1de
github.com/ksonnet/ksonnet/vendor/github.com/ksonnet/ksonnet-lib/ksonnet-gen/ksonnet.GenerateLib(0xc42085c030, 0xe, 0x0, 0x0, 0x0)
/Users/oren.shomron/go/src/github.com/ksonnet/ksonnet/vendor/github.com/ksonnet/ksonnet-lib/ksonnet-gen/ksonnet/ksonnet.go:25 +0xce
github.com/ksonnet/ksonnet/pkg/util/kslib.astBuilder(0xc42085c030, 0xe, 0xc42150a000, 0x4b73e0, 0x7ffe00, 0x0, 0x0, 0x0)
/Users/oren.shomron/go/src/github.com/ksonnet/ksonnet/pkg/util/kslib/ksonnet.go:87 +0x40
github.com/ksonnet/ksonnet/pkg/util/kslib.Ksonnet(0xc42150a000, 0x4b73e0, 0x7ffe00, 0x0, 0x0, 0x0)
/Users/oren.shomron/go/src/github.com/ksonnet/ksonnet/pkg/util/kslib/ksonnet.go:83 +0x3fd
github.com/ksonnet/ksonnet/pkg/lib.(*defaultKsLibGenerator).Generate(0xc4204b39c0, 0x4eeb1b, 0xc4209aba70, 0xc420a8ef60)
/Users/oren.shomron/go/src/github.com/ksonnet/ksonnet/pkg/lib/lib.go:61 +0x8e
github.com/ksonnet/ksonnet/pkg/lib.(*Manager).GenerateLibData(0xc42021ecc0, 0x10, 0x1f19de0)
/Users/oren.shomron/go/src/github.com/ksonnet/ksonnet/pkg/lib/lib.go:105 +0x4c
github.com/ksonnet/ksonnet/pkg/app.updateLibData(0x1f19de0, 0x2d46a70, 0xc4209aba70, 0x10, 0xc420a48bc0, 0x17, 0xc420a90258, 0xc420a8f230, 0x48364c, 0xc420a90258)
/Users/oren.shomron/go/src/github.com/ksonnet/ksonnet/pkg/app/app.go:149 +0x89
github.com/ksonnet/ksonnet/pkg/app.(*App010).AddEnvironment(0xc420a46440, 0xc420a630e0, 0xc4209aba70, 0x10, 0x2a24000, 0x3, 0x3)
/Users/oren.shomron/go/src/github.com/ksonnet/ksonnet/pkg/app/app010.go:81 +0x423
github.com/ksonnet/ksonnet/pkg/env.(*creator).Create(0xc4201e90e0, 0xc4201e90e0, 0x100)
/Users/oren.shomron/go/src/github.com/ksonnet/ksonnet/pkg/env/create.go:117 +0x9c3
github.com/ksonnet/ksonnet/pkg/env.Create(0x1f21920, 0xc420a46440, 0xc420995280, 0x1a, 0x19aa68a, 0x7, 0x19aa68a, 0x7, 0xc4209aba70, 0x10, ...)
/Users/oren.shomron/go/src/github.com/ksonnet/ksonnet/pkg/env/create.go:42 +0x157
github.com/ksonnet/ksonnet/pkg/appinit.(*initApp).Run(0xc420ab9900, 0x7fdcf6d826c8, 0x0)
/Users/oren.shomron/go/src/github.com/ksonnet/ksonnet/pkg/appinit/init.go:88 +0x452
github.com/ksonnet/ksonnet/pkg/appinit.Init(0x1f19de0, 0x2d46a70, 0x7fff053fdfe0, 0xe, 0xc420809ba0, 0x13, 0x0, 0x0, 0xc4209aba70, 0x10, ...)
/Users/oren.shomron/go/src/github.com/ksonnet/ksonnet/pkg/appinit/init.go:33 +0x14a
github.com/ksonnet/ksonnet/pkg/actions.(*Init).Run(0xc4201e8d80, 0xc4201e8d80, 0x0)
/Users/oren.shomron/go/src/github.com/ksonnet/ksonnet/pkg/actions/init.go:106 +0x121
github.com/ksonnet/ksonnet/pkg/actions.RunInit(0xc420a2e480, 0xc4206470b0, 0xe)
/Users/oren.shomron/go/src/github.com/ksonnet/ksonnet/pkg/actions/init.go:37 +0x5c
github.com/ksonnet/ksonnet/pkg/clicmd.runAction(0xe, 0xc420a2e480, 0x19c51e1, 0x17)
/Users/oren.shomron/go/src/github.com/ksonnet/ksonnet/pkg/clicmd/actions.go:115 +0x66
github.com/ksonnet/ksonnet/pkg/clicmd.newInitCmd.func1(0xc420825400, 0xc420734510, 0x1, 0x1, 0x0, 0x0)
/Users/oren.shomron/go/src/github.com/ksonnet/ksonnet/pkg/clicmd/init.go:153 +0x752
github.com/ksonnet/ksonnet/vendor/github.com/spf13/cobra.(*Command).execute(0xc420825400, 0xc4207344e0, 0x1, 0x1, 0xc420825400, 0xc4207344e0)
/Users/oren.shomron/go/src/github.com/ksonnet/ksonnet/vendor/github.com/spf13/cobra/command.go:753 +0x468
github.com/ksonnet/ksonnet/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc420796500, 0x7, 0x19a69fe, 0x4)
/Users/oren.shomron/go/src/github.com/ksonnet/ksonnet/vendor/github.com/spf13/cobra/command.go:843 +0x30a
github.com/ksonnet/ksonnet/vendor/github.com/spf13/cobra.(*Command).Execute(0xc420796500, 0x2d46a70, 0xc420048004)
/Users/oren.shomron/go/src/github.com/ksonnet/ksonnet/vendor/github.com/spf13/cobra/command.go:791 +0x2b
main.main()
/Users/oren.shomron/go/src/github.com/ksonnet/ksonnet/cmd/ks/main.go:48 +0x148

pipeline.zip

Thanks,
Aaditya Kurde

issue-label-bot · 2020-05-20T17:22:29Z

Issue-Label Bot is automatically applying the labels:

Label	Probability
area/docs	1.00

Please mark this comment with 👍 or 👎 to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

stale · 2020-08-19T06:42:28Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

llhuii · 2021-03-12T08:27:19Z

the example also can't support k8s with version >= v1.16.0, since its component ml-pipeline-kubeflow-deployer depends on ksonnect, and ksonnect can't support k8s with version >= v1.16.0:

v1.16.0/v1.7.0 with a lot memory consuming and 100%cpu, can't exit within a minute.
>=v1.18.0 reports the error 'No Major.Minor.Patch elements found' due to this k8s commit set version to unversioned in api/openapi-spec/swagger.json.

issue-label-bot bot added the kind/feature label Feb 7, 2020

k8s-ci-robot added the kind/bug label Feb 7, 2020

kubeflow-bot added this to To Do in Needs Triage Feb 7, 2020

sarahmaddox added the doc-sprint Issues to work on during the Kubeflow Doc Sprint label Feb 7, 2020

k8s-ci-robot assigned amygdala Feb 7, 2020

sarahmaddox added this to Sprint backlog in doc-sprint Feb 7, 2020

sarahmaddox moved this from Sprint backlog to Issues in progress in doc-sprint Feb 8, 2020

amygdala changed the title ~~Pipelines End-to-end on GCP: update screenshots~~ Pipelines End-to-end on GCP: fix example & update screenshots Feb 10, 2020

amygdala mentioned this issue Feb 10, 2020

pipelines example: images update #1647

Closed

amygdala mentioned this issue Feb 11, 2020

add note to indicate that mnist pipeline example does not currently work kubeflow/examples#726

Merged

joeliedtke mentioned this issue Feb 11, 2020

Removes the Pipelines GCP MNIST tutorial #1652

Merged

amygdala mentioned this issue Feb 11, 2020

create replacement gke pipelines example kubeflow/examples#727

Closed

janeman98 mentioned this issue Feb 26, 2020

Remove the Minikube installation guide #1662

Closed

k8s-ci-robot added the priority/p2 label Apr 7, 2020

jtfogarty moved this from To Do to Assigned to Area Owner For Triage in Needs Triage Apr 7, 2020

issue-label-bot bot added the area/docs label May 20, 2020

kubeflow-bot removed this from Assigned to Area Owner For Triage in Needs Triage May 20, 2020

stale bot added the lifecycle/stale label Aug 19, 2020

stale bot closed this as completed Aug 27, 2020

doc-sprint automation moved this from Issues in progress to Done Aug 27, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pipelines End-to-end on GCP: fix example & update screenshots #1611

Pipelines End-to-end on GCP: fix example & update screenshots #1611

amygdala commented Feb 7, 2020

issue-label-bot bot commented Feb 7, 2020

amygdala commented Feb 7, 2020

sarahmaddox commented Feb 7, 2020

sarahmaddox commented Feb 8, 2020

amygdala commented Feb 10, 2020 •

edited

Loading

amygdala commented Feb 10, 2020

amygdala commented Feb 10, 2020 •

edited

Loading

amygdala commented Feb 11, 2020

amygdala commented Feb 11, 2020 •

edited

Loading

joeliedtke commented Feb 11, 2020

jtfogarty commented Apr 7, 2020

renat-abbyazov commented Apr 10, 2020

aadityak33 commented May 20, 2020 •

edited

Loading

issue-label-bot bot commented May 20, 2020

stale bot commented Aug 19, 2020

llhuii commented Mar 12, 2021 •

edited

Loading

Pipelines End-to-end on GCP: fix example & update screenshots #1611

Pipelines End-to-end on GCP: fix example & update screenshots #1611

Comments

amygdala commented Feb 7, 2020

issue-label-bot bot commented Feb 7, 2020

amygdala commented Feb 7, 2020

sarahmaddox commented Feb 7, 2020

sarahmaddox commented Feb 8, 2020

amygdala commented Feb 10, 2020 • edited Loading

amygdala commented Feb 10, 2020

amygdala commented Feb 10, 2020 • edited Loading

amygdala commented Feb 11, 2020

amygdala commented Feb 11, 2020 • edited Loading

joeliedtke commented Feb 11, 2020

jtfogarty commented Apr 7, 2020

renat-abbyazov commented Apr 10, 2020

aadityak33 commented May 20, 2020 • edited Loading

issue-label-bot bot commented May 20, 2020

stale bot commented Aug 19, 2020

llhuii commented Mar 12, 2021 • edited Loading

amygdala commented Feb 10, 2020 •

edited

Loading

amygdala commented Feb 10, 2020 •

edited

Loading

amygdala commented Feb 11, 2020 •

edited

Loading

aadityak33 commented May 20, 2020 •

edited

Loading

llhuii commented Mar 12, 2021 •

edited

Loading