KEP-17: Pipe task implementation #1105

zen-dog · 2019-11-25T20:06:54Z

Summary:
as defined in KEP-0017 we introduce a new Pipe task. Given a task specification like:

tasks:
  - name: genfiles
    kind: Pipe
    spec:
      pod: pipe-pod.yaml
      pipe:
        - file: /tmp/foo.txt
          kind: Secret # or ConfigMap
          key: foo

KUDO will:

create a Pod with the provided pipe-pod.yaml
wait for the successful execution of the container
copy out specified pipe files
store them in the API server (in the above example as a Secret)
the secret can be referenced in the subsequent resources as {{Pipes.foo}}

For more information about the implementation please consult the KEP-0017.

Fixes: #774

Summary: as defined in KEP-0017 we introduce a new Pipe task. Given a task specification like: ``` tasks: - name: genfiles kind: Pipe spec: container: container.yaml pipe: - file: /tmp/foo.txt kind: Secret # or ConfigMap key: foo ``` KUDO will: - create a Pod with the provided `container.yaml` - wait for successful execution of the container - copy out specified pipe files - store them in the API server (in the above example as a Secret) - the secret can be referenced in the subsequent resources as `{{Pipes.foo}}` For more information about the implementation please consult the KEP-0017. Fixes: #774

alenkacz

I did not do a very deep review so far but I liked what I saw 👍 left couple of minor comments :)

pkg/engine/task/pod_exec.go

pkg/engine/task/task_pipe.go

ANeumann82 · 2019-11-26T10:11:05Z

pkg/engine/task/task.go

+}
+
+var (
+	pipeFileKeyRe  = regexp.MustCompile(`^[a-zA-Z0-9_\-]+$`)    //a-z, A-Z, 0-9, _ and - are allowed


This is basically a param-name, right? Do we have a specification/pattern for params somewhere else already? Might be good to reuse that, if we have it

As far as I can see we're only testing for invalid characters. Am I being too strict here? 🤔

Hmmmm. We should allow all parameter names that K8s allows. Otherwise we might run into a problem where some application expects a parameter name of "option=rw,size=2300". We just had that case in Marathon, where some application abused the parameter name to pass multiple things :-/

In any case, the validation should be the same as is used in the verify that you linked.

pkg/engine/task/task_pipe.go

nfnt

This looks great! Found a few nits though, all unrelated to the actual implementation.

pkg/engine/task/task_apply.go

go.mod

pkg/engine/task/task.go

pkg/engine/task/task_pipe.go

test/integration/upgrade-command/first-operator/operator.yaml

additionally, instance controller doesn't not reconcile on pipe-pod deletion events

kensipe

I would really like to see more tests around the mapping to a template file...
however this LGTM!

keps/0017-pipe-tasks.md

kensipe · 2019-12-04T02:42:16Z

keps/0017-pipe-tasks.md

@@ -107,21 +110,19 @@ spec:
  volumes:
  - name: cert
    secret:
-      secretName: {{.Pipes.Certificate}}
+      secretName: {{.Pipes.Mycertificate}}


it looks like pipe keys are generated to guarantee uniques based on phase and step... but then are flatten in the template... is that true?

I'm not sure I understand the question but:

pipe keys are not generated but chosen by the user e.g. {{.Pipes.Mycertificate}} in line 56 above

the actual name of the secret (where the file is stored) is generated by KUDO and is an implementation detail. I mentioned it here for completeness

pkg/controller/instance/instance_controller.go

pkg/engine/task/pod_exec.go

test/integration/upgrade-command/first-operator/templates/configmap.yaml

alenkacz

I guess nothing is really a blocker for me - there's definitely a lot of possible improvements, but we can iterate on that. The only part I am really concerned is the tricky code around streams and goroutines - it's not really tested that much and I personally would be really scared to touch it in any way. I wonder if we can still do better there :)

pkg/controller/instance/instance_controller.go

pkg/engine/task/pod_exec.go

pkg/engine/task/task_pipe.go

pkg/engine/task/pod_exec.go

pkg/engine/task/task_pipe.go

test/integration/upgrade-command/first-operator/templates/configmap.yaml

nfnt

This looks in a great shape already, my comments are mostly minor and around Goroutine use. Need to do another pass though tomorrow at pkg/engine/task/* because that's a lot of code.

pkg/controller/instance/instance_controller.go

pkg/engine/task/pod_exec.go

and with them goroutines and a lot of complexity.

nfnt

Finally could look at task_pipe.go. It looks great, though I'm concerned with leaking resources in some of the code.

pkg/engine/task/pod_exec.go

pkg/engine/task/task_pipe.go

additionally PodExec caller can now differentiate between command execution failures (exit code > 0) and other errors

pkg/engine/task/podexec/pod_exec.go

nfnt

Looks great!

zen-dog requested review from kensipe, alenkacz, gerred and ANeumann82 November 25, 2019 20:06

zen-dog added the WIP label Nov 25, 2019

alenkacz reviewed Nov 26, 2019

View reviewed changes

ANeumann82 reviewed Nov 26, 2019

View reviewed changes

pkg/engine/task/task_pipe.go Outdated Show resolved Hide resolved

zen-dog added 2 commits November 26, 2019 16:54

Added comments and TODOs

d58e063

Fixed a deadlock when executing a pod command when downloading a file

7b6d5f3

zen-dog requested a review from nfnt as a code owner November 27, 2019 16:25

zen-dog added 2 commits November 28, 2019 02:33

Added tests for validating pipe task and improved pipe artifact naming

8848320

Improved pipe artifact naming function

61dd838

nfnt requested changes Nov 28, 2019

View reviewed changes

zen-dog added 5 commits November 28, 2019 22:26

Instance controller now watches Pod events

3811c1a

Check pipe file size before downloading it

afb47ad

Ran go mod tidy

a921f6c

Added {{ .Pipes. }} config object to template rendering

fe78dd7

Fix test/integration/upgrade-command

d8589a9

zen-dog requested a review from jbarrick-mesosphere as a code owner December 2, 2019 15:23

zen-dog added needs review and removed WIP labels Dec 2, 2019

zen-dog commented Dec 2, 2019

View reviewed changes

test/integration/upgrade-command/first-operator/operator.yaml Show resolved Hide resolved

zen-dog added 4 commits December 3, 2019 16:58

Updated the pipe specification to take a pod instead of a container

f8e76c9

additionally, instance controller doesn't not reconcile on pipe-pod deletion events

Making linter happy

04c5870

Added more unit tests

67b9a1e

Minor comment improvement

ef7ce24

kensipe approved these changes Dec 4, 2019

View reviewed changes

alenkacz approved these changes Dec 4, 2019

View reviewed changes

zen-dog changed the title ~~KEP-0017: Pipe task implementation~~ KEP-17: Pipe task implementation Dec 4, 2019

nfnt requested changes Dec 4, 2019

View reviewed changes

pkg/controller/instance/instance_controller.go Outdated Show resolved Hide resolved

pkg/engine/task/pod_exec.go Outdated Show resolved Hide resolved

pkg/engine/task/pod_exec.go Outdated Show resolved Hide resolved

zen-dog added 2 commits December 4, 2019 19:32

First round of review feedback

e83fc64

Removed io.Pipe when executing pod commands

82be1a2

and with them goroutines and a lot of complexity.

nfnt requested changes Dec 5, 2019

View reviewed changes

pkg/engine/task/pod_exec.go Outdated Show resolved Hide resolved

pkg/engine/task/task_pipe.go Show resolved Hide resolved

zen-dog added 2 commits December 5, 2019 12:58

Moved pod_exec.go into a separate package

c1087ec

additionally PodExec caller can now differentiate between command execution failures (exit code > 0) and other errors

Making linter happy

a6bf10e

nfnt reviewed Dec 5, 2019

View reviewed changes

pkg/engine/task/podexec/pod_exec.go Show resolved Hide resolved

nfnt approved these changes Dec 5, 2019

View reviewed changes

Additional test for pipe-pod validation

6e38698

zen-dog merged commit ead3d4c into master Dec 5, 2019

zen-dog deleted the ad/pipe-task branch December 5, 2019 14:08

gerred removed the needs review label Jan 7, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KEP-17: Pipe task implementation #1105

KEP-17: Pipe task implementation #1105

zen-dog commented Nov 25, 2019 •

edited

Loading

alenkacz left a comment

ANeumann82 Nov 26, 2019

zen-dog Nov 26, 2019

ANeumann82 Nov 26, 2019

nfnt left a comment

kensipe left a comment

kensipe Dec 4, 2019

zen-dog Dec 4, 2019 •

edited

Loading

alenkacz left a comment

nfnt left a comment

nfnt left a comment •

edited

Loading

nfnt left a comment

KEP-17: Pipe task implementation #1105

KEP-17: Pipe task implementation #1105

Conversation

zen-dog commented Nov 25, 2019 • edited Loading

alenkacz left a comment

Choose a reason for hiding this comment

ANeumann82 Nov 26, 2019

Choose a reason for hiding this comment

zen-dog Nov 26, 2019

Choose a reason for hiding this comment

ANeumann82 Nov 26, 2019

Choose a reason for hiding this comment

nfnt left a comment

Choose a reason for hiding this comment

kensipe left a comment

Choose a reason for hiding this comment

kensipe Dec 4, 2019

Choose a reason for hiding this comment

zen-dog Dec 4, 2019 • edited Loading

Choose a reason for hiding this comment

alenkacz left a comment

Choose a reason for hiding this comment

nfnt left a comment

Choose a reason for hiding this comment

nfnt left a comment • edited Loading

Choose a reason for hiding this comment

nfnt left a comment

Choose a reason for hiding this comment

zen-dog commented Nov 25, 2019 •

edited

Loading

zen-dog Dec 4, 2019 •

edited

Loading

nfnt left a comment •

edited

Loading