fn sink and source reference docs #2013

droot · 2021-05-14T00:02:59Z

This PR does the following:

Add reference (help) docs for kpt fn source and kpt fn sink
Drops wrap-kind wrap-apiversion flags from kpt fn source. I don't think they are useful. We can introduce them when we see the need. Locking the API surface area.
Renamed function-config flag in fn source to fn-config to make it consistent with fn eval

site/reference/fn/sink/README.md

site/reference/fn/source/README.md

mengqiy · 2021-05-14T00:43:31Z

site/reference/fn/sink/README.md

+kpt fn sink DIR [flags]
+
+DIR:
+  Path to a local directory to resources to. Directory must exist.


Path to a local directory to resources to.

This doesn't read well.

Is DIR required? If so, this behavior is different

missed write :) fixed.

Yes, DIR is required (probably was done as part of CLI usability work).

if so, this behavior is different

what behavior are you referring to ? and what is different ?

DIR was optional before. If unspecified, it prints to stdout.

Shell32-Natsu · 2021-05-14T16:52:51Z

site/reference/fn/sink/README.md

 -->

-Implements a [sink function] by reading STDIN and writing configuration.
+`sink` reads resources from `stdin` and writes them to a local directory.


If there is no directory specified, sink will output to stdout. Will we change this in v1?

DIR is required argument now. This is changed in v1. Not sure output to stdout makes any sense. If that is the intention, output from source or eval should be good enough, no need of sink. Right ?

Shell32-Natsu · 2021-05-14T16:53:58Z

site/reference/fn/source/README.md

-Implements a [source function] by reading configuration and writing to STDOUT.
+`source` reads resources from a local directory and writes them in `functionSpec`
+wire format to `stdout`. The output of the `source` can be pipe'd to commands
+such as `kpt fn eval` that accepts `functionSpec` format. It is useful for


Maybe use the link to function spec here.

frankfarzan · 2021-05-14T18:09:14Z

site/reference/fn/sink/README.md

 -->

-Implements a [sink function] by reading STDIN and writing configuration.
+`sink` reads resources from `stdin` and writes them to a local directory.


The format of stdin for sink and stdout for source are currently underspecified.

Added brief note about the input for sink and output for source.

frankfarzan · 2021-05-14T18:12:29Z

site/reference/fn/source/README.md

 -->

-Implements a [source function] by reading configuration and writing to STDOUT.
+`source` reads resources from a local directory and writes them in `functionSpec`


Compare:

kpt fn source wordpress \ | less

This is wrapped in ResourceList.

kpt fn source wordpress \ | kpt fn eval --image gcr.io/kpt-fn/kubeval:v0.1 - \ | less

This is multi-object YAML.

Is there a reason, all wire format as passed through pipes should not be the same:

kpt fn source wordpress \ | kpt fn eval --image gcr.io/kpt-fn/set-namespace:v0.1 - -- namespace=mywordpress \ | kpt fn sink wordpress2

Need to also improve testing for ths.

Related to whether we want flexibility of both wrapped and unwrapped with flags:

Source can emit output wrapped or unwrapped

Eval can accept input as wrapped or unwrapped

Eval can emit output as wrapped or unwrapped

Sink can accept input as wrapped or unwrapped

We should think of reasons why one is better over the other. Starting with:

Unwrapped is probably easier to operate on with other tooling (e.g. sed or yq)?

Wrapped is better if we want to include other things in ResourceList (e.g. one of the OpenAPI options @natasha41575 has prototyped)

?

For the OpenAPI scenario specifically: if we choose not to wrap the output in ResourceList (e.g. ResourceList.OpenAPI), another option we'd discussed was wrapping the OpenAPI schema in its own KRM resource (that could be put in ResourceList.items).

I think either unwrapped or wrapped is fine, but I agree that it makes sense to have consistency across commands.

Is there a reason, all wire format as passed through pipes should not be the same:

I am not familiar with the reasoning. But when it comes to reading resources from a byte stream such as 'stdin, all these command use same kio.ByteReaderthat supports resource in both (wrapped/unwrapped) format without specifying the format, soevalandsinksupports both the formats. And when writing the output, they usekio.ByteWriterthat writes in multi-yaml object format by default unless specified to wrap. Forfn source`, it always write the output in wrapped format.

So looking at the use case of creating your own pipeline using source, eval, sink, source needs to support ResourceList output format. eval needs to support ResourceList input as default (that is does) and it SHOULD output ResourceList by default (and that's where the confusion arises). and sink accepts ResourceList as input.
So only change of default output of eval to ResourceList will bring consistency and will bring it closer to the pipeline execution.

Now output to multi-yaml object has its benefits of being able to pipe to kubectl etc., so that can be supported with a flag.

natasha41575 · 2021-05-14T19:44:21Z

site/reference/fn/sink/README.md

+# output to DIR directory.
+$ kpt fn source DIR |
+  kpt fn eval --image gcr.io/example.com/my-fn - |
+  kpt fn sink DIR


All of the examples here with fn eval also use source and sink together. Do they always have to be used in conjunction? What happens if one is used without the other, e.g. kpt fn source DIR | kpt fn eval, or kpt fn eval | kpt fn sink? It might be helpful to explicitly answer these questions in the documentation or have examples

source, 'eval' and sink can be used to create your own little pipeline where source acts as a generator (builtin) function, followed by set of eval functions and then final sink function that writes the output. So far, that is the only use-case I know of. I want to see more use-cases of these in the field.

natasha41575 · 2021-05-14T19:50:10Z

site/reference/fn/source/README.md

 -->

-Implements a [source function] by reading configuration and writing to STDOUT.
+`source` reads resources from a local directory and writes them in `functionSpec`


For the OpenAPI scenario specifically: if we choose not to wrap the output in ResourceList (e.g. ResourceList.OpenAPI), another option we'd discussed was wrapping the OpenAPI schema in its own KRM resource (that could be put in ResourceList.items).

I think either unwrapped or wrapped is fine, but I agree that it makes sense to have consistency across commands.

droot

Addressed the feedback. I am refraining from making behavioral changes to source, sink or eval as part of this PR. I have updated the reference to document current behavior. Can follow up with another PR for behavior change.

droot · 2021-05-19T01:00:21Z

site/reference/fn/sink/README.md

 -->

-Implements a [sink function] by reading STDIN and writing configuration.
+`sink` reads resources from `stdin` and writes them to a local directory.


DIR is required argument now. This is changed in v1. Not sure output to stdout makes any sense. If that is the intention, output from source or eval should be good enough, no need of sink. Right ?

droot · 2021-05-19T01:02:37Z

site/reference/fn/sink/README.md

 -->

-Implements a [sink function] by reading STDIN and writing configuration.
+`sink` reads resources from `stdin` and writes them to a local directory.


Added brief note about the input for sink and output for source.

droot · 2021-05-19T01:06:58Z

site/reference/fn/sink/README.md

+# output to DIR directory.
+$ kpt fn source DIR |
+  kpt fn eval --image gcr.io/example.com/my-fn - |
+  kpt fn sink DIR


source, 'eval' and sink can be used to create your own little pipeline where source acts as a generator (builtin) function, followed by set of eval functions and then final sink function that writes the output. So far, that is the only use-case I know of. I want to see more use-cases of these in the field.

droot · 2021-05-19T01:07:10Z

site/reference/fn/source/README.md

-Implements a [source function] by reading configuration and writing to STDOUT.
+`source` reads resources from a local directory and writes them in `functionSpec`
+wire format to `stdout`. The output of the `source` can be pipe'd to commands
+such as `kpt fn eval` that accepts `functionSpec` format. It is useful for


droot · 2021-05-19T01:38:45Z

site/reference/fn/source/README.md

 -->

-Implements a [source function] by reading configuration and writing to STDOUT.
+`source` reads resources from a local directory and writes them in `functionSpec`


Is there a reason, all wire format as passed through pipes should not be the same:

I am not familiar with the reasoning. But when it comes to reading resources from a byte stream such as 'stdin, all these command use same kio.ByteReaderthat supports resource in both (wrapped/unwrapped) format without specifying the format, soevalandsinksupports both the formats. And when writing the output, they usekio.ByteWriterthat writes in multi-yaml object format by default unless specified to wrap. Forfn source`, it always write the output in wrapped format.

So looking at the use case of creating your own pipeline using source, eval, sink, source needs to support ResourceList output format. eval needs to support ResourceList input as default (that is does) and it SHOULD output ResourceList by default (and that's where the confusion arises). and sink accepts ResourceList as input.
So only change of default output of eval to ResourceList will bring consistency and will bring it closer to the pipeline execution.

Now output to multi-yaml object has its benefits of being able to pipe to kubectl etc., so that can be supported with a flag.

frankfarzan

some nits

frankfarzan · 2021-05-19T17:10:21Z

site/reference/fn/eval/README.md

+
+  1. Multi object YAML where resources are separated by `---`.
+
+  2. [Function Specification] wire format where resources are wrapped in an object


nit: you're linking twice

Fixed. Removed the linking here because links are not being rendered in this section (may be because it's a code block)

frankfarzan · 2021-05-19T17:11:41Z

site/reference/fn/source/README.md

 ```

+#### Flags
+
 ```shell


nit: other commands don't use shell directive, so there's little bit of inconsistency in rendering

Thanks for point it out. Cleaned it up.

droot requested a review from frankfarzan as a code owner May 14, 2021 00:02

droot requested review from Shell32-Natsu and mengqiy May 14, 2021 00:03

Shell32-Natsu mentioned this pull request May 14, 2021

updated reference for fn source and sink #2001

Closed

mengqiy reviewed May 14, 2021

View reviewed changes

Shell32-Natsu reviewed May 14, 2021

View reviewed changes

frankfarzan reviewed May 14, 2021

View reviewed changes

frankfarzan requested a review from natasha41575 May 14, 2021 18:59

natasha41575 reviewed May 14, 2021

View reviewed changes

droot commented May 19, 2021

View reviewed changes

droot requested review from frankfarzan and Shell32-Natsu May 19, 2021 01:41

droot added 4 commits May 19, 2021 08:27

fn sink and source reference docs

6cdb1f1

addressed feedback

cfa566d

fixed chaining example

d4da665

addressed feedback

4ab5d91

frankfarzan approved these changes May 19, 2021

View reviewed changes

addressed feedback

ff7b290

droot force-pushed the fn-source-sink-reference branch from 4ae301f to ff7b290 Compare May 19, 2021 19:35

droot merged commit 54741cf into kptdev:next May 19, 2021

frankfarzan pushed a commit to frankfarzan/kpt that referenced this pull request Jun 3, 2021

fn sink and source reference docs (kptdev#2013)

5a03769

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fn sink and source reference docs #2013

fn sink and source reference docs #2013

droot commented May 14, 2021

mengqiy May 14, 2021

droot May 14, 2021

mengqiy May 14, 2021

Shell32-Natsu May 14, 2021

droot May 19, 2021

Shell32-Natsu May 14, 2021

droot May 19, 2021

frankfarzan May 14, 2021

droot May 19, 2021

frankfarzan May 14, 2021

frankfarzan May 14, 2021 •

edited

Loading

natasha41575 May 14, 2021 •

edited

Loading

droot May 19, 2021

natasha41575 May 14, 2021 •

edited

Loading

droot May 19, 2021

natasha41575 May 14, 2021 •

edited

Loading

droot left a comment

droot May 19, 2021

droot May 19, 2021

droot May 19, 2021

droot May 19, 2021

droot May 19, 2021

frankfarzan left a comment

frankfarzan May 19, 2021

droot May 19, 2021

frankfarzan May 19, 2021

droot May 19, 2021


		1. Multi object YAML where resources are separated by `---`.

		2. [Function Specification] wire format where resources are wrapped in an object

fn sink and source reference docs #2013

fn sink and source reference docs #2013

Conversation

droot commented May 14, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

frankfarzan May 14, 2021 • edited Loading

Choose a reason for hiding this comment

natasha41575 May 14, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

natasha41575 May 14, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

natasha41575 May 14, 2021 • edited Loading

Choose a reason for hiding this comment

droot left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

frankfarzan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

frankfarzan May 14, 2021 •

edited

Loading

natasha41575 May 14, 2021 •

edited

Loading

natasha41575 May 14, 2021 •

edited

Loading

natasha41575 May 14, 2021 •

edited

Loading