Is "composability" really a fundamental ytt limitation that jsonnet doesn't have?

[pmorch]

Hi,

Am I missing something? The following seems impossible in ytt, but is so easy in jsonnet: Composability - give the output of one template as the input to another template.

I hope I'm wrong because there are many other things I love about ytt over jsonnet and would prefer to stay with ytt.

It's as if the data.values mechanism is a single pass only. And that for me is a deal-breaker limitation that jsonnet doesn't seem to have. It's as if ytt is a programming language where you can call functions, but functions cannot call functions.

Example time:

values.yaml

#@data/values
---
foo: bar

template.yaml

#@ load("@ytt:data", "data")
field1: #! data.values.foo + "1"
field2: #! data.values.foo + "2"

This works (and is more complicated in real life). We have many users putting in all kinds of values in values.yaml.

Now somebody else wants to use our template.yaml unaltered, but wants to create the input for template.yaml with:

create_foo.yaml

#@ load("@ytt:data", "data")
foo: #! data.values.prefix+ "-yada-"

They want to use that and an unaltered template.yaml somehow to generate this output for prefix = a:

field1: a-yada-1
field2: a-yada-1

And this for prefix = b:

field1: b-yada-1
field2: b-yada-1

I don't see how this is possible with ytt. I can't use the output of one template as the input of another. I can't do "turles all the way down".

With jsonnet this is easy-peasy:

local template(foo) = {
  field1: foo + "1",
  field2: foo + "2",
};

// I can use template() directly
// template("test")

// But now I want to use it from another "template":
local another_template(prefix) = template(prefix + '-yada-');

another_template("a")

outputs:

field1: a-yada-1
field2: a-yada-1

Sure, with ytt we have skylark, and with skylark A can call B. But it is not possible to create a useful template.yaml in skylark, assuming template.yaml is > 70% YAML and < 30% logic. I can only use YAML in the last step.

I reported my first similar "issue" in #309. Really a feature request. Essentially: "How do I use data.values.foo to create data.values.bar". Answer: You can't. I now think I understand more fundamentally the issue I encountered back then, and that it is a fundamental ytt limitation that simply is not there in e.g. jsonnet.

Have I missed something?

[pivotaljohn]

Hey @pmorch! Thank you for taking the time to detail the situation and make so clear your question. 🙏🏼

There are a number of angles I see in here. I think I can best contribute here by drawing up the levels of composability present in ytt and then we can step back and look at it, together.

Level 0: A Function

Within a given invocation of ytt, we can capture any size chunk of YAML, wrap it in a function and then invoke it (Docs: Language > YAML Fragment).

Attempting to translate from the jsonnet example you gave, we'd have:

template.yml

#@ load("@ytt:template", "template")

#! First, capture the YAML document in a "YAML Fragment"
#!
#@ def my_template(foo):
---
field1: #@ foo + "1"
field2: #@ foo + "2"
#@ end

#! Later in the same file, I start a YAML document (`---`), and replace it with the document produced by `my_template()`, passing in the value for `foo`:
--- #@ template.replace(my_template("a-yada-"))

yielding:

field1: a-yada-1
field2: a-yada-2

So far, so good.

Now, we can write a second function in terms of this first function (for clarity, I'm repeating the whole example):

#@ load("@ytt:template", "template")

#! First, capture the YAML document in a "Fragment Function"
#!
#@ def my_template(foo):
---
field1: #@ foo + "1"
field2: #@ foo + "2"
#@ end

#@ def another_template(prefix):
#@   return my_template(prefix + "-yada-")
#@ end

#! Again, in the same file, I start a YAML document (`---`), and replace it with the document produced by `another_template()`, passing in the value for `foo`:
--- #@ template.replace(another_template("a"))
--- #@ template.replace(another_template("b"))

producing:

field1: a-yada-1
field2: a-yada-2
---
field1: b-yada-1
field2: b-yada-2

Playground

Level 1: A "Library" Function

The next tiny step would be to extract a function into a reusable chunk in a separate file. The key here is to add the .lib. extension. (Docs: Language > Load)

template.lib.yml

#! Define the "Fragment Function"
#!
#@ def my_template(foo):
---
field1: #@ foo + "1"
field2: #@ foo + "2"
#@ end

config.yml

#@ load("@ytt:template", "template")
#@ load("template.lib.yml", "my_template")

#@ def another_template(prefix):
#@   return my_template(prefix + "-yada-")
#@ end

#! showing off the `lambda` expression for creating function values wrapping an expression
#@ yet_another_template = lambda prefix: my_template(prefix + "-yada")

#! Again, in the same file, I start a YAML document (`---`), and replace it with the document produced by `another_template()`, passing in the value for `foo`:
--- #@ template.replace(another_template("a"))
--- #@ template.replace(yet_another_template("b"))

Playground

(More details on lambdas: Starlark spec > Lambda expressions)

---

Level 2: A ytt Library

You can also extract a whole set of templates into a completely isolated evaluation by putting them in a "Library" (Docs: Built-in @ytt Library > Library Module).

A ytt Library:

• declares its own data values
• evaluates completely isolated from all other libraries (including the main/root library)
• must be explicitly instantiated and evaluated to have any effect on the ytt run.

Example:

├── _ytt_lib
│   └── libby
│       ├── template.yml
│       └── values.yml
└── config.yml

where

• libby/ and all its contents (i.e. template.yml and values.yml) are considered a "Library" named "libby".
• config.yml is part of the "root" library.

_ytt_lib/libby/template.yml

#@ load("@ytt:data", "data")
---
field1: #@ data.values.foo + "1"
field2: #@ data.values.foo + "2"

_ytt_lib/libby/values.yml

#@data/values
---
foo: ""

config.yml

#@ load("@ytt:template", "template")
#@ load("@ytt:library", "library")
---

#! step 1: "instantiate" the library (just loaded, doesn't get evaluated, yet)
#@ libby = library.get("libby")

#! step 2: "instantiate" two more from that first definition
#@ libby_a = libby.with_data_values({"foo": "a-yada-"})
#@ libby_b = libby.with_data_values({"foo": "b-yada-"})

#! step 3: evaluate
--- #@ template.replace(libby_a.eval())
--- #@ template.replace(libby_b.eval())

(Unfortunately, one cannot include directories in gists. A hack is to use backslash \ and ask the recipient of the playground link to replace those backslashes with forward slashes. So, follow the link below and, replace all the \ in the filenames with /.)
Playground

Level 3: A Sourced Third-Party ytt Library

(This is likely beyond what you're pointing at, but I'm mentioning the level for completeness' sake.)

The "ultimate" in re-use, I guess, is being able to use someone else's ytt library in your own project.
Here's an example of such a library: https://github.com/vmware-tanzu/carvel-ytt-library-for-kubernetes

In a different meaning of "composability", we can pull another tool off the Carvel shelf: vendir to pull the dependency to local disk, first, if that is desired. At that point, the ytt portion of this solution is identical to "Level 2" above.

---

So, do any of these approaches seem to show the level of composition you're thinking about?

[pivotaljohn]

If it were made sufficiently obvious, you wouldn't have had to ask... 🤷🏻

Today, a lot of our documentation is primarily reference: an inventory of what exists. We're slowly adding to the two other fundamental categories: concepts and guides.

This discussion, here, is a great dress rehearsal for a guide on modularity.

Thanks for the opportunity! 🙏🏻

[pivotaljohn]

This same question in a different form here: https://kubernetes.slack.com/archives/CH8KCCKA5/p1622643291252400 which includes additional details that could be generalized into a guide.

[microwavables]

Hey @pmorch it's been awhile! I'm the community manager for Carvel and would love to connect with you to learn more about your use cases with Carvel. Would you be open to connecting and giving the team some feedback? And in case you hadn't heard, we recently became a Sandbox project with CNCF! Looking forward to knowing more about you and welcoming you, officially, to our community. :)

[pmorch]

Hi @microwavables,

After a while trying to wrangle ytt to do what I wanted, I eventually gave up and started using jsonnet exclusively. I was very satisfied with that choice. I've since left that employer, so am no longer using ytt nor jsonnet, but would use jsonnet again. So I'm not ideal to hook up with. Thanks for reaching out though.

Peter

[microwavables]

Thank you for the quick reply @pmorch !! Let us know if you have any feedback. Would love to know how we could improve in the future to make things easier. :)