Functions/datasources should accept Outputs

[lukehoban]

One of the most common usability issues I've seen new users hit with Pulumi is that datasources/functions take and return raw values instead of Inputs/Outputs, meaning they are very difficult to compose with the rest of the Pulumi resource model. We encourage folks to get used to writing code like this in the normal Pulumi usage:

const a = new ResourceA("name");
const b = new ResourceB("name", { prop: a.id }); // pass an `Output` as an `Input`

But then when you want to do the same with a Pulumi function - it doesn't work.

const a = new ResourceA("name");
const b = someFunction({ prop: a.id }); // cannot pass `Output` where `string` is expected

Instead - you have to write something like this:

const a = new ResourceA("name");
const b = a.id.apply(v => someFunction({ prop: v }));

This is not obvious, and not pleasant.

The original motivation was to provide the additional flexibility of being able to use these functions in cases where you don't need to get caught up in Outputs. And it may be we feel we have to continue to support those use cases.

A non-breaking way to improve this would be to project two copies of every function somFunc and someFuncOutput for example - where the latter accepts Inputs and returns Outputs.

---

Engineering Work Items

M64

• Update conceptual docs to use $fnOutput function forms by default #7946 (In review)
• Update examples to use new fn_output forms for fun calls where appropriate #7948 (In progress)
  • Update C# examples to use the new F.Invoke feature examples#1093
  • Update Go examples to use $fn.Output style of invokes examples#1094
  • Update Python examples to use fn_output forms where appropriate examples#1095
  • Update Node JS examples to use the new $fnOutput feature examples#1096
• Update program gen to employ new $fn_output forms #7949 (In review)
• Update docs to use new fn_output forms for funcalls so that this shows up in API reference documentation #7947
• Failing preview when f_output receives unknowns #8172
• Output-versioned function invoke fails to preview in C#/.NET #8322 (released in 3.17.1)

M63

• Implement 5758 for C#/.NET #7945
  PR: 5758 for C#/.NET #7899

M62

• Implement 5758 for Node JS #7944
  PR: 5758 for Node JS #8047
• Codegen testing upgrades #7989
• Simplify output-funcs codegen test #8039

M61

• 5758 for Go #7784
• Implement 5758 {fn}_output codgen for Python #7825
• Add CodegenUtilities to C# SDK in prep for 5758 #7934

Future Iterations

• Convert Go Azure examples to fnOutput form examples#1115 (blocked)
• Enable fnOutput function form generation on the Go build of pulumi-azure-native pulumi-azure-native#1279

[justinvp]

Semi-related to #5385. For multi-lang components, we want to be able to support functions referencing resources. To do so, if a function takes a resource as an argument, we'll update codegen to emit the function as accepting Input<TheResource> along with any other args as inputs, and have the function return an Output.

[mikhailshilkov]

Related to both comments above, I was playing with an idea of yet another form

const a = new ResourceA("name");
const b = a.someFunction();

This is based on Azure NextGen where invokes ~always "belong" to a parent resource and the relationship is easy to derive. WDYT?

[justinvp]

const b = a.someFunction();

We are planning to add support for "resource methods" as part of the multi-lang component work. I send you a link to the design doc.

[pgavlin]

A non-breaking way to improve this would be to project two copies of every function somFunc and someFuncOutput for example - where the latter accepts Inputs and returns Outputs.

I'm not sure that I agree that this is an improvement, especially for the first-time experience. How do we describe when to use which? Do we update examples to use *Output? We have apply for a reason; let's figure out how to explain that better rather than muddying our API surface area.

[lukehoban]

How do we describe when to use which?

~all documentation would point at the *Output form.

Do we update examples to use *Output?

Yes - almost all examples, and all code-generated examples, would use the *Output form - which will almost universally make the example code much cleaner as today our standard codegen pattern is to wrap these in apply anyway because we have to work around the lack of theses functions.

let's figure out how to explain that better rather than muddying our API surface area.

Having tried to explain this hundreds of times, at this point I believe we have to make a product change. Open to suggestions on alternative solutions, but this is just too significant of a usability issue to leave to just "explaining".

[pgavlin]

Do you believe that it's most common to call functions using outputs from resources and then use them as pure inputs to other resources (as opposed to using the results for control flow)?

[mikhailshilkov]

Do you believe that it's most common to call functions using outputs from resources and then use them as pure inputs to other resources (as opposed to using the results for control flow)?

FWIW, this is true for my experience with Azure NextGen and beyond.

[lukehoban]

As a concrete example - this is the code we will be adding to our azure Python template:

# Export the primary key of the Storage Account
primary_key = pulumi.Output.all(resource_group.name, account.name) \
    .apply(lambda args: storage.list_storage_account_keys(
        resource_group_name=args[0],
        account_name=args[1]
    )).apply(lambda accountKeys: accountKeys.keys[0].value)

If we had an Output form of the function - that whole thing would be:

# Export the primary key of the Storage Account
primary_key =  
storage.list_storage_account_keys_output(resource_group_name=resource_group.name, account_name=account.name).keys[0].value

[t0yv0]

Tangentially related, this style of composition is what Haskellers eat for breakfast under the name of Applicative functors.

 pure    :: a -> f a
 ( <*> ) :: f (a -> b) -> f a -> f b

 liftA2 f x y = f <$> x <*> y
    
 pure storage.list_storage_account_keys <*> resource_group.name <*> account.name  ==
      storage.list_storage_account_keys <$> resource_group.name <*> account.name  ==
      liftA2 storage.list_storage_account_keys resource_group.name account.name

This version of lift_a2 checks under mypy:

from pulumi import Output
from typing import Optional, Callable, TypeVar, List, Any
import pulumi


class InvokeOptions:
    pass

class Key:
    value: str

class ListStorageAccountKeysResult:
    keys: List[Key]


def list_storage_account_keys(account_name: Optional[str] = None,
                              expand: Optional[str] = None,
                              resource_group_name: Optional[str] = None,
                              opts: Optional[InvokeOptions] = None) -> ListStorageAccountKeysResult:
    ...


T1 = TypeVar('T1')
T2 = TypeVar('T2')
T3 = TypeVar('T3')


def lift_a2(f: Callable[[T1, T2], T3]) -> Callable[[Output[T1], Output[T2]], Output[T3]]:

    def f_any(xy: Any) -> T3:
        return f(xy[0], xy[1])

    return lambda x, y: pulumi.Output.all([x, y]).apply(f_any)


def example(account_name: Output[Optional[str]], resource_group_name: Output[Optional[str]]) -> Output[str]:
    return lift_a2(list_storage_account_keys)(account_name, resource_group_name) \
        .apply(lambda x: x.keys[0].value)

[t0yv0]

There is also a code style for this more suitable to imperative languages, I think I first saw it in Knockout JS library. The idea is that there is a magic "Output builder" function that accepts a pure (this is crucial) lambda that is allowed to evaluate any Output[T] to a T inside the body. This presents no problem for mypy checking again:

class Evaluator:
    def __call__(self, x: Output[T1]) -> T1:
        ...


def output(f: Callable[[Evaluator], T1]) -> Output[T1]:
    ...


def example2(account_name: Output[Optional[str]], resource_group_name: Output[Optional[str]]) -> Output[str]:
    return output(lambda e: list_storage_account_keys(e(account_name), e(resource_group_name)).keys[0].value)

The output function can be made general enough to support functor, applicative functor, and monadic composition. Applicative functor was just demonstrated. Functor is simple:

def a_to_b(x: A) -> B:
    ...

def functor_example(x: Output[A]) -> Output[B]:
    return output(lambda e: a_to_b(e(x)))

For monadic composition we need a variation that lets the body return a nested Output[T] and unwraps it later:

def output(f: Callable[[Evaluator], Output[T1]]) -> Output[T1]:
    ...

Purity of the builder lambda is important because the implementation needs to call the lambda at least twice, once with junk arguments, to introspect its body. The constraint is more than purity, in fact, it is assumed that the lambda does not introspect and branch on the arguments. This makes it slightly controversial. The notation is certainly super convenient but these restrictions may fly in the way of the programmer intuition. Again Knockout JS lib got away with this.

[t0yv0]

Chatting with @pgavlin (thanks a ton), concrete steps for me here are:

• table imperative output proposal into a separate issue for later discussion (low priority)

• lift_a2 style combinators allowed as an implementation detail but not public API (no teaching the world Haskell)

• provide a PR to edit SDK codegen in Pulumi to add the new form of functions

• spilt out a ticket and follow up with a PR to update hcl2 codegen to take advantage of the new form in the code generated out of documentation - this might be a little more involved

[lukehoban]

BTW - a sketch of what I expected this to look like for TypeScript is in 54dad85.

[t0yv0]

Quick update, I'm sorry for some delay getting all the subtasks done here. After chatting with @emiliza I've broken down the remaining work so it's easier to see in our planning so we see where we stand on this.

Go and Python are done.

Remainder:

C# - PR done with tests but waiting on providers to up the SDK

• Implement 5758 for C#/.NET #7945

Node

• Implement 5758 for Node JS #7944

Update conceptual docs

• Update conceptual docs to use $fnOutput function forms by default #7946

Update docs gen so that this shows up in API reference documentation

• Update docs to use new fn_output forms for funcalls so that this shows up in API reference documentation #7947

Update examples to use this style

• Update examples to use new fn_output forms for fun calls where appropriate #7948

Update program gen so that generated examples use this style

• Update program gen to employ new $fn_output forms #7949

Looks like I've already opened a set of similar issues before, let me clean this up as dups right now.

@ericrudder the list above may help answer your questions on the "big picture view" here, if not please let me know if you have further questions.