Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support better previews in Python by mocking out Unknown values #1482

Merged
merged 1 commit into from
Jun 11, 2018

Conversation

swgillespie
Copy link
Contributor

@swgillespie swgillespie commented Jun 9, 2018

This PR is a little wacky, and I totally understand if we don't want to take it, but it results in a much better preview experience for Python.

@pgavlin suggested the other day that the Python Unknown value be just the string <computed>. This PR is basically an elaboration on this idea that addresses some of the practical concerns that come out of it.

In JavaScript, we use Output to hide the existence of Unknown values. In this way we can prevent the user from ever observing an unknown value, since we don't run apply callbacks on previews. Python does not have Output (yet) so there's nothing stopping unknown values from leaking into user programs during previews.

It's not an awful thing for users to observe unknown values. An unknown value, though, should

  1. Be able to be passed to other resources without modification (e.g. the scenario of creating a resource and then creating another resource that references the first resource's ID)
  2. Be able to be printed as if it were a string
  3. Pass type checks imposed by tfgen-generated code - usually asserting that a value is a string, a list, or a dict
  4. Be able to be identified as Unknown elsewhere in the runtime, so it can be turned in to the unknown sentinel GUID before being sent to the engine.

This PR accomplishes all but the second two parts of 3 (masquerading as a list or dict) by defining an Unknown class that inherits from unicode and lies about its internals:

  1. Converting an Unknown to a string (e.g. by printing it or calling str on it) produces the string "computed". This allows users to safely treat Unknown as a string for concatenation, printing, or formatting.
  2. Converting an Unknown to a boolean (e.g. by using not with it or using it in a conditional. This passes all required argument presence checks in tfgen-generated code.
  3. Reading properties off of an Unknown produces another Unknown, regardless of the property. This allows Unknown to masquerade as a resource.
  4. Using the index operator on an Unknown produces another Unknown, which allows Unknown to masquerade as lists and dicts.
  5. Iterating over an Unknown produces an empty iterator.

With this object, we're able to do nontrivial previews on Python programs, as long as the preview does not attempt to use Unknown as a list or dict. Unknown inherits from unicode, the unicode string type, so it passes all tfgen checks that a property is a string, but it does not pass isinstance(..., list) or isinstance(..., dict).

@swgillespie
Copy link
Contributor Author

Here's an example where I preview a use of awsinfra.Network with private subnets over 2 availability zones.

Before:

$ pulumi preview
Previewing update of stack 'aws-infra-py'
Previewing changes:

     Type                             Name                       Plan          Info
 *   global                           global                     no change     1 error, 1 info message
 +   └─ pulumi:pulumi:Stack           aws-infra-py-aws-infra-py  create
 +      └─ aws-infra:network:Network  mynet                      create
 +         └─ aws:ec2:Vpc             mynet                      create

Diagnostics:
  global: global
    info: Traceback (most recent call last):
      File "/usr/local/pulumi/bin/pulumi-language-python-exec", line 55, in <module>
        pulumi.runtime.run_in_stack(lambda: runpy.run_path(args.PROGRAM, run_name='__main__'))
      File "/Users/swgillespie/go/src/github.com/pulumi/pulumi/sdk/python/env/src/pulumi/runtime/stack.py", line 28, in run_in_stack
        Stack(func)
      File "/Users/swgillespie/go/src/github.com/pulumi/pulumi/sdk/python/env/src/pulumi/runtime/stack.py", line 47, in __init__
        func()
      File "/usr/local/pulumi/bin/pulumi-language-python-exec", line 55, in <lambda>
        pulumi.runtime.run_in_stack(lambda: runpy.run_path(args.PROGRAM, run_name='__main__'))
      File "/usr/local/Cellar/python@2/2.7.15/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 276, in run_path
        run_name, fname, loader, pkg_name).copy()
      File "/usr/local/Cellar/python@2/2.7.15/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
        exec code in run_globals
      File "/Users/swgillespie/Documents/workspace/pulumi/aws-infra/__main__.py", line 6, in <module>
        network = Network("mynet", number_of_availability_zones=2, use_private_subnets=True)
      File "./network.py", line 58, in __init__
        print vpc.id
    AttributeError: 'Vpc' object has no attribute 'id'

  global: global
    error: an unhandled error occurred: Program exited with non-zero exit code: 1

error: an error occurred while advancing the preview

After:

$ pulumi preview
Previewing update of stack 'aws-infra-py'
Previewing changes:

     Type                                    Name                       Plan          Info
 *   global                                  global                     no change     1 info message
 +   └─ pulumi:pulumi:Stack                  aws-infra-py-aws-infra-py  create
 +      └─ aws-infra:network:Network         mynet                      create
 +         ├─ aws:ec2:Vpc                    mynet                      create
 +         ├─ aws:ec2:InternetGateway        mynet                      create
 +         ├─ aws:ec2:RouteTable             mynet                      create
 +         ├─ aws:ec2:Subnet                 mynet-0                    create
 +         ├─ aws:ec2:Subnet                 mynet-nat-0                create
 +         ├─ aws:ec2:RouteTableAssociation  mynet-nat-0                create
 +         ├─ aws:ec2:Eip                    mynet-nat-0                create
 +         ├─ aws:ec2:NatGateway             mynet-nat-0                create
 +         ├─ aws:ec2:RouteTable             mynet-nat-0                create
 +         ├─ aws:ec2:RouteTableAssociation  mynet-0                    create
 +         ├─ aws:ec2:Subnet                 mynet-1                    create
 +         ├─ aws:ec2:Subnet                 mynet-nat-1                create
 +         ├─ aws:ec2:RouteTableAssociation  mynet-nat-1                create
 +         ├─ aws:ec2:Eip                    mynet-nat-1                create
 +         ├─ aws:ec2:NatGateway             mynet-nat-1                create
 +         ├─ aws:ec2:RouteTable             mynet-nat-1                create
 +         └─ aws:ec2:RouteTableAssociation  mynet-1                    create

Diagnostics:
  global: global
    info: <computed> <--- added bonus - this is what you get if you print an Unknown value

info: 19 changes previewed:
    + 19 resources to create

@joeduffy
Copy link
Member

joeduffy commented Jun 9, 2018

Not having looked at the code yet, I strongly vote for taking this.

Seems like a huge step forward in the absence of having true outputs.

I shall take a look at the code now.

Copy link
Member

@joeduffy joeduffy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, let's get this in for the final release.

@swgillespie
Copy link
Contributor Author

let's get this in for the final release.

Agreed. I think we should also take pulumi/pulumi-terraform#171, it's a major annoyance for Python use.

Copy link
Member

@pgavlin pgavlin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@swgillespie swgillespie merged commit 8a00cb4 into master Jun 11, 2018
@swgillespie
Copy link
Contributor Author

thanks!

@swgillespie swgillespie deleted the swgillespie/python-unknown branch June 11, 2018 17:30
@mmdriley
Copy link
Contributor

this is a practical and Pythonic solution. Nice! lgtm.

@lindydonna lindydonna added kind/bug Some behavior is incorrect or out of spec impact/changelog labels Jun 15, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Some behavior is incorrect or out of spec
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants