Add Job awaiter

[lblackstone]

Fixes #449.

TODO:

• Report errors
• Add tests
• Await deletion

Here's how things look at this point:

Failed Job:

[0/2] Waiting for Job "foo" to start
[1/2] Waiting for Job "foo" to succeed
warning: [Pod foo-vk6wg]: containers with unready status: [pi]

warning: [Pod foo-vk6wg]: containers with unready status: [pi] -- [ContainerCannotRun] OCI runtime create failed: container_linux.go:345: starting container process caused "exec: \"perly\": executable file not found in $PATH": unknown

error: Job has reached the specified backoff limit -- [BackoffLimitExceeded] Job has reached the specified backoff limit
 
error: Plan apply failed: 4 errors occurred:
	* resource foo was successfully created, but the Kubernetes API server reported that it failed to fully initialize or become live: Resource 'foo' was created but failed to initialize
	* [Pod foo-vk6wg]: containers with unready status: [pi]
	* [Pod foo-vk6wg]: containers with unready status: [pi] -- [ContainerCannotRun] OCI runtime create failed: container_linux.go:345: starting container process caused "exec: \"perly\": executable file not found in $PATH": unknown
	* Job has reached the specified backoff limit -- [BackoffLimitExceeded] Job has reached the specified backoff limit

Successful Job:

[0/2] Waiting for Job "foo" to start
[1/2] Waiting for Job "foo" to succeed
warning: [Pod foo-wsq8r]: containers with unready status: [pi]
Job ready

[nesl247]

Looking forward to this. Should really simplify our code base. Any idea when this is targeted to be completed?

[lblackstone]

Any idea when this is targeted to be completed?

I'd guess that it will be in a dev build within the next week.

[nesl247]

Awesome, so glad to hear that. This is a HUGE feature for us, and pretty much anyone using pulumi to deploy applications that require stuff like DB migrations, etc.

[lblackstone]

Here's how things look at this point:

Failed Job:

[0/2] Waiting for Job "foo" to start
[1/2] Waiting for Job "foo" to succeed
warning: [Pod foo-vk6wg]: containers with unready status: [pi]

warning: [Pod foo-vk6wg]: containers with unready status: [pi] -- [ContainerCannotRun] OCI runtime create failed: container_linux.go:345: starting container process caused "exec: \"perly\": executable file not found in $PATH": unknown

error: Job has reached the specified backoff limit -- [BackoffLimitExceeded] Job has reached the specified backoff limit
 
error: Plan apply failed: 4 errors occurred:
	* resource foo was successfully created, but the Kubernetes API server reported that it failed to fully initialize or become live: Resource 'foo' was created but failed to initialize
	* [Pod foo-vk6wg]: containers with unready status: [pi]
	* [Pod foo-vk6wg]: containers with unready status: [pi] -- [ContainerCannotRun] OCI runtime create failed: container_linux.go:345: starting container process caused "exec: \"perly\": executable file not found in $PATH": unknown
	* Job has reached the specified backoff limit -- [BackoffLimitExceeded] Job has reached the specified backoff limit

Successful Job:

[0/2] Waiting for Job "foo" to start
[1/2] Waiting for Job "foo" to succeed
warning: [Pod foo-wsq8r]: containers with unready status: [pi]
Job ready

[metral]

Really digging the test scenarios here 🙂. I know we do similar, extended coverage for all other resources with fixtures, but am not sure to what extent. How is the parity in this space across all resources?

[lblackstone]

This is similar to the way we're testing Pod, and is the general direction I'm taking the test coverage. Existing tests for other resources are basically black box tests, and generally make it harder to reason about correctness.

There's room for another layer of tests to make sure the awaiter channels/timeouts are wired up properly, but I think that's a lot less critical and error prone than the state checking logic I'm testing here.

[metral]

Overall LGTM - A couple of logical changes would be nice to see to reduce on complexity

[metral]

LGTM

cc @hausdorff - PTAL and review.

[hausdorff]

Adding await semantics to Job simply enriches the existing functionality, so I think this is the right move.

But, before we click the merge button, I just want to make sure we're all on the same page about the (very weird) implications of using Job in Pulumi. All of these were true before, but we never had this conversation before, so let me state some things I believe to be true about the Job, and if any of them are wrong, please correct me:

• If you run pulumi up with a Job, it will stick around until you delete it. So subsequent runs of pulumi up will not cause the job to re-run.
• Users should be very cautious of including Job in Pulumi programs! Unlike other resource types, Job is intended to run once (e.g., for a DB schema migration), so when and how it runs really matters. Once you add a Job to your Pulumi project, ordering suddenly matters a lot—so if you run a fresh pulumi up and your Job does not run exactly when it is supposed to, it could fail the whole deployment.
• We make no attempt to be smart about automated cleanup from the TTL controller. So, if a user sets .spec.ttlSecondsAfterFinished and the Job gets cleaned up, another run of pulumi up after the TTL will re-deploy the Job.

Like I said, I think all of this is fine, especially since we support it all already, but I just want us to go in with eyes open.

[hausdorff]

We say that these are the conditions required to determine a job is "ready", but it sounds below like we're describing jobs that have completed?

[lblackstone]

Yes, I meant "ready" in the sense that we're done waiting on the resource. For Job, that would mean it is complete.

[lukehoban]

If you run pulumi up with a Job, it will stick around until you delete it. So subsequent runs of pulumi up will not cause the job to re-run.

Yes - that is expected and desired - unless you do something to force it to replace.

Users should be very cautious of including Job in Pulumi programs! Unlike other resource types, Job is intended to run once (e.g., for a DB schema migration), so when and how it runs really matters. Once you add a Job to your Pulumi project, ordering suddenly matters a lot—so if you run a fresh pulumi up and your Job does not run exactly when it is supposed to, it could fail the whole deployment.

I think this behaves exactly as you want for scenarios where it is useful - as long as you can force it to replace when the thing that should trigger it to re-run changes.

[hausdorff]

This is simple enough we can probably merge now. I'm fully confident we'll find more bugs, but I think we're well within our risk tolerance here.

I left a couple comments. The biggest thing that is missing is the error reporting is not super great in interactive mode. We can follow up with that though.