RFC: migrating provider build-time example translation to pulumi convert

[t0yv0]

We are considering to enforce using pulumi convert for the build-time Terraform example translation. Requesting feedback from bridged provider maintainers as this change will impact bridged provider builds and CI.

Background

As part of running make tfgen in a bridged provider, the build will attempt to locate Terraform examples, correlate them with the relevant Resource or Data Source documentation, and translate them to Pulumi examples. The result of this process is published in the Pulumi Package Schema informing the documentation rendering in the Pulumi registry, and also informs generated code for the SDKs in each Pulumi- supported language (Node, Python, Go, .NET and Java).

Currently the default method of translating the examples relied on linking in deprecated translation code into the build process directly. The improved method uses pulumi convert which invokes pulumi-converter-terraform under the hood. This change is looking at deprecating the default method as it is falling behind in functionality compared to pulumi convert and is not receiving fixes.

All bridged providers managed by Pulumi have finished migrating to the new method.

Preparing the provider build environment

A provider maintainer needs to:

• ensure pulumi CLI is installed (required)
• ensure Terraform converter plugin is installed (required)
• ensure Terraform converter plugin and any plugins referenced from examples are pinned (recommended)

Specifically pins are recommended for providers that build frequently and commit generated SDKs such as sdk/python to source control. If plugin references are left floating, out of band changes such as new releases of Pulumi providers may render the generated SDK code stale with respect to re-running make tfgen, which may negatively affect build hermeticity. These concerns may be ignored for providers that build infrequently or do not strictly enforce hermetic SDK builds.

Testing provider builds

Builds utilizing pulumi convert for example rendering can be currently tested by setting the PULUMI_CONVERT=1 environment variable. In the future bridged providers will opt into this behavior by default instead of gating it by an environment variable.

Example provider builds

Pulumi has finished migrating all internally managed bridged providers to use pulumi convert.

An example in pulumi-aws provider demonstrates pinning:

https://github.com/pulumi/pulumi-aws/blob/master/Makefile#L114

The Makefile pins Pulumi CLI version, the Terraform converter version and several related plugins as in:

	.pulumi/bin/pulumi plugin install converter terraform 1.0.17

It also opts into pulumi convert example rendering with the PULUMI_CONVERT=1 environment variable. In the future bridged providers will opt into this behavior by default instead of gating it by an environment variable.

Impact

Based on the experience of rolling the change out internally, the change is beneficial to the quality of the example conversion process:

• more examples convert successfully
• the generated code is more often correct
• the converter is supported going forward

There are a few known minor regressions:

• Skip autonamed fields pulumi-converter-terraform#122
• Boring resource names generated by the converter pulumi-converter-terraform#147

Issues can be filed against https://github.com/pulumi/pulumi-converter-terraform for further improvements in this area.

[t0yv0]

CC @guineveresaenger