[Epic] Unblock provider use of Plugin Framework alongside SDKv2

[t0yv0]

Support providers that serve a mix of SDKv2 and Plugin Framework based resources, targeting these three Pulumi providers:

Requirements

• no unexplained changes in schema
• tests passing on a mix of new and old resources/functions, new tests added as necessary
• functionality for the majority of resources/functions that continue to be implemented against TF SDKv2 upstream continues to use previous production code paths, so we gradually expose new code paths into production

Work Items ⚙️

These providers combine Plugin Framework and SDKv2 based resources with a gradual transition:

• CloudFlare Status: in prod v5.1.1
• Gitlab Status: in prod v5.0.0
• Linode Status: in prod v4.1.0
• Artifactory Status: in prod v3.5.0
• Datadog Status: in prod v4.18.0
• GCP Status: in prod v6.57.0
• AWS [Plugin Framework] Support bridging providers which incorporate both TFPF and SDK resources #830 WIP WIP: expose PF-based resources pulumi-aws#2464 Status: awaiting GCP to deploy to prod first

The following providers are fully based on Plugin Framework:

• Random - Status: in prod
• TLS - Status: awaiting upstream release based on PF

M0.87

• [Muxer] Implement Support for GetMappings #946
• Muxed AWS Provider fails to start #958
• Fix muxer so that AWS provider starts again #972
• Add runtime muxer #978
• Implement GetMapping for Muxed Providers #986
• Fixes an failure to derive Config encoder encountered in pulumi-aws #989
• Implemented TfBridge PF Muxing via shim.Provider merging #996
• Change Muxer API to take a single tfbridge.ProviderInfo #997
• [Muxer] Simplify GetMapping & Respect Flags #1001

M0.88

• User-provided secret bits are not preserved through resource properties #1028
• Plugin-framework based providers fail to support non-string configuration properties #1032
• AWS skip_metadata_api_check not working with Plugin Framework #1074

M0.89

• v4.18.0 produces error on pulumi up or preview using config for apiKey and appKey pulumi-datadog#276
• upgrading 4.9.0 to 5.0.0+ causes a problem with the provider pulumi-gitlab#273
• [x/muxer]: Log instead of failing on divergent CheckConfig results #1142
• Unexpected logs showing through Pulumi UI #1135

[t0yv0]

Some progress, got a build that has strictly additive schema changes and preserves ExtraTypes and is able to deploy resources.

pulumi/pulumi-aws#2464
#988

name: check
runtime: yaml
description: A minimal Pulumi YAML program
outputs: {}
resources:
  bucket1:
    type: aws:s3:Bucket
  rexpl1:
    type: aws:resourceexplorer/index:Index
    properties:
      type: LOCAL
  cidrcol1:
    type: aws:route53/cidrCollection:CidrCollection
    properties:
     name: collection-2
variables: {}

To make this mergeable:

• verify GetMapping
• verify cmd-line flags to dump MasrhsallableProviderInfo work as before
• verify sdk generation works
• verify tests pass in CI
• do an API simplification pass to tuck away internals from the provider itself
• break apart and check in changes to pulumi-terraform-bridge into the repo to release

[t0yv0]

A lot more progress pulumi/pulumi-aws#2464 - this doesn't have any known issues, we're working through CI issues to get a clean passing mark on all tests, and adding some tests specific to PF.

[t0yv0]

More progress still, most tests passing except aliased legacy LoadBalancer is failing with, I think @iwahbe is taking this, then we re-test.

    error: unrecognized resource type (Check): aws:elasticloadbalancingv2/loadBalancer:LoadBalancer

[AaronFriel]

@t0yv0 @iwahbe Can we add issues to track the rollout for Cloudflare, GCP & Azure (stage 1 & stage 2), cross-linking those with the other epics?

[t0yv0]

Per our conversations this issue is now repurposed as an epic encompassing all 3 providers rather than an AWS-specific task. I've added links to known WIP PRs and tracking issues. Feel free to edit. I've less context on what needs to happen for Stage 2, fork removal perhaps? If we want to track it here please populate that section.

[t0yv0]

#1032 is "almost" ready to review. The plan is:

https://github.com/pulumi/home/issues/2849 first cover prod code at ProgramTest level (postmortem follow, reduce risk to editing sdkv2 code)
https://github.com/pulumi/pulumi-terraform-bridge/pull/1086 then merge refactor of prod code
https://github.com/pulumi/pulumi-terraform-bridge/pull/1065 then rebase, review and merge actual fix

However 1032 by itself does NOT unblock pulumi-aws as I previously thought, which is unfortunate.

The underlying remaining issue #1074 is deeper.

I hope that #1086 fixes this deeper problem, but have not verified yet that this fixes it.

Also want to call out that lack of AutoName support is possible blocker if we move resources that used to rely on AutoName to PF and they lose AutoName that's breaking.

Support for provider-info level default values and autoname still on track in the iteration.

[t0yv0]

Notes from @iwahbe : it's helpful to install autoaliasing before these updates to speed up the process. We're unlocked now and starting to make releases.

[t0yv0]

Notes from project sync: AWS has extensive backports; likely first PF release of AWS (+ new version alpha) will not do any of that but simply add net-new PF resources and datasources. Removing backports and using straight PF resources will be left for later, and preferably picked up by pulumi/providers team.

[t0yv0]

Possible reasons to do backports sooner: if there's some incompatible changes over the resources in question that coincide with the major version bump; backported resources may defer these breaking changes.

[t0yv0]

Closing as complete per conversation with @AaronFriel . Remaining AWS rollout is happening as 6.0-alpha and tracked in pulumi/pulumi-aws#2539

[pulumi-bot]

Cannot close issue without required labels: resolution/