Pass PlanResourceChange through downstream tests #1962
Comments
VenelinMartinov
added
the
kind/engineering
|
This would be much appreciated since any bugs discovered here also probably affect AWS users through a handful of enrolled resources, and would also be happy-path simple bugs, not corner-case bugs discovered by randomized testing. |
|
I ran a round of downstream tests but we'd get more valuable results after #1849, so I'll finish that up first. |
VenelinMartinov
changed the title
|
Got pulumi/pulumi-aws#4013 to pass AWS test suite on P/R/C. |
|
Two new issues downstream
|
|
Ran downstream tests on #2065 with PRC enabled by default. skipped really means failed
AWS I believe the RDS failures here are explained in pulumi/pulumi-aws#4013 GCP After rerunning, a lot of the issues look like the known differences around unknowns discovered in #2054 Opened pulumi/pulumi-gcp#2079 for the GCP tests Auth0 yeah, the test was overfitting, fixed in pulumi/pulumi-auth0#572 Hcloud Azure Some unexpected updates, will investigate. Github Unexpected contents of a detailed diff in a regression test - will investigate. This was the test overfitting on GRPC output fixed in pulumi/pulumi-github#693 |
|
Confirmed, for AWS the test is not actually testing what it says in this case, there's on actual problem. |
|
pulumi/pulumi#16146 changed the results here - a few of the dirty refreshes present before no longer show up in tests. |
|
a ci-mgmt issue means that we don't currently run tests against latest pulumi CLI: pulumi/ci-mgmt#973 Will rerun downstream tests after that merges to see if the new refresh behaviour changes anything. |
|
The AWS failures are because AWS has not picked up ci-mgmt so has not picked up the new refresh from the CLI... retesting in pulumi/pulumi-aws#4081. will update manually EDIT: AWS looks good, barring pulumi/pulumi-aws#4080, which is present on master. |
|
Azure is fine too, just GCP left - failures in the tags tests, investigating. |
|
The GCP example is quite odd: GRPC logs{
"method": "/pulumirpc.ResourceProvider/Read",
"request": {
"id": "demo-bucket-ec59f09",
"urn": "urn:pulumi:p-it-venelins-m-labels-com-9668704a::labels-combinations-go::gcp:storage/bucket:Bucket::demo-bucket",
"properties": {
"__meta": "{\"e2bfb730-ecaa-11e6-8f88-34363bc7c4c0\":{\"create\":600000000000,\"read\":240000000000,\"update\":240000000000},\"schema_version\":\"1\"}",
"autoclass": null,
"cors": [],
"customPlacementConfig": null,
"defaultEventBasedHold": false,
"effectiveLabels": {
"y": "s"
},
"enableObjectRetention": false,
"encryption": null,
"forceDestroy": false,
"id": "demo-bucket-ec59f09",
"labels": {
"x": ""
},
"lifecycleRules": [],
"location": "US",
"logging": null,
"name": "demo-bucket-ec59f09",
"project": "pulumi-development",
"projectNumber": 921927215178,
"publicAccessPrevention": "inherited",
"pulumiLabels": {
"y": "s"
},
"requesterPays": false,
"retentionPolicy": null,
"rpo": "DEFAULT",
"selfLink": "https://www.googleapis.com/storage/v1/b/demo-bucket-ec59f09",
"softDeletePolicy": {
"effectiveTime": "2024-06-17T15:17:37.175Z",
"retentionDurationSeconds": 604800
},
"storageClass": "STANDARD",
"uniformBucketLevelAccess": false,
"url": "gs://demo-bucket-ec59f09",
"versioning": null,
"website": null
},
"inputs": {
"__defaults": [
"forceDestroy",
"name",
"storageClass"
],
"forceDestroy": false,
"labels": {
"x": ""
},
"location": "US",
"name": "demo-bucket-ec59f09",
"storageClass": "STANDARD"
}
},
"response": {
"id": "demo-bucket-ec59f09",
"properties": {
"__meta": "{\"e2bfb730-ecaa-11e6-8f88-34363bc7c4c0\":{\"create\":600000000000,\"read\":240000000000,\"update\":240000000000},\"schema_version\":\"1\"}",
"autoclass": null,
"cors": [],
"customPlacementConfig": null,
"defaultEventBasedHold": false,
"effectiveLabels": {
"y": "s"
},
"enableObjectRetention": false,
"encryption": null,
"forceDestroy": false,
"id": "demo-bucket-ec59f09",
"labels": {
"x": ""
},
"lifecycleRules": [],
"location": "US",
"logging": null,
"name": "demo-bucket-ec59f09",
"project": "pulumi-development",
"projectNumber": 921927215178,
"publicAccessPrevention": "inherited",
"pulumiLabels": {
"y": "s"
},
"requesterPays": false,
"retentionPolicy": null,
"rpo": "DEFAULT",
"selfLink": "https://www.googleapis.com/storage/v1/b/demo-bucket-ec59f09",
"softDeletePolicy": {
"effectiveTime": "2024-06-17T15:17:37.175Z",
"retentionDurationSeconds": 604800
},
"storageClass": "STANDARD",
"uniformBucketLevelAccess": false,
"url": "gs://demo-bucket-ec59f09",
"versioning": null,
"website": null
},
"inputs": {
"__defaults": [
"forceDestroy",
"name",
"storageClass"
],
"forceDestroy": false,
"labels": {
"x": ""
},
"location": "US",
"name": "demo-bucket-ec59f09",
"storageClass": "STANDARD"
}
},
"metadata": {
"kind": "resource",
"mode": "client",
"name": "gcp"
}
}
{
"method": "/pulumirpc.ResourceProvider/Diff",
"request": {
"id": "demo-bucket-ec59f09",
"urn": "urn:pulumi:p-it-venelins-m-labels-com-9668704a::labels-combinations-go::gcp:storage/bucket:Bucket::demo-bucket",
"olds": {
"__meta": "{\"e2bfb730-ecaa-11e6-8f88-34363bc7c4c0\":{\"create\":600000000000,\"read\":240000000000,\"update\":240000000000},\"schema_version\":\"1\"}",
"autoclass": null,
"cors": [],
"customPlacementConfig": null,
"defaultEventBasedHold": false,
"effectiveLabels": {
"y": "s"
},
"enableObjectRetention": false,
"encryption": null,
"forceDestroy": false,
"id": "demo-bucket-ec59f09",
"labels": {
"x": ""
},
"lifecycleRules": [],
"location": "US",
"logging": null,
"name": "demo-bucket-ec59f09",
"project": "pulumi-development",
"projectNumber": 921927215178,
"publicAccessPrevention": "inherited",
"pulumiLabels": {
"y": "s"
},
"requesterPays": false,
"retentionPolicy": null,
"rpo": "DEFAULT",
"selfLink": "https://www.googleapis.com/storage/v1/b/demo-bucket-ec59f09",
"softDeletePolicy": {
"effectiveTime": "2024-06-17T15:17:37.175Z",
"retentionDurationSeconds": 604800
},
"storageClass": "STANDARD",
"uniformBucketLevelAccess": false,
"url": "gs://demo-bucket-ec59f09",
"versioning": null,
"website": null
},
"news": {
"__defaults": [
"forceDestroy",
"name",
"storageClass"
],
"forceDestroy": false,
"labels": {
"x": ""
},
"location": "US",
"name": "demo-bucket-ec59f09",
"storageClass": "STANDARD"
},
"oldInputs": {
"__defaults": [
"forceDestroy",
"name",
"storageClass"
],
"forceDestroy": false,
"labels": {
"x": ""
},
"location": "US",
"name": "demo-bucket-ec59f09",
"storageClass": "STANDARD"
}
},
"response": {
"stables": [
"enableObjectRetention",
"location",
"name",
"project"
],
"changes": "DIFF_SOME",
"diffs": [
"effectiveLabels",
"pulumiLabels"
],
"detailedDiff": {
"effectiveLabels": {
"kind": "UPDATE"
},
"pulumiLabels": {
"kind": "UPDATE"
}
},
"hasDetailedDiff": true
},
"metadata": {
"kind": "resource",
"mode": "client",
"name": "gcp"
}
}The labels in the Read response is exactly the same as what was in the inputs but we return a diff. Neither of the properties which has a diff is in olds or news, so not sure why it is being flagged up. Might be a bridge workaround, investigating... |
|
pulumi/pulumi-gcp#2079 is now done so PRC passes all downstream tests. |
What happened?
@t0yv0 recently suggested running PlanResourceChange through downstream tests as a short-term way to gain more confidence in it.
This should help with #1785
We already know about some issues (GCP bucket) but it might be useful to learn about other problematic resources
Example
.
Output of
pulumi about.
Additional context
No response
Contributing
Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).
The text was updated successfully, but these errors were encountered: