[Bug]: Secret never Sync #1128

daniel-maganto · 2024-02-06T10:17:58Z

Is there an existing issue for this?

I have searched the existing issues

Affected Resource(s)

secretsmanager.aws.upbound.io/v1beta1 - Secret

Resource MRs required to reproduce the bug

apiVersion: secretsmanager.aws.upbound.io/v1beta1
kind: Secret
metadata:
  name: test-dmaganto1
spec:
  forProvider:
    name: test-dmaganto1
    region: eu-central-1
    recoveryWindowInDays: 0
  providerConfigRef:
    name: aws-admin-my-account-upbound

Steps to Reproduce

Apply this resource in the cluster

What happened?

It never reach the state Sync: true

Relevant Error Output Snippet

- lastTransitionTime: "2024-02-06T10:12:27Z"
    message: 'observe failed: cannot compute the instance diff: failed to compute
      the customized terraform.InstanceDiff: could not read replica block from config'

Crossplane Version

1.14.5

Provider Version

1.0.0

Kubernetes Version

v1.28.2

Kubernetes Distribution

EKS

Additional Info

No response

The text was updated successfully, but these errors were encountered:

turkenf · 2024-02-06T11:53:24Z

@daniel-maganto, thank you for raising this issue but I could not reproduce the issue with the provided information. Please check your example MR and let us know if there are other fields/parameters on the MR.

apiVersion: secretsmanager.aws.upbound.io/v1beta1
kind: Secret
metadata:
  annotations:
    crossplane.io/external-create-pending: "2024-02-06T11:44:56Z"
    crossplane.io/external-create-succeeded: "2024-02-06T11:44:56Z"
    crossplane.io/external-name: arn:aws:secretsmanager:us-west-1::secret:example-test-112-OnXjOd
    meta.upbound.io/example-id: secretsmanager/v1beta1/secret
    upjet.upbound.io/test: "true"
  creationTimestamp: "2024-02-06T11:44:53Z"
  finalizers:
  - finalizer.managedresource.crossplane.io
  generation: 2
  labels:
    testing.upbound.io/example-name: secretsmanager
  name: example
  resourceVersion: "1419"
  uid: <...>
spec:
  deletionPolicy: Delete
  forProvider:
    name: example-test-112
    recoveryWindowInDays: 0
    region: us-west-1
    tags:
      crossplane-kind: secret.secretsmanager.aws.upbound.io
      crossplane-name: example
      crossplane-providerconfig: default
  initProvider: {}
  managementPolicies:
  - '*'
  providerConfigRef:
    name: default
status:
  atProvider:
    arn: arn:aws:secretsmanager:us-west-1:<..>:secret:example-test-112-OnXjOd
    description: ""
    forceOverwriteReplicaSecret: false
    id: arn:aws:secretsmanager:us-west-1:<..>:secret:example-test-112-OnXjOd
    kmsKeyId: ""
    name: example-test-112
    policy: ""
    recoveryWindowInDays: 0
    tags:
      crossplane-kind: secret.secretsmanager.aws.upbound.io
      crossplane-name: example
      crossplane-providerconfig: default
    tagsAll:
      crossplane-kind: secret.secretsmanager.aws.upbound.io
      crossplane-name: example
      crossplane-providerconfig: default
  conditions:
  - lastTransitionTime: "2024-02-06T11:45:02Z"
    reason: Available
    status: "True"
    type: Ready
  - lastTransitionTime: "2024-02-06T11:44:56Z"
    reason: ReconcileSuccess
    status: "True"
    type: Synced
  - lastTransitionTime: "2024-02-06T11:44:57Z"
    reason: Success
    status: "True"
    type: LastAsyncOperation
  - lastTransitionTime: "2024-02-06T11:45:23Z"
    reason: UpToDate
    status: "True"
    type: Test

dmaganto · 2024-02-06T19:10:03Z

The provider's pods were stuck, after rebooting both, it started working successfully. It is funny because they were installed today so there is no overload at all

daniel-maganto · 2024-02-07T09:15:36Z

It happens again, here the log traces.

Cannot observe external resource	{"controller": "managed/secretsmanager.aws.upbound.io/v1beta1, kind=secret", "request": {"name":"service-test"}, "uid": "3e8c81da-0c2c-4a4b-b0b3-682ca0df0a4a", "version": "183689711", "external-name": "arn:aws:secretsmanager:eu-central-1:XXXXXXXXX:secret:service-test-dK6yoH", "error": "cannot compute the instance diff: failed to compute the customized terraform.InstanceDiff: could not read replica block from config", "errorVerbose": "could not read replica block from config
github.com/upbound/provider-aws/config/secretsmanager.Configure.func1.2
    github.com/upbound/provider-aws/config/secretsmanager/config.go:44
github.com/crossplane/upjet/pkg/controller.(*terraformPluginSDKExternal).getResourceDataDiff
    github.com/crossplane/upjet@v1.1.0/pkg/controller/external_tfpluginsdk.go:427
github.com/crossplane/upjet/pkg/controller.(*terraformPluginSDKExternal).Observe
    github.com/crossplane/upjet@v1.1.0/pkg/controller/external_tfpluginsdk.go:498
github.com/crossplane/upjet/pkg/controller.(*terraformPluginSDKAsyncExternal).Observe
    github.com/crossplane/upjet@v1.1.0/pkg/controller/external_async_tfpluginsdk.go:126
github.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile
    github.com/crossplane/crossplane-runtime@v1.15.0-rc.0.0.20231215091746-d23a82b3a2f5/pkg/reconciler/managed/reconciler.go:903
github.com/crossplane/crossplane-runtime/pkg/ratelimiter.(*Reconciler).Reconcile
    github.com/crossplane/crossplane-runtime@v1.15.0-rc.0.0.20231215091746-d23a82b3a2f5/pkg/ratelimiter/reconciler.go:54
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227
runtime.goexit
    runtime/asm_amd64.s:1650
failed to compute the customized terraform.InstanceDiff
github.com/crossplane/upjet/pkg/controller.(*terraformPluginSDKExternal).getResourceDataDiff
    github.com/crossplane/upjet@v1.1.0/pkg/controller/external_tfpluginsdk.go:429
github.com/crossplane/upjet/pkg/controller.(*terraformPluginSDKExternal).Observe
    github.com/crossplane/upjet@v1.1.0/pkg/controller/external_tfpluginsdk.go:498
github.com/crossplane/upjet/pkg/controller.(*terraformPluginSDKAsyncExternal).Observe
    github.com/crossplane/upjet@v1.1.0/pkg/controller/external_async_tfpluginsdk.go:126
github.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile
    github.com/crossplane/crossplane-runtime@v1.15.0-rc.0.0.20231215091746-d23a82b3a2f5/pkg/reconciler/managed/reconciler.go:903
github.com/crossplane/crossplane-runtime/pkg/ratelimiter.(*Reconciler).Reconcile
    github.com/crossplane/crossplane-runtime@v1.15.0-rc.0.0.20231215091746-d23a82b3a2f5/pkg/ratelimiter/reconciler.go:54
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227
runtime.goexit
    runtime/asm_amd64.s:1650
cannot compute the instance diff
github.com/crossplane/upjet/pkg/controller.(*terraformPluginSDKExternal).Observe
    github.com/crossplane/upjet@v1.1.0/pkg/controller/external_tfpluginsdk.go:500
github.com/crossplane/upjet/pkg/controller.(*terraformPluginSDKAsyncExternal).Observe
    github.com/crossplane/upjet@v1.1.0/pkg/controller/external_async_tfpluginsdk.go:126
github.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile
    github.com/crossplane/crossplane-runtime@v1.15.0-rc.0.0.20231215091746-d23a82b3a2f5/pkg/reconciler/managed/reconciler.go:903
github.com/crossplane/crossplane-runtime/pkg/ratelimiter.(*Reconciler).Reconcile
    github.com/crossplane/crossplane-runtime@v1.15.0-rc.0.0.20231215091746-d23a82b3a2f5/pkg/ratelimiter/reconciler.go:54
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227
runtime.goexit
    runtime/asm_amd64.s:1650"}

turkenf · 2024-02-07T15:09:10Z

I tried a few times and I can successfully create and delete the resource, but I can confirm that diff is detected in the logs:

2024-02-07T18:00:30+03:00	DEBUG	provider-aws	Diff detected	{"uid": "de62d41e-a87f-4373-9c4d-95c11c383a57", "name": "example", "gvk": "secretsmanager.aws.upbound.io/v1beta1, Kind=Secret", "instanceDiff": "*terraform.InstanceDiff{mu:sync.Mutex{state:0, sema:0x0}, Attributes:map[string]*terraform.ResourceAttrDiff{\"arn\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"force_overwrite_replica_secret\":*terraform.ResourceAttrDiff{Old:\"\", New:\"false\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"name\":*terraform.ResourceAttrDiff{Old:\"\", New:\"example-test1\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}, \"name_prefix\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}, \"recovery_window_in_days\":*terraform.ResourceAttrDiff{Old:\"\", New:\"0\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"replica.#\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"tags.%\":*terraform.ResourceAttrDiff{Old:\"0\", New:\"3\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"tags.crossplane-kind\":*terraform.ResourceAttrDiff{Old:\"\", New:\"secret.secretsmanager.aws.upbound.io\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"tags.crossplane-name\":*terraform.ResourceAttrDiff{Old:\"\", New:\"example\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"tags.crossplane-providerconfig\":*terraform.ResourceAttrDiff{Old:\"\", New:\"default\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"tags_all.%\":*terraform.ResourceAttrDiff{Old:\"0\", New:\"3\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"tags_all.crossplane-kind\":*terraform.ResourceAttrDiff{Old:\"\", New:\"secret.secretsmanager.aws.upbound.io\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"tags_all.crossplane-name\":*terraform.ResourceAttrDiff{Old:\"\", New:\"example\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"tags_all.crossplane-providerconfig\":*terraform.ResourceAttrDiff{Old:\"\", New:\"default\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}}, Destroy:false, DestroyDeposed:false, DestroyTainted:false, RawConfig:cty.NilVal, RawState:cty.NilVal, RawPlan:cty.NilVal, Meta:map[string]interface {}(nil)}"}

@erhancagirici, do you have any idea about this?

erhancagirici · 2024-02-08T08:02:33Z

Hi, This looks like an issue with the custom diff function for handling replica field changes, affecting the configs that do not have replica set. I will send a fix 👍 Thanks for reporting @daniel-maganto

mbbush · 2024-02-15T02:40:58Z

Specific impact of this bug, as far as I can tell:

New secrets can be created
Secrets can be deleted.
Secrets which already exist in AWS can be observed/imported, as long as there is no diff whatsoever between the spec and the observed values.
Updating existing secrets trigger the bug, which causes the update to fail. This includes things like setting or changing the crossplane-managed tags

daniel-maganto added bug Something isn't working needs:triage labels Feb 6, 2024

github-actions bot added the community label Feb 6, 2024

turkenf added the needs:information label Feb 6, 2024

turkenf added is:triaged Indicates that an issue has been reviewed. and removed needs:information needs:triage labels Feb 7, 2024

erhancagirici self-assigned this Feb 8, 2024

erhancagirici mentioned this issue Feb 9, 2024

fix Secret/secretmanager.aws custom diff logic when replica config is empty #1144

Merged

1 task

mbbush mentioned this issue Feb 15, 2024

Add examples with update parameter for secretsmanager secret #1154

Closed

1 task

erhancagirici closed this as completed in #1144 Feb 15, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: Secret never Sync #1128

[Bug]: Secret never Sync #1128

daniel-maganto commented Feb 6, 2024

turkenf commented Feb 6, 2024

dmaganto commented Feb 6, 2024

daniel-maganto commented Feb 7, 2024 •

edited

turkenf commented Feb 7, 2024

erhancagirici commented Feb 8, 2024 •

edited

mbbush commented Feb 15, 2024

[Bug]: Secret never Sync #1128

[Bug]: Secret never Sync #1128

Comments

daniel-maganto commented Feb 6, 2024

Is there an existing issue for this?

Affected Resource(s)

Resource MRs required to reproduce the bug

Steps to Reproduce

What happened?

Relevant Error Output Snippet

Crossplane Version

Provider Version

Kubernetes Version

Kubernetes Distribution

Additional Info

turkenf commented Feb 6, 2024

dmaganto commented Feb 6, 2024

daniel-maganto commented Feb 7, 2024 • edited

turkenf commented Feb 7, 2024

erhancagirici commented Feb 8, 2024 • edited

mbbush commented Feb 15, 2024

daniel-maganto commented Feb 7, 2024 •

edited

erhancagirici commented Feb 8, 2024 •

edited