Error when running destroy on application custom tags

[mattBaumBeneva]

Describe the bug
We apply custom tags to our applications, in order to correctly classify them by management zone.
When running tests, which create and then destroy resources, we obtain an error on the custom tag:
"Error: Unable to find tag TAG_NAME:TAG_VALUE"
Where TAG_NAME and TAG_VALUE are strings specific to our Dynatrace environment.

To Reproduce
Steps to reproduce the behavior:

1. Apply a module du create the following resources:

• dynatrace_web_application
• dynatrace_application_detection_rule
• dynatrace_custom_tags

2. Destroy the created elements
3. The error occurs upon Terraform attempting to delete the tag

Expected behavior
All configurations are deleted without error.

Additional context
Dynatrace provider version: 1.28.0
Terraform version: 1.0.8

Sample custom tag code:

resource "dynatrace_custom_tags" "application_custom_tags" {
  count           = var.proprietaire == "" ? 0 : 1
  entity_selector = "entityId(\"${dynatrace_web_application.application.id}\")"
  tags {
    filter {
      context = "CONTEXTLESS"
      key     = "globale-proprietaire"
      value   = var.proprietaire
    }
  }
}

Sample Jenkins log:

10:43:24  Plan: 0 to add, 0 to change, 3 to destroy.
10:43:24  module.Application_test-de-base.dynatrace_custom_tags.application_custom_tags[0]: Destroying... [id=ea88daff-529e-47aa-891f-9e57533f366c]
10:43:24  module.Application_test-de-base.dynatrace_application_detection_rule.application[0]: Destroying... [id=5f4a028c-b72d-3249-88a0-4ba4b48998e3]
10:43:24  module.Application_test-de-base.dynatrace_application_detection_rule.application[0]: Destruction complete after 0s
10:43:24  ╷
10:43:24  │ Error: Unable to find tag globale-proprietaire:equipe-monitoring

[mattBaumBeneva]

We notice this error does not occur 100% of the time. It may depend on a timing/sequencing issue.

[Dynatrace-Reinhard-Pilz]

Hello @mattBaumBeneva,

The resource dynatrace_custom_tags was using the endpoint /api/v2/tags without specifying the from and to parameters until now - which essentially resulted into queries that implicitly targeted entities from the last 24 hours.
At least for that use case it was immediately possible to reproduce the issue you've reported.

I cannot rule out completely, that also timing/sequencing plays a role here - I just wasn't able to create a situation where it mattered during local tests.

We're planning for an update release for Wednesday. I'd appreciate if I could get feedback from about you whether it takes care completely of the issues in your environment aswell.

best regards,
Reinhard

[mattBaumBeneva]

Thank you. I'll test the new version on Wednesday and report back.

[mattBaumBeneva]

Hi, @Dynatrace-Reinhard-Pilz .

I still reproduced this error using v1.30.2, when running a Jenkins pipeline which creates Applications, applies Custom tags to them, and then destroys the resources.

09:32:59  │ Error: Unable to find tag globale-horaire_surveillance:jour_semaine
09:32:59  │ 
09:32:59  │ 
09:32:59  ╵
09:32:59  ╷
09:32:59  │ Error: Unable to find tag beneva-domaine_affaires:INFRA
09:32:59  │ 
09:32:59  │ 
09:32:59  ╵
09:32:59  ╷
09:32:59  │ Error: Unable to find tag globale-proprietaire:equipe-monitoring
09:32:59  │ 

[Dynatrace-Reinhard-Pilz]

That's very unfortunate. Looks like there are other factors than the time frame involved here. I will investigate further.

[mattBaumBeneva]

Thank you.

[Dynatrace-Reinhard-Pilz]

Hello @mattBaumBeneva, we're planning to release another version of the provider still this week. Included is a solution that tackles this issue at another level than just assuming that it's related to the time frame where entities are supposed to match.

I would have to see your whole Terraform module in order to confirm this, but one way to end up with error messages like that is having multiple dynatrace_custom_tag resources that have different entity_selectors but eventually apply to the same application nevertheless. We've been aware of that danger when we implemented the resource. The REST endpoint we're utilizing here doesn't work like others. It doesn't provide a unique ID after the settings have been created. The only semi-unique part of that resource is the entity selector itself. That fact leaves plenty of room for contradictions within Terraform Modules (and other REST clients or users of the WebUI).

The solution within the upcoming release cannot address these potential contradictions. But it won't throw any Unable to find custom tag messages back into your face anymore. It will accept that something or someone has gotten rid of that tag already.

Like earlier, in case you still experience the same issue after the next release, just add a comment and I'll re-open the ticket.

[mattBaumBeneva]

Hi, @Dynatrace-Reinhard-Pilz . Is a release still planned for this week for us to test?

[Dynatrace-Reinhard-Pilz]

Sorry Matt, we will very likely have to push the release to Tuesday. May I ask for patience until then?

[mattBaumBeneva]

Sure. Of course.

[mattBaumBeneva]

Hi, @Dynatrace-Reinhard-Pilz . Any news on a date for this release?
Thanks again.

[Dynatrace-Reinhard-Pilz]

Sorry for the delay, Matt. We're going to release still today. The changes relating to this issue are already checked in. Thank you for your patience.