Bug: panos_panorama_address_object, panos_panorama_service_object cannot be deleted because of references from:

[lanzrein]

Describe the bug

When running terraform apply that would delete some objects, the program can not progress because it is blocked by empty references.
Moreover it becomes impossible to delete the object even manually without doing a manual restart of the management server.

Expected behavior

Objects should be removed.

Current behavior

Pipeline crashes because of empty reference errors.

Possible solution

This bug has already been documented on how to resolve in the palo alto knowledge base.
However this issue only happened to me when working with terraform/panos, so it may be due to the provider.
It is not practical to have to restart panorama whenever it occurs.

Also after further debugging, when restarting the management server, the rules are effectively deleted sometimes.

Steps to reproduce

TF Code :

locals {
  # the data files...
  rules     = yamldecode(file("data/rules.yaml"))
  tags      = yamldecode(file("data/tags.yaml"))
  services  = yamldecode(file("data/services.yaml"))
  addresses = yamldecode(file("data/addresses.yaml"))

}

variable "DG1_GLOBAL" {
  default = "DG1_GLOBAL"
}


variable "TestPA" {
  default = "XXXX" # some PA VM. 
}

terraform {
  required_providers {
    panos = {
      source  = "PaloAltoNetworks/panos"
      version = "1.6.3"

    }
  }
}

resource "panos_panorama_administrative_tag" "tags" {
  for_each     = { for t in local.tags : t.Name => t }
  name         = each.key
  device_group = var.DG1_GLOBAL
  comment      = "Automatic tag for testing."
}

resource "panos_panorama_service_object" "services" {
  for_each         = { for s in local.services : s.Name => s }
  name             = each.key
  protocol         = each.value.Protocol
  destination_port = each.value.Destination
  device_group     = var.DG1_GLOBAL
}


resource "panos_panorama_address_object" "addresses" {
  for_each = { for a in local.addresses : a.Name => a }

  name         = each.key
  value        = each.value.Value
  type         = each.value.Type
  device_group = var.DG1_GLOBAL # the device group where to put the rule.
  tags         = each.value.Tags

  depends_on = [panos_panorama_administrative_tag.tags]
}



resource "panos_panorama_security_rule_group" "rules" {
  count = length(local.rules) > 0 ? 1 : 0

  position_keyword   = "before"
  position_reference = "Denyall"# put it *before* the rule in position_reference.
  rulebase           = "post-rulebase"
  device_group       = var.TestPA # chose your firewall here. TestPA is a test firewall...


  dynamic "rule" {
    # iterate over every rule and fill it.
    for_each = local.rules
    content {
      name                  = rule.value.Name
      applications          = rule.value.Application
      categories            = ["any"]
      destination_addresses = rule.value.Destination.Addresses
      destination_zones     = rule.value.Destination.Zones
      hip_profiles          = ["any"]
      services              = rule.value.Service
      source_addresses      = rule.value.Source.Addresses
      source_zones          = rule.value.Source.Zones
      source_users          = ["any"]
      tags                  = rule.value.Tag

      # some values optional but better for cleanliness
      disable_server_response_inspection = false
      disabled                           = false
      icmp_unreachable                   = false
      log_end                            = true
      log_start                          = false
      negate_destination                 = false
      negate_source                      = false
      negate_target                      = false
      group                              = "XXX"
    }
  }
  # depends on means that this block will wait for the tags, services, addresses to be created before creating the rules.
  # this means that we do not run into issues like referencing a tag that has not yet been created.
  depends_on = [panos_panorama_administrative_tag.tags, panos_panorama_service_object.services, panos_panorama_address_object.addresses]

}

Yaml templates :

• tags.yaml:

- Name: my_tag1
- Name: my_tag1

• addresses.yaml:

- Name: 10.0.0.0-32
  Type: ip-netmask
  Value: 10.0.0.0/32
- Name: URL_example.com
  Type: fqdn
  Value: www.example.com

• services.yaml

- Name: service-Tcp_22
  Protocol: tcp
  Destination: 22
- Name: service-Udp_2223
  Protocol: udp
  Destination: 2223

• rules.yaml

- Name: rule_example1
  Tag:
    - my_tag1
  Source:
    Zones :
      - Trust
    Addresses :
      - '10.0.0.0-32'
  Destination:
    Zones :
      - Untrust
    Addresses:
      - any
  Service:
    - service-Tcp_22
  Application:
    - any
- Name: rule_example2
  Tag:
    - my_tag2
  Source:
    Zones :
      - Trust
    Addresses :
      - '10.0.0.0-32'
  Destination:
    Zones :
      - Untrust
    Addresses:
      - URL_example.com
  Service:
    - service-Udp_2223
  Application:
    - any

0. set up the yaml files in a data/ directory
1. terraform plan and then terraform apply
2. remove one rule and the objects related to the rule..
3. terraform plan and then terraform apply
4. The object cannot be deleted.

Screenshots

The screenshots are not of this exact example. But the workflow is the exact same and yields this result.

Context

I am trying to use terraform to automate deployment security group rules. The tags, services and addresses are needed for those rules to be deployed.

This makes it hard to work automatically as I always need to restart the management panorama..

Your Environment

• Version used: terraform version 0.13.2 , panos provider version 1.6.3, panorama version 9.1.3
  EDIT :
  I tried on panorama 9.1.4 and the issue is also present.

[shinmog]

This seems like a duplicate of #234 ..? Because at the end of the day you have one resource, panos_panorama_security_rule_group.rules, that needs to be updated by Terraform, and a few other resources that need to be deleted, but Terraform needs to remove the reference from the security rule group first and then remove the rule sub objects second.

[lanzrein]

Hello,
Yes I thought it was a similar to that issue indeed.
However, I tried to re-run the program again this morning to make sure and it actually is not the same.
When I try to replicate the issue with a small number of rules it does not arise. It seems that it happens when you delete/replace a large amount of data.

From the output that terraform does indeed first delete the rules and then destroys the addresses/services/tags. As expected but there is still something that blocks deletion for addresses/services/tags. Even if the rules have been properly deleted.

Sample of the output log ( shortened for brievety ) :

lanzrein@win:/home/lanzrein/panorama$ terraform apply -auto-approve plan 
panos_panorama_administrative_tag.tags["cgsa-migration-all-aws-sgp"]: Creating...
panos_panorama_security_rule_group.rules[0]: Destroying... [id=PA-VM-NOC:post-rulebase:1:Denyall:cnVsZTEKcnVsZTIKcnVsZTM=]
panos_panorama_security_rule_group.rules[0]: Destruction complete after 1s # <-- the rules have beed successfully destroyed ! 
panos_panorama_address_object.addresses["10.177.130.6-32"]: Destroying... [id=DG1_GLOBAL:10.177.130.6-32]
panos_panorama_address_object.addresses["10.192.25.102-32"]: Destroying... [id=DG1_GLOBAL:10.192.25.102-32]
[ ... and lot of other things trying to be destroyed ]
panos_panorama_address_object.addresses["10.177.130.8-32"]: Destroying... [id=DG1_GLOBAL:10.177.130.8-32]


Error:   10.192.17.36-32 cannot be deleted because of references from:



Error:   10.192.17.91-32 cannot be deleted because of references from:



Error:   service-Tcp_8080 cannot be deleted because of references from:

[... and a lot of other errors with same output]


Error:   service-Udp_2049 cannot be deleted because of references from:

And now it's impossible to delete allof those resources, even manually, without rebooting the management plan of panorama.

On Panorama 9.1.4, it seems that after restarting the management server the resources are deleted.

[shinmog]

Similar to #239 and #234, please see #234 for further information.

Closing this out as this is no longer a provider issue.

[lanzrein]

I have tried to use the flag create_before_destroy when creating new resources as you say in #234. The error still appears.

I have also tested by having different runs for each type of object : first delete the rules, then the addresses. The error still appears.
I think this may be an issue with Panorama as the objects are just blocked by phantom references which means that they could never be deleted without restarting the software.

For some reason, Panorama has a lock on those object. It looks like it believes the rules still exist somewhere and they have not been totally erased ?

[olbees]

I agree, this is not resolved by using the create_before_destroy flag. I have redeployed with this flag and have the same issue. The address group cannot be deleted without restarting the management services on panorama. I am running the latest version of Terraform.

Can this issue be reopened? It is not the same as #234