azurerm_storage_data_lake_gen2_path: ACE support shows weird behavior

[favoretti]

TF: 0.14.7
azurerm: 2.46.0

When running an apply on a complete codebase, plan wants changes, which are essentially just re-shuffling ace's ordering:

  # azurerm_storage_data_lake_gen2_path.generic_sto_acl4["fdx-qa"] will be updated in-place
  ~ resource "azurerm_storage_data_lake_gen2_path" "generic_sto_acl4" {
        id                 = "xxx"
        # (6 unchanged attributes hidden)

      - ace {
          - permissions = "---" -> null
          - scope       = "access" -> null
          - type        = "other" -> null
        }
      - ace {
          - permissions = "rwx" -> null
          - scope       = "access" -> null
          - type        = "group" -> null
        }
      - ace {
          - permissions = "rwx" -> null
          - scope       = "access" -> null
          - type        = "mask" -> null
        }
      - ace {
          - permissions = "rwx" -> null
          - scope       = "access" -> null
          - type        = "user" -> null
        }
      - ace {
          - id          = "c4aa5e31-aefd-458d-a347-249e6c09b77f" -> null
          - permissions = "-wx" -> null
          - scope       = "access" -> null
          - type        = "group" -> null
        }
      + ace {
          + id          = "c4aa5e31-aefd-458d-a347-249e6c09b77f"
          + permissions = "-wx"
          + scope       = "access"
          + type        = "user"
        }
      + ace {
          + permissions = "---"
          + scope       = "access"
          + type        = "other"
        }
      + ace {
          + permissions = "rwx"
          + scope       = "access"
          + type        = "group"
        }
      + ace {
          + permissions = "rwx"
          + scope       = "access"
          + type        = "mask"
        }
      + ace {
          + permissions = "rwx"
          + scope       = "access"
          + type        = "user"
        }

        # (1 unchanged block hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Apply returns an error:

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

azurerm_storage_data_lake_gen2_path.generic_sto_acl4["fdx-qa"]: Modifying... [id=xxx]

Error: Error setting access control for Path "data-layer/raw_zone/lookup-master/xxx" in File System "data-layer" in Storage Account "xxx": datalakestore.Client#SetAccessControl: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="AuthorizationPermissionMismatch" Message="This request is not authorized to perform this operation using this permission.\nRequestId:14f20235-901f-0040-0c5b-1008f2000000\nTime:2021-03-03T18:33:55.2048251Z"

  on storage_acls_ean.tf line 166, in resource "azurerm_storage_data_lake_gen2_path" "generic_sto_acl4":
 166: resource "azurerm_storage_data_lake_gen2_path" "generic_sto_acl4" {


Releasing state lock. This may take a few moments...

Subsequent plan results in no changes required:

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

// cc @stuartleeks

[favoretti]

Mostly for my own tracking, since I can't really find a proper reproduction scenario, but just in case someone can think of something.. Might be an artifact of 2.46.0's reordering (since ACEs became a Set there).

[kharkevich]

I got a pretty similar bug - in case if I want to add something to ACL change is failed:

  # module.storage.azurerm_storage_data_lake_gen2_path.landing_directory will be updated in-place
    ~ resource "azurerm_storage_data_lake_gen2_path" "some_directory" {
        id                 = "https://storageaccountname.dfs.core.windows.net/some/directory"
        # (6 unchanged attributes hidden)

      - ace {
          - permissions = "---" -> null
          - scope       = "access" -> null
          - type        = "other" -> null
        }
      - ace {
          - permissions = "---" -> null
          - scope       = "default" -> null
          - type        = "other" -> null
        }
      - ace {
          - permissions = "rwx" -> null
          - scope       = "access" -> null
          - type        = "group" -> null
        }
      - ace {
          - permissions = "rwx" -> null
          - scope       = "access" -> null
          - type        = "mask" -> null
        }
      - ace {
          - permissions = "rwx" -> null
          - scope       = "access" -> null
          - type        = "user" -> null
        }
      - ace {
          - permissions = "rwx" -> null
          - scope       = "default" -> null
          - type        = "group" -> null
        }
      - ace {
          - permissions = "rwx" -> null
          - scope       = "default" -> null
          - type        = "mask" -> null
        }
      - ace {
          - permissions = "rwx" -> null
          - scope       = "default" -> null
          - type        = "user" -> null
        }
      + ace {
          + id          = "11111111-ed3f-4d39-a039-75cc81f261dc"
          + permissions = "r-x"
          + scope       = "access"
          + type        = "group"
        }
      + ace {
          + id          = "11111111-310c-4e7b-81d9-f90666eafa29"
          + permissions = "r-x"
          + scope       = "access"
          + type        = "group"
        }
      + ace {
          + id          = "11111111-758a-4cb3-a83a-d7cd6d0f5ebe"
          + permissions = "r-x"
          + scope       = "access"
          + type        = "group"
        }
      + ace {
          + id          = "11111111-4336-bbbb-9996-4733aaa515b9"
          + permissions = "r-x"
          + scope       = "access"
          + type        = "group"
        }
      + ace {
          + id          = "11111111-4061-bbbb-94e2-22729a133244"
          + permissions = "r-x"
          + scope       = "access"
          + type        = "group"
        }
      + ace {
          + id          = "11111111-c620-bbbb-901b-7979ea77162c"
          + permissions = "r-x"
          + scope       = "access"
          + type        = "group"
        }
      + ace {
          + id          = "11111111-b441-bbbb-b48b-b67a095e92dd"
          + permissions = "r-x"
          + scope       = "access"
          + type        = "group"
        }
      + ace {
          + permissions = "---"
          + scope       = "access"
          + type        = "other"
        }
      + ace {
          + permissions = "---"
          + scope       = "default"
          + type        = "other"
        }
      + ace {
          + permissions = "rwx"
          + scope       = "access"
          + type        = "group"
        }
      + ace {
          + permissions = "rwx"
          + scope       = "access"
          + type        = "mask"
        }
      + ace {
          + permissions = "rwx"
          + scope       = "access"
          + type        = "user"
        }
      + ace {
          + permissions = "rwx"
          + scope       = "default"
          + type        = "group"
        }
      + ace {
          + permissions = "rwx"
          + scope       = "default"
          + type        = "mask"
        }
      + ace {
          + permissions = "rwx"
          + scope       = "default"
          + type        = "user"
        }
        # (18 unchanged blocks hidden)
    }

result:

Error: Error setting access control for Path "directory" in File System "some" in Storage Account "storageaccountname": datalakestore.Client#SetAccessControl: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="AuthorizationPermissionMismatch" Message="This request is not authorized to perform this operation using this permission.\nRequestId:XXXX-XXX-XXX-XXX-XXXXX000000\nTime:2021-04-20T15:54:50.5273952Z"

[stuartleeks]

@kharkevich - which version of the azurerm provider are you using? (and which version was the ACL originally created with?)

As @favoretti mentioned, in 2.46.0 there were changes to make ACEs became a Set rather than a List

[favoretti]

@stuartleeks i logged this against 2.46 :) since apparently sets didn't help.

[kharkevich]

@stuartleeks @favoretti exactly for this case is reproduced for terrafrom 0.15.0 and azurerm 2.56.0

full test case for reproducing:

1. create azurerm_storage_data_lake_gen2_path with some ace
2. add new ace

[kharkevich]

Finally i solved this issue:

Error: Error setting access control for Path "directory" in File System "some" in Storage Account "storageaccountname": datalakestore.Client#SetAccessControl: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="AuthorizationPermissionMismatch" Message="This request is not authorized to perform this operation using this permission.\nRequestId:XXXX-XXX-XXX-XXX-XXXXX000000\nTime:2021-04-20T15:54:50.5273952Z"

My service principal in test environment have owner permissions on subscription, but this still not enough in case of using azurerm_storage_data_lake_gen2_path

In case if I add Storage Blob Data Owner permissions on service principal for this storage account, acl management works like a charm

Probably we should just update documentation

@favoretti @stuartleeks

[github-actions]

This functionality has been released in v3.39.0 of the Terraform Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.