Updates azurerm_storage_blob on content_md5 changes #7786

Humoiz · 2020-07-16T18:31:06Z

Implementing feature mentioned in the feature request: #1990

Problem statement: Currently in order to update the blob, user has to change the blob name to trigger the update.

This PR adds new field content_md5 to have azurerm_storage_blob update on content_md5 changes. It forces a new deployment if the value for content_md5 is changed. This is only supported for blob type Block.

(fixes #1990)

azurerm/internal/services/storage/resource_arm_storage_blob.go

Stanpol · 2020-07-21T08:31:39Z

@Humoiz could you please request a review from @katbyte?
It would be great to have this PR merged in the coming milestones.

Humoiz · 2020-07-21T15:25:56Z

@katbyte - Can you please review this PR?

jackofallops · 2020-09-10T06:45:52Z

Hi @Humoiz - Apologies, I somehow missed the notification on this one. Could you add a passing test to cover this new functionality and show it working? (there's a few examples elsewhere in the provider for including a small file as testdata)

Thanks

3mard · 2020-09-14T14:35:19Z

Hi @Humoiz
Thanks for your great work, if you are busy I can take care of the testing

Humoiz · 2020-09-14T16:22:03Z

Hi @Humoiz
Thanks for your great work, if you are busy I can take care of the testing

Thanks! I will address the testing this week.

Humoiz · 2020-09-19T07:00:07Z

Hi @Humoiz - Apologies, I somehow missed the notification on this one. Could you add a passing test to cover this new functionality and show it working? (there's a few examples elsewhere in the provider for including a small file as testdata)

Thanks

Added the tests.

katbyte

Thanks for the pr @Humoiz - overall this looks good but these tests are failing due to public accessnot being allowed on the storage account:


TestAccAzureRMStorageBlob_blockFromInlineContent
--
TestAccAzureRMStorageBlob_blockFromPublicBlob
TestAccAzureRMStorageBlob_requiresImport
TestAccAzureRMStorageBlob_updateBlockFromInlineContent

should be a simple config change to fix this 🙂

Humoiz · 2020-09-21T00:15:02Z

TestAccAzureRMStorageBlob_updateBlockFromInlineContent

@katbyte - Thank you for pointing this out. I have updated TestAccAzureRMStorageBlob_updateBlockFromInlineContent test to have the container blob level access. I have not updated the other tests as they are existing tests and not in scope for this PR.

katbyte

@Humoiz - i think its the flag that needs to be changed:

=== CONT  TestAccAzureRMStorageBlob_blockEmptyAzureADAuth
    TestAccAzureRMStorageBlob_updateBlockFromInlineContent: testing.go:684: Step 0 error: errors during apply:
        
        Error: failed creating container: containers.Client#Create: Failure sending request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="PublicAccessNotPermitted" Message="Public access is not permitted on this storage account.\nRequestId:2ee4abf8-601e-0005-1f61-903875000000\nTime:2020-09-21T21:53:23.8541083Z"
        
          on /opt/teamcity-agent/temp/buildTmp/tf-test589256406/main.tf line 16:
          (source code not available)
        
        
    TestAccAzureRMStorageBlob_blockFromInlineContent: testing.go:684: Step 0 error: errors during apply:
        
        Error: failed creating container: containers.Client#Create: Failure sending request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="PublicAccessNotPermitted" Message="Public access is not permitted on this storage account.\nRequestId:3ad9340a-001e-0075-2f61-90d775000000\nTime:2020-09-21T21:53:23.9088950Z"
        
          on /opt/teamcity-agent/temp/buildTmp/tf-test588258110/main.tf line 16:
          (source code not available)
        
        
    TestAccAzureRMStorageBlob_blockFromPublicBlob: testing.go:684: Step 0 error: errors during apply:
        
        Error: failed creating container: containers.Client#Create: Failure sending request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="PublicAccessNotPermitted" Message="Public access is not permitted on this storage account.\nRequestId:e3cfbba0-101e-0054-0661-90cddc000000\nTime:2020-09-21T21:53:23.8093032Z"
        
          on /opt/teamcity-agent/temp/buildTmp/tf-test884844187/main.tf line 16:
          (source code not available)
        
        
    TestAccAzureRMStorageBlob_blockFromLocalFileWithContentMd5: testing.go:684: Step 0 error: errors during apply:
        
        Error: failed creating container: containers.Client#Create: Failure sending request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="PublicAccessNotPermitted" Message="Public access is not permitted on this storage account.\nRequestId:70ee7031-e01e-00c9-6f61-90f196000000\nTime:2020-09-21T21:53:48.5732074Z"
        
          on /opt/teamcity-agent/temp/buildTmp/tf-test894169348/main.tf line 16:
          (source code not available)
        
        
    TestAccAzureRMStorageBlob_requiresImport: testing.go:684: Step 0 error: errors during apply:
        
        Error: failed creating container: containers.Client#Create: Failure sending request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="PublicAccessNotPermitted" Message="Public access is not permitted on this storage account.\nRequestId:68d53229-101e-0038-3561-90c398000000\nTime:2020-09-21T21:53:48.4836020Z"
        
          on /opt/teamcity-agent/temp/buildTmp/tf-test072062594/main.tf line 16:
          (source code not available)

Humoiz · 2020-09-22T06:00:41Z

@Humoiz - i think its the flag that needs to be changed:

=== CONT  TestAccAzureRMStorageBlob_blockEmptyAzureADAuth
    TestAccAzureRMStorageBlob_updateBlockFromInlineContent: testing.go:684: Step 0 error: errors during apply:
        
        Error: failed creating container: containers.Client#Create: Failure sending request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="PublicAccessNotPermitted" Message="Public access is not permitted on this storage account.\nRequestId:2ee4abf8-601e-0005-1f61-903875000000\nTime:2020-09-21T21:53:23.8541083Z"
        
          on /opt/teamcity-agent/temp/buildTmp/tf-test589256406/main.tf line 16:
          (source code not available)
        
        
    TestAccAzureRMStorageBlob_blockFromInlineContent: testing.go:684: Step 0 error: errors during apply:
        
        Error: failed creating container: containers.Client#Create: Failure sending request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="PublicAccessNotPermitted" Message="Public access is not permitted on this storage account.\nRequestId:3ad9340a-001e-0075-2f61-90d775000000\nTime:2020-09-21T21:53:23.9088950Z"
        
          on /opt/teamcity-agent/temp/buildTmp/tf-test588258110/main.tf line 16:
          (source code not available)
        
        
    TestAccAzureRMStorageBlob_blockFromPublicBlob: testing.go:684: Step 0 error: errors during apply:
        
        Error: failed creating container: containers.Client#Create: Failure sending request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="PublicAccessNotPermitted" Message="Public access is not permitted on this storage account.\nRequestId:e3cfbba0-101e-0054-0661-90cddc000000\nTime:2020-09-21T21:53:23.8093032Z"
        
          on /opt/teamcity-agent/temp/buildTmp/tf-test884844187/main.tf line 16:
          (source code not available)
        
        
    TestAccAzureRMStorageBlob_blockFromLocalFileWithContentMd5: testing.go:684: Step 0 error: errors during apply:
        
        Error: failed creating container: containers.Client#Create: Failure sending request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="PublicAccessNotPermitted" Message="Public access is not permitted on this storage account.\nRequestId:70ee7031-e01e-00c9-6f61-90f196000000\nTime:2020-09-21T21:53:48.5732074Z"
        
          on /opt/teamcity-agent/temp/buildTmp/tf-test894169348/main.tf line 16:
          (source code not available)
        
        
    TestAccAzureRMStorageBlob_requiresImport: testing.go:684: Step 0 error: errors during apply:
        
        Error: failed creating container: containers.Client#Create: Failure sending request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="PublicAccessNotPermitted" Message="Public access is not permitted on this storage account.\nRequestId:68d53229-101e-0038-3561-90c398000000\nTime:2020-09-21T21:53:48.4836020Z"
        
          on /opt/teamcity-agent/temp/buildTmp/tf-test072062594/main.tf line 16:
          (source code not available)

@katbyte - The existing and new storage tests fails on my local environment. Unfortunately the failure does not give me any details on why it failed so I can not tests my new test changes. However, the same configuration works for me when ran outside of the tests. It would be helpful if you can exactly point me in the code on what needs to be changed to have these tests pass.

This is the configuration I am using in the new tests and it works when I run the configuration outside of the tests.

resource "azurerm_resource_group" "test" {
  name     = "acctestRG"
  location = "West US"
}

resource "azurerm_storage_account" "test" {
  name                     = "acctestacc2de4"
  resource_group_name      = azurerm_resource_group.test.name
  location                 = azurerm_resource_group.test.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

resource "azurerm_storage_container" "test" {
  name                  = "test"
  storage_account_name  = azurerm_storage_account.test.name
  container_access_type = "blob"
}

resource "azurerm_storage_blob" "test" {
  name                   = "example.vhd"
  storage_account_name   = azurerm_storage_account.test.name
  storage_container_name = azurerm_storage_container.test.name
  type                   = "Block"
  source_content         = "Wubba Lubba Dub Dub"
  content_md5            = "${md5("Wubba Lubba Dub Dub")}"
}

jackofallops · 2020-11-10T17:28:06Z

Apologies this has taken so long to get back to. I'll be looking at the this later this week.

katbyte

This LGTM @jackofallops 👍

tombuildsstuff

hey @Humoiz

Apologies for the delayed re-review here!

Taking a look through this mostly LGTM - based on the Giovanni SDK conditionally setting the checksum based on whether the value is non-nil - I believe we'll need to conditionally set the ContentMD5 field rather than always setting this to an empty string, incase the behaviour of the API changes - WDYT? That said, if we can fix up the remaining minor comments then this otherwise LGTM 👍

Thanks!

azurerm/internal/services/storage/blobs.go

azurerm/internal/services/storage/resource_arm_storage_blob.go

website/docs/r/storage_blob.html.markdown

azurerm/internal/services/storage/blobs.go

azurerm/internal/services/storage/resource_arm_storage_blob.go

* Conditionally setting the ContentMD5 field in requests, whilst overkill this avoids an expectation on the behaviour of the API ignoring an empty string, since Giovanni conditionally sends this if it's not-nil * Fixes a typo in the docs * Wraps a couple of exceptions to be clearer

tombuildsstuff · 2020-11-19T11:00:08Z

@Humoiz I hope you don't mind, but so that we can get this merged into this release, I've pushed a commit to resolve the comments above

dismissing since changes have been pushed

jackofallops

LGTM 👍

tombuildsstuff · 2020-11-19T11:28:52Z

Tests look good 👍

jpveritone · 2020-11-19T21:28:17Z

I'm using azure provider version 1.44.0. Will this only be available in latest 2.x version?

ghost · 2020-11-20T01:46:34Z

This has been released in version 2.37.0 of the provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading. As an example:

provider "azurerm" {
    version = "~> 2.37.0"
}
# ... other configuration ...

3mard · 2020-11-20T10:08:29Z

@jpveritone I believe so you need to be on version >= 2.37.0

ghost · 2020-12-19T14:50:13Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!

ghost added size/M documentation labels Jul 16, 2020

Humoiz commented Jul 16, 2020

View reviewed changes

azurerm/internal/services/storage/resource_arm_storage_blob.go Show resolved Hide resolved

Humoiz requested a review from jackofallops August 17, 2020 17:38

jackofallops added enhancement service/storage waiting-response labels Sep 10, 2020

ghost removed the waiting-response label Sep 14, 2020

ghost added size/L and removed size/M labels Sep 19, 2020

katbyte requested changes Sep 19, 2020

View reviewed changes

katbyte added the waiting-response label Sep 19, 2020

ghost removed the waiting-response label Sep 21, 2020

Humoiz requested a review from katbyte September 21, 2020 00:16

katbyte requested changes Sep 21, 2020

View reviewed changes

Humoiz requested a review from katbyte October 8, 2020 07:19

This comment has been minimized.

Sign in to view

Hussain added 2 commits November 11, 2020 14:35

Adds functionality to update blob based on content_md5 property

8d4b349

Updated doc and added validation for blob type page and append

c69fd8e

Hussain and others added 6 commits November 11, 2020 14:35

Removed Computed from the content_md5 field

954d3f4

Added test

599db20

fmt

c77195c

fix failing test

2f493d6

Updated the storage container access from private to blob

7b2ded5

rebase and small tweaks from manual testing

6cd7a49

ghost added size/M and removed size/L labels Nov 11, 2020

jackofallops requested a review from tombuildsstuff November 16, 2020 06:40

katbyte approved these changes Nov 17, 2020

View reviewed changes

tombuildsstuff previously requested changes Nov 19, 2020

View reviewed changes

jackofallops approved these changes Nov 19, 2020

View reviewed changes

tombuildsstuff added this to the v2.37.0 milestone Nov 19, 2020

r/storage_blob: force-enabling allow_blob_public_access for now

b928264

tombuildsstuff merged commit f6b8185 into hashicorp:master Nov 19, 2020

tombuildsstuff added a commit that referenced this pull request Nov 19, 2020

updating to include #7786

20c1b41

Humoiz deleted the hussain/storage_blob_etag_changes branch November 19, 2020 13:59

hashicorp locked as resolved and limited conversation to collaborators Dec 19, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Updates azurerm_storage_blob on content_md5 changes #7786

Updates azurerm_storage_blob on content_md5 changes #7786

Humoiz commented Jul 16, 2020 •

edited by katbyte

Stanpol commented Jul 21, 2020

Humoiz commented Jul 21, 2020

jackofallops commented Sep 10, 2020

3mard commented Sep 14, 2020

Humoiz commented Sep 14, 2020

Humoiz commented Sep 19, 2020 •

edited

katbyte left a comment

Humoiz commented Sep 21, 2020 •

edited

katbyte left a comment

Humoiz commented Sep 22, 2020 •

edited

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

jackofallops commented Nov 10, 2020

katbyte left a comment

tombuildsstuff left a comment

tombuildsstuff commented Nov 19, 2020

jackofallops left a comment

tombuildsstuff commented Nov 19, 2020

jpveritone commented Nov 19, 2020

ghost commented Nov 20, 2020

3mard commented Nov 20, 2020

ghost commented Dec 19, 2020

Updates azurerm_storage_blob on content_md5 changes #7786

Updates azurerm_storage_blob on content_md5 changes #7786

Conversation

Humoiz commented Jul 16, 2020 • edited by katbyte

Stanpol commented Jul 21, 2020

Humoiz commented Jul 21, 2020

jackofallops commented Sep 10, 2020

3mard commented Sep 14, 2020

Humoiz commented Sep 14, 2020

Humoiz commented Sep 19, 2020 • edited

katbyte left a comment

Choose a reason for hiding this comment

Humoiz commented Sep 21, 2020 • edited

katbyte left a comment

Choose a reason for hiding this comment

Humoiz commented Sep 22, 2020 • edited

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

jackofallops commented Nov 10, 2020

katbyte left a comment

Choose a reason for hiding this comment

tombuildsstuff left a comment

Choose a reason for hiding this comment

tombuildsstuff commented Nov 19, 2020

jackofallops left a comment

Choose a reason for hiding this comment

tombuildsstuff commented Nov 19, 2020

jpveritone commented Nov 19, 2020

ghost commented Nov 20, 2020

3mard commented Nov 20, 2020

ghost commented Dec 19, 2020

Humoiz commented Jul 16, 2020 •

edited by katbyte

Humoiz commented Sep 19, 2020 •

edited

Humoiz commented Sep 21, 2020 •

edited

Humoiz commented Sep 22, 2020 •

edited