[filebeat][ABS] - Fix Input failing with Storage Blob Data Reader RBAC role

[ShourieG]

Type of change

• Bug

Proposed commit message

[Fix] azure-blob-storage: Input failing with Storage Blob Data Reader RBAC role

Remove `Tags: true` from the `ListBlobsInclude` options in the blob
listing call. This option requires the `blobs/tags/read` DataAction,
which is not granted by the Storage Blob Data Reader or Storage Blob
Data Contributor RBAC roles, causing an AuthorizationPermissionMismatch
error when using Entra ID (OAuth2) authentication.

The blob index tags were never consumed by the input after being fetched,
so removing this option has no functional impact.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works. Where relevant, I have used the stresstest.sh script to run them under stress conditions and race detector to verify their stability.
• I have added an entry in ./changelog/fragments using the changelog tool.

Disruptive User Impact

Author's Checklist

• [ ]

How to test this PR locally

Related issues

• Closes [filebeat][ABS] - Input fails with AuthorizationPermissionMismatch when using Storage Blob Data Reader RBAC role #48890

Use cases

Screenshots

Logs

[github-actions]

🤖 GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[efd6]

For the future (to bear in mind, not do anything now), can we construct a testing approach that would avoid the issues that caused this?

[ShourieG]

For the future (to bear in mind, not do anything now), can we construct a testing approach that would avoid the issues that caused this?

Yes definitely, the main reason it passed the initial testing was the propagation delay (upto 10 mins) that made the test seem to pass with the data reader permissions. Also lack of clear documentation at the time of development did not help. You can check the linked issue for a detailed reason on why this happened.

[github-actions]

@Mergifyio backport 8.19 9.2 9.3

[mergify]

backport 8.19 9.2 9.3

✅ Backports have been created

Details

• #48910 [8.19](backport #48886) [filebeat][ABS] - Fix Input failing with Storage Blob Data Reader RBAC role has been created for branch 8.19
• #48911 [9.2](backport #48886) [filebeat][ABS] - Fix Input failing with Storage Blob Data Reader RBAC role has been created for branch 9.2
• #48912 [9.3](backport #48886) [filebeat][ABS] - Fix Input failing with Storage Blob Data Reader RBAC role has been created for branch 9.3

[efd6]

Yeah, that's understood. I'm just wondering how we would avoid that in the future.

[ShourieG]

Yeah, that's understood. I'm just wondering how we would avoid that in the future.

I think having a terraform test bed in our integrations would help out a lot, like we have for AWS. We should also make it mandatory to test with various permission levels in that test bed.