Logs automated export into Azure Storage Account and/or Log Analytics Workspace

[o-l-a-v]

Hello!

• Vote on this issue by adding a 👍 reaction
• If you want to implement this feature, comment to let us know (we'll work with you on design, scheduling, etc.)

Issue details

Context

(@dirien asked to be tagged on this one: https://www.reddit.com/r/AZURE/comments/1en0b8g/comment/lh91ae7)

As I've already posted on:

• Reddit r/Azure: https://www.reddit.com/r/AZURE/comments/1en0b8g
• GitHub repo for Azure Sentinel Connectors: [Feature] Pulumi Cloud - Connector that pulls logs from API, ingest to Log Analytics Azure/Azure-Sentinel#10944

There's currently no native or ready made solution for getting logs from Pulumi Cloud to Azure Log Analytics Workspace.

Current situation:

• Pulumi Cloud supports automated export to AWS S3, but we don't use AWS.
  • https://www.pulumi.com/docs/pulumi-cloud/audit-logs/#automated-export
• Pulumi Cloud has an API endpoint for pulling logs, but this endpoint is "rate-limited and only intended for occasional use".
  • https://www.pulumi.com/docs/pulumi-cloud/audit-logs/#exporting-audit-logs-using-the-api

Feature request

1. Support automated export to Azure Storage Account blob storage, which shouldn't require much work when you already support AWS S3.
   • Least amount of work required, I assume.
   • Cloudflare continous export named "Logpush" supports both targets.
     • AWS S3: https://developers.cloudflare.com/logs/get-started/enable-destinations/aws-s3/
     • Azure Storage Account: https://developers.cloudflare.com/logs/get-started/enable-destinations/azure/
2. Make API for log pull able to handle continous pull/export, for the ability to write custom integrations easily.
3. Support automated export straight to Log Analytics Workspace.
   • More work required, but would be great for Azure users to have this supported natively.
   • If you go this route, remember to use the new API: https://learn.microsoft.com/en-us/azure/azure-monitor/logs/logs-ingestion-api-overview

Affected area/feature

Pulumi Cloud

[justinvp]

Thanks for opening the issue, @o-l-a-v. I'm going to transfer this to the pulumi-cloud-requests repo and someone will take a look.