Skip to content

Conversation

@joshcoughlan
Copy link

Summary

This PR migrates the azure_blob sink to the official Azure SDK and adds per-sink proxy support, while preserving auth parity with the previous implementation.

Highlights:

  • Official SDK: azure_storage_blob with azure_core.
  • Per-sink proxy: uses an aliased reqwest 0.12 client (reqwest_0_12) injected into azure_core::ClientOptions transport, so proxy settings are scoped to the sink (no reliance on environment variables).
  • Auth parity:
    • SAS connection strings (BlobEndpoint + SharedAccessSignature) are supported.
    • AccountKey connection strings (AccountName + AccountKey) are supported by deriving a scoped, time-bound account-signed SAS at runtime and rotating it in the background. Note: the storage account must allow shared key access.
  • Isolation: The rest of Vector continues using reqwest 0.11 + http 0.2. The new reqwest 0.12 is isolated to the Azure path to avoid cross-crate type conflicts.
  • Documentation and changelog updated accordingly.

Vector configuration

Examples used for local verification.

  1. SAS connection string + per-sink proxy
[sinks.azure_blob]
type = "azure_blob"
inputs = ["in"]
container_name = "my-container"
connection_string = "BlobEndpoint=https://<account>.blob.core.windows.net/;SharedAccessSignature=sv=...&ss=b&..."

# partitioning/encoding examples
blob_prefix = "blob/%F/"
blob_time_format = "%s"
blob_append_uuid = true
encoding.codec = "json"
compression = "gzip"

# per-sink proxy (example)
[sinks.azure_blob.proxy]
enabled = true
http = "http://proxy.example:3128"
https = "http://proxy.example:3128"
no_proxy = "localhost,.internal"
  1. AccountKey connection string + per-sink proxy
[sinks.azure_blob]
type = "azure_blob"
inputs = ["in"]
container_name = "my-container"
connection_string = "DefaultEndpointsProtocol=https;AccountName=<name>;AccountKey=<base64key>;EndpointSuffix=core.windows.net"

# The sink derives a scoped, time‑bound account-signed SAS at runtime and rotates it in the background.
# The storage account must allow shared key access (Allow shared key access = Enabled).

blob_prefix = "blob/%F/"
blob_time_format = "%s"
blob_append_uuid = true
encoding.codec = "json"
compression = "gzip"

[sinks.azure_blob.proxy]
enabled = true
http = "http://proxy.example:3128"
https = "http://proxy.example:3128"
no_proxy = "localhost,.internal"

How did you test this PR?

Change Type

  • Bug fix
  • New feature
  • Non-functional (chore, refactoring, docs)
  • Performance

Is this a breaking change?

  • Yes
  • No

Does this PR include user facing changes?

  • Yes. Please add a changelog fragment based on our guidelines.
  • No. A maintainer will apply the no-changelog label to this PR.

References

Notes

  • Please read our Vector contributor resources.
  • Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
  • Some CI checks run only after we manually approve them.
    • We recommend adding a pre-push hook, please see this template.
    • Alternatively, we recommend running the following locally before pushing to the remote branch:
      • make fmt
      • make check-clippy (if there are failures it's possible some of them can be fixed with make clippy-fix)
      • make test
  • After a review is requested, please avoid force pushes to help us review incrementally.
    • Feel free to push as many commits as you want. They will be squashed into one before merging.
    • For example, you can run git merge origin master and git push.
  • If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
    run make build-licenses to regenerate the license inventory and commit the changes (if any). More details here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants