[exporter/azureblob] add implementation for azure blob exporter #38028

hgaol · 2025-02-18T16:22:26Z

Description

Implementation of new component, azure blob exporter.

Link to tracking issue

Testing

Validated using different authentication, and tested using multiple azure blob accounts.
example config.

processors:
  batch:
  batch/2:
    send_batch_size: 10000
    timeout: 10s

exporters:
  debug:
    verbosity: detailed
  azureblob/conn-string:
    # for connection string auth, no need to specify url, because it's already included in connection string
    auth:
      type: "connection_string"
      connection_string: "DefaultEndpointsProtocol=https;AccountName=ghaccoujt;AccountKey=<secret>;EndpointSuffix=core.windows.net"
    container:
      metrics: "test"
      logs: "test"
      traces: "test"
  azureblob/url:
    url: "https://cs110032000abaee117.blob.core.windows.net/"
    auth:
      type: "service_principal"
      tenant_id: "4d94074b-32c4-4a70-949c-c7bd7ff7af89"
      client_id: "5ee98aba-dee7-4fcb-9a79-fd6797f752cf"
      client_secret: "<secret>"
    container:
      metrics: "filesystem"
      logs: "filesystem"
      traces: "filesystem"

service:
  pipelines:
    metrics/1:
      receivers: [otlp]
      exporters: [azureblob/conn-string]
    traces/1:
      receivers: [otlp]
      exporters: [azureblob/conn-string]
    logs/1:
      receivers: [otlp]
      exporters: [azureblob/conn-string]
    metrics/2:
      receivers: [otlp]
      processors: [batch/2]
      exporters: [azureblob/url]
    traces/2:
      receivers: [otlp]
      processors: [batch/2]
      exporters: [azureblob/url]
    logs/2:
      receivers: [otlp]
      processors: [batch/2]
      exporters: [azureblob/url]

output.

Documentation

See README.md

MovieStoreGuy

Couple of things from me:

Could you please include some tests around this functionality?
The consume methods looks like duplicated code, could they be collapse into one method?

My initial thoughts by doing something like:

func (t *T) ConsumeTraces(ctx context.Context, pt ptrace.Traces) error {
  buf, err := t.marshalTraces(pt)
  if err != nil {
     return err
  }
  return t.consumeBuffer(ctx, buf)
}

func (t *T) consumeBuffer(ctx context.Context, buf []byte) error {
  // blob creationtion here and uploading
  return nil
}

MovieStoreGuy · 2025-02-20T05:50:42Z

exporter/azureblobexporter/exporter.go

+var fileExtensionMap = map[string]string{
+	formatTypeJSON:  "json",
+	formatTypeProto: "pb",
 }


In the event that I have configured an extension to perform the marshalling, it may be straight up raw text, or some other format that doesn't match json or protobuf.

Is it possible for a user to define their own extension for the data written?

Supported, it can support other encoding extension with below config. And updated descriptions in README.md.

azureblob/1: encodings: logs: text_encoding

MovieStoreGuy · 2025-02-20T05:52:06Z

exporter/azureblobexporter/exporter.go

+	client           *azblob.Client
+	signal           pipeline.Signal
+	marshaller       *marshaller
+	blobNameTemplate *template.Template


Could you expand why there is a template here?

removed template.

MovieStoreGuy · 2025-02-20T05:52:29Z

exporter/azureblobexporter/exporter.go

+		return fmt.Errorf("failed to parse template: %w", err)
+	}
+
+	// create client based on auth type


can you add a comment here to explain that this is only required until there is an azure auth extension?

Added comments in Config.URL.

MovieStoreGuy · 2025-02-20T05:55:42Z

exporter/azureblobexporter/exporter.go

+func (e *azureBlobExporter) generateBlobName(data map[string]any) (string, error) {
+	// Get current time
+	now := time.Now()
+
+	// Execute the template
+	var result bytes.Buffer
+	if err := e.blobNameTemplate.Execute(&result, data); err != nil {
+		return "", fmt.Errorf("failed to execute template: %w", err)
+	}
+
+	blobStr := result.String()
+	return now.Format(blobStr), nil
+}


My assumption here is that this works in two parts?

Provide a directory key, and ensure the blob has a unique name to avoid any collisions with other exporters?

I am mildly concerned that using a template might be slow in the hot path, and it might be better to use a strings builder instead.

removed template and use string operations directly.

MovieStoreGuy · 2025-02-20T05:56:27Z

exporter/azureblobexporter/exporter.go

+		return fmt.Errorf("failed to upload metrics data: %w", err)
+	}
+
+	e.logger.Info("Successfully exported metrics to Azure Blob Storage",


I suggest changing this to debug by default, it is going to be very noisey

MovieStoreGuy · 2025-02-20T05:57:31Z

exporter/azureblobexporter/exporter.go

+	}
+
+	blobContentReader := bytes.NewReader(data)
+	_, err = e.client.UploadStream(context.TODO(), e.config.Container.Traces, blobName, blobContentReader, nil)


Please use the provided context instead of using context.TODO

…g support

…tor-contrib into azureblobexporter-second-PR

hgaol · 2025-03-01T12:53:28Z

Couple of things from me:

Could you please include some tests around this functionality?

The consume methods looks like duplicated code, could they be collapse into one method?

My initial thoughts by doing something like:
func (t *T) ConsumeTraces(ctx context.Context, pt ptrace.Traces) error {
  buf, err := t.marshalTraces(pt)
  if err != nil {
     return err
  }
  return t.consumeBuffer(ctx, buf)
}

func (t *T) consumeBuffer(ctx context.Context, buf []byte) error {
  // blob creationtion here and uploading
  return nil
}

Thanks @MovieStoreGuy for the careful review! I've did some changes as below

collapsed into one method.
added tests for exporter, marshaller.
support specifying encoding extension for logs, metrics and traces.
deprecated using string template to generate blob name.

I also validated using 2 Azure blob storage account. Pls help to take a look at your convenience. Thanks!

…tor-contrib into azureblobexporter-second-PR

atoulme · 2025-03-04T17:35:45Z

Please make tidy

dehaansa · 2025-03-04T21:26:09Z

exporter/azureblobexporter/config.go

+	Container *Container      `mapstructure:"container"`
+	Auth      *Authentication `mapstructure:"auth"`
+
+	// BlobNameFormat is the format of the blob name. It controls the uploaded blob name, e.g. "2006/01/02/metrics_15_04_05_{{.SerialNum}}.json"


Comment is no longer accurate.

dehaansa · 2025-03-04T21:34:23Z

exporter/azureblobexporter/exporter.go

+		return fmt.Errorf("failed to upload traces data: %w", err)
+	}
+
+	e.logger.Debug("Successfully exported traces to Azure Blob Storage",


Message leftover from refactor, remove signal type from message and add it as a log attribute?

Making it an attribute will make querying the data easier

I found there's data_type attribute which contains signal type info. I just updated the message and not adding signal attribute.

dehaansa · 2025-03-04T21:35:18Z

exporter/azureblobexporter/exporter_test.go

+	assert.NoError(t, te.start(context.Background(), componenttest.NewNopHost()))
+}
+
+func TestExporterConsumeMetrics(t *testing.T) {


Suggested change

func TestExporterConsumeMetrics(t *testing.T) {

func TestExporterConsumeTelemetry(t *testing.T) {

updated the test function name

dehaansa · 2025-03-04T21:36:05Z

exporter/azureblobexporter/factory.go

-			MetricsFormat:  "2006/01/02/metrics_15_04_05_{{.SerialNum}}.{{.FileExtension}}",
-			LogsFormat:     "2006/01/02/logs_15_04_05_{{.SerialNum}}.{{.FileExtension}}",
-			TracesFormat:   "2006/01/02/traces_15_04_05_{{.SerialNum}}.{{.FileExtension}}",
+			MetricsFormat:  "2006/01/02/metrics_15_04_05.json",


Should default have millisecond precision? For large workloads.

There's also serial number, maybe minutes is enough as default?

I also plan to support append blob feature, but will do it in another PR for future.

dehaansa · 2025-03-04T21:37:28Z

exporter/azureblobexporter/marshaller.go

+		if encoding == nil {
+			return nil, fmt.Errorf("unknown encoding %q", config.Encodings.Logs)
+		}
+		// cast with ok to avoid panics.


If we're casting to avoid panics, we should check the ok and return an error.

MovieStoreGuy · 2025-03-04T22:50:51Z

exporter/azureblobexporter/marshaller.go

+	if m.tracesMarshaler == nil {
+		return nil, errors.New("traces are not supported by encoding")
+	}
+	buf, err := m.tracesMarshaler.MarshalTraces(td)


You can return here.

good catch! updated

MovieStoreGuy

These changes looks safe to merge as is, anything extra can be added in future Prs.

MovieStoreGuy · 2025-03-04T22:53:25Z

Please run go mod tidy to resolve the issues in packages.

…tor-contrib into azureblobexporter-second-PR

hgaol · 2025-03-05T01:22:29Z

Thank you all for the careful review! Updated and pls take a look!

dehaansa

Approved with one docs question. Marked as ready to merge as I'm fine with fixing it as a quick followup if a maintainer gets in here first and merges.

dehaansa · 2025-03-05T14:24:46Z

exporter/azureblobexporter/README.md

@@ -36,18 +36,38 @@ The following settings can be optionally configured and have default values:
  - traces_format (default `2006/01/02/traces_15_04_05_{{.SerialNum}}.{{.FileExtension}}`): blob name format.


Are these documented defaults still correct?

good catch, I'll fix them in future PR.

hgaol added 2 commits February 19, 2025 00:18

add implementation for azure blob exporter

b5a8d33

add change log

0a4707d

hgaol requested a review from a team as a code owner February 18, 2025 16:22

hgaol requested a review from dehaansa February 18, 2025 16:22

github-actions bot assigned dehaansa Feb 18, 2025

github-actions bot added the exporter/azureblob label Feb 18, 2025

hgaol added 2 commits February 19, 2025 08:36

fix lint issue

3ee4b78

fix CI issue

c563fa9

MovieStoreGuy reviewed Feb 20, 2025

View reviewed changes

hgaol added 4 commits February 28, 2025 09:46

feat(azureblobexporter): enhance blob name formatting and add encodin…

7558c1f

…g support

Merge branch 'main' of github.com:open-telemetry/opentelemetry-collec…

726df2d

…tor-contrib into azureblobexporter-second-PR

correct comments and improve test assertions for data validation

51cf556

fix component test

0422094

hgaol added 4 commits March 2, 2025 16:58

[azureblobexporter] add retry configuration options for failure handling

ef94262

Merge branch 'main' of github.com:open-telemetry/opentelemetry-collec…

6d561fa

…tor-contrib into azureblobexporter-second-PR

udpate go.mod

6ee3ce7

update crosslink

d2a37cd

dehaansa requested changes Mar 4, 2025

View reviewed changes

MovieStoreGuy reviewed Mar 4, 2025

View reviewed changes

MovieStoreGuy approved these changes Mar 4, 2025

View reviewed changes

hgaol added 5 commits March 5, 2025 08:33

fix CI issue

c1805db

update based on comments

a9d0178

Merge branch 'main' of github.com:open-telemetry/opentelemetry-collec…

d78a88c

…tor-contrib into azureblobexporter-second-PR

mod tidy

62211c1

remove redundant signal log attribute

1660f08

hgaol requested a review from dehaansa March 5, 2025 01:22

dehaansa approved these changes Mar 5, 2025

View reviewed changes

dehaansa added the ready to merge label Mar 5, 2025

atoulme merged commit b8f05c0 into open-telemetry:main Mar 5, 2025
167 checks passed

github-actions bot added this to the next release milestone Mar 5, 2025

	func TestExporterConsumeMetrics(t *testing.T) {
	func TestExporterConsumeTelemetry(t *testing.T) {

		@@ -36,18 +36,38 @@ The following settings can be optionally configured and have default values:
		- traces_format (default `2006/01/02/traces_15_04_05_{{.SerialNum}}.{{.FileExtension}}`): blob name format.

[exporter/azureblob] add implementation for azure blob exporter #38028

[exporter/azureblob] add implementation for azure blob exporter #38028

Conversation

hgaol commented Feb 18, 2025 • edited Loading

Description

Link to tracking issue

Testing

Documentation

MovieStoreGuy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hgaol commented Mar 1, 2025

atoulme commented Mar 4, 2025

Choose a reason for hiding this comment

hgaol Mar 5, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hgaol Mar 5, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MovieStoreGuy left a comment

Choose a reason for hiding this comment

MovieStoreGuy commented Mar 4, 2025

hgaol commented Mar 5, 2025

dehaansa left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hgaol commented Feb 18, 2025 •

edited

Loading

hgaol Mar 5, 2025 •

edited

Loading

hgaol Mar 5, 2025 •

edited

Loading

dehaansa left a comment •

edited

Loading