AWS s3 exporter component

CHANGELOG.md

@@ -13,6 +13,7 @@ Main (unreleased)
 ### Features
 
 - (_Public preview_) Add support for setting GOMEMLIMIT based on cgroup setting. (@mattdurham)
+- (_Experimental_) A new `otelcol.exporter.awss3` component for sending telemetry data to a S3 bucket. (@Imshelledin21)
 
 ### Enhancements
 

docs/sources/reference/components/otelcol.exporter.awss3.md

@@ -0,0 +1,190 @@
+---
+canonical: https://grafana.com/docs/alloy/latest/reference/components/otelcol.exporter.awss3/
+description: Learn about otelcol.exporter.awss3
+title: otelcol.exporter.awss3
+---
+
+<span class="badge docs-labels__stage docs-labels__item">Experimental</span>
+
+# otelcol.exporter.awss3
+
+{{< docs/shared lookup="stability/experimental.md" source="alloy" version="<ALLOY_VERSION>" >}}
+
+`otelcol.exporter.awss3` accepts telemetry data from other `otelcol` components
+and writes them to an AWS S3 Bucket.
+
+> **NOTE**: `otelcol.exporter.awss3` is a wrapper over the upstream
+> OpenTelemetry Collector Contrib `awss3` exporter. Bug reports or feature requests will
+> be redirected to the upstream repository, if necessary.
+
+Multiple `otelcol.exporter.awss3` components can be specified by giving them
+different labels.
+
+## Usage
+
+```alloy
+otelcol.exporter.awss3 "LABEL" {
+  s3_uploader {
+    region = "REGION"
+    s3_bucket = "BUCKET_NAME"
+    s3_prefix = "PREFIX"
+  }
+}
+```
+
+## Arguments
+
+`otelcol.exporter.awss3` supports the following arguments:
+
+Name       | Type       | Description                                      | Default | Required
+-----------|------------|--------------------------------------------------|---------|---------
+`encoding` | `string`   | Encoding extension to use to marshal data. Overrides the `marshaler` configuration option if set. | `""`  | no
+`encoding_file_ext` | `string` | file format extension suffix when using the `encoding` configuration option. May be left empty for no suffix to be appended. | `""` | no
+
+## Blocks
+
+The following blocks are supported inside the definition of
+`otelcol.exporter.awss3`:
+
+Hierarchy              | Block                | Description                                                                          | Required
+-----------------------|----------------------|--------------------------------------------------------------------------------------|---------
+s3_uploader            | [s3_uploader][]      | Configures the AWS S3 bucket details to send telemetry data to.                      | yes
+marshaler              | [marshaler][]        | Marshaler used to produce output data.                                               | no
+debug_metrics          | [debug_metrics][]    | Configures the metrics that this component generates to monitor its state.           | no
+
+[s3_uploader]: #s3_uploader-block
+[marshaler]: #marshaler-block
+[debug_metrics]: #debug_metrics-block
+
+### s3_uploader block
+
+The `s3_uploader` block configures the AWS S3 bucket details used by the component.
+
+The following arguments are supported:
+
+Name                  | Type                       | Description                                                                      | Default      | Required
+----------------------|----------------------------|----------------------------------------------------------------------------------|--------------|---------
+`region`              | `string`                   | AWS region.                                                                      | `"us-east-1"`| no
+`s3_bucket`           | `string`                   | S3 bucket                                                                        |              | yes
+`s3_prefix`           | `string`                   | Prefix for the S3 key (root directory inside bucket).                            |              | yes
+`s3_partition`        | `string`                   | Time granularity of S3 key: hour or minute                                       | `"minute"`   | no
+`role_arn`            | `string`                   | The Role ARN to be assumed                                                       |              | no
+`file_prefix`         | `string`                   | File prefix defined by user                                                      |              | no
+`endpoint`            | `string`                   | Overrides the endpoint used by the exporter instead of constructing it from `region` and `s3_bucket` |      | no
+`s3_force_path_style` | `boolean`                  | [Set this to `true` to force the request to use path-style addressing](http://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html) | `false`             | no
+`disable_ssl`         | `boolean`                  | Set this to `true` to disable SSL when sending requests           |              | `false`
+`compression`         | `string`                   | should the file be compressed                                                    | `none`      | no
+
+### marshaler block
+
+Marshaler determines the format of data sent to AWS S3. Currently, the following marshalers are implemented:
+
+- `otlp_json` (default): the [OpenTelemetry Protocol format](https://github.com/open-telemetry/opentelemetry-proto), represented as json.
+- `otlp_proto`: the [OpenTelemetry Protocol format](https://github.com/open-telemetry/opentelemetry-proto), represented as Protocol Buffers. A single protobuf message is written into each object.
+- `sumo_ic`: the [Sumo Logic Installed Collector Archive format](https://help.sumologic.com/docs/manage/data-archiving/archive/).
+  **This format is supported only for logs.**
+- `body`: export the log body as string.
+  **This format is supported only for logs.**
+
+The following arguments are supported:
+
+Name                    | Type       | Description                                                                                | Default | Required
+------------------------|------------|--------------------------------------------------------------------------------------------|---------|---------
+`type`                  | `string`   | Marshaler used to produce output data                                                      | `"otlp_json"`   | no
+
+### debug_metrics block
+
+{{< docs/shared lookup="reference/components/otelcol-debug-metrics-block.md" source="alloy" version="<ALLOY_VERSION>" >}}
+
+### Encoding
+
+Encoding overrides marshaler if present and sets to use an encoding extension defined in the collector configuration.
+
+See [https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/encoding](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/encoding).
+
+### Compression
+
+- `none` (default): No compression will be applied
+- `gzip`: Files will be compressed with gzip. **This does not support `sumo_ic`marshaler.**
+
+## Exported fields
+
+The following fields are exported and can be referenced by other components:
+
+Name    | Type               | Description
+--------|--------------------|-----------------------------------------------------------------
+`input` | `otelcol.Consumer` | A value that other components can use to send telemetry data to.
+
+`input` accepts `otelcol.Consumer` data for any telemetry signal (metrics,
+logs, or traces).
+
+## Component health
+
+`otelcol.exporter.awss3` is only reported as unhealthy if given an invalid
+configuration.
+
+## Debug information
+
+`otelcol.exporter.awss3` does not expose any component-specific debug
+information.
+
+## Debug metrics
+
+* `exporter_sent_spans_ratio_total` (counter): Number of spans successfully sent to destination.
+* `exporter_send_failed_spans_ratio_total` (counter): Number of spans in failed attempts to send to destination.
+* `exporter_queue_capacity_ratio` (gauge): Fixed capacity of the retry queue (in batches)
+* `exporter_queue_size_ratio` (gauge): Current size of the retry queue (in batches)
+* `rpc_client_duration_milliseconds` (histogram): Measures the duration of inbound RPC.
+* `rpc_client_request_size_bytes` (histogram): Measures size of RPC request messages (uncompressed).
+* `rpc_client_requests_per_rpc` (histogram): Measures the number of messages received per RPC. Should be 1 for all non-streaming RPCs.
+* `rpc_client_response_size_bytes` (histogram): Measures size of RPC response messages (uncompressed).
+* `rpc_client_responses_per_rpc` (histogram): Measures the number of messages received per RPC. Should be 1 for all non-streaming RPCs.
+
+## Example
+
+This example forwards scrape logs to an AWS S3 Bucket:
+
+```alloy
+local.file_match "logs" {
+  path_targets = [{
+    __address__ = "localhost",
+    __path__    = "/var/log/{syslog,messages,*.log}",
+    instance    = constants.hostname,
+    job         = "integrations/node_exporter",
+  }]
+}
+
+loki.source.file "logs" {
+  targets    = local.file_match.logs.targets
+  forward_to = [otelcol.receiver.loki.default.receiver]
+}
+
+otelcol.receiver.loki "default" {
+  output {
+    logs = [otelcol.exporter.awss3.logs.input]
+  }
+}
+
+otelcol.exporter.awss3 "logs" {
+  s3_uploader {
+    region = "us-east-1"
+    s3_bucket = "logs_bucket"
+    s3_prefix = "logs"
+  }
+}
+```
+
+<!-- START GENERATED COMPATIBLE COMPONENTS -->
+
+## Compatible components
+
+`otelcol.exporter.otlp` has exports that can be consumed by the following components:
+
+- Components that consume [OpenTelemetry `otelcol.Consumer`](../../compatibility/#opentelemetry-otelcolconsumer-consumers)
+
+{{< admonition type="note" >}}
+Connecting some components may not be sensible or components may require further configuration to make the connection work correctly.
+Refer to the linked documentation for more details.
+{{< /admonition >}}
+
+<!-- END GENERATED COMPATIBLE COMPONENTS -->

go.mod

@@ -261,6 +261,7 @@ require (
 	github.com/KimMachineGun/automemlimit v0.6.0
 	github.com/Shopify/sarama v1.38.1
 	github.com/grafana/kafka_exporter v0.0.0-20240409084445-5e3488ad9f9a
+	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awss3exporter v0.96.0
 )
 
 require (

go.sum

@@ -1072,8 +1072,6 @@ github.com/grafana/pyroscope-go/godeltaprof v0.1.7 h1:C11j63y7gymiW8VugJ9ZW0pWfx
 github.com/grafana/pyroscope-go/godeltaprof v0.1.7/go.mod h1:Tk376Nbldo4Cha9RgiU7ik8WKFkNpfds98aUzS8omLE=
 github.com/grafana/pyroscope/api v0.4.0 h1:J86DxoNeLOvtJhB1Cn65JMZkXe682D+RqeoIUiYc/eo=
 github.com/grafana/pyroscope/api v0.4.0/go.mod h1:MFnZNeUM4RDsDOnbgKW3GWoLSBpLzMMT9nkvhHHo81o=
-github.com/grafana/pyroscope/ebpf v0.4.6-0.20240426163159-8575ca65f70b h1:BXLaXB9gPvlbPeiwOkauVFvf7iaZwqS+BYCmHiAue64=
-github.com/grafana/pyroscope/ebpf v0.4.6-0.20240426163159-8575ca65f70b/go.mod h1:0iOWpGm2M6KXiP2nGa4wf02knSSjEtu11vpUOdQT5AY=
 github.com/grafana/pyroscope/ebpf v0.4.6 h1:8A6vddwCF2q8XXrOebFuDczr9xOHCfqPN2QwIkXKxgU=
 github.com/grafana/pyroscope/ebpf v0.4.6/go.mod h1:0iOWpGm2M6KXiP2nGa4wf02knSSjEtu11vpUOdQT5AY=
 github.com/grafana/regexp v0.0.0-20221123153739-15dc172cd2db h1:7aN5cccjIqCLTzedH7MZzRZt5/lsAHch6Z3L2ZGn5FA=
@@ -1708,6 +1706,8 @@ github.com/open-telemetry/opentelemetry-collector-contrib/connector/servicegraph
 github.com/open-telemetry/opentelemetry-collector-contrib/connector/servicegraphconnector v0.96.0/go.mod h1:/NA9T4O1WOlkUwvTXBz5wmuddpC0cc2cDLEBH5ck9eM=
 github.com/open-telemetry/opentelemetry-collector-contrib/connector/spanmetricsconnector v0.96.0 h1:KAlAzuzvYq0xZWRR+N2qUJhE7/pvmNFYlcN5yW8Km60=
 github.com/open-telemetry/opentelemetry-collector-contrib/connector/spanmetricsconnector v0.96.0/go.mod h1:KcZjtSdoelUWRwGtVaiEX16Hw8mFH+JnYrN+r4Ox550=
+github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awss3exporter v0.96.0 h1:xCxngHclXqvvEZ1M2mwrYI50X5ZiyXOaGuRYo31uVBQ=
+github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awss3exporter v0.96.0/go.mod h1:j/h1Y3drOchJKdbGueHzEBRzvzy4QdzVKJ8GpvQX3jw=
 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/kafkaexporter v0.96.0 h1:2FnXGN9xxIcIz7f4hdX+OgsGowWC1D35oNtX5ErnLBc=
 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/kafkaexporter v0.96.0/go.mod h1:VPyawEuVpqKg3oemeDnYwDfBbh9gjGbrVVXl4OeHK60=
 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/loadbalancingexporter v0.96.0 h1:3+Ca2P/XLCSSc3299+4fjQf2sPMepiewR+KSBzzIGvg=

internal/component/all/all.go

@@ -67,6 +67,7 @@ import (
 	_ "github.com/grafana/alloy/internal/component/otelcol/connector/servicegraph"           // Import otelcol.connector.servicegraph
 	_ "github.com/grafana/alloy/internal/component/otelcol/connector/spanlogs"               // Import otelcol.connector.spanlogs
 	_ "github.com/grafana/alloy/internal/component/otelcol/connector/spanmetrics"            // Import otelcol.connector.spanmetrics
+	_ "github.com/grafana/alloy/internal/component/otelcol/exporter/awss3"                   // Import otelcol.exporter.awss3exporter
 	_ "github.com/grafana/alloy/internal/component/otelcol/exporter/loadbalancing"           // Import otelcol.exporter.loadbalancing
 	_ "github.com/grafana/alloy/internal/component/otelcol/exporter/logging"                 // Import otelcol.exporter.logging
 	_ "github.com/grafana/alloy/internal/component/otelcol/exporter/loki"                    // Import otelcol.exporter.loki

internal/component/otelcol/exporter/awss3/awss3.go

@@ -0,0 +1,118 @@
+// Package awss3 provides an otelcol.exporter.awss3 component
+package awss3
+
+import (
+	"github.com/grafana/alloy/internal/component"
+	"github.com/grafana/alloy/internal/component/otelcol"
+	"github.com/grafana/alloy/internal/component/otelcol/exporter"
+	"github.com/grafana/alloy/internal/featuregate"
+	"github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awss3exporter"
+	otelcomponent "go.opentelemetry.io/collector/component"
+	"go.opentelemetry.io/collector/config/configcompression"
+	otelextension "go.opentelemetry.io/collector/extension"
+)
+
+func init() {
+	component.Register(component.Registration{
+		Name:      "otelcol.exporter.awss3",
+		Stability: featuregate.StabilityGenerallyAvailable,
+		Args:      Arguments{},
+		Exports:   otelcol.ConsumerExports{},
+
+		Build: func(opts component.Options, args component.Arguments) (component.Component, error) {
+			fact := awss3exporter.NewFactory()
+			return exporter.New(opts, fact, args.(Arguments), exporter.TypeAll)
+		},
+	})
+}
+
+// Arguments configures the otelcol.exporter.awss3 component.
+type Arguments struct {
+	Encoding              string `alloy:"encoding,attr,optional"`
+	EncodingFileExtension string `alloy:"encoding_file_ext,attr,optional"`
+
+	S3Uploader    S3Uploader    `alloy:"s3_uploader,block"`
+	MarshalerName MarshalerType `alloy:"marshaler,block,optional"`
+
+	// DebugMetrics configures component internal metrics. Optional
+	DebugMetrics otelcol.DebugMetricsArguments `alloy:"debug_metrics,block,optional"`
+}
+
+var _ exporter.Arguments = Arguments{}
+
+func (args *Arguments) SetToDefault() {
+	args.MarshalerName.SetToDefault()
+	args.S3Uploader.SetToDefault()
+}
+
+func (args Arguments) Convert() (otelcomponent.Config, error) {
+	var result awss3exporter.Config
+
+	result.S3Uploader = args.S3Uploader.Convert()
+	result.MarshalerName = args.MarshalerName.Convert()
+
+	return &result, nil
+}
+
+func (args Arguments) Extensions() map[otelcomponent.ID]otelextension.Extension {
+	return nil
+}
+
+func (args Arguments) Exporters() map[otelcomponent.DataType]map[otelcomponent.ID]otelcomponent.Component {
+	return nil
+}
+
+func (args Arguments) DebugMetricsConfig() otelcol.DebugMetricsArguments {
+	return args.DebugMetrics
+}
+
+// S3 Uploader Arguments Block
+type S3Uploader struct {
+	Region           string                 `alloy:"region,attr,optional"`
+	S3Bucket         string                 `alloy:"s3_bucket,attr"`
+	S3Prefix         string                 `alloy:"s3_prefix,attr"`
+	S3Partition      string                 `alloy:"s3_partition,attr,optional"`
+	RoleArn          string                 `alloy:"role_arn,attr,optional"`
+	FilePrefix       string                 `alloy:"file_prefix,attr,optional"`
+	Endpoint         string                 `alloy:"endpoint,attr,optional"`
+	S3ForcePathStyle bool                   `alloy:"s3_force_path_style,attr,optional"`
+	DisableSSL       bool                   `alloy:"disable_ssl,attr,optional"`
+	Compression      configcompression.Type `alloy:"compression,attr,optional"`
+}
+
+func (args *S3Uploader) SetToDefault() {
+	*args = S3Uploader{
+		Region:           "us-east-1",
+		S3ForcePathStyle: false,
+		DisableSSL:       false,
+	}
+}
+
+func (args *S3Uploader) Convert() awss3exporter.S3UploaderConfig {
+	return awss3exporter.S3UploaderConfig{
+		Region:           args.Region,
+		S3Bucket:         args.S3Bucket,
+		S3Prefix:         args.S3Prefix,
+		S3Partition:      args.S3Partition,
+		FilePrefix:       args.FilePrefix,
+		Endpoint:         args.Endpoint,
+		RoleArn:          args.RoleArn,
+		S3ForcePathStyle: args.S3ForcePathStyle,
+		DisableSSL:       args.DisableSSL,
+	}
+}
+
+// MarshalerType Argument Block
+type MarshalerType struct {
+	Type string `alloy:"type,attr,optional"`
+}
+
+func (args *MarshalerType) SetToDefault() {
+	*args = MarshalerType{
+		Type: "otlp_json",
+	}
+}
+
+func (args MarshalerType) Convert() awss3exporter.MarshalerType {
+	return awss3exporter.MarshalerType(args.Type)
+}