diff --git a/docs/reference/_snippets/find-motlp-endpoint.md b/docs/reference/_snippets/find-motlp-endpoint.md new file mode 100644 index 00000000..db25e6c5 --- /dev/null +++ b/docs/reference/_snippets/find-motlp-endpoint.md @@ -0,0 +1,20 @@ +To retrieve your {{motlp}} endpoint address and API key, follow these steps: + +::::{applies-switch} +:::{applies-item} serverless: +1. In {{ecloud}}, create an Observability project or open an existing one. +2. Go to **Add data**, select **Applications** and then select **OpenTelemetry**. +3. Copy the endpoint and authentication headers values. + +Alternatively, you can retrieve the endpoint from the **Manage project** page and create an API key manually from the **API keys** page. +::: + +:::{applies-item} ess: +{applies_to}`stack: preview 9.2` +1. In {{ecloud}}, create an {{ech}} deployment or open an existing one. +2. Go to **Add data**, select **Applications** and then select **OpenTelemetry**. +3. Copy the endpoint and authentication headers values. + +Alternatively, you can retrieve the endpoint from the **Manage project** page and create an API key manually from the **API keys** page. +::: +:::: \ No newline at end of file diff --git a/docs/reference/edot-cloud-forwarder/aws.md b/docs/reference/edot-cloud-forwarder/aws.md index fd4f436c..46037f75 100644 --- a/docs/reference/edot-cloud-forwarder/aws.md +++ b/docs/reference/edot-cloud-forwarder/aws.md @@ -4,8 +4,9 @@ description: Set up the EDOT Cloud Forwarder for AWS to bring your AWS logs to E applies_to: serverless: observability: preview -# deployment: -# ess: preview + deployment: + ess: preview + self: unavailable product: edot_cf_aws: preview products: @@ -16,31 +17,29 @@ products: # EDOT Cloud Forwarder for AWS -{{edot-cf}} (CF) for AWS provides the EDOT Collector as a Lambda function that collects and forwards logs to Elastic Observability on {{serverless-full}}. +{{edot-cf}} (CF) for AWS provides the EDOT Collector as a Lambda function that collects and forwards logs to {{product.observability}} on {{serverless-full}}. {{edot-cf}} for AWS supports the following log sources: | AWS Service | Telemetry Description | | --- | --- | -| Virtual Private Cloud (VPC) | [VPC Flow Logs](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html) to capture information about IP traffic | -| Elastic Load Balancer (ELB) | [Access logs](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html) for your Application Load Balancer | -% | CloudWatch {applies_to}`product: planned` | Logs generated by AWS CloudWatch | +| Virtual Private Cloud (VPC) | [VPC Flow Logs](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html) to capture information about IP traffic. | +| Elastic Load Balancer (ELB) | [Access logs](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html) for your Application Load Balancer. | +| AWS CloudTrail | [CloudTrail Logs](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-log-file-examples.html) to record account activity. | +% | CloudWatch {applies_to}`product: planned` | Logs generated by AWS CloudWatch. | Read on to learn how to set up {{edot-cf}} for AWS. -::::{note} -We are working to support other popular log types and sources. Get in touch to let us know of any specific requirements that could influence our plans. -:::: +:::{note} +We are working to support other popular log types and sources. [Contact us](docs-content://troubleshoot/ingest/opentelemetry/contact-support.md) to let us know of any specific requirements that could influence our plans. +::: ## Prerequisites ::::{important} -{{edot-cf}} for AWS requires a Managed OTLP endpoint and an API key. Managed OTLP is available for {{serverless-full}} and will soon be available for {{ech}}. - -For self-managed deployments, set up an EDOT Collector in [Gateway mode](elastic-agent://reference/edot-collector/config/default-config-standalone.md#gateway-mode) that ingests OTel data from the edge setup into the self-managed Elastic Stack. +{{edot-cf}} for AWS requires a Managed OTLP endpoint and an API key. Managed OTLP is available for {{serverless-full}} and {{ech}}. :::: - To collect logs using {{edot-cf}} for AWS, you need: ::::{tab-set} @@ -65,6 +64,15 @@ To collect Elastic Load Balancer (ELB) Access logs, you need: ::: +:::{tab-item} CloudTrail + +To collect AWS CloudTrail logs, you need: + +- A trail that delivers account events as log files to an Amazon S3 bucket +- An S3 bucket to store the trail logs + +::: + :::: -In addition, you need to know the URL of the managed OTLP endpoint and the API key for authentication. +You also need to know the URL of the managed OTLP endpoint and the API key for authentication. -::::{dropdown} Steps to retrieve the OTLP endpoint and API key - -:::{include} ../_snippets/serverless-endpoint-api.md +:::{include} ../_snippets/find-motlp-endpoint.md ::: In the CloudFormation templates, the OTLP endpoint is set as `OTLPEndpoint`, and the API key is set as `ElasticAPIKey`. @@ -90,28 +96,44 @@ Trim the API key from `Authorization=ApiKey MYKEYVALUE...` to just `MYKEYVALUE.. ::: :::: +## Deployment methods + +{{edot-cf}} for AWS can be deployed using any of the following methods: + +| Deployment Method | Description | +| --- | --- | +| CloudFormation (AWS CLI) | Deploy using AWS CLI commands with CloudFormation templates. | +| CloudFormation (AWS Console) | Deploy using the AWS Management Console UI. | + + +Each method achieves the same result and uses CloudFormation templates. Choose the method that best adapts to your workflow. + ## Deployment considerations -Before deploying {{edot-cf}} for AWS, keep these points in mind: +Before deploying {{edot-cf}} for AWS, consider the following actions: - Deploy a separate CloudFormation stack for each log type, for example VPC Flow Logs or ELB Access Logs. Each CloudFormation stack can only process one log type and format at a time. - Logs stored in S3 must be placed in separate buckets. Each log type should reside in its own dedicated bucket. - The CloudFormation stack deployment region must match the region of the S3 bucket. -## Download the template [download-templates] +## CloudFormation templates -Download the CloudFormation template to deploy the appropriate stack based on your log source: +The CloudFormation templates are hosted in a public Amazon S3 bucket and can be accessed through HTTPS URL. You can reference these templates directly during deployment or download them for local use. -| Log Source | CloudFormation template | -| --- | ------------------------------------------------ | -| S3 logs | `https://edot-cloud-forwarder.s3.amazonaws.com/v0/latest/cloudformation/s3_logs-cloudformation.yaml` | +| Log type | Log source | CloudFormation template | +| --- | --- | ------------------------------------------------ | +| VPC | S3 | `https://edot-cloud-forwarder.s3.amazonaws.com/v0/latest/cloudformation/s3_logs-cloudformation.yaml` | +| ELB | S3 | `https://edot-cloud-forwarder.s3.amazonaws.com/v0/latest/cloudformation/s3_logs-cloudformation.yaml` | +| CloudTrail | S3 | `https://edot-cloud-forwarder.s3.amazonaws.com/v0/latest/cloudformation/s3_logs-cloudformation.yaml` | % | CloudWatch logs | `https://edot-cloud-forwarder.s3.amazonaws.com/v0/latest/cloudformation/cloudwatch_logs-cloudformation.yaml` | For specific versions, edit `latest` in the URL to the required version in the format `vX.Y.Z`. ## Configure the template -{{edot-cf}} for AWS uses a CloudFormation template to deploy the EDOT Collector as a Lambda function. +Before deploying {{edot-cf}} for AWS, configure the CloudFormation template parameters based on your specific requirements. The template uses the following settings to deploy and configure the EDOT Collector Lambda function. ### Required settings @@ -119,10 +141,9 @@ These are the required settings you need: | Setting | Description | | -------------------------------------- | ----------- | -| `region` | AWS region where the CloudFormation stack is to be deployed | -| `stack-name` | Name of the CloudFormation stack, for example, `vpc-edot-cf`
Do not use the same name for different sources. | -| `OTLPEndpoint` | The OTLP endpoint URL used for data ingestion | -| `ElasticApiKey` | API key for authentication with Elastic | +| `stack-name` | Name of the CloudFormation stack, for example, `vpc-edot-cf`
Do not use the same name for different stacks. | +| `OTLPEndpoint` | The OTLP endpoint URL used for data ingestion, obtained from {{serverless-full}}. | +| `ElasticApiKey` | API key for authentication with Elastic, obtained from {{serverless-full}}. | ### Log source settings @@ -132,11 +153,11 @@ Set the following settings based on the log source: ::::{tab-item} S3 -For S3 logs, use the following settings: +For logs sourced from S3, use the following settings: | Setting | Description | | ------------------ | --- | -| `EdotCloudForwarderS3LogsType` | The encoding format for logs in the S3 bucket. Supported options:
- `vpcflow`: VPC Flow Logs
- `elbaccess`: ELB Access logs| +| `EdotCloudForwarderS3LogsType` | The encoding format for logs in the S3 bucket. Supported options:
- `vpcflow`: VPC Flow Logs
- `elbaccess`: ELB Access logs
- `cloudtrail`: CloudTrail Logs| | `SourceS3BucketARN` | Amazon Resource Name (ARN) of the S3 bucket where logs are stored. This bucket will trigger the `edot-cloud-forwarder` Lambda function automatically. | % | `S3LogsJsonEncodingMode` | _(Required if `EdotCloudForwarderS3LogsType` is `json`)_
Defines how JSON logs are structured:
- `body` _(default)_: Stores logs in the request body
- `body_with_inline_attributes`: Logs include inline attributes | @@ -145,7 +166,7 @@ For S3 logs, use the following settings: ## CloudFormation stack resources @@ -471,20 +532,36 @@ The CloudWatch Log Subscription Filter, `CloudWatchLogSubscriptionFilter`, ensur CloudWatch Log Groups help monitor execution performance and debug issues. IAM permissions (`LambdaExecutionRole`, `LambdaPermissionCloudWatch`) control interactions between CloudWatch and Lambda, while the failure bucket, `S3FailureBucketARN`, helps prevent data loss in case of processing errors. --> + +## Datastreams + +Logs collected by {{edot-cf}} for AWS are stored in {{es}} datastreams in OpenTelemetry native format. The following table shows which datastreams are used for each log type: + +| **AWS Log Type** | **Datastream Dataset** | **Description** | +|------------------|------------------------|-----------------| +| VPC Flow Logs | `aws.vpcflow.otel` | VPC Flow Log records | +| ELB Access Logs | `aws.elbaccess.otel` | ELB Access Log records (ALB, NLB, CLB) | + +The logs are produced in OpenTelemetry native format. For detailed information about the field mappings and structure of each log type, refer to the following documentation: + +- **VPC Flow Logs**: See [VPC Flow Log record fields](https://github.com/occamshub-dev/opentelemetry-collector-contrib/blob/main/extension/encoding/awslogsencodingextension/README.md#vpc-flow-log-record-fields) for the complete field mapping. +- **ELB Access Logs**: See [ELB Access Log fields](https://github.com/occamshub-dev/opentelemetry-collector-contrib/blob/main/extension/encoding/awslogsencodingextension/README.md#elb-access-log-fields) for the complete field mapping. + ## Kibana integration setup -After {{edot-cf}} for AWS is successfully running and forwarding logs to Elastic Observability, install the {{kib}} integrations to visualize your data with out-of-the-box dashboards and visualizations. +After {{edot-cf}} for AWS is successfully running and forwarding logs to {{product.observability}}, install the {{kib}} integrations to visualize your data with out-of-the-box dashboards and visualizations. To set up data visualization in {{kib}}: -1.Log into your Elastic Cloud deployment and open Kibana. -2. Go to **Management** → **Integrations** in the Kibana navigation menu. +1.Log into your {{ecloud}} deployment and open {{kib}} +2. Go to **Management** → **Integrations** in the {{kib}} navigation menu. 3. Search for the appropriate integration based on your log type and install it: | **AWS Log Type** | **Integration Name** | **Description** | |------------------|---------------------|-----------------| | ELB Access Logs | **AWS ELB OpenTelemetry Assets** | Dashboards and visualizations for Elastic Load Balancer logs | | VPC Flow Logs | **AWS VPC Flow Logs OpenTelemetry Assets** | Dashboards and visualizations for VPC flow log data | +| CloudTrail Logs | **AWS CloudTrail Logs OpenTelemetry Assets** | Dashboards and visualizations for CloudTrail log data | 4. Once installed, navigate to **Dashboard** to view the pre-built dashboards for your AWS log data. @@ -523,31 +600,28 @@ With AWS CLI, you can use `--timeout` to increase currently configured Lambda ti However, if a timeout occurs, you need to run the custom event multiple times to fully process all error events from the bucket. ::: -## Delete a CloudFormation stack +## Remove a CloudFormation stack -If you no longer need a deployed stack and want to clean up all associated resources, you can delete it using the following command: +If you no longer need a deployed stack and want to clean up all associated resources, you can remove it using either the AWS CLI or the AWS Console. -```sh -aws cloudformation delete-stack \ - --stack-name \ - --region -``` +### Important considerations -Deleting a stack will remove all AWS resources created by that stack. However, if you allowed the stack to automatically create a dedicated S3 bucket for failed Lambda invocations, that bucket will not be deleted if it contains objects, because CloudFormation doesn't force-delete non-empty buckets. To remove the bucket entirely, you must empty it manually before deleting it. +Deleting a stack removes all AWS resources created by that stack. However: -If you specified an existing bucket through the `S3FailureBucketARN` parameter instead, that bucket will not be deleted because it is not managed by the stack. +- If you allowed the stack to automatically create a dedicated S3 bucket for failed Lambda invocations, that bucket is not removed if it contains objects, because CloudFormation doesn't force-remove non-empty buckets. To remove the bucket entirely, you must empty it manually before deleting it. +- If you specified an existing bucket through the `S3FailureBucketARN` parameter, that bucket is not removed because it is not managed by the stack. -:::{dropdown} Example: Deleting a stack +### Remove using AWS CLI -The following command deletes the `edot-cloud-forwarder-vpc` stack: +Use the following command to remove a stack: ```sh -aws cloudformation delete-stack \ - --stack-name edot-cloud-forwarder-vpc \ - --region eu-central-1 +aws cloudformation remove-stack \ + --stack-name \ + --region ``` -You can monitor the deletion progress in the AWS Console under **CloudFormation** → **Stacks**, or through this command: +You can monitor the deletion progress through this command: ```sh aws cloudformation describe-stacks \ @@ -558,11 +632,57 @@ aws cloudformation describe-stacks \ If the stack deletion fails and remains in a `DELETE_FAILED` state, you can retry the deletion with force mode: ```sh -aws cloudformation delete-stack \ - --stack-name edot-cloud-forwarder-vpc \ - --region eu-central-1 \ +aws cloudformation remove-stack \ + --stack-name \ + --region \ --deletion-mode FORCE_DELETE_STACK ``` -This forcibly deletes the stack's resources, except any that cannot be deleted, like the failure S3 bucket if it still contains objects. For a complete cleanup, empty the bucket manually before retrying deletion. +This forcibly removes the stack's resources, except any that cannot be removed, like the failure S3 bucket if it still contains objects. For a complete cleanup, empty the bucket manually before retrying deletion. + +:::{dropdown} Example: Deleting a stack using AWS CLI + +The following command removes the `edot-cloud-forwarder-vpc` stack: + +```sh +aws cloudformation remove-stack \ + --stack-name edot-cloud-forwarder-vpc \ + --region eu-central-1 +``` + +Monitor the deletion progress: + +```sh +aws cloudformation describe-stacks \ + --stack-name edot-cloud-forwarder-vpc \ + --region eu-central-1 +``` + ::: + +### Remove using AWS Console + +To remove a stack using the AWS Management Console: + +1. Navigate to **CloudFormation** in the AWS Management Console. +2. Select the stack you want to remove from the list. +3. Click **Remove** at the top of the stack details page. +4. Monitor the deletion progress on the **Events** tab or wait until the stack disappears from the stack list (indicating deletion is complete). + +## Monitoring and troubleshooting + +To monitor your {{edot-cf}} Lambda function performance and troubleshoot issues: + +1. **CloudWatch Metrics Explorer**: View Lambda metrics such as: + - Duration + - ConcurrentExecutions + - Errors + - Throttles + - Invocations + +2. **CloudWatch Logs**: Check the Lambda function logs for: + - Processing errors + - Configuration issues + - Data export failures + +The `LambdaLogGroup` resource created by the CloudFormation stack stores all Lambda execution logs. Look for error messages or warnings that indicate configuration or performance issues. diff --git a/docs/reference/edot-cloud-forwarder/azure.md b/docs/reference/edot-cloud-forwarder/azure.md index e917deae..2a87c75c 100644 --- a/docs/reference/edot-cloud-forwarder/azure.md +++ b/docs/reference/edot-cloud-forwarder/azure.md @@ -89,7 +89,7 @@ To find out the URL of the managed OTLP endpoint and the API key for authenticat ::::{dropdown} Steps to retrieve the OTLP endpoint and API key -:::{include} ../_snippets/serverless-endpoint-api.md +:::{include} ../_snippets/find-motlp-endpoint.md ::: In the Bicep templates, the OTLP endpoint is set as `elasticsearchOtlpEndpoint`, and the API key is set as `elasticsearchApiKey`. diff --git a/docs/reference/motlp.md b/docs/reference/motlp.md index cf432316..70b7444d 100644 --- a/docs/reference/motlp.md +++ b/docs/reference/motlp.md @@ -57,25 +57,7 @@ To send data to Elastic through the {{motlp}}, follow the [Send data to the Elas ### Find your {{motlp}} endpoint -To retrieve your {{motlp}} endpoint address, follow these steps: - -::::{applies-switch} -:::{applies-item} serverless: -1. In {{ecloud}}, select the project name and then select **Manage project**. -2. Locate the **Connection alias** and select **Edit**. -3. Copy the Managed OTLP endpoint URL. -::: - -:::{applies-item} ess: -{applies_to}`stack: preview 9.2` -1. In {{ecloud}}, select **Add data** and then select **Application: OpenTelemetry** -2. Copy the Managed OTLP endpoint URL from `OTEL_EXPORTER_OTLP_ENDPOINT`. You can use this endpoint for both OpenTelemetry collectors and SDKs. -::: -:::: - -:::{note} -:applies_to: { ess:, stack: preview 9.2 } -The Managed OTLP endpoint might not be available in all {{ech}} regions during the Technical Preview. +:::{include} _snippets/find-motlp-endpoint.md ::: ### Configure SDKs to send data directly