Skip to content

Commit

Permalink
Merge branch 'main' into main
Browse files Browse the repository at this point in the history
  • Loading branch information
@bonclay7
bonclay7 committed Jan 31, 2024
2 parents ef483f9 + ada16d5 commit e5aee6e
Show file tree
Hide file tree
Showing 49 changed files with 503 additions and 840 deletions.
190 changes: 17 additions & 173 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,140 +11,38 @@ AWS-managed observability services such as Amazon Managed Service for Prometheus
Amazon Managed Grafana, AWS Distro for OpenTelemetry (ADOT) and Amazon CloudWatch.

We provide curated metrics, logs, traces collection, alerting rules and Grafana
dashboards for your EKS infrastructure, Java/JMX, NGINX based workloads and
your custom applications.

You also can monitor your Amazon Managed Service for Prometheus workspaces ingestion,
costs, active series with [this module](./modules/managed-prometheus-monitoring).
dashboards for your AWS infrastructure and custom applications.

![image](https://github.com/aws-observability/terraform-aws-observability-accelerator/assets/10175027/e83f8709-f754-4192-90f2-e3de96d2e26c)


## Documentation

To explore the complete project documentation, please visit our [documentation site.](https://aws-observability.github.io/terraform-aws-observability-accelerator/)

## ⚠️ Dropping base module

## ⚠️ Migration to v2.5

If you are migrating from earlier versions to v2.5, please follow this guide.

v2.5.0 removes the dependency to the Terraform Grafana provider in the EKS
monitoring module. As Grafana Operator manages and syncs the Grafana contents,
Terraform is not required anymore in this context.

However, if you migrate from earlier versions, you might leave some data orphans
as the Grafana provider is dropped. Terraform will throw an error. We have
released [v2.5.0-rc.1](https://github.com/aws-observability/terraform-aws-observability-accelerator/releases/tag/v2.5.0-rc.1)
which removes all the Grafana resources provisioned by Terraform in the EKS
context, without removing the provider configurations.

- Step 1: migrate to [v2.5.0-rc.1](https://github.com/aws-observability/terraform-aws-observability-accelerator/releases/tag/v2.5.0-rc.1)
and run `apply`
- Step 2: migrate to `v2.5.0` or above


## Getting started

To quick start with a complete workflow and view Amazon EKS infrastructure dashboards,
visit the [Amazon EKS cluster monitoring documentation](https://aws-observability.github.io/terraform-aws-observability-accelerator/eks/)

## How it works

The sections below demonstrate how you can leverage AWS Observability Accelerator
to enable monitoring to an existing EKS cluster.


### Base Module

The base module allows you to configure the AWS Observability services for your
cluster and the AWS Distro for OpenTelemetry (ADOT) Operator as the signals
collection mechanism.

This is the minimum configuration to have a new Amazon Managed Service for
Prometheus Workspace and ADOT Operator deployed for you and ready to receive
your data. The base module serve as an anchor to the workload modules and
cannot run on its own.

```hcl
module "aws_observability_accelerator" {
# use release tags and check for the latest versions
# https://github.com/aws-observability/terraform-aws-observability-accelerator/releases
source = "github.com/aws-observability/terraform-aws-observability-accelerator?ref=v2.1.0"
aws_region = "eu-west-1"
eks_cluster_id = "my-eks-cluster"
# As Grafana shares a different lifecycle, we recommend using an existing workspace.
managed_grafana_workspace_id = var.managed_grafana_workspace_id
}
```

You can optionally reuse an existing Amazon Managed Servce for Prometheus Workspace:

```hcl
module "aws_observability_accelerator" {
# use release tags and check for the latest versions
# https://github.com/aws-observability/terraform-aws-observability-accelerator/releases
source = "github.com/aws-observability/terraform-aws-observability-accelerator?ref=v2.1.0"
Starting in v2.12.0, we have removed the [base module](https://github.com/aws-observability/terraform-aws-observability-accelerator/blob/v2.10.3/docs/concepts.md#base-module)
which previously served as a glue between modules. However, the modules were quite
independent and the base module provided limited functionality.

aws_region = "eu-west-1"
eks_cluster_id = "my-eks-cluster"
We have moved the creation of Managed Prometheus Workspaces into the
[eks-monitoring module](https://github.com/aws-observability/terraform-aws-observability-accelerator/tree/main/modules/eks-monitoring).

# prevents creation of a new Amazon Managed Prometheus workspace
enable_managed_prometheus = false
See our [examples](https://github.com/aws-observability/terraform-aws-observability-accelerator/tree/main/examples)
to understand the changes.

# reusing existing Amazon Managed Prometheus Workspace
managed_prometheus_workspace_id = "ws-abcd123..."
managed_grafana_workspace_id = "g-abcdef123"
}
```

View all the configuration options in the module documentation below.

### Workload modules

[Workloads modules](./modules) are provided, which essentially provide curated
metrics, logs, traces collection, alerting rules and Grafana dashboards.

#### Amazon EKS monitoring

```hcl
module "eks_monitoring" {
source = "github.com/aws-observability/terraform-aws-observability-accelerator//modules/eks-monitoring?ref=v2.1.0"
eks_cluster_id = module.eks_observability_accelerator.eks_cluster_id
dashboards_folder_id = module.eks_observability_accelerator.grafana_dashboards_folder_id
managed_prometheus_workspace_id = module.eks_observability_accelerator.managed_prometheus_workspace_id
managed_prometheus_workspace_endpoint = module.eks_observability_accelerator.managed_prometheus_workspace_endpoint
managed_prometheus_workspace_region = module.eks_observability_accelerator.managed_prometheus_workspace_region
enable_logs = true
enable_tracing = true
}
```
## Modules

#### Amazon ECS monitoring
ECS cluster with VPC and EC2 can be created using the example [here](./examples/ecs_cluster_with_vpc)
This accelerator provides the following observability modules:

```hcl
module "ecs_monitoring" {
source = "github.com/aws-observability/terraform-aws-observability-accelerator//modules/ecs-monitoring"
- [EKS Monitoring with AWS-Managed Open Source](https://aws-observability.github.io/terraform-aws-observability-accelerator/eks/) - Get Prometheus metrics, CloudWatch logs collection, and X-Ray traces (with OTLP support) for your EKS cluster. Visualize key metrics and logs with provided Grafana dashboards and get pre-built alerting rules.

aws_ecs_cluster_name = module.ecs_cluster.cluster_name
task_role_arn = module.ecs_cluster.task_exec_iam_role_arn
execution_role_arn = module.ecs_cluster.task_exec_iam_role_arn
}
```
Grafana Dashboards
- [EKS Monitoring with Enhanced CloudWatch Container Insights](https://aws-observability.github.io/terraform-aws-observability-accelerator/container-insights/eks/) - Get deep visibility into EKS using Amazon CloudWatch for metrics collection, aggregation, and insights summaries. Includes support for [CloudWatch Application Signals (preview)](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Application-Monitoring-Sections.html)

<img width="2056" alt="image" src="https://user-images.githubusercontent.com/10175027/199110753-9bc7a9b7-1b45-4598-89d3-32980154080e.png">
- [ECS Monitoring on EC2 with AWS-Managed Open Source](https://aws-observability.github.io/terraform-aws-observability-accelerator/ecs/ecs-monitoring-on-ec2/) - Collect metrics, traces, and logs for ECS on EC2 and send them to a Managed Prometheus workspace , X-Ray, and CloudWatch Logs. Includes pre-built Grafana dashboards for key metrics and logs.

- [Managed Prometheus Monitoring](https://aws-observability.github.io/terraform-aws-observability-accelerator/workloads/managed-prometheus/) - This module sets up automated observability for Amazon Managed Service for Prometheus workspaces, including a Grafana dashboard, CloudWatch monitoring, and service quota alarms.

Check the the [complete example](./examples/existing-cluster-with-base-and-infra/)

## Motivation

Expand All @@ -156,9 +54,8 @@ Grafana and Amazon OpenSearch.
AWS customers have asked for best-practices and guidance to collect metrics, logs
and traces from their containerized applications and microservices with ease of
deployment. Customers can use the AWS Observability Accelerator to configure their
metrics and traces collection, leveraging [AWS Distro for OpenTelemetry](https://aws-otel.github.io/),
to have opinionated dashboards and alerts available in only minutes.

metrics and traces collection to have opinionated dashboards and alerts
available in only minutes.

## Support & Feedback

Expand All @@ -173,59 +70,6 @@ section of this GitHub repo.
If you are interested in contributing, see the
[Contribution guide](https://github.com/aws-observability/terraform-aws-observability-accelerator/blob/main/CONTRIBUTING.md).

---

<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
## Requirements

| Name | Version |
|------|---------|
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.1.0 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 4.0.0 |
| <a name="requirement_awscc"></a> [awscc](#requirement\_awscc) | >= 0.24.0 |

## Providers

| Name | Version |
|------|---------|
| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 4.0.0 |

## Modules

No modules.

## Resources

| Name | Type |
|------|------|
| [aws_prometheus_alert_manager_definition.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/prometheus_alert_manager_definition) | resource |
| [aws_prometheus_workspace.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/prometheus_workspace) | resource |
| [aws_grafana_workspace.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/grafana_workspace) | data source |
| [aws_region.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/region) | data source |

## Inputs

| Name | Description | Type | Default | Required |
|------|-------------|------|---------|:--------:|
| <a name="input_aws_region"></a> [aws\_region](#input\_aws\_region) | AWS Region | `string` | n/a | yes |
| <a name="input_enable_alertmanager"></a> [enable\_alertmanager](#input\_enable\_alertmanager) | Creates Amazon Managed Service for Prometheus AlertManager for all workloads | `bool` | `false` | no |
| <a name="input_enable_managed_prometheus"></a> [enable\_managed\_prometheus](#input\_enable\_managed\_prometheus) | Creates a new Amazon Managed Service for Prometheus Workspace | `bool` | `true` | no |
| <a name="input_managed_grafana_workspace_id"></a> [managed\_grafana\_workspace\_id](#input\_managed\_grafana\_workspace\_id) | Amazon Managed Grafana Workspace ID | `string` | n/a | yes |
| <a name="input_managed_prometheus_workspace_id"></a> [managed\_prometheus\_workspace\_id](#input\_managed\_prometheus\_workspace\_id) | Amazon Managed Service for Prometheus Workspace ID | `string` | `""` | no |
| <a name="input_managed_prometheus_workspace_region"></a> [managed\_prometheus\_workspace\_region](#input\_managed\_prometheus\_workspace\_region) | Region where Amazon Managed Service for Prometheus is deployed | `string` | `null` | no |
| <a name="input_tags"></a> [tags](#input\_tags) | Additional tags (e.g. `map('BusinessUnit`,`XYZ`) | `map(string)` | `{}` | no |

## Outputs

| Name | Description |
|------|-------------|
| <a name="output_aws_region"></a> [aws\_region](#output\_aws\_region) | AWS Region |
| <a name="output_managed_grafana_workspace_endpoint"></a> [managed\_grafana\_workspace\_endpoint](#output\_managed\_grafana\_workspace\_endpoint) | Amazon Managed Grafana workspace endpoint |
| <a name="output_managed_prometheus_workspace_endpoint"></a> [managed\_prometheus\_workspace\_endpoint](#output\_managed\_prometheus\_workspace\_endpoint) | Amazon Managed Prometheus workspace endpoint |
| <a name="output_managed_prometheus_workspace_id"></a> [managed\_prometheus\_workspace\_id](#output\_managed\_prometheus\_workspace\_id) | Amazon Managed Prometheus workspace ID |
| <a name="output_managed_prometheus_workspace_region"></a> [managed\_prometheus\_workspace\_region](#output\_managed\_prometheus\_workspace\_region) | Amazon Managed Prometheus workspace region |
<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->

## Contributing

See [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) for more information.
Expand Down
74 changes: 5 additions & 69 deletions docs/concepts.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,6 @@ We have setup a [GitRepository](https://fluxcd.io/flux/components/source/gitrepo
We have placed our declarative code snippet to create an Amazon Managed Service For Promethes datasource and Grafana Dashboard in Amazon Managed Grafana in our [AWS Observabiity Accelerator GitHub Repository](https://github.com/aws-observability/aws-observability-accelerator). We have setup a GitRepository to point to the AWS Observabiity Accelerator GitHub Repository and `Kustomization` for flux to sync Git Repository with artifacts in `./artifacts/grafana-operator-manifests/*` path in the AWS Observabiity Accelerator GitHub Repository. You can use this extension of our solution to point your own Kubernetes manifests to create Grafana Datasources and personified Grafana Dashboards of your choice using GitOps with Grafana Operator and Flux in Kubernetes native way with altering and redeploying this solution for changes to Grafana resources.



## Release notes

We encourage you to use our [release versions](https://github.com/aws-observability/terraform-aws-observability-accelerator/releases)
Expand All @@ -57,82 +56,19 @@ module "eks_monitoring" {
}
```

## Modules

## Base module

The base module allows you to configure the AWS Observability services for your cluster and
the AWS Distro for OpenTelemetry (ADOT) Operator as the signals collection mechanism.

This is the minimum configuration to have a new Amazon Managed Service for Prometheus Workspace
and ADOT Operator deployed for you and ready to receive your data.
The base module serve as an anchor to the workload modules and cannot run on its own.

```hcl
module "aws_observability_accelerator" {
# use release tags and check for the latest versions
# https://github.com/aws-observability/terraform-aws-observability-accelerator/releases
source = "github.com/aws-observability/terraform-aws-observability-accelerator?ref=v1.6.1"
aws_region = "eu-west-1"
eks_cluster_id = "my-eks-cluster"
# As Grafana shares a different lifecycle, we recommend using an existing workspace.
managed_grafana_workspace_id = var.managed_grafana_workspace_id
}
```

You can optionally reuse an existing Amazon Managed Service for Prometheus Workspace:

```hcl
module "aws_observability_accelerator" {
# use release tags and check for the latest versions
# https://github.com/aws-observability/terraform-aws-observability-accelerator/releases
source = "github.com/aws-observability/terraform-aws-observability-accelerator?ref=v1.6.1"
aws_region = "eu-west-1"
eks_cluster_id = "my-eks-cluster"
# prevents creation of a new Amazon Managed Prometheus workspace
enable_managed_prometheus = false
# reusing existing Amazon Managed Prometheus Workspace
managed_prometheus_workspace_id = "ws-abcd123..."
managed_grafana_workspace_id = "g-abcdef123"
}
```

View all the configuration options in the [module's documentation](https://github.com/aws-observability/terraform-aws-observability-accelerator#requirements)

## Workload modules

Workloads modules are focused Terraform modules provided in this repository. They essentially provide curated metrics collection, alerts and Grafana dashboards according to the use case. Most of those modules require the base module.

You can check the full workload modules list and their documentation [here](https://github.com/aws-observability/terraform-aws-observability-accelerator/tree/main/modules/).

[Modules](https://github.com/aws-observability/terraform-aws-observability-accelerator/tree/main/modules/)
are set of functionalities (ex: Managed Open-Source EKS monitoring, CloudWatch Container Insights, ...)
packaged together that can be used to add Observability to your environments.
All the modules come with end-to-end deployable examples.

## Examples

[Examples](https://github.com/aws-observability/terraform-aws-observability-accelerator/tree/main/examples) put modules together in a ready to deploy terraform configuration as a starting point. With little to no configuration, you can run `terraform apply` and use the deployed resources on your AWS Account.
[Examples](https://github.com/aws-observability/terraform-aws-observability-accelerator/tree/main/examples) put [modules](https://github.com/aws-observability/terraform-aws-observability-accelerator/tree/main/modules/) together in a ready to deploy terraform configuration as a starting point. With little to no configuration, you can run `terraform apply` and use the deployed resources on your AWS Account.

You can find **workload** examples like [Amazon EKS infrastructure monitoring](https://aws-observability.github.io/terraform-aws-observability-accelerator/eks/) or [monitoring your Amazon Managed Service for Prometheus workspace](https://aws-observability.github.io/terraform-aws-observability-accelerator/workloads/managed-prometheus/) and more.

```mermaid
classDiagram
Example <|-- Base Module
Example <|-- Workload Module
class Base Module{
Amazon Managed Prometheus
Amazon Managed Grafana Data Sources
}
class Workload Module{
Amazon Distro for Open Telemetry Config
Amazon Managed Prometheus Alerts
Amazon Managed Grafana Dashboards
}
```


## Getting started with AWS Observability services

Expand Down
Loading

0 comments on commit e5aee6e

Please sign in to comment.