Control Plane Logging as a default in all patterns (#129)

* Added documentation on how control plane logs can be accessed and made documentation updates for all patterns * Removed upstream testing dependencies * Made requested changes; tactfully edited documentation and ensured for all existing deployments, enabling control plane logging remains a user choice. * Resolved comments from @elamaras about working around adding .enableControlPlaneLogging()
aws-observability · Dec 19, 2023 · 69f19cf · 69f19cf
1 parent 3b9c55d
commit 69f19cf
Show file tree

Hide file tree

Showing 24 changed files with 94 additions and 8 deletions.
diff --git a/docs/logs.md b/docs/logs.md
@@ -18,6 +18,13 @@ Then you can view the results of your query:
 
 ![logs-results](./patterns/images/mixed-log-results.png)
 
+You can also find control plane logs in CloudWatch with Control Plane logs visible under the `/cluster` log group:
+
+![control-plane-logs](./patterns/images/control-plane-logs.png)
+
+You can then filter by patterns within the logs to find specific events like so:
+![control-plane-insights](./patterns/images/control-plane-insights.png)
+
 ## Viewing Logs in Grafana
 
 ### Using CloudWatch Logs as data source in Grafana

diff --git a/...existing-eks-observability-accelerators/existing-eks-apiserver-observability.md b/...existing-eks-observability-accelerators/existing-eks-apiserver-observability.md
@@ -20,6 +20,17 @@ You will also need:
 
 ## Deploying
 
+!!! note If control plane logging is not enabled in the existing cluster, edit 
+`lib/existing-eks-opensource-observability-pattern/index.ts` to include `.enableControlPlaneLogging()` as shown below:
+```typescript
+    ObservabilityBuilder.builder()
+        // some properties
+        .enableControlPlaneLogging()
+        // other properties
+        .build(scope, stackId);
+```
+
+
 1. Edit `~/.cdk.json` by setting the name of your existing cluster:
 
 ```json

diff --git a/...existing-eks-observability-accelerators/existing-eks-awsnative-observability.md b/...existing-eks-observability-accelerators/existing-eks-awsnative-observability.md
@@ -31,6 +31,16 @@ You will also need:
 2. An OpenID Connect (OIDC) provider, associated to the above EKS cluster (Note: Single EKS Cluster Pattern takes care of that for you)
 
 ## Deploying
+!!! note If control plane logging is not enabled in the existing cluster, edit 
+`lib/existing-eks-awsnative-observability-pattern/index.ts` to include `.enableControlPlaneLogging()` as shown below:
+```typescript
+    ObservabilityBuilder.builder()
+        // some properties
+        .enableControlPlaneLogging()
+        // other properties
+        .build(scope, stackId);
+```
+
 
 1. Edit `~/.cdk.json` by setting the name of your existing cluster:
 

diff --git a/...rns/existing-eks-observability-accelerators/existing-eks-mixed-observability.md b/...rns/existing-eks-observability-accelerators/existing-eks-mixed-observability.md
@@ -2,19 +2,29 @@
 
 ## Architecture
 
-The following figure illustrates the architecture of the pattern we will be deploying for Existing EKS Cluster AWS Mixed Observability pattern, using AWS native tools such as CloudWatch and X-Ray and Open Source tools such as AWS Distro for OpenTelemetry (ADOT) and Prometheus Node Exporter.
+The following figure illustrates the architecture of the pattern we will be deploying for Existing EKS Cluster AWS Mixed
+Observability pattern, using AWS native tools such as CloudWatch and X-Ray and Open Source tools such as AWS Distro for
+OpenTelemetry (ADOT) and Prometheus Node Exporter.
 
 ![Architecture](../images/mixed-diagram.png)
 
-This example makes use of CloudWatch, as a metric and log aggregation layer, while X-Ray is used as a trace-aggregation layer. In order to collect the metrics and traces, we use the Open Source ADOT collector. Fluent Bit is used to export the logs to CloudWatch Logs.
+This example makes use of CloudWatch, as a metric and log aggregation layer, while X-Ray is used as a trace-aggregation
+layer. In order to collect the metrics and traces, we use the Open Source ADOT collector. Fluent Bit is used to export
+the logs to CloudWatch Logs.
 
-In this architecture, AWS X-Ray provides a complete view of requests as they travel through your application and filters visual data across payloads, functions, traces, services, and APIs. X-Ray also allows you to perform analytics, to gain powerful insights about your distributed trace data.
+In this architecture, AWS X-Ray provides a complete view of requests as they travel through your application and filters
+visual data across payloads, functions, traces, services, and APIs. X-Ray also allows you to perform analytics, to gain
+powerful insights about your distributed trace data.
 
-Utilizing CloudWatch and X-Ray as an aggregation layer allows for a fully-managed scalable telemetry backend. In this example we get those benefits while still having the flexibility and rapid development of the Open Source collection tools.
+Utilizing CloudWatch and X-Ray as an aggregation layer allows for a fully-managed scalable telemetry backend. In this
+example we get those benefits while still having the flexibility and rapid development of the Open Source collection
+tools.
 
 ## Objective
 
-This pattern aims to add Observability on top of an existing EKS cluster, with a mixture of AWS native and open source managed AWS services.
+
+This pattern aims to add Observability on top of an existing EKS cluster, with a mixture of AWS native and open source
+managed AWS services.
 
 ## Prerequisites:
 
@@ -32,6 +42,16 @@ You will also need:
 
 ## Deploying
 
+!!! note If control plane logging is not enabled in the existing cluster, edit `lib/existing-eks-mixed-observability-pattern/index.ts` to include `.enableControlPlaneLogging()` as shown below:
+```typescript
+    ObservabilityBuilder.builder()
+        // some properties
+        .enableControlPlaneLogging()
+        // other properties
+        .build(scope, stackId);
+```
+
+
 1. Edit `~/.cdk.json` by setting the name of your existing cluster:
 
 ```json

diff --git a/...rns/existing-eks-observability-accelerators/existing-eks-nginx-observability.md b/...rns/existing-eks-observability-accelerators/existing-eks-nginx-observability.md
@@ -28,6 +28,16 @@ You will also need:
 
 ## Deploying
 
+!!! note If control plane logging is not enabled in the existing cluster, edit 
+`lib/existing-eks-opensource-observability-pattern/index.ts` to include `.enableControlPlaneLogging()` as shown below:
+```typescript
+    ObservabilityBuilder.builder()
+        // some properties
+        .enableControlPlaneLogging()
+        // other properties
+        .build(scope, stackId);
+```
+
 1. Edit `~/.cdk.json` by setting the name of your existing cluster:
 
 ```json

diff --git a/...xisting-eks-observability-accelerators/existing-eks-opensource-observability.md b/...xisting-eks-observability-accelerators/existing-eks-opensource-observability.md
@@ -36,6 +36,16 @@ You will also need:
 
 ## Deploying
 
+!!! note If control plane logging is not enabled in the existing cluster, edit 
+`lib/existing-eks-opensource-observability-pattern/index.ts` to include `.enableControlPlaneLogging()` as shown below:
+```typescript
+    ObservabilityBuilder.builder()
+        // some properties
+        .enableControlPlaneLogging()
+        // other properties
+        .build(scope, stackId);
+```
+
 1. Edit `~/.cdk.json` by setting the name of your existing cluster:
 
 ```json

diff --git a/docs/patterns/images/control-plane-insights.png b/docs/patterns/images/control-plane-insights.png
diff --git a/docs/patterns/images/control-plane-logs.png b/docs/patterns/images/control-plane-logs.png
diff --git a/...lti-new-eks-observability-accelerators/multi-acc-new-eks-mixed-observability.md b/...lti-new-eks-observability-accelerators/multi-acc-new-eks-mixed-observability.md
@@ -8,7 +8,7 @@ The following figure illustrates the architecture of the pattern we will be depl
 
 ## Objective
 
-1. Deploying two production grade Amazon EKS cluster across two AWS Accounts (Prod1, Prod2 account) in two different regions through a Continuous Deployment infrastructure pipeline triggered upon a commit to the repository that holds the pipeline configuration in another AWS account (pipeline account).
+1. Deploying two production grade Amazon EKS cluster with control plane logging across two AWS Accounts (Prod1, Prod2 account) in two different regions through a Continuous Deployment infrastructure pipeline triggered upon a commit to the repository that holds the pipeline configuration in another AWS account (pipeline account).
 2. Deploying ADOT add-on, AMP add-on to Prod 1 Amazon EKS Cluster to remote-write metrics to AMP workspace in Prod 1 AWS Account.
 3. Deploying ADOT add-on, CloudWatch add-on to Prod 2 Amazon EKS Cluster to write metrics to CloudWatch in Prod 2 AWS Account.
 4. Configuring GitOps tooling (Argo CD add-on) to support deployment of [ho11y](https://github.com/aws-observability/aws-o11y-recipes/tree/main/sandbox/ho11y) and [yelb](https://github.com/mreferre/yelb) sample applications, in a way that restricts each application to be deployed only into the team namespace, by using Argo CD projects.

diff --git a/...ks-observability-accelerators/single-new-eks-awsnative-fargate-observability.md b/...ks-observability-accelerators/single-new-eks-awsnative-fargate-observability.md
@@ -20,7 +20,8 @@ please read the [service documentation](https://docs.aws.amazon.com/eks/latest/u
 
 ## Objective
 
-- Deploys one production grade Amazon EKS Fargate cluster.
+- Deploys one production grade Amazon EKS Fargate cluster
+- Enables Control Plane Logging. 
 - Logs with CloudWatch Logs
 - Enables CloudWatch Container Insights.
 - Installs Prometheus Node Exporter and Metrics Server for infrastructure metrics.

diff --git a/...le-new-eks-observability-accelerators/single-new-eks-awsnative-observability.md b/...le-new-eks-observability-accelerators/single-new-eks-awsnative-observability.md
@@ -18,6 +18,7 @@ please read the [service documentation](https://docs.aws.amazon.com/eks/latest/u
 ## Objective
 
 - Deploys one production grade Amazon EKS cluster.
+- Enables Control Plane Logging.
 - AWS Distro For OpenTelemetry Operator and Collector
 - Logs with [AWS for FluentBit](https://github.com/aws/aws-for-fluent-bit) and CloudWatch Logs
 - Enables CloudWatch Container Insights.

diff --git a/...w-eks-observability-accelerators/single-new-eks-gpu-opensource-observability.md b/...w-eks-observability-accelerators/single-new-eks-gpu-opensource-observability.md
@@ -12,7 +12,7 @@ Amazon Managed Grafana is a managed service for Grafana, a popular open-source a
 
 ## Objective
 
-This pattern deploys an Amazon EKS cluster with a node group that includes instance types featuring NVIDIA GPUs.
+This pattern deploys an Amazon EKS cluster and a node group that includes instance types featuring NVIDIA GPUs.
 
 The AMI type of the node group is `AL2_x86_64_GPU AMI`, which uses the [Amazon EKS-optimized Linux AMI with GPU support](https://aws.amazon.com/marketplace/pp/prodview-nwwwodawoxndm). In addition to the standard Amazon EKS-optimized AMI configuration, the GPU AMI includes the NVIDIA drivers.
 
@@ -25,6 +25,8 @@ Data is visualised in Amazon Managed Grafana by the [NVIDIA DCGM Exporter Dashbo
 
 The rest of the setup to collect and visualise metrics with Amazon Managed Service for Prometheus and Amazon Managed Grafana, is similar to that used in other open-source based patterns included in this repository.
 
+It also enables control plane logging on the EKS cluster for a comprehensive overview of cluster health.
+
 ## Prerequisites:
 
 Ensure that you have installed the following tools on your machine.

diff --git a/...-observability-accelerators/single-new-eks-graviton-opensource-observability.md b/...-observability-accelerators/single-new-eks-graviton-opensource-observability.md
@@ -21,6 +21,7 @@ Visit our [EKS Blueprints docs](https://github.com/aws-quickstart/cdk-eks-bluepr
 ## Objective
 
 - Deploys one production grade Amazon EKS cluster running on a Graviton3 Processor
+- Enables control plane logging
 - AWS Distro For OpenTelemetry Operator and Collector for Metrics and Traces
 - Logs with [AWS for FluentBit](https://github.com/aws/aws-for-fluent-bit)
 - Installs Grafana Operator to add AWS data sources and create Grafana Dashboards to Amazon Managed Grafana.

diff --git a/...-eks-observability-accelerators/single-new-eks-java-opensource-observability.md b/...-eks-observability-accelerators/single-new-eks-java-opensource-observability.md
@@ -4,6 +4,8 @@
 
 This pattern demonstrates how to use the _New EKS Cluster Open Source Observability Accelerator_ with Java based workloads.
 
+It also enables control plane logging for comprehensive monitoring on the EKS cluster.
+
 ## Prerequisites
 
 Ensure that you have installed the following tools on your machine.

diff --git a/...single-new-eks-observability-accelerators/single-new-eks-mixed-observability.md b/...single-new-eks-observability-accelerators/single-new-eks-mixed-observability.md
@@ -15,6 +15,7 @@ Utilizing CloudWatch and X-Ray as an aggregation layer allows for a fully-manage
 ## Objective
 
 - Deploys one production grade Amazon EKS cluster.
+- Enables Control Plane logging
 - AWS Distro For OpenTelemetry Operator and Collector configured to collect metrics and traces.
 - Logs with [AWS for FluentBit](https://github.com/aws/aws-for-fluent-bit) and CloudWatch Logs
 - Aggregate Metrics in CloudWatch

diff --git a/...eks-observability-accelerators/single-new-eks-nginx-opensource-observability.md b/...eks-observability-accelerators/single-new-eks-nginx-opensource-observability.md
@@ -4,6 +4,8 @@
 
 This pattern demonstrates how to use the _New EKS Cluster Open Source Observability Accelerator_ with Nginx based workloads.
 
+It also enables control plane logging to provide comprehensive overview of cluster health.
+
 ## Prerequisites
 
 Ensure that you have installed the following tools on your machine.

diff --git a/...e-new-eks-observability-accelerators/single-new-eks-opensource-observability.md b/...e-new-eks-observability-accelerators/single-new-eks-opensource-observability.md
@@ -15,6 +15,7 @@ please read the [service documentation](https://docs.aws.amazon.com/eks/latest/u
 ## Objective
 
 - Deploys one production grade Amazon EKS cluster.
+- Enables Control Plane logging.
 - AWS Distro For OpenTelemetry Operator and Collector for Metrics and Traces
 - Logs with [AWS for FluentBit](https://github.com/aws/aws-for-fluent-bit)
 - Installs Grafana Operator to add AWS data sources and create Grafana Dashboards to Amazon Managed Grafana.

diff --git a/lib/single-new-eks-awsnative-fargate-observability-construct/index.ts b/lib/single-new-eks-awsnative-fargate-observability-construct/index.ts
@@ -64,6 +64,7 @@ export default class SingleNewEksAWSNativeFargateobservabilityConstruct {
             .withCertManagerProps(certManagerAddOnProps)
             .withCoreDnsProps(coreDnsAddOnProps)
             .enableFargatePatternAddOns()
+            .enableControlPlaneLogging()
             .clusterProvider(fargateClusterProvider)
             .addOns(...addOns)
             .build(scope, stackId);

diff --git a/lib/single-new-eks-awsnative-observability-pattern/index.ts b/lib/single-new-eks-awsnative-observability-pattern/index.ts
@@ -18,6 +18,7 @@ export default class SingleNewEksClusterAWSNativeobservabilityPattern {
             .region(region)
             .version('auto')
             .enableNativePatternAddOns()
+            .enableControlPlaneLogging()
             .addOns(...addOns)
             .build(scope, stackId);
     }

diff --git a/lib/single-new-eks-cluster-pattern/index.ts b/lib/single-new-eks-cluster-pattern/index.ts
@@ -17,6 +17,7 @@ export default class SingleNewEksPattern {
             .account(account)
             .region(region)
             .version('auto')
+            .enableControlPlaneLogging()
             .addOns(...addOns)
             .build(scope, stackId);
     }

diff --git a/lib/single-new-eks-mixed-observability-pattern/index.ts b/lib/single-new-eks-mixed-observability-pattern/index.ts
@@ -32,6 +32,7 @@ export default class SingleNewEksMixedobservabilityPattern {
             .region(region)
             .version('auto')
             .enableMixedPatternAddOns()
+            .enableControlPlaneLogging()
             .addOns(...addOns)
             .build(scope, stackId);
     }

diff --git a/lib/single-new-eks-opensource-observability-pattern/gpu-index.ts b/lib/single-new-eks-opensource-observability-pattern/gpu-index.ts
@@ -77,6 +77,7 @@ export default class SingleNewEksGpuOpenSourceObservabilityPattern {
             .version('auto')
             .withAmpProps(ampAddOnProps)
             .enableOpenSourcePatternAddOns()
+            .enableControlPlaneLogging()
             .resourceProvider(ampWorkspaceName, new blueprints.CreateAmpProvider(ampWorkspaceName, ampWorkspaceName))
             .clusterProvider(
                 new blueprints.GenericClusterProvider({

diff --git a/lib/single-new-eks-opensource-observability-pattern/graviton-index.ts b/lib/single-new-eks-opensource-observability-pattern/graviton-index.ts
@@ -106,6 +106,7 @@ export default class SingleNewEksGravitonOpenSourceObservabilityPattern {
             .version('auto')
             .withAmpProps(ampAddOnProps)
             .enableOpenSourcePatternAddOns()
+            .enableControlPlaneLogging()
             .resourceProvider(ampWorkspaceName, new blueprints.CreateAmpProvider(ampWorkspaceName, ampWorkspaceName))
             .clusterProvider(new blueprints.MngClusterProvider(mngProps))
             .addOns(...addOns)

diff --git a/lib/single-new-eks-opensource-observability-pattern/index.ts b/lib/single-new-eks-opensource-observability-pattern/index.ts
@@ -115,6 +115,7 @@ export default class SingleNewEksOpenSourceobservabilityPattern {
             .version('auto')
             .withAmpProps(ampAddOnProps)
             .enableOpenSourcePatternAddOns()
+            .enableControlPlaneLogging()
             .addOns(...addOns)
             .build(scope, stackId);
     }