Support Monitoring DynamoDB.

.github/workflows/skywalking.yaml

@@ -672,6 +672,8 @@ jobs:
             config: test/e2e-v2/cases/win/e2e.yaml
           - name: AWS Cloud S3
             config: test/e2e-v2/cases/aws/s3/e2e.yaml
+          - name: AWS Cloud DynamoDB
+            config: test/e2e-v2/cases/aws/dynamodb/e2e.yaml
     steps:
       - uses: actions/checkout@v3
         with:

docs/en/changes/changes.md

@@ -99,6 +99,7 @@
 * Refactor http-based alarm plugins and extract common logic to `HttpAlarmCallback`.
 * Support Amazon Simple Storage Service (Amazon S3) metrics monitoring
 * Support process Sum metrics with AGGREGATION_TEMPORALITY_DELTA case
+* Support Amazon DynamoDB monitoring.
 
 #### UI
 

docs/en/setup/backend/aws-firehose-receiver.md

@@ -19,6 +19,8 @@ CloudWatch metrics with S3 -->  CloudWatch Metric Stream (OpenTelemetry formart)
 | Description                             | Configuration File                  | Data Source                                                                                                                                       |
 |-----------------------------------------|-------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------|
 | Metrics of AWS Cloud S3                 | otel-rules/aws-s3/s3-service.yaml   | AWS CloudWatcher Metrics Stream -> AWS Firehose delivery stream -> SkyWalking OAP Server with [AWS Firehose receiver](./aws-firehose-receiver.md) |
+| Metrics of AWS DynamoDB | otel-rules/aws-dynamodb/dynamodb-service.yaml  | AWS CloudWatcher Metrics Stream -> AWS Firehose delivery stream -> SkyWalking OAP Server with [AWS Firehose receiver](./aws-firehose-receiver.md) |
+| Metrics of AWS DynamoDB | otel-rules/aws-dynamodb/dynamodb-endpoint.yaml | AWS CloudWatcher Metrics Stream -> AWS Firehose delivery stream -> SkyWalking OAP Server with [AWS Firehose receiver](./aws-firehose-receiver.md) |
 
 ## Notice
 

docs/en/setup/backend/backend-aws-dynamodb-monitoring.md

@@ -0,0 +1,36 @@
+# AWS DynamoDb monitoring
+SkyWalking leverages Amazon Kinesis Data Filehose with [Amazon CloudWatch](https://aws.amazon.com/cn/cloudwatch/) to transfer the metrics into the [Meter System](./../../concepts-and-designs/meter.md).
+
+### Data flow
+1. Amazon CloudWatch fetches metrics from DynamoDB and pushes metrics to SkyWalking OAP Server via Amazon Kinesis data firehose.
+2. The SkyWalking OAP Server parses the expression with [MAL](../../concepts-and-designs/mal.md) to filter/calculate/aggregate and store the results.
+
+### Set up
+1. Create CloudWatch metrics configuration for DynamoDB, refer to [DynamoDB metrics configuration](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/monitoring-cloudwatch.html)
+   Create an Amazon Kinesis Data Firehose Delivery Stream, and set [AWS Kinesis Data Firehose receiver](./aws-firehose-receiver.md)'s address as HTTP(s) Destination, refer to [Create Delivery Stream](https://docs.aws.amazon.com/firehose/latest/dev/basic-create.html)3. Create a [metric stream](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Metric-Streams.html), set namespace to DynanoDB, and set `Kinesis Data Firehose` to the firehose you just created.
+2. Config [aws-firehose-receiver](aws-firehose-receiver.md) to receive data.
+3. Create CloudWatch metric stream, and select the Firehose Delivery Stream which has been created above, set `Select namespaces` to `AWS/DynamoDB`, `Select output format` to `OpenTelemetry 0.7`. refer to [CloudWatch Metric Streams](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Metric-Streams.html)
+
+### DynamoDB Monitoring
+DynamoDB monitoring provides monitoring of the status and resources of the DynamoDB server. AWS user id is cataloged as a `Layer: AWS_DYNAMODB` `Service` in OAP.
+Each DynamoDB table is cataloged as an `Endpoint` in OAP.
+
+#### Supported Metrics
+| Monitoring Panel | Unit | Metric Name | Description | Data Source |
+|-----|-----|-----|-----|--|
+| Read Usage |   unit/s  | consumed_read_capacity_units <br /> provisioned_read_capacity_units| The situation of read capacity units consumed and provisioned over the specified time period | Amazon CloudWatch |
+| Write Usage |   unit/s  | consumed_write_capacity_units <br /> provisioned_write_capacity_units| The situation of write capacity units consumed and provisioned over the specified time period | Amazon CloudWatch |
+| Successful Request Latency |  ms   | get_successful_request_latency <br /> put_successful_request_latency <br /> query_successful_request_latency <br /> scan_successful_request_latency | The latency of successful request | Amazon CloudWatch |
+| TTL Deleted Item count |     | time_to_live_deleted_item_count | The count of items deleted by TTL | Amazon CloudWatch |
+| Throttle Events|     | read_throttle_events <br /> write_throttle_events | Requests to DynamoDB that exceed the provisioned read/write capacity units for a table or a global secondary index. | Amazon CloudWatch |
+| Throttled Requests |     | read_throttled_requests <br /> write_throttled_requests | Requests to DynamoDB that exceed the provisioned throughput limits on a resource (such as a table or an index). | Amazon CloudWatch |
+| Scan/Query Operation Returned Item Ccount |     | scan_returned_item_count <br/>query_returned_item_count<br /> | The number of items returned by Query, Scan or ExecuteStatement (select) operations during the specified time period. | Amazon CloudWatch |
+| System Errors |    | read_system_errors<br />write_system_errors | The requests to DynamoDB or Amazon DynamoDB Streams that generate an HTTP 500 status code during the specified time period. | Amazon CloudWatch |
+| User Errors |    | user_errors | Requests to DynamoDB or Amazon DynamoDB Streams that generate an HTTP 400 status code during the specified time period.| Amazon CloudWatch |
+| Condition Checked Fail Requests |     | conditional_check_failed_requests  | The number of failed attempts to perform conditional writes.  | Amazon CloudWatch |
+| Transaction Conflict |    | transaction_conflict | Rejected item-level requests due to transactional conflicts between concurrent requests on the same items.  | Amazon CloudWatch |
+
+### Customizations
+You can customize your own metrics/expression/dashboard panel.
+The metrics definition and expression rules are found in `/config/otel-rules/aws-dynamodb`.
+The DynamoDB dashboard panel configurations are found in `/config/ui-initialized-templates/aws_dynamodb`.

docs/menu.yml

@@ -191,6 +191,8 @@ catalog:
                 path: "/en/setup/backend/backend-aws-eks-monitoring"
               - name: "S3 Monitoring"
                 path: "/en/setup/backend/backend-aws-s3-monitoring"
+              - name: "AWS DynamoDB"
+                path: "/en/setup/backend/backend-aws-dynamodb-monitoring"
           - name: "Browser Monitoring"
             path: "/en/setup/service-agent/browser-agent"
           - name: "Gateway Monitoring"
@@ -203,6 +205,8 @@ catalog:
                 path: "/en/setup/backend/backend-mysql-monitoring"
               - name: "PostgreSQL Server"
                 path: "/en/setup/backend/backend-postgresql-monitoring"
+              - name: "AWS DynamoDB"
+                path: "/en/setup/backend/backend-aws-dynamodb-monitoring"
           - name: "Self Observability"
             catalog:
               - name: "OAP self telemetry"

oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/Layer.java

@@ -151,7 +151,13 @@ public enum Layer {
     /**
      * Amazon Simple Storage Service (Amazon S3) is an object storage service provided by AWS Cloud
      */
-    AWS_S3(24, true);
+    AWS_S3(24, true),
+
+    /*
+     * Amazon DynamoDB is a fully managed NoSQL database service that provides
+     * fast and predictable performance with seamless scalability.
+     */
+    AWS_DYNAMODB(25, true);
 
     private final int value;
     /**

oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/management/ui/template/UITemplateInitializer.java

@@ -63,6 +63,7 @@ public class UITemplateInitializer {
         Layer.AWS_EKS.name(),
         Layer.OS_WINDOWS.name(),
         Layer.AWS_S3.name(),
+        Layer.AWS_DYNAMODB.name(),
         "custom"
     };
     private final UITemplateManagementService uiTemplateManagementService;

oap-server/server-starter/src/main/resources/application.yml

@@ -356,7 +356,7 @@ receiver-otel:
   selector: ${SW_OTEL_RECEIVER:default}
   default:
     enabledHandlers: ${SW_OTEL_RECEIVER_ENABLED_HANDLERS:"oc,otlp"}
-    enabledOtelRules: ${SW_OTEL_RECEIVER_ENABLED_OTEL_RULES:"apisix,k8s/*,istio-controlplane,vm,mysql/*,postgresql/*,oap,aws-eks/*,windows,aws-s3/*"}
+    enabledOtelRules: ${SW_OTEL_RECEIVER_ENABLED_OTEL_RULES:"apisix,k8s/*,istio-controlplane,vm,mysql/*,postgresql/*,oap,aws-eks/*,windows,aws-s3/*,aws-dynamodb/*"}
 
 receiver-zipkin:
   selector: ${SW_RECEIVER_ZIPKIN:-}

oap-server/server-starter/src/main/resources/otel-rules/aws-dynamodb/dynamodb-endpoint.yaml

@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This will parse a textual representation of a duration. The formats
+# accepted are based on the ISO-8601 duration format {@code PnDTnHnMn.nS}
+# with days considered to be exactly 24 hours.
+# <p>
+# Examples:
+# <pre>
+#    "PT20.345S" -- parses as "20.345 seconds"
+#    "PT15M"     -- parses as "15 minutes" (where a minute is 60 seconds)
+#    "PT10H"     -- parses as "10 hours" (where an hour is 3600 seconds)
+#    "P2D"       -- parses as "2 days" (where a day is 24 hours or 86400 seconds)
+#    "P2DT3H4M"  -- parses as "2 days, 3 hours and 4 minutes"
+#    "P-6H3M"    -- parses as "-6 hours and +3 minutes"
+#    "-P6H3M"    -- parses as "-6 hours and -3 minutes"
+#    "-P-6H+3M"  -- parses as "+6 hours and -3 minutes"
+# </pre>
+
+filter: "{ tags -> tags.Namespace == 'AWS/DynamoDB' }" # The OpenTelemetry job name
+expPrefix: tag({tags -> tags.host_name = 'aws-dynamodb::' + tags.cloud_account_id})
+expSuffix: service(['host_name'], Layer.AWS_DYNAMODB).endpoint(['host_name'], ['TableName'], Layer.AWS_DYNAMODB)
+metricPrefix: aws_dynamodb
+metricsRules:
+  # table metrics
+  - name: endpoint_consumed_write_capacity_units
+    exp: amazonaws_com_AWS_DynamoDB_ConsumedWriteCapacityUnits
+
+  - name: endpoint_consumed_read_capacity_units
+    exp: amazonaws_com_AWS_DynamoDB_ConsumedReadCapacityUnits
+
+  - name: endpoint_provisioned_read_capacity_units
+    exp: amazonaws_com_AWS_DynamoDB_ProvisionedReadCapacityUnits
+
+  - name: endpoint_provisioned_write_capacity_units
+    exp: amazonaws_com_AWS_DynamoDB_ProvisionedWriteCapacityUnits
+
+  # table operation metrics
+  - name: endpoint_get_successful_request_latency
+    exp: amazonaws_com_AWS_DynamoDB_SuccessfulRequestLatency.tagMatch('Operation','GetItem|BatchGetItem')
+  - name: endpoint_put_successful_request_latency
+    exp: amazonaws_com_AWS_DynamoDB_SuccessfulRequestLatency.tagMatch('Operation','PutItem|BatchWriteItem')
+  - name: endpoint_query_successful_request_latency
+    exp: amazonaws_com_AWS_DynamoDB_SuccessfulRequestLatency.tagEqual('Operation','Query')
+  - name: endpoint_scan_successful_request_latency
+    exp: amazonaws_com_AWS_DynamoDB_SuccessfulRequestLatency.tagEqual('Operation','Scan')
+
+  - name: endpoint_scan_returned_item_count
+    exp: amazonaws_com_AWS_DynamoDB_ReturnedItemCount.tagEqual('Operation','Scan')
+  - name: endpoint_query_returned_item_count
+    exp: amazonaws_com_AWS_DynamoDB_ReturnedItemCount.tagEqual('Operation','Query')
+
+  - name: endpoint_time_to_live_deleted_item_count
+    exp: amazonaws_com_AWS_DynamoDB_TimeToLiveDeletedItemCount
+
+  - name: endpoint_read_throttled_requests
+    exp: amazonaws_com_AWS_DynamoDB_ThrottledRequests.tagMatch('Operation','GetItem|Scan|Query|BatchGetItem')
+  - name: endpoint_write_throttled_requests
+    exp: amazonaws_com_AWS_DynamoDB_ThrottledRequests.tagMatch('Operation','PutItem|UpdateItem|DeleteItem|BatchWriteItem')
+
+  - name: endpoint_read_throttle_events
+    exp: amazonaws_com_AWS_DynamoDB_ReadThrottleEvents
+  - name: endpoint_write_throttle_events
+    exp: amazonaws_com_AWS_DynamoDB_WriteThrottleEvents
+  - name: endpoint_read_system_errors
+    exp: amazonaws_com_AWS_DynamoDB_SystemErrors.tagMatch('Operation','GetItem|Scan|Query|BatchGetItem|TransactGetItems')
+  - name: endpoint_write_system_errors
+    exp: amazonaws_com_AWS_DynamoDB_SystemErrors.tagMatch('Operation','PutItem|UpdateItem|DeleteItem|BatchWriteItem|TransactWriteItems')
+  - name: endpoint_user_errors
+    exp: amazonaws_com_AWS_DynamoDB_UserErrors
+  - name: endpoint_conditional_check_failed_requests
+    exp: amazonaws_com_AWS_DynamoDB_ConditionalCheckFailedRequests
+  - name: endpoint_transaction_conflict
+    exp: amazonaws_com_AWS_DynamoDB_TransactionConflict

oap-server/server-starter/src/main/resources/otel-rules/aws-dynamodb/dynamodb-service.yaml

@@ -0,0 +1,108 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This will parse a textual representation of a duration. The formats
+# accepted are based on the ISO-8601 duration format {@code PnDTnHnMn.nS}
+# with days considered to be exactly 24 hours.
+# <p>
+# Examples:
+# <pre>
+#    "PT20.345S" -- parses as "20.345 seconds"
+#    "PT15M"     -- parses as "15 minutes" (where a minute is 60 seconds)
+#    "PT10H"     -- parses as "10 hours" (where an hour is 3600 seconds)
+#    "P2D"       -- parses as "2 days" (where a day is 24 hours or 86400 seconds)
+#    "P2DT3H4M"  -- parses as "2 days, 3 hours and 4 minutes"
+#    "P-6H3M"    -- parses as "-6 hours and +3 minutes"
+#    "-P6H3M"    -- parses as "-6 hours and -3 minutes"
+#    "-P-6H+3M"  -- parses as "+6 hours and -3 minutes"
+# </pre>
+
+filter: "{ tags -> tags.Namespace == 'AWS/DynamoDB' }" # The OpenTelemetry job name
+expPrefix: tag({tags -> tags.host_name = 'aws-dynamodb::' + tags.cloud_account_id})
+expSuffix: service(['host_name'], Layer.AWS_DYNAMODB)
+metricPrefix: aws_dynamodb
+metricsRules:
+  # account metrics
+  - name: account_max_writes
+    exp: amazonaws_com_AWS_DynamoDB_AccountMaxWrites
+  - name: account_max_reads
+    exp: amazonaws_com_AWS_DynamoDB_AccountMaxReads
+
+  - name: account_max_table_level_writes
+    exp: amazonaws_com_AWS_DynamoDB_AccountMaxTableLevelWrites
+  - name: account_max_table_level_reads
+    exp: amazonaws_com_AWS_DynamoDB_AccountMaxTableLevelReads
+
+  - name: max_provisioned_write_capacity_utilization
+    exp: amazonaws_com_AWS_DynamoDB_MaxProvisionedTableWriteCapacityUtilization
+  - name: max_provisioned_read_capacity_utilization
+    exp: amazonaws_com_AWS_DynamoDB_MaxProvisionedTableReadCapacityUtilization
+
+  - name: account_provisioned_read_capacity_utilization
+    exp: amazonaws_com_AWS_DynamoDB_AccountProvisionedReadCapacityUtilization
+  - name: account_provisioned_write_capacity_utilization
+    exp: amazonaws_com_AWS_DynamoDB_AccountProvisionedWriteCapacityUtilization
+
+  # table metrics
+  - name: consumed_write_capacity_units
+    exp: amazonaws_com_AWS_DynamoDB_ConsumedWriteCapacityUnits
+
+  - name: consumed_read_capacity_units
+    exp: amazonaws_com_AWS_DynamoDB_ConsumedReadCapacityUnits
+
+  - name: provisioned_read_capacity_units
+    exp: amazonaws_com_AWS_DynamoDB_ProvisionedReadCapacityUnits
+
+  - name: provisioned_write_capacity_units
+    exp: amazonaws_com_AWS_DynamoDB_ProvisionedWriteCapacityUnits
+
+  # table operation metrics
+  - name: get_successful_request_latency
+    exp: amazonaws_com_AWS_DynamoDB_SuccessfulRequestLatency.tagMatch('Operation','GetItem|BatchGetItem')
+  - name: put_successful_request_latency
+    exp: amazonaws_com_AWS_DynamoDB_SuccessfulRequestLatency.tagMatch('Operation','PutItem|BatchWriteItem')
+  - name: query_successful_request_latency
+    exp: amazonaws_com_AWS_DynamoDB_SuccessfulRequestLatency.tagEqual('Operation','Query')
+  - name: scan_successful_request_latency
+    exp: amazonaws_com_AWS_DynamoDB_SuccessfulRequestLatency.tagEqual('Operation','Scan')
+  - name: scan_returned_item_count
+    exp: amazonaws_com_AWS_DynamoDB_ReturnedItemCount.tagEqual('Operation','Scan')
+  - name: query_returned_item_count
+    exp: amazonaws_com_AWS_DynamoDB_ReturnedItemCount.tagEqual('Operation','Query')
+
+  - name: time_to_live_deleted_item_count
+    exp: amazonaws_com_AWS_DynamoDB_TimeToLiveDeletedItemCount
+
+  - name: read_throttled_requests
+    exp: amazonaws_com_AWS_DynamoDB_ThrottledRequests.tagMatch('Operation','GetItem|Scan|Query|BatchGetItem')
+  - name: write_throttled_requests
+    exp: amazonaws_com_AWS_DynamoDB_ThrottledRequests.tagMatch('Operation','PutItem|UpdateItem|DeleteItem|BatchWriteItem')
+
+  - name: read_throttle_events
+    exp: amazonaws_com_AWS_DynamoDB_ReadThrottleEvents
+  - name: write_throttle_events
+    exp: amazonaws_com_AWS_DynamoDB_WriteThrottleEvents
+  - name: read_system_errors
+    exp: amazonaws_com_AWS_DynamoDB_SystemErrors.tagMatch('Operation','GetItem|Scan|Query|BatchGetItem|TransactGetItems')
+  - name: write_system_errors
+    exp: amazonaws_com_AWS_DynamoDB_SystemErrors.tagMatch('Operation','PutItem|UpdateItem|DeleteItem|BatchWriteItem|TransactWriteItems')
+  - name: user_errors
+    exp: amazonaws_com_AWS_DynamoDB_UserErrors
+  - name: conditional_check_failed_requests
+    exp: amazonaws_com_AWS_DynamoDB_ConditionalCheckFailedRequests
+  - name: transaction_conflict
+    exp: amazonaws_com_AWS_DynamoDB_TransactionConflict
+
+