diff --git a/.gitattributes b/.gitattributes deleted file mode 100644 index b858c864..00000000 --- a/.gitattributes +++ /dev/null @@ -1 +0,0 @@ -flink-application-properties-dev.json filter=arn-filter diff --git a/.github/workflows/check-arns.yml b/.github/workflows/check-arns.yml new file mode 100644 index 00000000..e858080b --- /dev/null +++ b/.github/workflows/check-arns.yml @@ -0,0 +1,24 @@ +name: Check for Exposed ARNs + +on: + pull_request: + +jobs: + check-arns: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + + - name: Check for exposed ARNs + run: | + # Find files containing ARN patterns with actual account IDs + # Exclude .git directory, markdown files, and this workflow file itself + if grep -r --include="*" --exclude="*.md" --exclude-dir=".git" --exclude=".github/workflows/check-arns.yml" -E 'arn:aws:[^:]+:[^:]+:[0-9]{12}:' .; then + echo "ERROR: Found unsanitized ARNs in the repository" + echo "Please replace account IDs with a placeholder such as " + echo "Files with exposed ARNs:" + grep -r --include="*" --exclude="*.md" --exclude-dir=".git" --exclude=".github/workflows/check-arns.yml" -l -E 'arn:aws:[^:]+:[^:]+:[0-9]{12}:' . + exit 1 + fi + + echo "All files checked - no exposed ARNs found" \ No newline at end of file diff --git a/.gitignore b/.gitignore index 94085cc1..49ecf155 100644 --- a/.gitignore +++ b/.gitignore @@ -13,7 +13,7 @@ venv/ .java-version /pyflink/ __pycache__/ - +.vscode/ /.run/ clean.sh diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index b2ce7d9a..361a68d7 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -59,6 +59,7 @@ The AWS team managing the repository reserves the right to modify or reject new versions, external dependencies, permissions, and runtime configuration. Use [this example](https://github.com/aws-samples/amazon-managed-service-for-apache-flink-examples/tree/main/java/KafkaConfigProviders/Kafka-SASL_SSL-ConfigProviders) as a reference. * Make sure the example works with what explained in the README, and without any implicit dependency or configuration. +* Add an entry for the new example in the top-level [README](README.md) or in the README of the subfolder, if the example is in a subfolder such as `java/FlinkCDC` or `java/Iceberg` #### AWS authentication and credentials @@ -66,10 +67,19 @@ The AWS team managing the repository reserves the right to modify or reject new * Any permissions must be provided from the IAM Role assigned to the Managed Apache Flink application. When running locally, leverage the IDE AWS plugins. #### Dependencies in PyFlink examples - * Use the pattern illustrated by [this example](https://github.com/aws-samples/amazon-managed-service-for-apache-flink-examples/tree/main/python/GettingStarted) - to provide JAR dependencies and build the ZIP using Maven. - * If the application also requires Python dependencies, use the pattern illustrated by [this example](https://github.com/aws-samples/amazon-managed-service-for-apache-flink-examples/tree/main/python/PythonDependencies) - leveraging `requirements.txt`. + +* Use the pattern illustrated by [this example](https://github.com/aws-samples/amazon-managed-service-for-apache-flink-examples/tree/main/python/GettingStarted) + to provide JAR dependencies and build the ZIP using Maven. +* If the application also requires Python dependencies used for UDF and data processing in general, use the pattern illustrated by [this example](https://github.com/aws-samples/amazon-managed-service-for-apache-flink-examples/tree/main/python/PythonDependencies) + leveraging `requirements.txt`. +* Only if the application requires Python dependencies used during the job initialization, in the main(), use the pattern + illustrated in [this other example](https://github.com/aws-samples/amazon-managed-service-for-apache-flink-examples/tree/main/python/PackagedPythonDependencies), + packaging dependencies in the ZIP artifact. + +#### Top POM-file for Java examples + +* Add the new Java example also to the `pom.xml` file in the `java/` folder + ## Reporting Bugs/Feature Requests diff --git a/README.md b/README.md index 4a86784a..39ad86ac 100644 --- a/README.md +++ b/README.md @@ -2,14 +2,89 @@ Example applications in Java, Python, Scala and SQL for Amazon Managed Service for Apache Flink (formerly known as Amazon Kinesis Data Analytics), illustrating various aspects of Apache Flink applications, and simple "getting started" base projects. -* [Java examples](./java) -* [Python examples](./python) -* [Scala examples](/scala) -* [Operational utilities and infrastructure code](./infrastructure) +## Table of Contents -## Security +### Java Examples -See [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) for more information. +#### Getting Started +- [**Getting Started - DataStream API**](./java/GettingStarted) - Skeleton project for a basic Flink Java application using DataStream API +- [**Getting Started - Table API & SQL**](./java/GettingStartedTable) - Basic Flink Java application using Table API & SQL with DataStream API + +#### Connectors +- [**Kinesis Connectors**](./java/KinesisConnectors) - Examples of Flink Kinesis Connector source and sink (standard and EFO) +- [**Kinesis Source Deaggregation**](./java/KinesisSourceDeaggregation) - Handling Kinesis record deaggregation in the Kinesis source +- [**Kafka Connectors**](./java/KafkaConnectors) - Examples of Flink Kafka Connector source and sink +- [**Kafka Config Providers**](./java/KafkaConfigProviders) - Examples of using Kafka Config Providers for secure configuration management +- [**DynamoDB Stream Source**](./java/DynamoDBStreamSource) - Reading from DynamoDB Streams as a source +- [**Kinesis Firehose Sink**](./java/KinesisFirehoseSink) - Writing data to Amazon Kinesis Data Firehose +- [**SQS Sink**](./java/SQSSink) - Writing data to Amazon SQS +- [**Prometheus Sink**](./java/PrometheusSink) - Sending metrics to Prometheus +- [**Flink CDC**](./java/FlinkCDC) - Change Data Capture examples using Flink CDC + +#### Reading and writing files and transactional data lake formats +- [**Iceberg**](./java/Iceberg) - Working with Apache Iceberg and Amazon S3 Tables +- [**S3 Sink**](./java/S3Sink) - Writing JSON data to Amazon S3 +- [**S3 Avro Sink**](./java/S3AvroSink) - Writing Avro format data to Amazon S3 +- [**S3 Avro Source**](./java/S3AvroSource) - Reading Avro format data from Amazon S3 +- [**S3 Parquet Sink**](./java/S3ParquetSink) - Writing Parquet format data to Amazon S3 +- [**S3 Parquet Source**](./java/S3ParquetSource) - Reading Parquet format data from Amazon S3 + +#### Data Formats & Schema Registry +- [**Avro with Glue Schema Registry - Kinesis**](./java/AvroGlueSchemaRegistryKinesis) - Using Avro format with AWS Glue Schema Registry and Kinesis +- [**Avro with Glue Schema Registry - Kafka**](./java/AvroGlueSchemaRegistryKafka) - Using Avro format with AWS Glue Schema Registry and Kafka + +#### Stream Processing Patterns +- [**Serialization**](./java/Serialization) - Serialization of record and state +- [**Windowing**](./java/Windowing) - Time-based window aggregation examples +- [**Side Outputs**](./java/SideOutputs) - Using side outputs for data routing and filtering +- [**Async I/O**](./java/AsyncIO) - Asynchronous I/O patterns with retries for external API calls\ +- [**Custom Metrics**](./java/CustomMetrics) - Creating and publishing custom application metrics + +#### Utilities +- [**Fink Data Generator (JSON)**](java/FlinkDataGenerator) - How to use a Flink application as data generator, for functional and load testing. + +### Python Examples + +#### Getting Started +- [**Getting Started**](./python/GettingStarted) - Basic PyFlink application Table API & SQL + +#### Handling Python dependencies +- [**Python Dependencies**](./python/PythonDependencies) - Managing Python dependencies in PyFlink applications using `requirements.txt` +- [**Packaged Python Dependencies**](./python/PackagedPythonDependencies) - Managing Python dependencies packaged with the PyFlink application at build time + +#### Connectors +- [**Datastream Kafka Connector**](./python/DatastreamKafkaConnector) - Using Kafka connector with PyFlink DataStream API +- [**Kafka Config Providers**](./python/KafkaConfigProviders) - Secure configuration management for Kafka in PyFlink +- [**S3 Sink**](./python/S3Sink) - Writing data to Amazon S3 using PyFlink +- [**Firehose Sink**](./python/FirehoseSink) - Writing data to Amazon Kinesis Data Firehose +- [**Iceberg Sink**](./python/IcebergSink) - Writing data to Apache Iceberg tables + +#### Stream Processing Patterns +- [**Windowing**](./python/Windowing) - Time-based window aggregation examples with PyFlink/SQL +- [**User Defined Functions (UDF)**](./python/UDF) - Creating and using custom functions in PyFlink + +#### Utilities +- [**Data Generator**](./python/data-generator) - Python script for generating sample data to Kinesis Data Streams +- [**Local Development on Apple Silicon**](./python/LocalDevelopmentOnAppleSilicon) - Setup guide for local development of Flink 1.15 on Apple Silicon Macs (not required with Flink 1.18 or later) + + +### Scala Examples + +#### Getting Started +- [**Getting Started - DataStream API**](./scala/GettingStarted) - Skeleton project for a basic Flink Scala application using DataStream API + +### Infrastructure & Operations + +- [**Auto Scaling**](./infrastructure/AutoScaling) - Custom autoscaler for Amazon Managed Service for Apache Flink +- [**Scheduled Scaling**](./infrastructure/ScheduledScaling) - Scale applications up and down based on daily time schedules +- [**Monitoring**](./infrastructure/monitoring) - Extended CloudWatch Dashboard examples for monitoring applications +- [**Scripts**](./infrastructure/scripts) - Useful shell scripts for interacting with Amazon Managed Service for Apache Flink control plane API + +--- + +## Contributing + +See [Contributing Guidelines](CONTRIBUTING.md#security-issue-notifications) for more information. ## License Summary diff --git a/infrastructure/AutoScaling/.gitignore b/infrastructure/AutoScaling/.gitignore new file mode 100644 index 00000000..4fe961d9 --- /dev/null +++ b/infrastructure/AutoScaling/.gitignore @@ -0,0 +1,5 @@ +node_modules +cdk.out +/cdk.context.json +Autoscaler-*.yaml +/.DS_Store diff --git a/infrastructure/AutoScaling/README.md b/infrastructure/AutoScaling/README.md index aead3c5c..381e289a 100644 --- a/infrastructure/AutoScaling/README.md +++ b/infrastructure/AutoScaling/README.md @@ -1,73 +1,202 @@ -# Automatic Scaling for Amazon Managed Service for Apache Flink Applications +# Amazon Managed Service for Apache Flink - Custom autoscaler -* Python 3.9 for Lambda Function +This tool helps creating a custom autoscaler for your Amazon Managed Service for Apache Flink application. +The custom autoscaller allows you defining using metrics other than `containerCPUUtilization`, defining custom thresholds, max and min scale, and customize the autoscale cycle. -**IMPORTANT:** We strongly recommend that you disable autoscaling within your Amazon Managed Service for Apache Flink application if using the approach described here. -This sample illustrates how to scale your Amazon Managed Service for Apache Flink application using a different CloudWatch Metric from the Apache Flink Application, Amazon MSK, Amazon Kinesis Data Stream. Here's the high level approach: +Setting up the autoscaler is a two-steps process: +1. Generate a CloudFormation (CFN) template for an autoscaler for on a specific metric and statistic +2. Use the generated CFN template to deploy a CFN Stack which control autoscaling of a specific Managed Service for Apache Flink application. -- Using Amazon CloudWatch Alarms for the select metric in order to trigger Scaling Step Function Logic -- Step Functions that triggers the AWS Lambda Scaling Function, which verifies if the alarm after metric has gone into OK status -- AWS Lambda Function to Scale Up/Down the Managed Flink Application, as well as verifying the Application doesn't go above or below maximum/minimum KPU. -## Deploying the AutoScaling for Managed Flink Applications +## Process overview -Follow the instructions to deploy the autoscaling solution in your AWS Account -1. Clone this repository -2. Go to CloudFormation -3. Click Create stack -4. Select `Upload a template file` -5. Upload the template from this repository -6. This deployment takes the following CFN Parameters - 1. **Amazon Managed Service for Apache Flink AutoScaling Configuration:** +1. Decide metric and statistic for your autoscaler (see [Supported metric](#supported-metrics)). +2. Run the `generateautoscaler.sh` script to generate the CFN template. The generate template is named `Autoscaler-*.yaml` (see [Generate CFN template](#generate-the-cloudformation-template-using-the-script-recommended)) +3. Use the CFN template to create the autoscaler Stack. When you create the stack specify all remaining [autoscaler parameters](#autoscaler-parameters), including the application name (see [Deploying the autoscaler template](#deploying-the-autoscaler-template)) - 1. *Amazon Managed Service for Apache Flink Application Name*: The name of the Amazon Managed Apache Flink Application you would want to Auto Scale - 2. *Auto Scale Metric*: Available metrics to use for Autoscaling. - 3. *Custom Metric Name*: If you choose custom metric to do scaling, please provide its name. Remember that it will only work if you add as dimension to the Metric group **kinesisAnalytics** - 3. *Maximum KPU*: Maximum number of KPUs you want the Managed Flink Application to Scale - 4. *Minimum KPU*: Minimum number of KPUs you want the Managed Flink Application to Scale - 2. [**CloudWatch Alarm Configuration:**](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html#alarm-evaluation) - 5. *Evaluation Period for Metric*: Period to be used in minutes for the evaluation for scaling in or out (Example: 5) - 6. *Number of Data points to Trigger Alarm*: Number of Data Points (60 seconds each data point) during Evaluation Period where Metric has to be over threshold for rule to be in Alarm State. (Example: 2 This would mean that alarm would trigger if during a 5 minute window, 2 data points are above threshold) - 7. Waiting time for Application to finish updating (Seconds) - 7. *Grace Period for Alarm*: Time given in seconds to application after scaling to have alarms go to OK status (Example: 120) - 3. **Scale In/Out Configuration:** +Notes +* A generated CFN Template hardwires a specific metric and statistics. +* An autoscaler Stack controls a single Managed Flink application. +* If you want to create autoscaler Stacks for multiple applications, using the same metric and statistics, you can reuse the same CFN Template. - 8. *Scale Out/In Operation*: Scale Out/In Operation (Multiply/Divide or Add/Substract) - 9. *Scale In Factor*: Factor by which you want to reduce the number of KPUs in your Flink Application - 10. *Threshold for Metric to Scale Down*: Choose the threshold for when the Scale In Rule should be in Alarm State - 11. *Scale Out Factor*: Factor by which you want to increase the number of KPUs in your Flink Application - 12. *Threshold for Metric to Scale Up*: Choose the threshold for when the Scale Out Rule should be in Alarm State - 4. **Kafka Configuration:** +The CFN Stack creates two CloudWatch Alarms, a StepFunction and a Lambda function which automatically control the scaling of your Managed Flink application. - 13. *Amazon MSK Cluster Name*: If you choose topic metrics (MaxOffsetLag, SumOffsetLag or EstimatedMaxTimeLag) as metric to scale, you need to provide the name of the MSK Cluster for monitoring - 14. *Kafka Topic Name*: If you choose topic metrics (MaxOffsetLag, SumOffsetLag or EstimatedMaxTimeLag) as metric to scale, you need to provide the Kafka Topic for monitoring - 15. *Kafka Consumer Group*: If you choose topic metrics (MaxOffsetLag, SumOffsetLag or EstimatedMaxTimeLag) as metric to scale, you need to provide the Consumer Group name for monitoring - 5. **Kinesis Configuration:** - 16. *Kinesis Data Streams Name*: If you choose MillisBehindLatest as metric to scale, you need to provide the Kinesis Data Stream Name for monitoring +![Process of generating the CFN template and creating the autoscaler](img/generating-autoscaler.png) -## Scaling logic -As alluded to above, the scaling logic is a bit more involved than simply calling `UpdateApplication` on your Amazon Managed Service for Apache Flink application. Here are the steps involved: +> ⚠️ Neither the autoscaler nor CloudWatch Alarm validate the metric name. +> If you type a wrong metric name or select a metric type inconsistent with the metric, the CFN template will create the autoscaler stack but the CloudWatch Alarms controlling the scale will never trigger. -The Solution will trigger an alarm based on the threshold set in the parameter of the CloudFormation Template for the selected metric. This will activate an AWS Step Functions Workflow which will -* Scale the Managed Flink Application In or Out using an increase factor defined in the CloudFormation Template. -* Once the application has finished updating, it will verify if it has reached the minimum or maximum value for KPUs that the application can have. If it has it will finish the scaling event. -* If the application hasn't reached the max/min values of allocated KPU’s, the workflow will wait for a given period of time, to allow the metric to fall within threshold and have the Amazon CloudWatch Rule from ALARM status to OK. -* If the rule is still in ALARM status, the workflow will scale again the application. If the rule is now in OK, it will finish the scaling event. +--- +## Step 1: Generate the CloudFormation template using the script (recommended) -NOTE: In this sample, we assume that the parallelism/KPU is 1. For more background on parallelism and parallelism/KPU, please see [Application Scaling in Amazon Managed Service for Apache Flink](https://docs.aws.amazon.com/kinesisanalytics/latest/java/how-scaling.html). +1. Ensure you have installed AWS CDK in the current directory: `npm install aws-cdk-lib` +2. Decide which metric and statistics you want to use. See [Supported metrics](#supported-metrics) and [Supported statistics](#supported-statistics) +3. Execute the generator script to generate the YAML file with the CloudFormation template -## References +``` +generateautoscaler.sh type= stat= +``` -- [Amazon Managed Service for Apache Flink developer guide](https://docs.aws.amazon.com/kinesisanalytics/latest/java/what-is.html). -- [Application Scaling in Amazon Managed Service for Apache Flink](https://docs.aws.amazon.com/kinesisanalytics/latest/java/how-scaling.html). -- [KinesisAnalyticsV2 boto3 reference](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/kinesisanalyticsv2.html). \ No newline at end of file +The script generates a CFN template file named `Autoscaler--.yaml` in the same directory. + +Script parameters: +* `type` [`KinesisAnalytics`, `MSK`, `Kinesis`, `KinesisEFO`]: determines the metric type (default: `KinesisAnalytics`). +* `metric`: (mandatory) name of the metric. See [Supported metric](#supported-metrics). +* `stat` [`Average`, `Minimum`, `Maximum`, `Sum`] : metric statistics (default: `Maximum`). See [Supported statistics](#supported-statistics). + + +Examples of valid generator commands: + +``` +./generateautoscaler.sh type=KinesisAnalytics metric=containerCPUUtilization stat=Average + +./generateautoscaler.sh type=MSK metric=MaxOffsetLag stat=Maximum + +./generateautoscaler.sh type=Kinesis metric=GetRecords.IteratorAgeMilliseconds stat=Maximum + +./generateautoscaler.sh type=KinesisEFO metric=SubscribeToShardEvent.MillisBehindLates stat=Maximum +``` + +### Supported metrics + +When you generate the CFN template you specify +1. Metric type (see below) +2. Metric name +3. Statistic (Maximum, Average etc) + +The metric type determines the Namespace and Dimensions of the CloudWatch metric to use for autoscaling. + +> The name *KinesisAnalytics* refers to metrics exposed by Amazon Managed Service for Apache Flink. This is due to the CloudWatch Namespace that is called `AWS/KinesisAnalytics` for legacy reasons. + + +| Metric Type | Supported Metrics | Namespace | Dimensions | +|--------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------|-------------------------------------------| +| `KinesisAnalytics` | Any metrics exposed by Managed Service for Apache Flink which have only the `Applicaton` dimension. This also includes custom metrics when *Monitoring metrics level* is set to *Application*. | `AWS/KinesisAnalytics` | `Application` | +| `MSK` | MSK Consumer Group metrics, such as `EstimatedMaxTimeLag`, `EstimatedTimeLag`, `MaxOffsetLag`, `OffsetLag`, and `SumOffsetLag`. | `AWS/Kafka` | `Cluster Name`, `Consumer Group`, `Topic` | +| `Kinesis` | Any basic stream-level metrics exposed by Kinesis Data Streams with `StreamName` dimension only (i.e. excluding metric related to EFO consumers. | `AWS/Kinesis` | `StreamName` | +| `KinesisEFO` | Any basic stream-level metrics exposed by Kinesis Data Streams for EFO consumers, with `StreamName` and `ConsumerName` dimensions. For example `SubscribeToShardEvent.MillisBehindLatest`. | `AWS/Kinesis` | `StreamName`, `ConsumerName` | + +Reference docs +* [Metrics and dimensions in Managed Service for Apache Flink](https://docs.aws.amazon.com/managed-flink/latest/java/metrics-dimensions.html) +* [Use custom metrics with Amazon Managed Service for Apache Flink](https://docs.aws.amazon.com/managed-flink/latest/java/monitoring-metrics-custom.html) +* MSK: [Monitoring consumer lags](https://docs.aws.amazon.com/msk/latest/developerguide/consumer-lag.html) +* [Monitor the Amazon Kinesis Data Streams service with Amazon CloudWatch](https://docs.aws.amazon.com/streams/latest/dev/monitoring-with-cloudwatch.html#kinesis-metrics-stream) + + +### Supported statistics + +The autoscaler supports the following metric statistics: +* `Average` +* `Maximum` +* `Minumum` +* `Sum` + +The period of calculation of the statistic is defined when you create the stack (default: 60 sec). + +### Requirements for template generation + +To generate the CFN template you need: +1. AWS CLI +2. AWS CDK (Node.js). See [Getting started with AWS CDK](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html). + +This code has been tested with the following versions: +* Node v22.15.0 +* aws-cdk@2.1001.0 +* aws-cdk-lib@2.181.1 (constructs@10.4.2) +* aws-cli 2.24.14 + +The script will install any required Node dependency in the `./cdk` directory, using `npm install`. + +--- + +## Step 2: Deploying the autoscaler template + +You can use the generated autocaler CFN template to create an autoscaler stack that uses the metric and stats you specified when you generated the script, with any Managed Flink application. + +To deploy it (to create the stack) you can either use the AWS console or [CloudFormation CLI](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/cloudformation/create-stack.html). + +> ⚠️ The provided CDK code is not designed to deploy the autoscaler stack. It only generates the CFN YAML template, locally. + + +When you deploy the CFN Template you need to specify additional parameters, to specify the application to control and other variables of the autoscaler, such as thesholds and scaling factors. The parameters depends on the metric type you chose, when you generated the template. + +### Paramers requested for all metric types + +Paramters you probably want to customize, for each application: + +| Parameter | Parameter name | Default | Description | +|-----------------|----------------|---------|--------------| +| Application Name | `ApplicationName` | (none) | Name of the Managed Flink application to control. | +| Maximum KPU | `MaxKPU` | 10 | Maximum number of KPU the autoscaler may scale out to. | +| Minimum KPU | `MinKPU` | 1 | Minimum number of KPU the autoscaler may scale in to. | +| Scale operation | `ScaleOperation` | (none) | Operation to calculate the new parallelism, when scaling out/in. `Multiply/Divide` to multiply or divide the current parallelism by the scaling factor, `Add/Subtract` to add to or substract from the current parallelism. | +| Scale-out factor | `ScaleOutFactor` | 2 | Factor added to (Add) or multiplied by (Multiply) the current parallelism, on a scale-out event. | +| Scale-out metric threshold | `ScaleOutThreshold` | 80 | Upper threshold for the metric to trigger the scaling-out alarm. | +| Scale-in factor | `ScaleInFactor` | 2 | Factor subtracted to (Subtract) or divided by (Divide) the current parallelism, on a scale-in event. | +| Scale-in metric threshold | `ScaleInThreshold` | 20 | Lower threshold for the metric to trigger the scaling-in alarm. | +| Cooling down period (grace period) after scaling | `ScalingCoolingDownPeriod` | 300 (seconds) | Cool-down time, after the application has scaled in or out, before reconsidering the scaling alarm. | +| Scaling Alarm evaluation datapoints | `ScalingEvaluationPeriod` | 5 | Number of datapoints (metric periods) considered to evaluate the scaling alarm. This is in terms of datapoints. The actual duration depends on the metric period. | +| Number of datapoints to trigger the alarm | `DataPointsToTriggerAlarm` | 5 | Number of datapoints (metric periods) beyond threshold that trigger the scaling alarm. | + +Notes +* *Cooling down period (grace period) after scaling* should be long enough to let the application stabilize after scaling in or out. + If the grace period is too short (and the scaling factor is too small), the backlog accumulated in the downtime while + the application is scaling may cause consecutive "bounce up" when the application scales up on a workload peal, + or "bouncing down and up" when the application scales down +* If you choose *Scale operation* = *Add/Subtract* and you are too conservative in the Scale-in or Scale-out factors, + the autoscaler may trigger many consecutive scaling events increasing the overall downtime. + + +Parameters you seldom want to change: + +| Parameter | Parameter name | Default | Description | +|-----------------|----------------|---------|--------------| +| Waiting time while updating | `UpdateApplicationWaitingTime` | 60 (seconds) | This is the polling interval during the scaling operation, to check whether the application is still in `UPDATING` state. We recommend not to change this unless your application takes very long time to scale, due to a very big state. | +| Metric data point duration (metric period) | `MetricPeriod` | 60 (seconds) | Duration of each individual metric datapoints stats used for the alarm. This is the period over which the statistics is calculated. You probably do not want to change the default 60 seconds. | + + +### Parameters requested for `MSK` metric type only + +These parameters are requested only if you selected metric type `MSK` at template generation. + +| Parameter | Parameter name | Default | Description | +|-----------------|----------------|---------|--------------| +| MSK cluster name | `MSKClusterName`| (none) | Name of the MSK cluster. | +| Kafka Consumer Group name | `KafkaConsumerGroupName` | (none) | Name of the Kafka Consumer Group. | +| Kafka topic name | `KafkaTopicName` | none) | Name of the topic. | + +### Parameters requested for `Kinesis` and `KinesisEFO` metric types only + +These parameters are requested only if you selected metric type `Kinesis` or `KinesisEFO` at template generation. + +| Parameter | Parameter name | Metric type | Default | Description | +|-----------------|----------------|-------------|---------|--------------| +| Kinesis Data Stream name | `KinesisDataStreamName` | `Kinesis` and `KinesisEFO` | (none) | Name of the Kinesis stream. | +| Kinesis EFO Consumer name | `KinesisConsumerName` | `KinesisEFO` | (none) | Consumer Name, for metrics like `SubscribeToShardEvent.MillisBehindLatest`. | + + +--- + +## Known limitation + +The autoscaler only supports for Managed Service for Apache Flink metrics with `Application` dimension. + +If your application uses *Monitoring metrics level* = *Application* all metrics are published to CloudWatch with `Application` +dimension only, and the autoscaler can use any metrics. +However, when *Monitoring metrics level* is set to higher than *Application*, the autoscaler can only use metrics exposed +at Application level, such as `containerCPUUtilization`, `containerMemoryUtilization`, or `heapMemoryUtilization`. + +When *Monitoring metrics level* is set to higher than *Application*, [custom metrics]((https://docs.aws.amazon.com/managed-flink/latest/java/monitoring-metrics-custom.html)) +are also published to CloudWatch with additional dimensions and cannot be used by the autoscaler. + +The autoscaler does not support defining autoscaling based on math expression. Only simple statistics are supported. diff --git a/infrastructure/AutoScaling/cdk/bin/kda-autoscaling.d.ts b/infrastructure/AutoScaling/cdk/bin/kda-autoscaling.d.ts new file mode 100644 index 00000000..c5f71d11 --- /dev/null +++ b/infrastructure/AutoScaling/cdk/bin/kda-autoscaling.d.ts @@ -0,0 +1,2 @@ +#!/usr/bin/env node +import 'source-map-support/register'; diff --git a/infrastructure/AutoScaling/cdk/bin/kda-autoscaling.js b/infrastructure/AutoScaling/cdk/bin/kda-autoscaling.js new file mode 100644 index 00000000..7b3a51ae --- /dev/null +++ b/infrastructure/AutoScaling/cdk/bin/kda-autoscaling.js @@ -0,0 +1,20 @@ +#!/usr/bin/env node +"use strict"; +Object.defineProperty(exports, "__esModule", { value: true }); +require("source-map-support/register"); +const cdk = require("aws-cdk-lib"); +const kda_autoscaling_stack_1 = require("../lib/kda-autoscaling-stack"); +const app = new cdk.App(); +new kda_autoscaling_stack_1.KdaAutoscalingStack(app, 'KdaAutoscalingStack', { +/* If you don't specify 'env', this stack will be environment-agnostic. + * Account/Region-dependent features and context lookups will not work, + * but a single synthesized template can be deployed anywhere. */ +/* Uncomment the next line to specialize this stack for the AWS Account + * and Region that are implied by the current CLI configuration. */ +// env: { account: process.env.CDK_DEFAULT_ACCOUNT, region: process.env.CDK_DEFAULT_REGION }, +/* Uncomment the next line if you know exactly what Account and Region you + * want to deploy the stack to. */ +// env: { account: '123456789012', region: 'us-east-1' }, +/* For more information, see https://docs.aws.amazon.com/cdk/latest/guide/environments.html */ +}); +//# sourceMappingURL=data:application/json;base64,eyJ2ZXJzaW9uIjozLCJmaWxlIjoia2RhLWF1dG9zY2FsaW5nLmpzIiwic291cmNlUm9vdCI6IiIsInNvdXJjZXMiOlsia2RhLWF1dG9zY2FsaW5nLnRzIl0sIm5hbWVzIjpbXSwibWFwcGluZ3MiOiI7OztBQUNBLHVDQUFxQztBQUNyQyxtQ0FBbUM7QUFDbkMsd0VBQW1FO0FBRW5FLE1BQU0sR0FBRyxHQUFHLElBQUksR0FBRyxDQUFDLEdBQUcsRUFBRSxDQUFDO0FBQzFCLElBQUksMkNBQW1CLENBQUMsR0FBRyxFQUFFLHFCQUFxQixFQUFFO0FBQ2xEOztpRUFFaUU7QUFFakU7bUVBQ21FO0FBQ25FLDZGQUE2RjtBQUU3RjtrQ0FDa0M7QUFDbEMseURBQXlEO0FBRXpELDhGQUE4RjtDQUMvRixDQUFDLENBQUMiLCJzb3VyY2VzQ29udGVudCI6WyIjIS91c3IvYmluL2VudiBub2RlXG5pbXBvcnQgJ3NvdXJjZS1tYXAtc3VwcG9ydC9yZWdpc3Rlcic7XG5pbXBvcnQgKiBhcyBjZGsgZnJvbSAnYXdzLWNkay1saWInO1xuaW1wb3J0IHsgS2RhQXV0b3NjYWxpbmdTdGFjayB9IGZyb20gJy4uL2xpYi9rZGEtYXV0b3NjYWxpbmctc3RhY2snO1xuXG5jb25zdCBhcHAgPSBuZXcgY2RrLkFwcCgpO1xubmV3IEtkYUF1dG9zY2FsaW5nU3RhY2soYXBwLCAnS2RhQXV0b3NjYWxpbmdTdGFjaycsIHtcbiAgLyogSWYgeW91IGRvbid0IHNwZWNpZnkgJ2VudicsIHRoaXMgc3RhY2sgd2lsbCBiZSBlbnZpcm9ubWVudC1hZ25vc3RpYy5cbiAgICogQWNjb3VudC9SZWdpb24tZGVwZW5kZW50IGZlYXR1cmVzIGFuZCBjb250ZXh0IGxvb2t1cHMgd2lsbCBub3Qgd29yayxcbiAgICogYnV0IGEgc2luZ2xlIHN5bnRoZXNpemVkIHRlbXBsYXRlIGNhbiBiZSBkZXBsb3llZCBhbnl3aGVyZS4gKi9cblxuICAvKiBVbmNvbW1lbnQgdGhlIG5leHQgbGluZSB0byBzcGVjaWFsaXplIHRoaXMgc3RhY2sgZm9yIHRoZSBBV1MgQWNjb3VudFxuICAgKiBhbmQgUmVnaW9uIHRoYXQgYXJlIGltcGxpZWQgYnkgdGhlIGN1cnJlbnQgQ0xJIGNvbmZpZ3VyYXRpb24uICovXG4gIC8vIGVudjogeyBhY2NvdW50OiBwcm9jZXNzLmVudi5DREtfREVGQVVMVF9BQ0NPVU5ULCByZWdpb246IHByb2Nlc3MuZW52LkNES19ERUZBVUxUX1JFR0lPTiB9LFxuXG4gIC8qIFVuY29tbWVudCB0aGUgbmV4dCBsaW5lIGlmIHlvdSBrbm93IGV4YWN0bHkgd2hhdCBBY2NvdW50IGFuZCBSZWdpb24geW91XG4gICAqIHdhbnQgdG8gZGVwbG95IHRoZSBzdGFjayB0by4gKi9cbiAgLy8gZW52OiB7IGFjY291bnQ6ICcxMjM0NTY3ODkwMTInLCByZWdpb246ICd1cy1lYXN0LTEnIH0sXG5cbiAgLyogRm9yIG1vcmUgaW5mb3JtYXRpb24sIHNlZSBodHRwczovL2RvY3MuYXdzLmFtYXpvbi5jb20vY2RrL2xhdGVzdC9ndWlkZS9lbnZpcm9ubWVudHMuaHRtbCAqL1xufSk7Il19 \ No newline at end of file diff --git a/infrastructure/AutoScaling/cdk/bin/kda-autoscaling.ts b/infrastructure/AutoScaling/cdk/bin/kda-autoscaling.ts new file mode 100644 index 00000000..740debc7 --- /dev/null +++ b/infrastructure/AutoScaling/cdk/bin/kda-autoscaling.ts @@ -0,0 +1,19 @@ +#!/usr/bin/env node +import 'source-map-support/register'; +import * as cdk from 'aws-cdk-lib'; +import {KdaAutoscalingStack} from '../lib/kda-autoscaling-stack'; +import {DefaultStackSynthesizer} from "aws-cdk-lib"; + +const app = new cdk.App(); + +const synthDate = new Date().toISOString().split('T')[0]; + +new KdaAutoscalingStack(app, 'KdaAutoscalingStack', { + description: `MSF autoscaler (${synthDate})`, + + synthesizer: new DefaultStackSynthesizer({ + generateBootstrapVersionRule: false + }) +}); + + diff --git a/infrastructure/AutoScaling/cdk/cdk.json b/infrastructure/AutoScaling/cdk/cdk.json new file mode 100644 index 00000000..b5183998 --- /dev/null +++ b/infrastructure/AutoScaling/cdk/cdk.json @@ -0,0 +1,53 @@ +{ + "app": "npx ts-node --prefer-ts-exts bin/kda-autoscaling.ts", + "watch": { + "include": [ + "**" + ], + "exclude": [ + "README.md", + "cdk*.json", + "**/*.d.ts", + "**/*.js", + "tsconfig.json", + "package*.json", + "yarn.lock", + "node_modules", + "test" + ] + }, + "context": { + "@aws-cdk/aws-lambda:recognizeLayerVersion": true, + "@aws-cdk/core:checkSecretUsage": true, + "@aws-cdk/core:target-partitions": [ + "aws", + "aws-cn" + ], + "@aws-cdk-containers/ecs-service-extensions:enableDefaultLogDriver": true, + "@aws-cdk/aws-ec2:uniqueImdsv2TemplateName": true, + "@aws-cdk/aws-ecs:arnFormatIncludesClusterName": true, + "@aws-cdk/aws-iam:minimizePolicies": true, + "@aws-cdk/core:validateSnapshotRemovalPolicy": true, + "@aws-cdk/aws-codepipeline:crossAccountKeyAliasStackSafeResourceName": true, + "@aws-cdk/aws-s3:createDefaultLoggingPolicy": true, + "@aws-cdk/aws-sns-subscriptions:restrictSqsDescryption": true, + "@aws-cdk/aws-apigateway:disableCloudWatchRole": true, + "@aws-cdk/core:enablePartitionLiterals": true, + "@aws-cdk/aws-events:eventsTargetQueueSameAccount": true, + "@aws-cdk/aws-iam:standardizedServicePrincipals": true, + "@aws-cdk/aws-ecs:disableExplicitDeploymentControllerForCircuitBreaker": true, + "@aws-cdk/aws-iam:importedRoleStackSafeDefaultPolicyName": true, + "@aws-cdk/aws-s3:serverAccessLogsUseBucketPolicy": true, + "@aws-cdk/aws-route53-patters:useCertificate": true, + "@aws-cdk/customresources:installLatestAwsSdkDefault": false, + "@aws-cdk/aws-rds:databaseProxyUniqueResourceName": true, + "@aws-cdk/aws-codedeploy:removeAlarmsFromDeploymentGroup": true, + "@aws-cdk/aws-apigateway:authorizerChangeDeploymentLogicalId": true, + "@aws-cdk/aws-ec2:launchTemplateDefaultUserData": true, + "@aws-cdk/aws-secretsmanager:useAttachedSecretResourcePolicyForSecretTargetAttachments": true, + "@aws-cdk/aws-redshift:columnId": true, + "@aws-cdk/aws-stepfunctions-tasks:enableEmrServicePolicyV2": true, + "@aws-cdk/aws-ec2:restrictDefaultSecurityGroup": true, + "@aws-cdk/aws-apigateway:requestValidatorUniqueId": true + } +} diff --git a/infrastructure/AutoScaling/cdk/jest.config.js b/infrastructure/AutoScaling/cdk/jest.config.js new file mode 100644 index 00000000..08263b89 --- /dev/null +++ b/infrastructure/AutoScaling/cdk/jest.config.js @@ -0,0 +1,8 @@ +module.exports = { + testEnvironment: 'node', + roots: ['/test'], + testMatch: ['**/*.test.ts'], + transform: { + '^.+\\.tsx?$': 'ts-jest' + } +}; diff --git a/infrastructure/AutoScaling/cdk/lib/kda-autoscaling-stack.d.ts b/infrastructure/AutoScaling/cdk/lib/kda-autoscaling-stack.d.ts new file mode 100644 index 00000000..f5b4bf55 --- /dev/null +++ b/infrastructure/AutoScaling/cdk/lib/kda-autoscaling-stack.d.ts @@ -0,0 +1,5 @@ +import * as cdk from 'aws-cdk-lib'; +import { Construct } from 'constructs'; +export declare class KdaAutoscalingStack extends cdk.Stack { + constructor(scope: Construct, id: string, props?: cdk.StackProps); +} diff --git a/infrastructure/AutoScaling/cdk/lib/kda-autoscaling-stack.js b/infrastructure/AutoScaling/cdk/lib/kda-autoscaling-stack.js new file mode 100644 index 00000000..a31c4d2e --- /dev/null +++ b/infrastructure/AutoScaling/cdk/lib/kda-autoscaling-stack.js @@ -0,0 +1,17 @@ +"use strict"; +Object.defineProperty(exports, "__esModule", { value: true }); +exports.KdaAutoscalingStack = void 0; +const cdk = require("aws-cdk-lib"); +// import * as sqs from 'aws-cdk-lib/aws-sqs'; +class KdaAutoscalingStack extends cdk.Stack { + constructor(scope, id, props) { + super(scope, id, props); + // The code that defines your stack goes here + // example resource + // const queue = new sqs.Queue(this, 'KdaAutoscalingQueue', { + // visibilityTimeout: cdk.Duration.seconds(300) + // }); + } +} +exports.KdaAutoscalingStack = KdaAutoscalingStack; +//# sourceMappingURL=data:application/json;base64,eyJ2ZXJzaW9uIjozLCJmaWxlIjoia2RhLWF1dG9zY2FsaW5nLXN0YWNrLmpzIiwic291cmNlUm9vdCI6IiIsInNvdXJjZXMiOlsia2RhLWF1dG9zY2FsaW5nLXN0YWNrLnRzIl0sIm5hbWVzIjpbXSwibWFwcGluZ3MiOiI7OztBQUFBLG1DQUFtQztBQUVuQyw4Q0FBOEM7QUFFOUMsTUFBYSxtQkFBb0IsU0FBUSxHQUFHLENBQUMsS0FBSztJQUNoRCxZQUFZLEtBQWdCLEVBQUUsRUFBVSxFQUFFLEtBQXNCO1FBQzlELEtBQUssQ0FBQyxLQUFLLEVBQUUsRUFBRSxFQUFFLEtBQUssQ0FBQyxDQUFDO1FBRXhCLDZDQUE2QztRQUU3QyxtQkFBbUI7UUFDbkIsNkRBQTZEO1FBQzdELGlEQUFpRDtRQUNqRCxNQUFNO0lBQ1IsQ0FBQztDQUNGO0FBWEQsa0RBV0MiLCJzb3VyY2VzQ29udGVudCI6WyJpbXBvcnQgKiBhcyBjZGsgZnJvbSAnYXdzLWNkay1saWInO1xuaW1wb3J0IHsgQ29uc3RydWN0IH0gZnJvbSAnY29uc3RydWN0cyc7XG4vLyBpbXBvcnQgKiBhcyBzcXMgZnJvbSAnYXdzLWNkay1saWIvYXdzLXNxcyc7XG5cbmV4cG9ydCBjbGFzcyBLZGFBdXRvc2NhbGluZ1N0YWNrIGV4dGVuZHMgY2RrLlN0YWNrIHtcbiAgY29uc3RydWN0b3Ioc2NvcGU6IENvbnN0cnVjdCwgaWQ6IHN0cmluZywgcHJvcHM/OiBjZGsuU3RhY2tQcm9wcykge1xuICAgIHN1cGVyKHNjb3BlLCBpZCwgcHJvcHMpO1xuXG4gICAgLy8gVGhlIGNvZGUgdGhhdCBkZWZpbmVzIHlvdXIgc3RhY2sgZ29lcyBoZXJlXG5cbiAgICAvLyBleGFtcGxlIHJlc291cmNlXG4gICAgLy8gY29uc3QgcXVldWUgPSBuZXcgc3FzLlF1ZXVlKHRoaXMsICdLZGFBdXRvc2NhbGluZ1F1ZXVlJywge1xuICAgIC8vICAgdmlzaWJpbGl0eVRpbWVvdXQ6IGNkay5EdXJhdGlvbi5zZWNvbmRzKDMwMClcbiAgICAvLyB9KTtcbiAgfVxufVxuIl19 \ No newline at end of file diff --git a/infrastructure/AutoScaling/cdk/lib/kda-autoscaling-stack.ts b/infrastructure/AutoScaling/cdk/lib/kda-autoscaling-stack.ts new file mode 100644 index 00000000..1848ccdb --- /dev/null +++ b/infrastructure/AutoScaling/cdk/lib/kda-autoscaling-stack.ts @@ -0,0 +1,520 @@ +import * as cdk from 'aws-cdk-lib'; +import {Aws, aws_events_targets, CfnCondition, CfnJson, CfnParameter, Fn, Token} from 'aws-cdk-lib'; +import {Construct} from 'constructs'; +import * as iam from 'aws-cdk-lib/aws-iam'; +import * as cloudwatch from 'aws-cdk-lib/aws-cloudwatch'; +import * as events from 'aws-cdk-lib/aws-events'; +import * as sfn from 'aws-cdk-lib/aws-stepfunctions'; +import * as tasks from 'aws-cdk-lib/aws-stepfunctions-tasks'; +import * as lambda from 'aws-cdk-lib/aws-lambda'; +import {readFileSync} from 'fs'; + +function removePrefix(str: string, prefix: string): string { + const regex = new RegExp(`^${prefix}`); + return str.replace(regex, ''); +} + +enum MetricStats { + Maximum = 'Maximum', + Minimum = 'Minimum', + Average = 'Average', + Sum = 'Sum', +} + +enum MetricType { + KinesisAnalytics = "KinesisAnalytics", + MSK = "MSK", + Kinesis = "Kinesis", + KinesisEFO = "KinesisEFO" +} + + +export class KdaAutoscalingStack extends cdk.Stack { + constructor(scope: Construct, id: string, props?: cdk.StackProps) { + super(scope, id, props); + + /// CDK CLI parameters (at synth) + + const metricTypeStr: string = this.node.tryGetContext("type") || "KinesisAnalytics"; + if (!Object.values(MetricType).includes(metricTypeStr as MetricType)) { + console.error(`Invalid metric type: "${metricTypeStr}". Expected one of ${Object.values(MetricType).join(", ")}`); + process.exit(1); + } + const metricType: MetricType = metricTypeStr as MetricType; + + const autoscaleMetricName_: string | undefined = this.node.tryGetContext("metric"); + const autoscaleMetricStat_: string = this.node.tryGetContext("stat") || "Maximum"; + const triggerMetricPeriod_: number = Number(this.node.tryGetContext('period') ?? 60); + + /// Validate CDK parameters + + if (autoscaleMetricName_ === undefined) { + console.error("Metric name not defined") + process.exit(1) + } + + + if (!(autoscaleMetricStat_ in MetricStats)) { + console.error(`Invalid metric stat. Must be one of ${Object.values(MetricStats).join(", ")}`) + process.exit(1) + } + + + /// Base CFN parameters, always present + + const flinkApplicationName = new CfnParameter(this, 'ApplicationName', { + type: "String", + description: 'The name of the Amazon Managed Apache Flink Application to control' + }); + + const maxKPU = new CfnParameter(this, 'MaxKPU', { + type: "Number", + description: 'Maximum number of KPUs the Managed Flink Application may scale-out to', + default: '10' + }); + + const minKPU = new CfnParameter(this, 'MinKPU', { + type: "Number", + description: 'Minimum number of KPUs the Managed Flink Application may scale-in to', + default: '1' + }); + + const scaleMetricPeriod = new CfnParameter(this, 'MetricPeriod', { + type: "Number", + description: "Duration of each individual data point for an alarm (Seconds)", + default: 60, + allowedValues: ["1", "5", "10", "30", "60", "120", "180", "240", "300"], + }); + + const scaleInMetricThreshold = new CfnParameter(this, 'ScaleInThreshold', { + type: "Number", + description: "Lower threshold for the metric to trigger the scaling-in alarm", + default: 20 + }); + + const scaleOutMetricThreshold = new CfnParameter(this, 'ScaleOutThreshold', { + type: "Number", + description: "Upper threshold for the metric to trigger the scaling-out alarm", + default: 80 + }); + + const autoscaleEvaluationPeriod = new CfnParameter(this, 'ScalingEvaluationPeriod', { + type: "Number", + description: "Number of datapoints (metric periods) considered to evaluate the scaling alarm", + default: 5 + }); + + const autoscaleDataPointsAlarm = new CfnParameter(this, 'DataPointsToTriggerAlarm', { + type: "Number", + description: "Number of datapoints (metric periods) beyond threshold that trigger the scaling alarm", + default: 5 + }); + + const autoscaleCoolingDownPeriod = new CfnParameter(this, 'ScalingCoolingDownPeriod', { + type: "Number", + description: "Cool-down time, after the application has scaled in or out, before reconsidering the scaling alarm (in seconds)", + default: 300 + }); + + const updateWaitingTime = new CfnParameter(this, 'UpdateApplicationWaitingTime', { + type: "Number", + description: "Time given to the application to complete updating, in seconds", + default: 60 + }); + + const scaleInFactor = new CfnParameter(this, 'ScaleInFactor', { + type: "Number", + description: "Factor subtracted to (Subtract) or divided by (Divide) the current parallelism, on a scale-in event", + default: 2 + }); + + const scaleOutFactor = new CfnParameter(this, 'ScaleOutFactor', { + type: "Number", + description: "Factor added to (Add) or multiplied by (Multiply) the current parallelism, on a scale-out event", + default: 2 + }); + + const scaleOperation = new CfnParameter(this, "ScaleOperation", { + type: "String", + description: "Operation to calculate the new parallelism (Multiply/Divide or Add/Subtract) when scaling out/in", + allowedValues: ["Multiply/Divide", "Add/Subtract"] + }); + + // Define CFN template interface with the base parameter, always present + this.templateOptions.metadata = { + 'AWS::CloudFormation::Interface': { + ParameterGroups: [ + { + Label: {default: `Amazon Managed Flink application scaling (metric type: ${metricType}, metric: ${autoscaleMetricName_}, stat: ${autoscaleMetricStat_})`}, + Parameters: [flinkApplicationName.logicalId, maxKPU.logicalId, minKPU.logicalId] + }, + { + Label: {default: 'CloudWatch Scaling Alarm'}, + Parameters: [scaleMetricPeriod.logicalId, autoscaleEvaluationPeriod.logicalId, autoscaleDataPointsAlarm.logicalId] + }, + { + Label: {default: 'Autoscaling cycle'}, + Parameters: [autoscaleCoolingDownPeriod.logicalId, updateWaitingTime.logicalId] + }, + { + Label: {default: 'Scaling parameters'}, + Parameters: [scaleOperation.logicalId, scaleOutFactor.logicalId, scaleOutMetricThreshold.logicalId, scaleInFactor.logicalId, scaleInMetricThreshold.logicalId] + }, + ], + ParameterLabels: { + ApplicationName: {default: 'Application Name'}, + MaxKPU: {default: 'Maximum KPU'}, + MinKPU: {default: 'Minimum KPU'}, + MetricPeriod: {default: 'Metric data point duration (sec)'}, + ScalingEvaluationPeriod: {default: 'Scaling alarm evaluation datapoints'}, + DataPointsToTriggerAlarm: {default: 'Number of datapoints beyond threshold to trigger the alarm'}, + ScalingCoolingDownPeriod: {default: 'Cooling down period (grace period) after scaling (sec)'}, + UpdateApplicationWaitingTime: {default: 'Waiting time while updating (sec)'}, + ScaleInFactor: {default: 'Scale-in factor'}, + ScaleOutFactor: {default: 'Scale-out factor'}, + ScaleOperation: {default: 'Scale operation (mandatory)'}, + ScaleInThreshold: {default: 'Scale-in metric threshold'}, + ScaleOutThreshold: {default: 'Scale-out metric threshold'}, + } + } + } + + const applicationName = flinkApplicationName.valueAsString.trim(); + let triggerMetricNamespace; + let kinesisDataStreamName: cdk.CfnParameter + let kinesisConsumerName: cdk.CfnParameter + let triggerMetricDimensions: cdk.aws_cloudwatch.DimensionsMap; + + // Conditionally add additional CFN parameters and labels, and generate the metric parameters, depending on the metric type + switch (metricType) { + + case MetricType.KinesisAnalytics: + // Metric parameters + triggerMetricNamespace = "AWS/KinesisAnalytics"; + triggerMetricDimensions = { + Application: applicationName + } + + break; + + case MetricType.Kinesis: + /// Conditionally add KinesisStreamName CFN Parameter + + // CFN parameters + kinesisDataStreamName = new CfnParameter(this, "KinesisDataStreamName", { + type: "String", + description: "Name of the Kinesis Data Stream" + }); + + + // CFN parameter group + this.templateOptions.metadata['AWS::CloudFormation::Interface'].ParameterGroups.push({ + Label: {default: 'Kinesis metrics'}, + Parameters: [kinesisDataStreamName.logicalId] + }); + + // CFN parameter labels + this.templateOptions.metadata['AWS::CloudFormation::Interface'].ParameterLabels.KinesisDataStreamName = { + default: 'Kinesis Data Stream name' + }; + + // Metric parameters + triggerMetricNamespace = "AWS/Kinesis"; + triggerMetricDimensions = { + StreamName: kinesisDataStreamName.valueAsString.trim() + } + break; + + case MetricType.KinesisEFO: + /// Conditionally add KinesisStreamName and ConsumerName CFN Parameters + + // CFN parameters + kinesisDataStreamName = new CfnParameter(this, "KinesisDataStreamName", { + type: "String", + description: "Name of the Kinesis Data Stream" + }); + kinesisConsumerName = new CfnParameter(this, "KinesisConsumerName", { + type: "String", + description: "EFO Consumer Name" + }); + + // CFN parameter group + this.templateOptions.metadata['AWS::CloudFormation::Interface'].ParameterGroups.push({ + Label: {default: 'Kinesis EFO consumer metrics'}, + Parameters: [kinesisDataStreamName.logicalId, kinesisConsumerName.logicalId] + }); + + // CFN parameter labels + this.templateOptions.metadata['AWS::CloudFormation::Interface'].ParameterLabels.KinesisDataStreamName = { + default: 'Kinesis Data Stream name' + }; + this.templateOptions.metadata['AWS::CloudFormation::Interface'].ParameterLabels.KinesisConsumerName = { + default: 'Kinesis EFO Consumer name (optional)' + }; + + + // Kinesis EFO consumer metric parameters + triggerMetricNamespace = "AWS/Kinesis"; + triggerMetricDimensions = { + StreamName: kinesisDataStreamName.valueAsString.trim(), + ConsumerName: kinesisConsumerName.valueAsString.trim() + } + break; + + case MetricType.MSK: + // Conditionally add CFN parameters and labels for MSK Consumer Group metrics + const mskClusterName = new CfnParameter(this, "MSKClusterName", { + type: "String", + description: "If you choose topic metrics (MaxOffsetLag, SumOffsetLag or EstimatedMaxTimeLag) as metric to scale, you need to provide the name of the MSK Cluster for monitoring" + }); + const kafkaTopicName = new CfnParameter(this, "KafkaTopicName", { + type: "String", + description: "If you choose topic metrics (MaxOffsetLag, SumOffsetLag or EstimatedMaxTimeLag) as metric to scale, you need to provide the Kafka Topic for monitoring" + }); + const kafkaConsumerGroup = new CfnParameter(this, "KafkaConsumerGroupName", { + type: "String", + description: "If you choose topic metrics (MaxOffsetLag, SumOffsetLag or EstimatedMaxTimeLag) as metric to scale, you need to provide the Consumer Group name for monitoring" + }); + + this.templateOptions.metadata['AWS::CloudFormation::Interface'].ParameterGroups.push({ + Label: {default: 'MSK Consumer Group metrics'}, + Parameters: [mskClusterName.logicalId, kafkaTopicName.logicalId, kafkaConsumerGroup.logicalId,] + }); + + this.templateOptions.metadata['AWS::CloudFormation::Interface'].ParameterLabels.MSKClusterName = { + default: 'MSK cluster name' + }; + this.templateOptions.metadata['AWS::CloudFormation::Interface'].ParameterLabels.KafkaConsumerGroupName = { + default: 'Kafka consumer group name' + }; + this.templateOptions.metadata['AWS::CloudFormation::Interface'].ParameterLabels.KafkaTopicName = { + default: 'Kafka topic name' + }; + + // MSK Consumer Group metric parameters + triggerMetricNamespace = "AWS/Kafka"; + triggerMetricDimensions = { + "Cluster Name": mskClusterName.valueAsString, + "Consumer Group": kafkaConsumerGroup.valueAsString, + "Topic": kafkaTopicName.valueAsString + } + break; + + default: + throw new Error("Invalid metric type: " + metricType); + + } + + + /// Metric + const triggerMetric: cdk.aws_cloudwatch.Metric = new cloudwatch.Metric({ + namespace: triggerMetricNamespace, + metricName: autoscaleMetricName_, + statistic: autoscaleMetricStat_, + period: cdk.Duration.seconds(triggerMetricPeriod_), + dimensionsMap: triggerMetricDimensions + } + ); + + + /// Alarms + const scaleOutAlarm: cdk.aws_cloudwatch.Alarm = new cloudwatch.Alarm(this, `Scale-out alarm on ${autoscaleMetricName_}`, { + comparisonOperator: cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD, + alarmName: `ScaleOutAlarm-${applicationName}`, + threshold: scaleOutMetricThreshold.valueAsNumber, + evaluationPeriods: autoscaleEvaluationPeriod.valueAsNumber, + datapointsToAlarm: autoscaleDataPointsAlarm.valueAsNumber, + metric: triggerMetric + }); + const scaleInAlarm: cdk.aws_cloudwatch.Alarm = new cloudwatch.Alarm(this, `Scale-in alarm on ${autoscaleMetricName_}`, { + comparisonOperator: cloudwatch.ComparisonOperator.LESS_THAN_OR_EQUAL_TO_THRESHOLD, + alarmName: `ScaleInAlarm-${applicationName}`, + threshold: scaleInMetricThreshold.valueAsNumber, + evaluationPeriods: autoscaleEvaluationPeriod.valueAsNumber, + datapointsToAlarm: autoscaleDataPointsAlarm.valueAsNumber, + metric: triggerMetric + }); + + + /// Rule + const scaleRule: cdk.aws_events.Rule = new events.Rule(this, `Rule on ${autoscaleMetricName_}`, { + ruleName: `ScalingRule-${applicationName}`, + eventPattern: { + source: ["aws.cloudwatch"], + detailType: ["CloudWatch Alarm State Change"], + resources: [scaleOutAlarm.alarmArn, scaleInAlarm.alarmArn], + detail: { + state: { + "value": ["ALARM"] + } + } + }, + }); + + + /// Lambda + + // Allow Lambda to log to CW + const accessCWLogsPolicy: cdk.aws_iam.PolicyDocument = new iam.PolicyDocument({ + statements: [ + new iam.PolicyStatement({ + resources: [`arn:aws:logs:${this.region}:${this.account}:log-group:/aws/lambda/*`], + actions: ['logs:CreateLogGroup', 'logs:CreateLogStream', 'logs:PutLogEvents'], + }), + ], + }); + + // Allow Lambda to describe and update kinesisanalytics application + const kdaAccessPolicy: cdk.aws_iam.PolicyDocument = new iam.PolicyDocument({ + statements: [ + new iam.PolicyStatement({ + resources: [`arn:aws:kinesisanalytics:${this.region}:${this.account}:application/${applicationName}`], + actions: ['kinesisanalytics:DescribeApplication', 'kinesisAnalytics:UpdateApplication'] + }), + ], + }); + + // Lambda IAM role + const lambdaRole: cdk.aws_iam.Role = new iam.Role(this, 'Scaling Function Role', { + assumedBy: new iam.ServicePrincipal('lambda.amazonaws.com'), + description: 'Lambda Scaling Role', + inlinePolicies: { + AccessCWLogsPolicy: accessCWLogsPolicy, + AccessKDA: kdaAccessPolicy, + }, + }); + + // Create Lambda function + const lambdaCode = readFileSync("resources/scaling/scaling.py", "utf-8") + const lambdaFunction: cdk.aws_lambda.Function = new lambda.Function(this, 'Scaling Function', { + runtime: lambda.Runtime.PYTHON_3_9, + handler: "index.handler", + role: lambdaRole, + code: lambda.Code.fromInline(lambdaCode), + environment: { + flinkApplicationName: applicationName, + scaleInFactor: scaleInFactor.valueAsString, + scaleOutFactor: scaleOutFactor.valueAsString, + scaleOperation: scaleOperation.valueAsString, + maxKPU: maxKPU.valueAsString, + minKPU: minKPU.valueAsString + } + }) + + + /// Step Functions + + // IAM Policy to be added to StepFunction to allow KinesisAnalytics DescribeApplication. + // Note that CallAwsService is supposed to add the correct IAM permissions automatically, based on `service`, + // `action`, and `iamResources`. However, because the service API is "kinesisanalyticsv2" while the IAM action + // prefix is "kinesisanalytics" this does not work properly + const kdaDescribeApplicationPolicyStatement: cdk.aws_iam.PolicyStatement = new iam.PolicyStatement({ + resources: [`arn:aws:kinesisanalytics:${this.region}:${this.account}:application/${applicationName}`], + actions: ['kinesisanalytics:DescribeApplication'] + }); + + const describeKdaApplicationTask: cdk.aws_stepfunctions_tasks.CallAwsService = new tasks.CallAwsService(this, "Describe Application", { + service: 'kinesisanalyticsv2', // API is "v2" + action: 'describeApplication', + iamResources: [`arn:aws:kinesisanalytics:${this.region}:${this.account}:application/${applicationName}`], + additionalIamStatements: [kdaDescribeApplicationPolicyStatement], // ensure IAM policy has the correct permissions + parameters: { + "ApplicationName": applicationName + }, + resultPath: "$.TaskResult" + }) + + const scalingChoice: cdk.aws_stepfunctions.Choice = new sfn.Choice(this, 'Still updating?'); + const lambdaChoice: cdk.aws_stepfunctions.Choice = new sfn.Choice(this, 'MinMax KPU Reached?'); + const alarmChoice: cdk.aws_stepfunctions.Choice = new sfn.Choice(this, 'Still in alarm?'); + + const waitUpdate: cdk.aws_stepfunctions.Wait = new sfn.Wait(this, "Waiting for application to finish updating", { + time: sfn.WaitTime.duration(cdk.Duration.seconds(updateWaitingTime.valueAsNumber)) + }); + const waitCoolingPeriod: cdk.aws_stepfunctions.Wait = new sfn.Wait(this, "Cooling Period for Alarm", { + time: sfn.WaitTime.duration(cdk.Duration.seconds(autoscaleCoolingDownPeriod.valueAsNumber)) + }); + + // Add conditions with .when() + const successState: cdk.aws_stepfunctions.Pass = new sfn.Pass(this, 'SuccessState'); + + const lambdaTask: cdk.aws_stepfunctions_tasks.LambdaInvoke = new tasks.LambdaInvoke(this, 'Invoke Lambda Function', { + lambdaFunction: lambdaFunction, + resultPath: "$.TaskResult" + }) + + const describeKdaApplicationAfterScalingTask: cdk.aws_stepfunctions_tasks.CallAwsService = new tasks.CallAwsService(this, "Describe Application after scaling", { + service: 'kinesisanalyticsv2', // API is "v2" + action: 'describeApplication', + iamResources: [`arn:aws:kinesisanalytics:${this.region}:${this.account}:application/${applicationName}`], + additionalIamStatements: [kdaDescribeApplicationPolicyStatement], // ensure IAM policy has the correct permissions + parameters: { + "ApplicationName": applicationName + }, + resultPath: "$.TaskResult", + }) + + + const describeCwAlarmTask: cdk.aws_stepfunctions_tasks.CallAwsService = new tasks.CallAwsService(this, "Describe Alarm after waiting", { + service: 'cloudwatch', + action: 'describeAlarms', + iamResources: [`arn:aws:cloudwatch:${this.region}:${this.account}:alarm:*`], + parameters: { + "AlarmNames.$": "States.Array($.detail.alarmName)", + "AlarmTypes": ['CompositeAlarm', 'MetricAlarm'] + }, + resultPath: "$.TaskResult" + }) + + const definition: sfn.Chain = describeKdaApplicationTask + .next(lambdaTask) + .next(lambdaChoice + .when(sfn.Condition.numberEquals('$.TaskResult.Payload.body', 1), successState) + .otherwise(waitUpdate + .next(describeKdaApplicationAfterScalingTask) + .next(scalingChoice + .when(sfn.Condition.stringEquals('$.TaskResult.ApplicationDetail.ApplicationStatus', 'UPDATING'), waitUpdate) + .otherwise(waitCoolingPeriod + .next(describeCwAlarmTask.next(alarmChoice + .when(sfn.Condition.stringEquals('$.TaskResult.MetricAlarms[0].StateValue', 'ALARM'), describeKdaApplicationTask) + .otherwise(successState))))))); + + const stateMachine: sfn.StateMachine = new sfn.StateMachine(this, 'StateMachine', { + definitionBody: sfn.DefinitionBody.fromChainable(definition) + }); + + const stepFunctionPolicy: cdk.aws_iam.PolicyDocument = new iam.PolicyDocument({ + statements: [ + new iam.PolicyStatement({ + resources: [stateMachine.stateMachineArn], + actions: ['states:StartExecution'] + }), + ], + }); + + const eventBridgeRole: cdk.aws_iam.Role = new iam.Role(this, 'EventBridge Role', { + assumedBy: new iam.ServicePrincipal('events.amazonaws.com'), + description: 'EventBridge Role', + inlinePolicies: { + AccessStepFunctionsPolicy: stepFunctionPolicy + }, + }); + + scaleRule.addTarget(new aws_events_targets.SfnStateMachine(stateMachine, { + role: eventBridgeRole + })); + + + /// CFN Outputs + new cdk.CfnOutput(this, "Autoscaler Metric Name", {value: autoscaleMetricName_}); + new cdk.CfnOutput(this, "Autoscaler Metric Statistic", {value: autoscaleMetricStat_}); + new cdk.CfnOutput(this, "Application name", {value: applicationName}); + new cdk.CfnOutput(this, "CW Metric Namespace", {value: triggerMetricNamespace}); + new cdk.CfnOutput(this, "CW Metric Dimensions", {value: JSON.stringify(triggerMetricDimensions)}); + + new cdk.CfnOutput(this, "Scale-in alarm name", {value: scaleInAlarm.alarmName}); + new cdk.CfnOutput(this, "Scale-out alarm name", {value: scaleOutAlarm.alarmName}); + + } +} + diff --git a/infrastructure/AutoScaling/cdk/package-lock.json b/infrastructure/AutoScaling/cdk/package-lock.json new file mode 100644 index 00000000..c6fb8458 --- /dev/null +++ b/infrastructure/AutoScaling/cdk/package-lock.json @@ -0,0 +1,4152 @@ +{ + "name": "kda-autoscaling", + "version": "0.1.0", + "lockfileVersion": 3, + "requires": true, + "packages": { + "": { + "name": "kda-autoscaling", + "version": "0.1.0", + "dependencies": { + "aws-cdk-lib": "^2.192.0", + "constructs": "^10.4.2", + "source-map-support": "^0.5.21" + }, + "bin": { + "kda-autoscaling": "bin/kda-autoscaling.js" + }, + "devDependencies": { + "@types/jest": "^29.5.1", + "@types/node": "20.1.7", + "aws-cdk": "^2.1001.0", + "jest": "^29.5.0", + "ts-jest": "^29.1.0", + "ts-node": "^10.9.1", + "typescript": "~5.0.4" + } + }, + "node_modules/@ampproject/remapping": { + "version": "2.3.0", + "resolved": "https://registry.npmjs.org/@ampproject/remapping/-/remapping-2.3.0.tgz", + "integrity": "sha512-30iZtAPgz+LTIYoeivqYo853f02jBYSd5uGnGpkFV0M3xOt9aN73erkgYAmZU43x4VfqcnLxW9Kpg3R5LC4YYw==", + "dev": true, + "dependencies": { + "@jridgewell/gen-mapping": "^0.3.5", + "@jridgewell/trace-mapping": "^0.3.24" + }, + "engines": { + "node": ">=6.0.0" + } + }, + "node_modules/@aws-cdk/asset-awscli-v1": { + "version": "2.2.233", + "resolved": "https://registry.npmjs.org/@aws-cdk/asset-awscli-v1/-/asset-awscli-v1-2.2.233.tgz", + "integrity": "sha512-OH5ZN1F/0wwOUwzVUSvE0/syUOi44H9the6IG16anlSptfeQ1fvduJazZAKRuJLtautPbiqxllyOrtWh6LhX8A==" + }, + "node_modules/@aws-cdk/asset-node-proxy-agent-v6": { + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/@aws-cdk/asset-node-proxy-agent-v6/-/asset-node-proxy-agent-v6-2.1.0.tgz", + "integrity": "sha512-7bY3J8GCVxLupn/kNmpPc5VJz8grx+4RKfnnJiO1LG+uxkZfANZG3RMHhE+qQxxwkyQ9/MfPtTpf748UhR425A==" + }, + "node_modules/@aws-cdk/cloud-assembly-schema": { + "version": "41.2.0", + "resolved": "https://registry.npmjs.org/@aws-cdk/cloud-assembly-schema/-/cloud-assembly-schema-41.2.0.tgz", + "integrity": "sha512-JaulVS6z9y5+u4jNmoWbHZRs9uGOnmn/ktXygNWKNu1k6lF3ad4so3s18eRu15XCbUIomxN9WPYT6Ehh7hzONw==", + "bundleDependencies": [ + "jsonschema", + "semver" + ], + "dependencies": { + "jsonschema": "~1.4.1", + "semver": "^7.7.1" + }, + "engines": { + "node": ">= 14.15.0" + } + }, + "node_modules/@aws-cdk/cloud-assembly-schema/node_modules/jsonschema": { + "version": "1.4.1", + "inBundle": true, + "license": "MIT", + "engines": { + "node": "*" + } + }, + "node_modules/@aws-cdk/cloud-assembly-schema/node_modules/semver": { + "version": "7.7.1", + "inBundle": true, + "license": "ISC", + "bin": { + "semver": "bin/semver.js" + }, + "engines": { + "node": ">=10" + } + }, + "node_modules/@babel/code-frame": { + "version": "7.26.2", + "resolved": "https://registry.npmjs.org/@babel/code-frame/-/code-frame-7.26.2.tgz", + "integrity": "sha512-RJlIHRueQgwWitWgF8OdFYGZX328Ax5BCemNGlqHfplnRT9ESi8JkFlvaVYbS+UubVY6dpv87Fs2u5M29iNFVQ==", + "dev": true, + "dependencies": { + "@babel/helper-validator-identifier": "^7.25.9", + "js-tokens": "^4.0.0", + "picocolors": "^1.0.0" + }, + "engines": { + "node": ">=6.9.0" + } + }, + "node_modules/@babel/compat-data": { + "version": "7.26.8", + "resolved": "https://registry.npmjs.org/@babel/compat-data/-/compat-data-7.26.8.tgz", + "integrity": "sha512-oH5UPLMWR3L2wEFLnFJ1TZXqHufiTKAiLfqw5zkhS4dKXLJ10yVztfil/twG8EDTA4F/tvVNw9nOl4ZMslB8rQ==", + "dev": true, + "engines": { + "node": ">=6.9.0" + } + }, + "node_modules/@babel/core": { + "version": "7.26.10", + "resolved": "https://registry.npmjs.org/@babel/core/-/core-7.26.10.tgz", + "integrity": "sha512-vMqyb7XCDMPvJFFOaT9kxtiRh42GwlZEg1/uIgtZshS5a/8OaduUfCi7kynKgc3Tw/6Uo2D+db9qBttghhmxwQ==", + "dev": true, + "dependencies": { + "@ampproject/remapping": "^2.2.0", + "@babel/code-frame": "^7.26.2", + "@babel/generator": "^7.26.10", + "@babel/helper-compilation-targets": "^7.26.5", + "@babel/helper-module-transforms": "^7.26.0", + "@babel/helpers": "^7.26.10", + "@babel/parser": "^7.26.10", + "@babel/template": "^7.26.9", + "@babel/traverse": "^7.26.10", + "@babel/types": "^7.26.10", + "convert-source-map": "^2.0.0", + "debug": "^4.1.0", + "gensync": "^1.0.0-beta.2", + "json5": "^2.2.3", + "semver": "^6.3.1" + }, + "engines": { + "node": ">=6.9.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/babel" + } + }, + "node_modules/@babel/generator": { + "version": "7.27.0", + "resolved": "https://registry.npmjs.org/@babel/generator/-/generator-7.27.0.tgz", + "integrity": "sha512-VybsKvpiN1gU1sdMZIp7FcqphVVKEwcuj02x73uvcHE0PTihx1nlBcowYWhDwjpoAXRv43+gDzyggGnn1XZhVw==", + "dev": true, + "dependencies": { + "@babel/parser": "^7.27.0", + "@babel/types": "^7.27.0", + "@jridgewell/gen-mapping": "^0.3.5", + "@jridgewell/trace-mapping": "^0.3.25", + "jsesc": "^3.0.2" + }, + "engines": { + "node": ">=6.9.0" + } + }, + "node_modules/@babel/helper-compilation-targets": { + "version": "7.27.0", + "resolved": "https://registry.npmjs.org/@babel/helper-compilation-targets/-/helper-compilation-targets-7.27.0.tgz", + "integrity": "sha512-LVk7fbXml0H2xH34dFzKQ7TDZ2G4/rVTOrq9V+icbbadjbVxxeFeDsNHv2SrZeWoA+6ZiTyWYWtScEIW07EAcA==", + "dev": true, + "dependencies": { + "@babel/compat-data": "^7.26.8", + "@babel/helper-validator-option": "^7.25.9", + "browserslist": "^4.24.0", + "lru-cache": "^5.1.1", + "semver": "^6.3.1" + }, + "engines": { + "node": ">=6.9.0" + } + }, + "node_modules/@babel/helper-module-imports": { + "version": "7.25.9", + "resolved": "https://registry.npmjs.org/@babel/helper-module-imports/-/helper-module-imports-7.25.9.tgz", + "integrity": "sha512-tnUA4RsrmflIM6W6RFTLFSXITtl0wKjgpnLgXyowocVPrbYrLUXSBXDgTs8BlbmIzIdlBySRQjINYs2BAkiLtw==", + "dev": true, + "dependencies": { + "@babel/traverse": "^7.25.9", + "@babel/types": "^7.25.9" + }, + "engines": { + "node": ">=6.9.0" + } + }, + "node_modules/@babel/helper-module-transforms": { + "version": "7.26.0", + "resolved": "https://registry.npmjs.org/@babel/helper-module-transforms/-/helper-module-transforms-7.26.0.tgz", + "integrity": "sha512-xO+xu6B5K2czEnQye6BHA7DolFFmS3LB7stHZFaOLb1pAwO1HWLS8fXA+eh0A2yIvltPVmx3eNNDBJA2SLHXFw==", + "dev": true, + "dependencies": { + "@babel/helper-module-imports": "^7.25.9", + "@babel/helper-validator-identifier": "^7.25.9", + "@babel/traverse": "^7.25.9" + }, + "engines": { + "node": ">=6.9.0" + }, + "peerDependencies": { + "@babel/core": "^7.0.0" + } + }, + "node_modules/@babel/helper-plugin-utils": { + "version": "7.26.5", + "resolved": "https://registry.npmjs.org/@babel/helper-plugin-utils/-/helper-plugin-utils-7.26.5.tgz", + "integrity": "sha512-RS+jZcRdZdRFzMyr+wcsaqOmld1/EqTghfaBGQQd/WnRdzdlvSZ//kF7U8VQTxf1ynZ4cjUcYgjVGx13ewNPMg==", + "dev": true, + "engines": { + "node": ">=6.9.0" + } + }, + "node_modules/@babel/helper-string-parser": { + "version": "7.25.9", + "resolved": "https://registry.npmjs.org/@babel/helper-string-parser/-/helper-string-parser-7.25.9.tgz", + "integrity": "sha512-4A/SCr/2KLd5jrtOMFzaKjVtAei3+2r/NChoBNoZ3EyP/+GlhoaEGoWOZUmFmoITP7zOJyHIMm+DYRd8o3PvHA==", + "dev": true, + "engines": { + "node": ">=6.9.0" + } + }, + "node_modules/@babel/helper-validator-identifier": { + "version": "7.25.9", + "resolved": "https://registry.npmjs.org/@babel/helper-validator-identifier/-/helper-validator-identifier-7.25.9.tgz", + "integrity": "sha512-Ed61U6XJc3CVRfkERJWDz4dJwKe7iLmmJsbOGu9wSloNSFttHV0I8g6UAgb7qnK5ly5bGLPd4oXZlxCdANBOWQ==", + "dev": true, + "engines": { + "node": ">=6.9.0" + } + }, + "node_modules/@babel/helper-validator-option": { + "version": "7.25.9", + "resolved": "https://registry.npmjs.org/@babel/helper-validator-option/-/helper-validator-option-7.25.9.tgz", + "integrity": "sha512-e/zv1co8pp55dNdEcCynfj9X7nyUKUXoUEwfXqaZt0omVOmDe9oOTdKStH4GmAw6zxMFs50ZayuMfHDKlO7Tfw==", + "dev": true, + "engines": { + "node": ">=6.9.0" + } + }, + "node_modules/@babel/helpers": { + "version": "7.27.0", + "resolved": "https://registry.npmjs.org/@babel/helpers/-/helpers-7.27.0.tgz", + "integrity": "sha512-U5eyP/CTFPuNE3qk+WZMxFkp/4zUzdceQlfzf7DdGdhp+Fezd7HD+i8Y24ZuTMKX3wQBld449jijbGq6OdGNQg==", + "dev": true, + "dependencies": { + "@babel/template": "^7.27.0", + "@babel/types": "^7.27.0" + }, + "engines": { + "node": ">=6.9.0" + } + }, + "node_modules/@babel/parser": { + "version": "7.27.0", + "resolved": "https://registry.npmjs.org/@babel/parser/-/parser-7.27.0.tgz", + "integrity": "sha512-iaepho73/2Pz7w2eMS0Q5f83+0RKI7i4xmiYeBmDzfRVbQtTOG7Ts0S4HzJVsTMGI9keU8rNfuZr8DKfSt7Yyg==", + "dev": true, + "dependencies": { + "@babel/types": "^7.27.0" + }, + "bin": { + "parser": "bin/babel-parser.js" + }, + "engines": { + "node": ">=6.0.0" + } + }, + "node_modules/@babel/plugin-syntax-async-generators": { + "version": "7.8.4", + "resolved": "https://registry.npmjs.org/@babel/plugin-syntax-async-generators/-/plugin-syntax-async-generators-7.8.4.tgz", + "integrity": "sha512-tycmZxkGfZaxhMRbXlPXuVFpdWlXpir2W4AMhSJgRKzk/eDlIXOhb2LHWoLpDF7TEHylV5zNhykX6KAgHJmTNw==", + "dev": true, + "dependencies": { + "@babel/helper-plugin-utils": "^7.8.0" + }, + "peerDependencies": { + "@babel/core": "^7.0.0-0" + } + }, + "node_modules/@babel/plugin-syntax-bigint": { + "version": "7.8.3", + "resolved": "https://registry.npmjs.org/@babel/plugin-syntax-bigint/-/plugin-syntax-bigint-7.8.3.tgz", + "integrity": "sha512-wnTnFlG+YxQm3vDxpGE57Pj0srRU4sHE/mDkt1qv2YJJSeUAec2ma4WLUnUPeKjyrfntVwe/N6dCXpU+zL3Npg==", + "dev": true, + "dependencies": { + "@babel/helper-plugin-utils": "^7.8.0" + }, + "peerDependencies": { + "@babel/core": "^7.0.0-0" + } + }, + "node_modules/@babel/plugin-syntax-class-properties": { + "version": "7.12.13", + "resolved": "https://registry.npmjs.org/@babel/plugin-syntax-class-properties/-/plugin-syntax-class-properties-7.12.13.tgz", + "integrity": "sha512-fm4idjKla0YahUNgFNLCB0qySdsoPiZP3iQE3rky0mBUtMZ23yDJ9SJdg6dXTSDnulOVqiF3Hgr9nbXvXTQZYA==", + "dev": true, + "dependencies": { + "@babel/helper-plugin-utils": "^7.12.13" + }, + "peerDependencies": { + "@babel/core": "^7.0.0-0" + } + }, + "node_modules/@babel/plugin-syntax-class-static-block": { + "version": "7.14.5", + "resolved": "https://registry.npmjs.org/@babel/plugin-syntax-class-static-block/-/plugin-syntax-class-static-block-7.14.5.tgz", + "integrity": "sha512-b+YyPmr6ldyNnM6sqYeMWE+bgJcJpO6yS4QD7ymxgH34GBPNDM/THBh8iunyvKIZztiwLH4CJZ0RxTk9emgpjw==", + "dev": true, + "dependencies": { + "@babel/helper-plugin-utils": "^7.14.5" + }, + "engines": { + "node": ">=6.9.0" + }, + "peerDependencies": { + "@babel/core": "^7.0.0-0" + } + }, + "node_modules/@babel/plugin-syntax-import-attributes": { + "version": "7.26.0", + "resolved": "https://registry.npmjs.org/@babel/plugin-syntax-import-attributes/-/plugin-syntax-import-attributes-7.26.0.tgz", + "integrity": "sha512-e2dttdsJ1ZTpi3B9UYGLw41hifAubg19AtCu/2I/F1QNVclOBr1dYpTdmdyZ84Xiz43BS/tCUkMAZNLv12Pi+A==", + "dev": true, + "dependencies": { + "@babel/helper-plugin-utils": "^7.25.9" + }, + "engines": { + "node": ">=6.9.0" + }, + "peerDependencies": { + "@babel/core": "^7.0.0-0" + } + }, + "node_modules/@babel/plugin-syntax-import-meta": { + "version": "7.10.4", + "resolved": "https://registry.npmjs.org/@babel/plugin-syntax-import-meta/-/plugin-syntax-import-meta-7.10.4.tgz", + "integrity": "sha512-Yqfm+XDx0+Prh3VSeEQCPU81yC+JWZ2pDPFSS4ZdpfZhp4MkFMaDC1UqseovEKwSUpnIL7+vK+Clp7bfh0iD7g==", + "dev": true, + "dependencies": { + "@babel/helper-plugin-utils": "^7.10.4" + }, + "peerDependencies": { + "@babel/core": "^7.0.0-0" + } + }, + "node_modules/@babel/plugin-syntax-json-strings": { + "version": "7.8.3", + "resolved": "https://registry.npmjs.org/@babel/plugin-syntax-json-strings/-/plugin-syntax-json-strings-7.8.3.tgz", + "integrity": "sha512-lY6kdGpWHvjoe2vk4WrAapEuBR69EMxZl+RoGRhrFGNYVK8mOPAW8VfbT/ZgrFbXlDNiiaxQnAtgVCZ6jv30EA==", + "dev": true, + "dependencies": { + "@babel/helper-plugin-utils": "^7.8.0" + }, + "peerDependencies": { + "@babel/core": "^7.0.0-0" + } + }, + "node_modules/@babel/plugin-syntax-jsx": { + "version": "7.25.9", + "resolved": "https://registry.npmjs.org/@babel/plugin-syntax-jsx/-/plugin-syntax-jsx-7.25.9.tgz", + "integrity": "sha512-ld6oezHQMZsZfp6pWtbjaNDF2tiiCYYDqQszHt5VV437lewP9aSi2Of99CK0D0XB21k7FLgnLcmQKyKzynfeAA==", + "dev": true, + "dependencies": { + "@babel/helper-plugin-utils": "^7.25.9" + }, + "engines": { + "node": ">=6.9.0" + }, + "peerDependencies": { + "@babel/core": "^7.0.0-0" + } + }, + "node_modules/@babel/plugin-syntax-logical-assignment-operators": { + "version": "7.10.4", + "resolved": "https://registry.npmjs.org/@babel/plugin-syntax-logical-assignment-operators/-/plugin-syntax-logical-assignment-operators-7.10.4.tgz", + "integrity": "sha512-d8waShlpFDinQ5MtvGU9xDAOzKH47+FFoney2baFIoMr952hKOLp1HR7VszoZvOsV/4+RRszNY7D17ba0te0ig==", + "dev": true, + "dependencies": { + "@babel/helper-plugin-utils": "^7.10.4" + }, + "peerDependencies": { + "@babel/core": "^7.0.0-0" + } + }, + "node_modules/@babel/plugin-syntax-nullish-coalescing-operator": { + "version": "7.8.3", + "resolved": "https://registry.npmjs.org/@babel/plugin-syntax-nullish-coalescing-operator/-/plugin-syntax-nullish-coalescing-operator-7.8.3.tgz", + "integrity": "sha512-aSff4zPII1u2QD7y+F8oDsz19ew4IGEJg9SVW+bqwpwtfFleiQDMdzA/R+UlWDzfnHFCxxleFT0PMIrR36XLNQ==", + "dev": true, + "dependencies": { + "@babel/helper-plugin-utils": "^7.8.0" + }, + "peerDependencies": { + "@babel/core": "^7.0.0-0" + } + }, + "node_modules/@babel/plugin-syntax-numeric-separator": { + "version": "7.10.4", + "resolved": "https://registry.npmjs.org/@babel/plugin-syntax-numeric-separator/-/plugin-syntax-numeric-separator-7.10.4.tgz", + "integrity": "sha512-9H6YdfkcK/uOnY/K7/aA2xpzaAgkQn37yzWUMRK7OaPOqOpGS1+n0H5hxT9AUw9EsSjPW8SVyMJwYRtWs3X3ug==", + "dev": true, + "dependencies": { + "@babel/helper-plugin-utils": "^7.10.4" + }, + "peerDependencies": { + "@babel/core": "^7.0.0-0" + } + }, + "node_modules/@babel/plugin-syntax-object-rest-spread": { + "version": "7.8.3", + "resolved": "https://registry.npmjs.org/@babel/plugin-syntax-object-rest-spread/-/plugin-syntax-object-rest-spread-7.8.3.tgz", + "integrity": "sha512-XoqMijGZb9y3y2XskN+P1wUGiVwWZ5JmoDRwx5+3GmEplNyVM2s2Dg8ILFQm8rWM48orGy5YpI5Bl8U1y7ydlA==", + "dev": true, + "dependencies": { + "@babel/helper-plugin-utils": "^7.8.0" + }, + "peerDependencies": { + "@babel/core": "^7.0.0-0" + } + }, + "node_modules/@babel/plugin-syntax-optional-catch-binding": { + "version": "7.8.3", + "resolved": "https://registry.npmjs.org/@babel/plugin-syntax-optional-catch-binding/-/plugin-syntax-optional-catch-binding-7.8.3.tgz", + "integrity": "sha512-6VPD0Pc1lpTqw0aKoeRTMiB+kWhAoT24PA+ksWSBrFtl5SIRVpZlwN3NNPQjehA2E/91FV3RjLWoVTglWcSV3Q==", + "dev": true, + "dependencies": { + "@babel/helper-plugin-utils": "^7.8.0" + }, + "peerDependencies": { + "@babel/core": "^7.0.0-0" + } + }, + "node_modules/@babel/plugin-syntax-optional-chaining": { + "version": "7.8.3", + "resolved": "https://registry.npmjs.org/@babel/plugin-syntax-optional-chaining/-/plugin-syntax-optional-chaining-7.8.3.tgz", + "integrity": "sha512-KoK9ErH1MBlCPxV0VANkXW2/dw4vlbGDrFgz8bmUsBGYkFRcbRwMh6cIJubdPrkxRwuGdtCk0v/wPTKbQgBjkg==", + "dev": true, + "dependencies": { + "@babel/helper-plugin-utils": "^7.8.0" + }, + "peerDependencies": { + "@babel/core": "^7.0.0-0" + } + }, + "node_modules/@babel/plugin-syntax-private-property-in-object": { + "version": "7.14.5", + "resolved": "https://registry.npmjs.org/@babel/plugin-syntax-private-property-in-object/-/plugin-syntax-private-property-in-object-7.14.5.tgz", + "integrity": "sha512-0wVnp9dxJ72ZUJDV27ZfbSj6iHLoytYZmh3rFcxNnvsJF3ktkzLDZPy/mA17HGsaQT3/DQsWYX1f1QGWkCoVUg==", + "dev": true, + "dependencies": { + "@babel/helper-plugin-utils": "^7.14.5" + }, + "engines": { + "node": ">=6.9.0" + }, + "peerDependencies": { + "@babel/core": "^7.0.0-0" + } + }, + "node_modules/@babel/plugin-syntax-top-level-await": { + "version": "7.14.5", + "resolved": "https://registry.npmjs.org/@babel/plugin-syntax-top-level-await/-/plugin-syntax-top-level-await-7.14.5.tgz", + "integrity": "sha512-hx++upLv5U1rgYfwe1xBQUhRmU41NEvpUvrp8jkrSCdvGSnM5/qdRMtylJ6PG5OFkBaHkbTAKTnd3/YyESRHFw==", + "dev": true, + "dependencies": { + "@babel/helper-plugin-utils": "^7.14.5" + }, + "engines": { + "node": ">=6.9.0" + }, + "peerDependencies": { + "@babel/core": "^7.0.0-0" + } + }, + "node_modules/@babel/plugin-syntax-typescript": { + "version": "7.25.9", + "resolved": "https://registry.npmjs.org/@babel/plugin-syntax-typescript/-/plugin-syntax-typescript-7.25.9.tgz", + "integrity": "sha512-hjMgRy5hb8uJJjUcdWunWVcoi9bGpJp8p5Ol1229PoN6aytsLwNMgmdftO23wnCLMfVmTwZDWMPNq/D1SY60JQ==", + "dev": true, + "dependencies": { + "@babel/helper-plugin-utils": "^7.25.9" + }, + "engines": { + "node": ">=6.9.0" + }, + "peerDependencies": { + "@babel/core": "^7.0.0-0" + } + }, + "node_modules/@babel/template": { + "version": "7.27.0", + "resolved": "https://registry.npmjs.org/@babel/template/-/template-7.27.0.tgz", + "integrity": "sha512-2ncevenBqXI6qRMukPlXwHKHchC7RyMuu4xv5JBXRfOGVcTy1mXCD12qrp7Jsoxll1EV3+9sE4GugBVRjT2jFA==", + "dev": true, + "dependencies": { + "@babel/code-frame": "^7.26.2", + "@babel/parser": "^7.27.0", + "@babel/types": "^7.27.0" + }, + "engines": { + "node": ">=6.9.0" + } + }, + "node_modules/@babel/traverse": { + "version": "7.27.0", + "resolved": "https://registry.npmjs.org/@babel/traverse/-/traverse-7.27.0.tgz", + "integrity": "sha512-19lYZFzYVQkkHkl4Cy4WrAVcqBkgvV2YM2TU3xG6DIwO7O3ecbDPfW3yM3bjAGcqcQHi+CCtjMR3dIEHxsd6bA==", + "dev": true, + "dependencies": { + "@babel/code-frame": "^7.26.2", + "@babel/generator": "^7.27.0", + "@babel/parser": "^7.27.0", + "@babel/template": "^7.27.0", + "@babel/types": "^7.27.0", + "debug": "^4.3.1", + "globals": "^11.1.0" + }, + "engines": { + "node": ">=6.9.0" + } + }, + "node_modules/@babel/types": { + "version": "7.27.0", + "resolved": "https://registry.npmjs.org/@babel/types/-/types-7.27.0.tgz", + "integrity": "sha512-H45s8fVLYjbhFH62dIJ3WtmJ6RSPt/3DRO0ZcT2SUiYiQyz3BLVb9ADEnLl91m74aQPS3AzzeajZHYOalWe3bg==", + "dev": true, + "dependencies": { + "@babel/helper-string-parser": "^7.25.9", + "@babel/helper-validator-identifier": "^7.25.9" + }, + "engines": { + "node": ">=6.9.0" + } + }, + "node_modules/@bcoe/v8-coverage": { + "version": "0.2.3", + "resolved": "https://registry.npmjs.org/@bcoe/v8-coverage/-/v8-coverage-0.2.3.tgz", + "integrity": "sha512-0hYQ8SB4Db5zvZB4axdMHGwEaQjkZzFjQiN9LVYvIFB2nSUHW9tYpxWriPrWDASIxiaXax83REcLxuSdnGPZtw==", + "dev": true + }, + "node_modules/@cspotcode/source-map-support": { + "version": "0.8.1", + "resolved": "https://registry.npmjs.org/@cspotcode/source-map-support/-/source-map-support-0.8.1.tgz", + "integrity": "sha512-IchNf6dN4tHoMFIn/7OE8LWZ19Y6q/67Bmf6vnGREv8RSbBVb9LPJxEcnwrcwX6ixSvaiGoomAUvu4YSxXrVgw==", + "dev": true, + "dependencies": { + "@jridgewell/trace-mapping": "0.3.9" + }, + "engines": { + "node": ">=12" + } + }, + "node_modules/@cspotcode/source-map-support/node_modules/@jridgewell/trace-mapping": { + "version": "0.3.9", + "resolved": "https://registry.npmjs.org/@jridgewell/trace-mapping/-/trace-mapping-0.3.9.tgz", + "integrity": "sha512-3Belt6tdc8bPgAtbcmdtNJlirVoTmEb5e2gC94PnkwEW9jI6CAHUeoG85tjWP5WquqfavoMtMwiG4P926ZKKuQ==", + "dev": true, + "dependencies": { + "@jridgewell/resolve-uri": "^3.0.3", + "@jridgewell/sourcemap-codec": "^1.4.10" + } + }, + "node_modules/@istanbuljs/load-nyc-config": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/@istanbuljs/load-nyc-config/-/load-nyc-config-1.1.0.tgz", + "integrity": "sha512-VjeHSlIzpv/NyD3N0YuHfXOPDIixcA1q2ZV98wsMqcYlPmv2n3Yb2lYP9XMElnaFVXg5A7YLTeLu6V84uQDjmQ==", + "dev": true, + "dependencies": { + "camelcase": "^5.3.1", + "find-up": "^4.1.0", + "get-package-type": "^0.1.0", + "js-yaml": "^3.13.1", + "resolve-from": "^5.0.0" + }, + "engines": { + "node": ">=8" + } + }, + "node_modules/@istanbuljs/schema": { + "version": "0.1.3", + "resolved": "https://registry.npmjs.org/@istanbuljs/schema/-/schema-0.1.3.tgz", + "integrity": "sha512-ZXRY4jNvVgSVQ8DL3LTcakaAtXwTVUxE81hslsyD2AtoXW/wVob10HkOJ1X/pAlcI7D+2YoZKg5do8G/w6RYgA==", + "dev": true, + "engines": { + "node": ">=8" + } + }, + "node_modules/@jest/console": { + "version": "29.7.0", + "resolved": "https://registry.npmjs.org/@jest/console/-/console-29.7.0.tgz", + "integrity": "sha512-5Ni4CU7XHQi32IJ398EEP4RrB8eV09sXP2ROqD4bksHrnTree52PsxvX8tpL8LvTZ3pFzXyPbNQReSN41CAhOg==", + "dev": true, + "dependencies": { + "@jest/types": "^29.6.3", + "@types/node": "*", + "chalk": "^4.0.0", + "jest-message-util": "^29.7.0", + "jest-util": "^29.7.0", + "slash": "^3.0.0" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + } + }, + "node_modules/@jest/core": { + "version": "29.7.0", + "resolved": "https://registry.npmjs.org/@jest/core/-/core-29.7.0.tgz", + "integrity": "sha512-n7aeXWKMnGtDA48y8TLWJPJmLmmZ642Ceo78cYWEpiD7FzDgmNDV/GCVRorPABdXLJZ/9wzzgZAlHjXjxDHGsg==", + "dev": true, + "dependencies": { + "@jest/console": "^29.7.0", + "@jest/reporters": "^29.7.0", + "@jest/test-result": "^29.7.0", + "@jest/transform": "^29.7.0", + "@jest/types": "^29.6.3", + "@types/node": "*", + "ansi-escapes": "^4.2.1", + "chalk": "^4.0.0", + "ci-info": "^3.2.0", + "exit": "^0.1.2", + "graceful-fs": "^4.2.9", + "jest-changed-files": "^29.7.0", + "jest-config": "^29.7.0", + "jest-haste-map": "^29.7.0", + "jest-message-util": "^29.7.0", + "jest-regex-util": "^29.6.3", + "jest-resolve": "^29.7.0", + "jest-resolve-dependencies": "^29.7.0", + "jest-runner": "^29.7.0", + "jest-runtime": "^29.7.0", + "jest-snapshot": "^29.7.0", + "jest-util": "^29.7.0", + "jest-validate": "^29.7.0", + "jest-watcher": "^29.7.0", + "micromatch": "^4.0.4", + "pretty-format": "^29.7.0", + "slash": "^3.0.0", + "strip-ansi": "^6.0.0" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + }, + "peerDependencies": { + "node-notifier": "^8.0.1 || ^9.0.0 || ^10.0.0" + }, + "peerDependenciesMeta": { + "node-notifier": { + "optional": true + } + } + }, + "node_modules/@jest/environment": { + "version": "29.7.0", + "resolved": "https://registry.npmjs.org/@jest/environment/-/environment-29.7.0.tgz", + "integrity": "sha512-aQIfHDq33ExsN4jP1NWGXhxgQ/wixs60gDiKO+XVMd8Mn0NWPWgc34ZQDTb2jKaUWQ7MuwoitXAsN2XVXNMpAw==", + "dev": true, + "dependencies": { + "@jest/fake-timers": "^29.7.0", + "@jest/types": "^29.6.3", + "@types/node": "*", + "jest-mock": "^29.7.0" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + } + }, + "node_modules/@jest/expect": { + "version": "29.7.0", + "resolved": "https://registry.npmjs.org/@jest/expect/-/expect-29.7.0.tgz", + "integrity": "sha512-8uMeAMycttpva3P1lBHB8VciS9V0XAr3GymPpipdyQXbBcuhkLQOSe8E/p92RyAdToS6ZD1tFkX+CkhoECE0dQ==", + "dev": true, + "dependencies": { + "expect": "^29.7.0", + "jest-snapshot": "^29.7.0" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + } + }, + "node_modules/@jest/expect-utils": { + "version": "29.7.0", + "resolved": "https://registry.npmjs.org/@jest/expect-utils/-/expect-utils-29.7.0.tgz", + "integrity": "sha512-GlsNBWiFQFCVi9QVSx7f5AgMeLxe9YCCs5PuP2O2LdjDAA8Jh9eX7lA1Jq/xdXw3Wb3hyvlFNfZIfcRetSzYcA==", + "dev": true, + "dependencies": { + "jest-get-type": "^29.6.3" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + } + }, + "node_modules/@jest/fake-timers": { + "version": "29.7.0", + "resolved": "https://registry.npmjs.org/@jest/fake-timers/-/fake-timers-29.7.0.tgz", + "integrity": "sha512-q4DH1Ha4TTFPdxLsqDXK1d3+ioSL7yL5oCMJZgDYm6i+6CygW5E5xVr/D1HdsGxjt1ZWSfUAs9OxSB/BNelWrQ==", + "dev": true, + "dependencies": { + "@jest/types": "^29.6.3", + "@sinonjs/fake-timers": "^10.0.2", + "@types/node": "*", + "jest-message-util": "^29.7.0", + "jest-mock": "^29.7.0", + "jest-util": "^29.7.0" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + } + }, + "node_modules/@jest/globals": { + "version": "29.7.0", + "resolved": "https://registry.npmjs.org/@jest/globals/-/globals-29.7.0.tgz", + "integrity": "sha512-mpiz3dutLbkW2MNFubUGUEVLkTGiqW6yLVTA+JbP6fI6J5iL9Y0Nlg8k95pcF8ctKwCS7WVxteBs29hhfAotzQ==", + "dev": true, + "dependencies": { + "@jest/environment": "^29.7.0", + "@jest/expect": "^29.7.0", + "@jest/types": "^29.6.3", + "jest-mock": "^29.7.0" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + } + }, + "node_modules/@jest/reporters": { + "version": "29.7.0", + "resolved": "https://registry.npmjs.org/@jest/reporters/-/reporters-29.7.0.tgz", + "integrity": "sha512-DApq0KJbJOEzAFYjHADNNxAE3KbhxQB1y5Kplb5Waqw6zVbuWatSnMjE5gs8FUgEPmNsnZA3NCWl9NG0ia04Pg==", + "dev": true, + "dependencies": { + "@bcoe/v8-coverage": "^0.2.3", + "@jest/console": "^29.7.0", + "@jest/test-result": "^29.7.0", + "@jest/transform": "^29.7.0", + "@jest/types": "^29.6.3", + "@jridgewell/trace-mapping": "^0.3.18", + "@types/node": "*", + "chalk": "^4.0.0", + "collect-v8-coverage": "^1.0.0", + "exit": "^0.1.2", + "glob": "^7.1.3", + "graceful-fs": "^4.2.9", + "istanbul-lib-coverage": "^3.0.0", + "istanbul-lib-instrument": "^6.0.0", + "istanbul-lib-report": "^3.0.0", + "istanbul-lib-source-maps": "^4.0.0", + "istanbul-reports": "^3.1.3", + "jest-message-util": "^29.7.0", + "jest-util": "^29.7.0", + "jest-worker": "^29.7.0", + "slash": "^3.0.0", + "string-length": "^4.0.1", + "strip-ansi": "^6.0.0", + "v8-to-istanbul": "^9.0.1" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + }, + "peerDependencies": { + "node-notifier": "^8.0.1 || ^9.0.0 || ^10.0.0" + }, + "peerDependenciesMeta": { + "node-notifier": { + "optional": true + } + } + }, + "node_modules/@jest/schemas": { + "version": "29.6.3", + "resolved": "https://registry.npmjs.org/@jest/schemas/-/schemas-29.6.3.tgz", + "integrity": "sha512-mo5j5X+jIZmJQveBKeS/clAueipV7KgiX1vMgCxam1RNYiqE1w62n0/tJJnHtjW8ZHcQco5gY85jA3mi0L+nSA==", + "dev": true, + "dependencies": { + "@sinclair/typebox": "^0.27.8" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + } + }, + "node_modules/@jest/source-map": { + "version": "29.6.3", + "resolved": "https://registry.npmjs.org/@jest/source-map/-/source-map-29.6.3.tgz", + "integrity": "sha512-MHjT95QuipcPrpLM+8JMSzFx6eHp5Bm+4XeFDJlwsvVBjmKNiIAvasGK2fxz2WbGRlnvqehFbh07MMa7n3YJnw==", + "dev": true, + "dependencies": { + "@jridgewell/trace-mapping": "^0.3.18", + "callsites": "^3.0.0", + "graceful-fs": "^4.2.9" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + } + }, + "node_modules/@jest/test-result": { + "version": "29.7.0", + "resolved": "https://registry.npmjs.org/@jest/test-result/-/test-result-29.7.0.tgz", + "integrity": "sha512-Fdx+tv6x1zlkJPcWXmMDAG2HBnaR9XPSd5aDWQVsfrZmLVT3lU1cwyxLgRmXR9yrq4NBoEm9BMsfgFzTQAbJYA==", + "dev": true, + "dependencies": { + "@jest/console": "^29.7.0", + "@jest/types": "^29.6.3", + "@types/istanbul-lib-coverage": "^2.0.0", + "collect-v8-coverage": "^1.0.0" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + } + }, + "node_modules/@jest/test-sequencer": { + "version": "29.7.0", + "resolved": "https://registry.npmjs.org/@jest/test-sequencer/-/test-sequencer-29.7.0.tgz", + "integrity": "sha512-GQwJ5WZVrKnOJuiYiAF52UNUJXgTZx1NHjFSEB0qEMmSZKAkdMoIzw/Cj6x6NF4AvV23AUqDpFzQkN/eYCYTxw==", + "dev": true, + "dependencies": { + "@jest/test-result": "^29.7.0", + "graceful-fs": "^4.2.9", + "jest-haste-map": "^29.7.0", + "slash": "^3.0.0" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + } + }, + "node_modules/@jest/transform": { + "version": "29.7.0", + "resolved": "https://registry.npmjs.org/@jest/transform/-/transform-29.7.0.tgz", + "integrity": "sha512-ok/BTPFzFKVMwO5eOHRrvnBVHdRy9IrsrW1GpMaQ9MCnilNLXQKmAX8s1YXDFaai9xJpac2ySzV0YeRRECr2Vw==", + "dev": true, + "dependencies": { + "@babel/core": "^7.11.6", + "@jest/types": "^29.6.3", + "@jridgewell/trace-mapping": "^0.3.18", + "babel-plugin-istanbul": "^6.1.1", + "chalk": "^4.0.0", + "convert-source-map": "^2.0.0", + "fast-json-stable-stringify": "^2.1.0", + "graceful-fs": "^4.2.9", + "jest-haste-map": "^29.7.0", + "jest-regex-util": "^29.6.3", + "jest-util": "^29.7.0", + "micromatch": "^4.0.4", + "pirates": "^4.0.4", + "slash": "^3.0.0", + "write-file-atomic": "^4.0.2" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + } + }, + "node_modules/@jest/types": { + "version": "29.6.3", + "resolved": "https://registry.npmjs.org/@jest/types/-/types-29.6.3.tgz", + "integrity": "sha512-u3UPsIilWKOM3F9CXtrG8LEJmNxwoCQC/XVj4IKYXvvpx7QIi/Kg1LI5uDmDpKlac62NUtX7eLjRh+jVZcLOzw==", + "dev": true, + "dependencies": { + "@jest/schemas": "^29.6.3", + "@types/istanbul-lib-coverage": "^2.0.0", + "@types/istanbul-reports": "^3.0.0", + "@types/node": "*", + "@types/yargs": "^17.0.8", + "chalk": "^4.0.0" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + } + }, + "node_modules/@jridgewell/gen-mapping": { + "version": "0.3.8", + "resolved": "https://registry.npmjs.org/@jridgewell/gen-mapping/-/gen-mapping-0.3.8.tgz", + "integrity": "sha512-imAbBGkb+ebQyxKgzv5Hu2nmROxoDOXHh80evxdoXNOrvAnVx7zimzc1Oo5h9RlfV4vPXaE2iM5pOFbvOCClWA==", + "dev": true, + "dependencies": { + "@jridgewell/set-array": "^1.2.1", + "@jridgewell/sourcemap-codec": "^1.4.10", + "@jridgewell/trace-mapping": "^0.3.24" + }, + "engines": { + "node": ">=6.0.0" + } + }, + "node_modules/@jridgewell/resolve-uri": { + "version": "3.1.2", + "resolved": "https://registry.npmjs.org/@jridgewell/resolve-uri/-/resolve-uri-3.1.2.tgz", + "integrity": "sha512-bRISgCIjP20/tbWSPWMEi54QVPRZExkuD9lJL+UIxUKtwVJA8wW1Trb1jMs1RFXo1CBTNZ/5hpC9QvmKWdopKw==", + "dev": true, + "engines": { + "node": ">=6.0.0" + } + }, + "node_modules/@jridgewell/set-array": { + "version": "1.2.1", + "resolved": "https://registry.npmjs.org/@jridgewell/set-array/-/set-array-1.2.1.tgz", + "integrity": "sha512-R8gLRTZeyp03ymzP/6Lil/28tGeGEzhx1q2k703KGWRAI1VdvPIXdG70VJc2pAMw3NA6JKL5hhFu1sJX0Mnn/A==", + "dev": true, + "engines": { + "node": ">=6.0.0" + } + }, + "node_modules/@jridgewell/sourcemap-codec": { + "version": "1.5.0", + "resolved": "https://registry.npmjs.org/@jridgewell/sourcemap-codec/-/sourcemap-codec-1.5.0.tgz", + "integrity": "sha512-gv3ZRaISU3fjPAgNsriBRqGWQL6quFx04YMPW/zD8XMLsU32mhCCbfbO6KZFLjvYpCZ8zyDEgqsgf+PwPaM7GQ==", + "dev": true + }, + "node_modules/@jridgewell/trace-mapping": { + "version": "0.3.25", + "resolved": "https://registry.npmjs.org/@jridgewell/trace-mapping/-/trace-mapping-0.3.25.tgz", + "integrity": "sha512-vNk6aEwybGtawWmy/PzwnGDOjCkLWSD2wqvjGGAgOAwCGWySYXfYoxt00IJkTF+8Lb57DwOb3Aa0o9CApepiYQ==", + "dev": true, + "dependencies": { + "@jridgewell/resolve-uri": "^3.1.0", + "@jridgewell/sourcemap-codec": "^1.4.14" + } + }, + "node_modules/@sinclair/typebox": { + "version": "0.27.8", + "resolved": "https://registry.npmjs.org/@sinclair/typebox/-/typebox-0.27.8.tgz", + "integrity": "sha512-+Fj43pSMwJs4KRrH/938Uf+uAELIgVBmQzg/q1YG10djyfA3TnrU8N8XzqCh/okZdszqBQTZf96idMfE5lnwTA==", + "dev": true + }, + "node_modules/@sinonjs/commons": { + "version": "3.0.1", + "resolved": "https://registry.npmjs.org/@sinonjs/commons/-/commons-3.0.1.tgz", + "integrity": "sha512-K3mCHKQ9sVh8o1C9cxkwxaOmXoAMlDxC1mYyHrjqOWEcBjYr76t96zL2zlj5dUGZ3HSw240X1qgH3Mjf1yJWpQ==", + "dev": true, + "dependencies": { + "type-detect": "4.0.8" + } + }, + "node_modules/@sinonjs/fake-timers": { + "version": "10.3.0", + "resolved": "https://registry.npmjs.org/@sinonjs/fake-timers/-/fake-timers-10.3.0.tgz", + "integrity": "sha512-V4BG07kuYSUkTCSBHG8G8TNhM+F19jXFWnQtzj+we8DrkpSBCee9Z3Ms8yiGer/dlmhe35/Xdgyo3/0rQKg7YA==", + "dev": true, + "dependencies": { + "@sinonjs/commons": "^3.0.0" + } + }, + "node_modules/@tsconfig/node10": { + "version": "1.0.11", + "resolved": "https://registry.npmjs.org/@tsconfig/node10/-/node10-1.0.11.tgz", + "integrity": "sha512-DcRjDCujK/kCk/cUe8Xz8ZSpm8mS3mNNpta+jGCA6USEDfktlNvm1+IuZ9eTcDbNk41BHwpHHeW+N1lKCz4zOw==", + "dev": true + }, + "node_modules/@tsconfig/node12": { + "version": "1.0.11", + "resolved": "https://registry.npmjs.org/@tsconfig/node12/-/node12-1.0.11.tgz", + "integrity": "sha512-cqefuRsh12pWyGsIoBKJA9luFu3mRxCA+ORZvA4ktLSzIuCUtWVxGIuXigEwO5/ywWFMZ2QEGKWvkZG1zDMTag==", + "dev": true + }, + "node_modules/@tsconfig/node14": { + "version": "1.0.3", + "resolved": "https://registry.npmjs.org/@tsconfig/node14/-/node14-1.0.3.tgz", + "integrity": "sha512-ysT8mhdixWK6Hw3i1V2AeRqZ5WfXg1G43mqoYlM2nc6388Fq5jcXyr5mRsqViLx/GJYdoL0bfXD8nmF+Zn/Iow==", + "dev": true + }, + "node_modules/@tsconfig/node16": { + "version": "1.0.4", + "resolved": "https://registry.npmjs.org/@tsconfig/node16/-/node16-1.0.4.tgz", + "integrity": "sha512-vxhUy4J8lyeyinH7Azl1pdd43GJhZH/tP2weN8TntQblOY+A0XbT8DJk1/oCPuOOyg/Ja757rG0CgHcWC8OfMA==", + "dev": true + }, + "node_modules/@types/babel__core": { + "version": "7.20.5", + "resolved": "https://registry.npmjs.org/@types/babel__core/-/babel__core-7.20.5.tgz", + "integrity": "sha512-qoQprZvz5wQFJwMDqeseRXWv3rqMvhgpbXFfVyWhbx9X47POIA6i/+dXefEmZKoAgOaTdaIgNSMqMIU61yRyzA==", + "dev": true, + "dependencies": { + "@babel/parser": "^7.20.7", + "@babel/types": "^7.20.7", + "@types/babel__generator": "*", + "@types/babel__template": "*", + "@types/babel__traverse": "*" + } + }, + "node_modules/@types/babel__generator": { + "version": "7.27.0", + "resolved": "https://registry.npmjs.org/@types/babel__generator/-/babel__generator-7.27.0.tgz", + "integrity": "sha512-ufFd2Xi92OAVPYsy+P4n7/U7e68fex0+Ee8gSG9KX7eo084CWiQ4sdxktvdl0bOPupXtVJPY19zk6EwWqUQ8lg==", + "dev": true, + "dependencies": { + "@babel/types": "^7.0.0" + } + }, + "node_modules/@types/babel__template": { + "version": "7.4.4", + "resolved": "https://registry.npmjs.org/@types/babel__template/-/babel__template-7.4.4.tgz", + "integrity": "sha512-h/NUaSyG5EyxBIp8YRxo4RMe2/qQgvyowRwVMzhYhBCONbW8PUsg4lkFMrhgZhUe5z3L3MiLDuvyJ/CaPa2A8A==", + "dev": true, + "dependencies": { + "@babel/parser": "^7.1.0", + "@babel/types": "^7.0.0" + } + }, + "node_modules/@types/babel__traverse": { + "version": "7.20.7", + "resolved": "https://registry.npmjs.org/@types/babel__traverse/-/babel__traverse-7.20.7.tgz", + "integrity": "sha512-dkO5fhS7+/oos4ciWxyEyjWe48zmG6wbCheo/G2ZnHx4fs3EU6YC6UM8rk56gAjNJ9P3MTH2jo5jb92/K6wbng==", + "dev": true, + "dependencies": { + "@babel/types": "^7.20.7" + } + }, + "node_modules/@types/graceful-fs": { + "version": "4.1.9", + "resolved": "https://registry.npmjs.org/@types/graceful-fs/-/graceful-fs-4.1.9.tgz", + "integrity": "sha512-olP3sd1qOEe5dXTSaFvQG+02VdRXcdytWLAZsAq1PecU8uqQAhkrnbli7DagjtXKW/Bl7YJbUsa8MPcuc8LHEQ==", + "dev": true, + "dependencies": { + "@types/node": "*" + } + }, + "node_modules/@types/istanbul-lib-coverage": { + "version": "2.0.6", + "resolved": "https://registry.npmjs.org/@types/istanbul-lib-coverage/-/istanbul-lib-coverage-2.0.6.tgz", + "integrity": "sha512-2QF/t/auWm0lsy8XtKVPG19v3sSOQlJe/YHZgfjb/KBBHOGSV+J2q/S671rcq9uTBrLAXmZpqJiaQbMT+zNU1w==", + "dev": true + }, + "node_modules/@types/istanbul-lib-report": { + "version": "3.0.3", + "resolved": "https://registry.npmjs.org/@types/istanbul-lib-report/-/istanbul-lib-report-3.0.3.tgz", + "integrity": "sha512-NQn7AHQnk/RSLOxrBbGyJM/aVQ+pjj5HCgasFxc0K/KhoATfQ/47AyUl15I2yBUpihjmas+a+VJBOqecrFH+uA==", + "dev": true, + "dependencies": { + "@types/istanbul-lib-coverage": "*" + } + }, + "node_modules/@types/istanbul-reports": { + "version": "3.0.4", + "resolved": "https://registry.npmjs.org/@types/istanbul-reports/-/istanbul-reports-3.0.4.tgz", + "integrity": "sha512-pk2B1NWalF9toCRu6gjBzR69syFjP4Od8WRAX+0mmf9lAjCRicLOWc+ZrxZHx/0XRjotgkF9t6iaMJ+aXcOdZQ==", + "dev": true, + "dependencies": { + "@types/istanbul-lib-report": "*" + } + }, + "node_modules/@types/jest": { + "version": "29.5.14", + "resolved": "https://registry.npmjs.org/@types/jest/-/jest-29.5.14.tgz", + "integrity": "sha512-ZN+4sdnLUbo8EVvVc2ao0GFW6oVrQRPn4K2lglySj7APvSrgzxHiNNK99us4WDMi57xxA2yggblIAMNhXOotLQ==", + "dev": true, + "dependencies": { + "expect": "^29.0.0", + "pretty-format": "^29.0.0" + } + }, + "node_modules/@types/node": { + "version": "20.1.7", + "resolved": "https://registry.npmjs.org/@types/node/-/node-20.1.7.tgz", + "integrity": "sha512-WCuw/o4GSwDGMoonES8rcvwsig77dGCMbZDrZr2x4ZZiNW4P/gcoZXe/0twgtobcTkmg9TuKflxYL/DuwDyJzg==", + "dev": true + }, + "node_modules/@types/stack-utils": { + "version": "2.0.3", + "resolved": "https://registry.npmjs.org/@types/stack-utils/-/stack-utils-2.0.3.tgz", + "integrity": "sha512-9aEbYZ3TbYMznPdcdr3SmIrLXwC/AKZXQeCf9Pgao5CKb8CyHuEX5jzWPTkvregvhRJHcpRO6BFoGW9ycaOkYw==", + "dev": true + }, + "node_modules/@types/yargs": { + "version": "17.0.33", + "resolved": "https://registry.npmjs.org/@types/yargs/-/yargs-17.0.33.tgz", + "integrity": "sha512-WpxBCKWPLr4xSsHgz511rFJAM+wS28w2zEO1QDNY5zM/S8ok70NNfztH0xwhqKyaK0OHCbN98LDAZuy1ctxDkA==", + "dev": true, + "dependencies": { + "@types/yargs-parser": "*" + } + }, + "node_modules/@types/yargs-parser": { + "version": "21.0.3", + "resolved": "https://registry.npmjs.org/@types/yargs-parser/-/yargs-parser-21.0.3.tgz", + "integrity": "sha512-I4q9QU9MQv4oEOz4tAHJtNz1cwuLxn2F3xcc2iV5WdqLPpUnj30aUuxt1mAxYTG+oe8CZMV/+6rU4S4gRDzqtQ==", + "dev": true + }, + "node_modules/acorn": { + "version": "8.14.1", + "resolved": "https://registry.npmjs.org/acorn/-/acorn-8.14.1.tgz", + "integrity": "sha512-OvQ/2pUDKmgfCg++xsTX1wGxfTaszcHVcTctW4UJB4hibJx2HXxxO5UmVgyjMa+ZDsiaf5wWLXYpRWMmBI0QHg==", + "dev": true, + "bin": { + "acorn": "bin/acorn" + }, + "engines": { + "node": ">=0.4.0" + } + }, + "node_modules/acorn-walk": { + "version": "8.3.4", + "resolved": "https://registry.npmjs.org/acorn-walk/-/acorn-walk-8.3.4.tgz", + "integrity": "sha512-ueEepnujpqee2o5aIYnvHU6C0A42MNdsIDeqy5BydrkuC5R1ZuUFnm27EeFJGoEHJQgn3uleRvmTXaJgfXbt4g==", + "dev": true, + "dependencies": { + "acorn": "^8.11.0" + }, + "engines": { + "node": ">=0.4.0" + } + }, + "node_modules/ansi-escapes": { + "version": "4.3.2", + "resolved": "https://registry.npmjs.org/ansi-escapes/-/ansi-escapes-4.3.2.tgz", + "integrity": "sha512-gKXj5ALrKWQLsYG9jlTRmR/xKluxHV+Z9QEwNIgCfM1/uwPMCuzVVnh5mwTd+OuBZcwSIMbqssNWRm1lE51QaQ==", + "dev": true, + "dependencies": { + "type-fest": "^0.21.3" + }, + "engines": { + "node": ">=8" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/ansi-regex": { + "version": "5.0.1", + "resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-5.0.1.tgz", + "integrity": "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ==", + "dev": true, + "engines": { + "node": ">=8" + } + }, + "node_modules/ansi-styles": { + "version": "4.3.0", + "resolved": "https://registry.npmjs.org/ansi-styles/-/ansi-styles-4.3.0.tgz", + "integrity": "sha512-zbB9rCJAT1rbjiVDb2hqKFHNYLxgtk8NURxZ3IZwD3F6NtxbXZQCnnSi1Lkx+IDohdPlFp222wVALIheZJQSEg==", + "dev": true, + "dependencies": { + "color-convert": "^2.0.1" + }, + "engines": { + "node": ">=8" + }, + "funding": { + "url": "https://github.com/chalk/ansi-styles?sponsor=1" + } + }, + "node_modules/anymatch": { + "version": "3.1.3", + "resolved": "https://registry.npmjs.org/anymatch/-/anymatch-3.1.3.tgz", + "integrity": "sha512-KMReFUr0B4t+D+OBkjR3KYqvocp2XaSzO55UcB6mgQMd3KbcE+mWTyvVV7D/zsdEbNnV6acZUutkiHQXvTr1Rw==", + "dev": true, + "dependencies": { + "normalize-path": "^3.0.0", + "picomatch": "^2.0.4" + }, + "engines": { + "node": ">= 8" + } + }, + "node_modules/arg": { + "version": "4.1.3", + "resolved": "https://registry.npmjs.org/arg/-/arg-4.1.3.tgz", + "integrity": "sha512-58S9QDqG0Xx27YwPSt9fJxivjYl432YCwfDMfZ+71RAqUrZef7LrKQZ3LHLOwCS4FLNBplP533Zx895SeOCHvA==", + "dev": true + }, + "node_modules/argparse": { + "version": "1.0.10", + "resolved": "https://registry.npmjs.org/argparse/-/argparse-1.0.10.tgz", + "integrity": "sha512-o5Roy6tNG4SL/FOkCAN6RzjiakZS25RLYFrcMttJqbdd8BWrnA+fGz57iN5Pb06pvBGvl5gQ0B48dJlslXvoTg==", + "dev": true, + "dependencies": { + "sprintf-js": "~1.0.2" + } + }, + "node_modules/async": { + "version": "3.2.6", + "resolved": "https://registry.npmjs.org/async/-/async-3.2.6.tgz", + "integrity": "sha512-htCUDlxyyCLMgaM3xXg0C0LW2xqfuQ6p05pCEIsXuyQ+a1koYKTuBMzRNwmybfLgvJDMd0r1LTn4+E0Ti6C2AA==", + "dev": true + }, + "node_modules/aws-cdk": { + "version": "2.1012.0", + "resolved": "https://registry.npmjs.org/aws-cdk/-/aws-cdk-2.1012.0.tgz", + "integrity": "sha512-C6jSWkqP0hkY2Cs300VJHjspmTXDTMfB813kwZvRbd/OsKBfTBJBbYU16VoLAp1LVEOnQMf8otSlaSgzVF0X9A==", + "dev": true, + "bin": { + "cdk": "bin/cdk" + }, + "engines": { + "node": ">= 14.15.0" + }, + "optionalDependencies": { + "fsevents": "2.3.2" + } + }, + "node_modules/aws-cdk-lib": { + "version": "2.192.0", + "resolved": "https://registry.npmjs.org/aws-cdk-lib/-/aws-cdk-lib-2.192.0.tgz", + "integrity": "sha512-Y9BAlr9a4QsEsamKc2cOGzX8DpVSOh94wsrMSGRXT0bZaqmixhhmT7WYCrT1KX4MU3gYk3OiwY2BbNyWaNE8Fg==", + "bundleDependencies": [ + "@balena/dockerignore", + "case", + "fs-extra", + "ignore", + "jsonschema", + "minimatch", + "punycode", + "semver", + "table", + "yaml", + "mime-types" + ], + "dependencies": { + "@aws-cdk/asset-awscli-v1": "^2.2.229", + "@aws-cdk/asset-node-proxy-agent-v6": "^2.1.0", + "@aws-cdk/cloud-assembly-schema": "^41.0.0", + "@balena/dockerignore": "^1.0.2", + "case": "1.6.3", + "fs-extra": "^11.3.0", + "ignore": "^5.3.2", + "jsonschema": "^1.5.0", + "mime-types": "^2.1.35", + "minimatch": "^3.1.2", + "punycode": "^2.3.1", + "semver": "^7.7.1", + "table": "^6.9.0", + "yaml": "1.10.2" + }, + "engines": { + "node": ">= 14.15.0" + }, + "peerDependencies": { + "constructs": "^10.0.0" + } + }, + "node_modules/aws-cdk-lib/node_modules/@balena/dockerignore": { + "version": "1.0.2", + "inBundle": true, + "license": "Apache-2.0" + }, + "node_modules/aws-cdk-lib/node_modules/ajv": { + "version": "8.17.1", + "inBundle": true, + "license": "MIT", + "dependencies": { + "fast-deep-equal": "^3.1.3", + "fast-uri": "^3.0.1", + "json-schema-traverse": "^1.0.0", + "require-from-string": "^2.0.2" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/epoberezkin" + } + }, + "node_modules/aws-cdk-lib/node_modules/ansi-regex": { + "version": "5.0.1", + "inBundle": true, + "license": "MIT", + "engines": { + "node": ">=8" + } + }, + "node_modules/aws-cdk-lib/node_modules/ansi-styles": { + "version": "4.3.0", + "inBundle": true, + "license": "MIT", + "dependencies": { + "color-convert": "^2.0.1" + }, + "engines": { + "node": ">=8" + }, + "funding": { + "url": "https://github.com/chalk/ansi-styles?sponsor=1" + } + }, + "node_modules/aws-cdk-lib/node_modules/astral-regex": { + "version": "2.0.0", + "inBundle": true, + "license": "MIT", + "engines": { + "node": ">=8" + } + }, + "node_modules/aws-cdk-lib/node_modules/balanced-match": { + "version": "1.0.2", + "inBundle": true, + "license": "MIT" + }, + "node_modules/aws-cdk-lib/node_modules/brace-expansion": { + "version": "1.1.11", + "inBundle": true, + "license": "MIT", + "dependencies": { + "balanced-match": "^1.0.0", + "concat-map": "0.0.1" + } + }, + "node_modules/aws-cdk-lib/node_modules/case": { + "version": "1.6.3", + "inBundle": true, + "license": "(MIT OR GPL-3.0-or-later)", + "engines": { + "node": ">= 0.8.0" + } + }, + "node_modules/aws-cdk-lib/node_modules/color-convert": { + "version": "2.0.1", + "inBundle": true, + "license": "MIT", + "dependencies": { + "color-name": "~1.1.4" + }, + "engines": { + "node": ">=7.0.0" + } + }, + "node_modules/aws-cdk-lib/node_modules/color-name": { + "version": "1.1.4", + "inBundle": true, + "license": "MIT" + }, + "node_modules/aws-cdk-lib/node_modules/concat-map": { + "version": "0.0.1", + "inBundle": true, + "license": "MIT" + }, + "node_modules/aws-cdk-lib/node_modules/emoji-regex": { + "version": "8.0.0", + "inBundle": true, + "license": "MIT" + }, + "node_modules/aws-cdk-lib/node_modules/fast-deep-equal": { + "version": "3.1.3", + "inBundle": true, + "license": "MIT" + }, + "node_modules/aws-cdk-lib/node_modules/fast-uri": { + "version": "3.0.6", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/fastify" + }, + { + "type": "opencollective", + "url": "https://opencollective.com/fastify" + } + ], + "inBundle": true, + "license": "BSD-3-Clause" + }, + "node_modules/aws-cdk-lib/node_modules/fs-extra": { + "version": "11.3.0", + "inBundle": true, + "license": "MIT", + "dependencies": { + "graceful-fs": "^4.2.0", + "jsonfile": "^6.0.1", + "universalify": "^2.0.0" + }, + "engines": { + "node": ">=14.14" + } + }, + "node_modules/aws-cdk-lib/node_modules/graceful-fs": { + "version": "4.2.11", + "inBundle": true, + "license": "ISC" + }, + "node_modules/aws-cdk-lib/node_modules/ignore": { + "version": "5.3.2", + "inBundle": true, + "license": "MIT", + "engines": { + "node": ">= 4" + } + }, + "node_modules/aws-cdk-lib/node_modules/is-fullwidth-code-point": { + "version": "3.0.0", + "inBundle": true, + "license": "MIT", + "engines": { + "node": ">=8" + } + }, + "node_modules/aws-cdk-lib/node_modules/json-schema-traverse": { + "version": "1.0.0", + "inBundle": true, + "license": "MIT" + }, + "node_modules/aws-cdk-lib/node_modules/jsonfile": { + "version": "6.1.0", + "inBundle": true, + "license": "MIT", + "dependencies": { + "universalify": "^2.0.0" + }, + "optionalDependencies": { + "graceful-fs": "^4.1.6" + } + }, + "node_modules/aws-cdk-lib/node_modules/jsonschema": { + "version": "1.5.0", + "inBundle": true, + "license": "MIT", + "engines": { + "node": "*" + } + }, + "node_modules/aws-cdk-lib/node_modules/lodash.truncate": { + "version": "4.4.2", + "inBundle": true, + "license": "MIT" + }, + "node_modules/aws-cdk-lib/node_modules/mime-db": { + "version": "1.52.0", + "inBundle": true, + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/aws-cdk-lib/node_modules/mime-types": { + "version": "2.1.35", + "inBundle": true, + "license": "MIT", + "dependencies": { + "mime-db": "1.52.0" + }, + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/aws-cdk-lib/node_modules/minimatch": { + "version": "3.1.2", + "inBundle": true, + "license": "ISC", + "dependencies": { + "brace-expansion": "^1.1.7" + }, + "engines": { + "node": "*" + } + }, + "node_modules/aws-cdk-lib/node_modules/punycode": { + "version": "2.3.1", + "inBundle": true, + "license": "MIT", + "engines": { + "node": ">=6" + } + }, + "node_modules/aws-cdk-lib/node_modules/require-from-string": { + "version": "2.0.2", + "inBundle": true, + "license": "MIT", + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/aws-cdk-lib/node_modules/semver": { + "version": "7.7.1", + "inBundle": true, + "license": "ISC", + "bin": { + "semver": "bin/semver.js" + }, + "engines": { + "node": ">=10" + } + }, + "node_modules/aws-cdk-lib/node_modules/slice-ansi": { + "version": "4.0.0", + "inBundle": true, + "license": "MIT", + "dependencies": { + "ansi-styles": "^4.0.0", + "astral-regex": "^2.0.0", + "is-fullwidth-code-point": "^3.0.0" + }, + "engines": { + "node": ">=10" + }, + "funding": { + "url": "https://github.com/chalk/slice-ansi?sponsor=1" + } + }, + "node_modules/aws-cdk-lib/node_modules/string-width": { + "version": "4.2.3", + "inBundle": true, + "license": "MIT", + "dependencies": { + "emoji-regex": "^8.0.0", + "is-fullwidth-code-point": "^3.0.0", + "strip-ansi": "^6.0.1" + }, + "engines": { + "node": ">=8" + } + }, + "node_modules/aws-cdk-lib/node_modules/strip-ansi": { + "version": "6.0.1", + "inBundle": true, + "license": "MIT", + "dependencies": { + "ansi-regex": "^5.0.1" + }, + "engines": { + "node": ">=8" + } + }, + "node_modules/aws-cdk-lib/node_modules/table": { + "version": "6.9.0", + "inBundle": true, + "license": "BSD-3-Clause", + "dependencies": { + "ajv": "^8.0.1", + "lodash.truncate": "^4.4.2", + "slice-ansi": "^4.0.0", + "string-width": "^4.2.3", + "strip-ansi": "^6.0.1" + }, + "engines": { + "node": ">=10.0.0" + } + }, + "node_modules/aws-cdk-lib/node_modules/universalify": { + "version": "2.0.1", + "inBundle": true, + "license": "MIT", + "engines": { + "node": ">= 10.0.0" + } + }, + "node_modules/aws-cdk-lib/node_modules/yaml": { + "version": "1.10.2", + "inBundle": true, + "license": "ISC", + "engines": { + "node": ">= 6" + } + }, + "node_modules/babel-jest": { + "version": "29.7.0", + "resolved": "https://registry.npmjs.org/babel-jest/-/babel-jest-29.7.0.tgz", + "integrity": "sha512-BrvGY3xZSwEcCzKvKsCi2GgHqDqsYkOP4/by5xCgIwGXQxIEh+8ew3gmrE1y7XRR6LHZIj6yLYnUi/mm2KXKBg==", + "dev": true, + "dependencies": { + "@jest/transform": "^29.7.0", + "@types/babel__core": "^7.1.14", + "babel-plugin-istanbul": "^6.1.1", + "babel-preset-jest": "^29.6.3", + "chalk": "^4.0.0", + "graceful-fs": "^4.2.9", + "slash": "^3.0.0" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + }, + "peerDependencies": { + "@babel/core": "^7.8.0" + } + }, + "node_modules/babel-plugin-istanbul": { + "version": "6.1.1", + "resolved": "https://registry.npmjs.org/babel-plugin-istanbul/-/babel-plugin-istanbul-6.1.1.tgz", + "integrity": "sha512-Y1IQok9821cC9onCx5otgFfRm7Lm+I+wwxOx738M/WLPZ9Q42m4IG5W0FNX8WLL2gYMZo3JkuXIH2DOpWM+qwA==", + "dev": true, + "dependencies": { + "@babel/helper-plugin-utils": "^7.0.0", + "@istanbuljs/load-nyc-config": "^1.0.0", + "@istanbuljs/schema": "^0.1.2", + "istanbul-lib-instrument": "^5.0.4", + "test-exclude": "^6.0.0" + }, + "engines": { + "node": ">=8" + } + }, + "node_modules/babel-plugin-istanbul/node_modules/istanbul-lib-instrument": { + "version": "5.2.1", + "resolved": "https://registry.npmjs.org/istanbul-lib-instrument/-/istanbul-lib-instrument-5.2.1.tgz", + "integrity": "sha512-pzqtp31nLv/XFOzXGuvhCb8qhjmTVo5vjVk19XE4CRlSWz0KoeJ3bw9XsA7nOp9YBf4qHjwBxkDzKcME/J29Yg==", + "dev": true, + "dependencies": { + "@babel/core": "^7.12.3", + "@babel/parser": "^7.14.7", + "@istanbuljs/schema": "^0.1.2", + "istanbul-lib-coverage": "^3.2.0", + "semver": "^6.3.0" + }, + "engines": { + "node": ">=8" + } + }, + "node_modules/babel-plugin-jest-hoist": { + "version": "29.6.3", + "resolved": "https://registry.npmjs.org/babel-plugin-jest-hoist/-/babel-plugin-jest-hoist-29.6.3.tgz", + "integrity": "sha512-ESAc/RJvGTFEzRwOTT4+lNDk/GNHMkKbNzsvT0qKRfDyyYTskxB5rnU2njIDYVxXCBHHEI1c0YwHob3WaYujOg==", + "dev": true, + "dependencies": { + "@babel/template": "^7.3.3", + "@babel/types": "^7.3.3", + "@types/babel__core": "^7.1.14", + "@types/babel__traverse": "^7.0.6" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + } + }, + "node_modules/babel-preset-current-node-syntax": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/babel-preset-current-node-syntax/-/babel-preset-current-node-syntax-1.1.0.tgz", + "integrity": "sha512-ldYss8SbBlWva1bs28q78Ju5Zq1F+8BrqBZZ0VFhLBvhh6lCpC2o3gDJi/5DRLs9FgYZCnmPYIVFU4lRXCkyUw==", + "dev": true, + "dependencies": { + "@babel/plugin-syntax-async-generators": "^7.8.4", + "@babel/plugin-syntax-bigint": "^7.8.3", + "@babel/plugin-syntax-class-properties": "^7.12.13", + "@babel/plugin-syntax-class-static-block": "^7.14.5", + "@babel/plugin-syntax-import-attributes": "^7.24.7", + "@babel/plugin-syntax-import-meta": "^7.10.4", + "@babel/plugin-syntax-json-strings": "^7.8.3", + "@babel/plugin-syntax-logical-assignment-operators": "^7.10.4", + "@babel/plugin-syntax-nullish-coalescing-operator": "^7.8.3", + "@babel/plugin-syntax-numeric-separator": "^7.10.4", + "@babel/plugin-syntax-object-rest-spread": "^7.8.3", + "@babel/plugin-syntax-optional-catch-binding": "^7.8.3", + "@babel/plugin-syntax-optional-chaining": "^7.8.3", + "@babel/plugin-syntax-private-property-in-object": "^7.14.5", + "@babel/plugin-syntax-top-level-await": "^7.14.5" + }, + "peerDependencies": { + "@babel/core": "^7.0.0" + } + }, + "node_modules/babel-preset-jest": { + "version": "29.6.3", + "resolved": "https://registry.npmjs.org/babel-preset-jest/-/babel-preset-jest-29.6.3.tgz", + "integrity": "sha512-0B3bhxR6snWXJZtR/RliHTDPRgn1sNHOR0yVtq/IiQFyuOVjFS+wuio/R4gSNkyYmKmJB4wGZv2NZanmKmTnNA==", + "dev": true, + "dependencies": { + "babel-plugin-jest-hoist": "^29.6.3", + "babel-preset-current-node-syntax": "^1.0.0" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + }, + "peerDependencies": { + "@babel/core": "^7.0.0" + } + }, + "node_modules/balanced-match": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-1.0.2.tgz", + "integrity": "sha512-3oSeUO0TMV67hN1AmbXsK4yaqU7tjiHlbxRDZOpH0KW9+CeX4bRAaX0Anxt0tx2MrpRpWwQaPwIlISEJhYU5Pw==", + "dev": true + }, + "node_modules/brace-expansion": { + "version": "1.1.11", + "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-1.1.11.tgz", + "integrity": "sha512-iCuPHDFgrHX7H2vEI/5xpz07zSHB00TpugqhmYtVmMO6518mCuRMoOYFldEBl0g187ufozdaHgWKcYFb61qGiA==", + "dev": true, + "dependencies": { + "balanced-match": "^1.0.0", + "concat-map": "0.0.1" + } + }, + "node_modules/braces": { + "version": "3.0.3", + "resolved": "https://registry.npmjs.org/braces/-/braces-3.0.3.tgz", + "integrity": "sha512-yQbXgO/OSZVD2IsiLlro+7Hf6Q18EJrKSEsdoMzKePKXct3gvD8oLcOQdIzGupr5Fj+EDe8gO/lxc1BzfMpxvA==", + "dev": true, + "dependencies": { + "fill-range": "^7.1.1" + }, + "engines": { + "node": ">=8" + } + }, + "node_modules/browserslist": { + "version": "4.24.4", + "resolved": "https://registry.npmjs.org/browserslist/-/browserslist-4.24.4.tgz", + "integrity": "sha512-KDi1Ny1gSePi1vm0q4oxSF8b4DR44GF4BbmS2YdhPLOEqd8pDviZOGH/GsmRwoWJ2+5Lr085X7naowMwKHDG1A==", + "dev": true, + "funding": [ + { + "type": "opencollective", + "url": "https://opencollective.com/browserslist" + }, + { + "type": "tidelift", + "url": "https://tidelift.com/funding/github/npm/browserslist" + }, + { + "type": "github", + "url": "https://github.com/sponsors/ai" + } + ], + "dependencies": { + "caniuse-lite": "^1.0.30001688", + "electron-to-chromium": "^1.5.73", + "node-releases": "^2.0.19", + "update-browserslist-db": "^1.1.1" + }, + "bin": { + "browserslist": "cli.js" + }, + "engines": { + "node": "^6 || ^7 || ^8 || ^9 || ^10 || ^11 || ^12 || >=13.7" + } + }, + "node_modules/bs-logger": { + "version": "0.2.6", + "resolved": "https://registry.npmjs.org/bs-logger/-/bs-logger-0.2.6.tgz", + "integrity": "sha512-pd8DCoxmbgc7hyPKOvxtqNcjYoOsABPQdcCUjGp3d42VR2CX1ORhk2A87oqqu5R1kk+76nsxZupkmyd+MVtCog==", + "dev": true, + "dependencies": { + "fast-json-stable-stringify": "2.x" + }, + "engines": { + "node": ">= 6" + } + }, + "node_modules/bser": { + "version": "2.1.1", + "resolved": "https://registry.npmjs.org/bser/-/bser-2.1.1.tgz", + "integrity": "sha512-gQxTNE/GAfIIrmHLUE3oJyp5FO6HRBfhjnw4/wMmA63ZGDJnWBmgY/lyQBpnDUkGmAhbSe39tx2d/iTOAfglwQ==", + "dev": true, + "dependencies": { + "node-int64": "^0.4.0" + } + }, + "node_modules/buffer-from": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/buffer-from/-/buffer-from-1.1.2.tgz", + "integrity": "sha512-E+XQCRwSbaaiChtv6k6Dwgc+bx+Bs6vuKJHHl5kox/BaKbhiXzqQOwK4cO22yElGp2OCmjwVhT3HmxgyPGnJfQ==" + }, + "node_modules/callsites": { + "version": "3.1.0", + "resolved": "https://registry.npmjs.org/callsites/-/callsites-3.1.0.tgz", + "integrity": "sha512-P8BjAsXvZS+VIDUI11hHCQEv74YT67YUi5JJFNWIqL235sBmjX4+qx9Muvls5ivyNENctx46xQLQ3aTuE7ssaQ==", + "dev": true, + "engines": { + "node": ">=6" + } + }, + "node_modules/camelcase": { + "version": "5.3.1", + "resolved": "https://registry.npmjs.org/camelcase/-/camelcase-5.3.1.tgz", + "integrity": "sha512-L28STB170nwWS63UjtlEOE3dldQApaJXZkOI1uMFfzf3rRuPegHaHesyee+YxQ+W6SvRDQV6UrdOdRiR153wJg==", + "dev": true, + "engines": { + "node": ">=6" + } + }, + "node_modules/caniuse-lite": { + "version": "1.0.30001715", + "resolved": "https://registry.npmjs.org/caniuse-lite/-/caniuse-lite-1.0.30001715.tgz", + "integrity": "sha512-7ptkFGMm2OAOgvZpwgA4yjQ5SQbrNVGdRjzH0pBdy1Fasvcr+KAeECmbCAECzTuDuoX0FCY8KzUxjf9+9kfZEw==", + "dev": true, + "funding": [ + { + "type": "opencollective", + "url": "https://opencollective.com/browserslist" + }, + { + "type": "tidelift", + "url": "https://tidelift.com/funding/github/npm/caniuse-lite" + }, + { + "type": "github", + "url": "https://github.com/sponsors/ai" + } + ] + }, + "node_modules/chalk": { + "version": "4.1.2", + "resolved": "https://registry.npmjs.org/chalk/-/chalk-4.1.2.tgz", + "integrity": "sha512-oKnbhFyRIXpUuez8iBMmyEa4nbj4IOQyuhc/wy9kY7/WVPcwIO9VA668Pu8RkO7+0G76SLROeyw9CpQ061i4mA==", + "dev": true, + "dependencies": { + "ansi-styles": "^4.1.0", + "supports-color": "^7.1.0" + }, + "engines": { + "node": ">=10" + }, + "funding": { + "url": "https://github.com/chalk/chalk?sponsor=1" + } + }, + "node_modules/char-regex": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/char-regex/-/char-regex-1.0.2.tgz", + "integrity": "sha512-kWWXztvZ5SBQV+eRgKFeh8q5sLuZY2+8WUIzlxWVTg+oGwY14qylx1KbKzHd8P6ZYkAg0xyIDU9JMHhyJMZ1jw==", + "dev": true, + "engines": { + "node": ">=10" + } + }, + "node_modules/ci-info": { + "version": "3.9.0", + "resolved": "https://registry.npmjs.org/ci-info/-/ci-info-3.9.0.tgz", + "integrity": "sha512-NIxF55hv4nSqQswkAeiOi1r83xy8JldOFDTWiug55KBu9Jnblncd2U6ViHmYgHf01TPZS77NJBhBMKdWj9HQMQ==", + "dev": true, + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/sibiraj-s" + } + ], + "engines": { + "node": ">=8" + } + }, + "node_modules/cjs-module-lexer": { + "version": "1.4.3", + "resolved": "https://registry.npmjs.org/cjs-module-lexer/-/cjs-module-lexer-1.4.3.tgz", + "integrity": "sha512-9z8TZaGM1pfswYeXrUpzPrkx8UnWYdhJclsiYMm6x/w5+nN+8Tf/LnAgfLGQCm59qAOxU8WwHEq2vNwF6i4j+Q==", + "dev": true + }, + "node_modules/cliui": { + "version": "8.0.1", + "resolved": "https://registry.npmjs.org/cliui/-/cliui-8.0.1.tgz", + "integrity": "sha512-BSeNnyus75C4//NQ9gQt1/csTXyo/8Sb+afLAkzAptFuMsod9HFokGNudZpi/oQV73hnVK+sR+5PVRMd+Dr7YQ==", + "dev": true, + "dependencies": { + "string-width": "^4.2.0", + "strip-ansi": "^6.0.1", + "wrap-ansi": "^7.0.0" + }, + "engines": { + "node": ">=12" + } + }, + "node_modules/co": { + "version": "4.6.0", + "resolved": "https://registry.npmjs.org/co/-/co-4.6.0.tgz", + "integrity": "sha512-QVb0dM5HvG+uaxitm8wONl7jltx8dqhfU33DcqtOZcLSVIKSDDLDi7+0LbAKiyI8hD9u42m2YxXSkMGWThaecQ==", + "dev": true, + "engines": { + "iojs": ">= 1.0.0", + "node": ">= 0.12.0" + } + }, + "node_modules/collect-v8-coverage": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/collect-v8-coverage/-/collect-v8-coverage-1.0.2.tgz", + "integrity": "sha512-lHl4d5/ONEbLlJvaJNtsF/Lz+WvB07u2ycqTYbdrq7UypDXailES4valYb2eWiJFxZlVmpGekfqoxQhzyFdT4Q==", + "dev": true + }, + "node_modules/color-convert": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/color-convert/-/color-convert-2.0.1.tgz", + "integrity": "sha512-RRECPsj7iu/xb5oKYcsFHSppFNnsj/52OVTRKb4zP5onXwVF3zVmmToNcOfGC+CRDpfK/U584fMg38ZHCaElKQ==", + "dev": true, + "dependencies": { + "color-name": "~1.1.4" + }, + "engines": { + "node": ">=7.0.0" + } + }, + "node_modules/color-name": { + "version": "1.1.4", + "resolved": "https://registry.npmjs.org/color-name/-/color-name-1.1.4.tgz", + "integrity": "sha512-dOy+3AuW3a2wNbZHIuMZpTcgjGuLU/uBL/ubcZF9OXbDo8ff4O8yVp5Bf0efS8uEoYo5q4Fx7dY9OgQGXgAsQA==", + "dev": true + }, + "node_modules/concat-map": { + "version": "0.0.1", + "resolved": "https://registry.npmjs.org/concat-map/-/concat-map-0.0.1.tgz", + "integrity": "sha512-/Srv4dswyQNBfohGpz9o6Yb3Gz3SrUDqBH5rTuhGR7ahtlbYKnVxw2bCFMRljaA7EXHaXZ8wsHdodFvbkhKmqg==", + "dev": true + }, + "node_modules/constructs": { + "version": "10.4.2", + "resolved": "https://registry.npmjs.org/constructs/-/constructs-10.4.2.tgz", + "integrity": "sha512-wsNxBlAott2qg8Zv87q3eYZYgheb9lchtBfjHzzLHtXbttwSrHPs1NNQbBrmbb1YZvYg2+Vh0Dor76w4mFxJkA==" + }, + "node_modules/convert-source-map": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/convert-source-map/-/convert-source-map-2.0.0.tgz", + "integrity": "sha512-Kvp459HrV2FEJ1CAsi1Ku+MY3kasH19TFykTz2xWmMeq6bk2NU3XXvfJ+Q61m0xktWwt+1HSYf3JZsTms3aRJg==", + "dev": true + }, + "node_modules/create-jest": { + "version": "29.7.0", + "resolved": "https://registry.npmjs.org/create-jest/-/create-jest-29.7.0.tgz", + "integrity": "sha512-Adz2bdH0Vq3F53KEMJOoftQFutWCukm6J24wbPWRO4k1kMY7gS7ds/uoJkNuV8wDCtWWnuwGcJwpWcih+zEW1Q==", + "dev": true, + "dependencies": { + "@jest/types": "^29.6.3", + "chalk": "^4.0.0", + "exit": "^0.1.2", + "graceful-fs": "^4.2.9", + "jest-config": "^29.7.0", + "jest-util": "^29.7.0", + "prompts": "^2.0.1" + }, + "bin": { + "create-jest": "bin/create-jest.js" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + } + }, + "node_modules/create-require": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/create-require/-/create-require-1.1.1.tgz", + "integrity": "sha512-dcKFX3jn0MpIaXjisoRvexIJVEKzaq7z2rZKxf+MSr9TkdmHmsU4m2lcLojrj/FHl8mk5VxMmYA+ftRkP/3oKQ==", + "dev": true + }, + "node_modules/cross-spawn": { + "version": "7.0.6", + "resolved": "https://registry.npmjs.org/cross-spawn/-/cross-spawn-7.0.6.tgz", + "integrity": "sha512-uV2QOWP2nWzsy2aMp8aRibhi9dlzF5Hgh5SHaB9OiTGEyDTiJJyx0uy51QXdyWbtAHNua4XJzUKca3OzKUd3vA==", + "dev": true, + "dependencies": { + "path-key": "^3.1.0", + "shebang-command": "^2.0.0", + "which": "^2.0.1" + }, + "engines": { + "node": ">= 8" + } + }, + "node_modules/debug": { + "version": "4.4.0", + "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.0.tgz", + "integrity": "sha512-6WTZ/IxCY/T6BALoZHaE4ctp9xm+Z5kY/pzYaCHRFeyVhojxlrm+46y68HA6hr0TcwEssoxNiDEUJQjfPZ/RYA==", + "dev": true, + "dependencies": { + "ms": "^2.1.3" + }, + "engines": { + "node": ">=6.0" + }, + "peerDependenciesMeta": { + "supports-color": { + "optional": true + } + } + }, + "node_modules/dedent": { + "version": "1.5.3", + "resolved": "https://registry.npmjs.org/dedent/-/dedent-1.5.3.tgz", + "integrity": "sha512-NHQtfOOW68WD8lgypbLA5oT+Bt0xXJhiYvoR6SmmNXZfpzOGXwdKWmcwG8N7PwVVWV3eF/68nmD9BaJSsTBhyQ==", + "dev": true, + "peerDependencies": { + "babel-plugin-macros": "^3.1.0" + }, + "peerDependenciesMeta": { + "babel-plugin-macros": { + "optional": true + } + } + }, + "node_modules/deepmerge": { + "version": "4.3.1", + "resolved": "https://registry.npmjs.org/deepmerge/-/deepmerge-4.3.1.tgz", + "integrity": "sha512-3sUqbMEc77XqpdNO7FRyRog+eW3ph+GYCbj+rK+uYyRMuwsVy0rMiVtPn+QJlKFvWP/1PYpapqYn0Me2knFn+A==", + "dev": true, + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/detect-newline": { + "version": "3.1.0", + "resolved": "https://registry.npmjs.org/detect-newline/-/detect-newline-3.1.0.tgz", + "integrity": "sha512-TLz+x/vEXm/Y7P7wn1EJFNLxYpUD4TgMosxY6fAVJUnJMbupHBOncxyWUG9OpTaH9EBD7uFI5LfEgmMOc54DsA==", + "dev": true, + "engines": { + "node": ">=8" + } + }, + "node_modules/diff": { + "version": "4.0.2", + "resolved": "https://registry.npmjs.org/diff/-/diff-4.0.2.tgz", + "integrity": "sha512-58lmxKSA4BNyLz+HHMUzlOEpg09FV+ev6ZMe3vJihgdxzgcwZ8VoEEPmALCZG9LmqfVoNMMKpttIYTVG6uDY7A==", + "dev": true, + "engines": { + "node": ">=0.3.1" + } + }, + "node_modules/diff-sequences": { + "version": "29.6.3", + "resolved": "https://registry.npmjs.org/diff-sequences/-/diff-sequences-29.6.3.tgz", + "integrity": "sha512-EjePK1srD3P08o2j4f0ExnylqRs5B9tJjcp9t1krH2qRi8CCdsYfwe9JgSLurFBWwq4uOlipzfk5fHNvwFKr8Q==", + "dev": true, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + } + }, + "node_modules/ejs": { + "version": "3.1.10", + "resolved": "https://registry.npmjs.org/ejs/-/ejs-3.1.10.tgz", + "integrity": "sha512-UeJmFfOrAQS8OJWPZ4qtgHyWExa088/MtK5UEyoJGFH67cDEXkZSviOiKRCZ4Xij0zxI3JECgYs3oKx+AizQBA==", + "dev": true, + "dependencies": { + "jake": "^10.8.5" + }, + "bin": { + "ejs": "bin/cli.js" + }, + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/electron-to-chromium": { + "version": "1.5.144", + "resolved": "https://registry.npmjs.org/electron-to-chromium/-/electron-to-chromium-1.5.144.tgz", + "integrity": "sha512-eJIaMRKeAzxfBSxtjYnoIAw/tdD6VIH6tHBZepZnAbE3Gyqqs5mGN87DvcldPUbVkIljTK8pY0CMcUljP64lfQ==", + "dev": true + }, + "node_modules/emittery": { + "version": "0.13.1", + "resolved": "https://registry.npmjs.org/emittery/-/emittery-0.13.1.tgz", + "integrity": "sha512-DeWwawk6r5yR9jFgnDKYt4sLS0LmHJJi3ZOnb5/JdbYwj3nW+FxQnHIjhBKz8YLC7oRNPVM9NQ47I3CVx34eqQ==", + "dev": true, + "engines": { + "node": ">=12" + }, + "funding": { + "url": "https://github.com/sindresorhus/emittery?sponsor=1" + } + }, + "node_modules/emoji-regex": { + "version": "8.0.0", + "resolved": "https://registry.npmjs.org/emoji-regex/-/emoji-regex-8.0.0.tgz", + "integrity": "sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A==", + "dev": true + }, + "node_modules/error-ex": { + "version": "1.3.2", + "resolved": "https://registry.npmjs.org/error-ex/-/error-ex-1.3.2.tgz", + "integrity": "sha512-7dFHNmqeFSEt2ZBsCriorKnn3Z2pj+fd9kmI6QoWw4//DL+icEBfc0U7qJCisqrTsKTjw4fNFy2pW9OqStD84g==", + "dev": true, + "dependencies": { + "is-arrayish": "^0.2.1" + } + }, + "node_modules/escalade": { + "version": "3.2.0", + "resolved": "https://registry.npmjs.org/escalade/-/escalade-3.2.0.tgz", + "integrity": "sha512-WUj2qlxaQtO4g6Pq5c29GTcWGDyd8itL8zTlipgECz3JesAiiOKotd8JU6otB3PACgG6xkJUyVhboMS+bje/jA==", + "dev": true, + "engines": { + "node": ">=6" + } + }, + "node_modules/escape-string-regexp": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/escape-string-regexp/-/escape-string-regexp-2.0.0.tgz", + "integrity": "sha512-UpzcLCXolUWcNu5HtVMHYdXJjArjsF9C0aNnquZYY4uW/Vu0miy5YoWvbV345HauVvcAUnpRuhMMcqTcGOY2+w==", + "dev": true, + "engines": { + "node": ">=8" + } + }, + "node_modules/esprima": { + "version": "4.0.1", + "resolved": "https://registry.npmjs.org/esprima/-/esprima-4.0.1.tgz", + "integrity": "sha512-eGuFFw7Upda+g4p+QHvnW0RyTX/SVeJBDM/gCtMARO0cLuT2HcEKnTPvhjV6aGeqrCB/sbNop0Kszm0jsaWU4A==", + "dev": true, + "bin": { + "esparse": "bin/esparse.js", + "esvalidate": "bin/esvalidate.js" + }, + "engines": { + "node": ">=4" + } + }, + "node_modules/execa": { + "version": "5.1.1", + "resolved": "https://registry.npmjs.org/execa/-/execa-5.1.1.tgz", + "integrity": "sha512-8uSpZZocAZRBAPIEINJj3Lo9HyGitllczc27Eh5YYojjMFMn8yHMDMaUHE2Jqfq05D/wucwI4JGURyXt1vchyg==", + "dev": true, + "dependencies": { + "cross-spawn": "^7.0.3", + "get-stream": "^6.0.0", + "human-signals": "^2.1.0", + "is-stream": "^2.0.0", + "merge-stream": "^2.0.0", + "npm-run-path": "^4.0.1", + "onetime": "^5.1.2", + "signal-exit": "^3.0.3", + "strip-final-newline": "^2.0.0" + }, + "engines": { + "node": ">=10" + }, + "funding": { + "url": "https://github.com/sindresorhus/execa?sponsor=1" + } + }, + "node_modules/exit": { + "version": "0.1.2", + "resolved": "https://registry.npmjs.org/exit/-/exit-0.1.2.tgz", + "integrity": "sha512-Zk/eNKV2zbjpKzrsQ+n1G6poVbErQxJ0LBOJXaKZ1EViLzH+hrLu9cdXI4zw9dBQJslwBEpbQ2P1oS7nDxs6jQ==", + "dev": true, + "engines": { + "node": ">= 0.8.0" + } + }, + "node_modules/expect": { + "version": "29.7.0", + "resolved": "https://registry.npmjs.org/expect/-/expect-29.7.0.tgz", + "integrity": "sha512-2Zks0hf1VLFYI1kbh0I5jP3KHHyCHpkfyHBzsSXRFgl/Bg9mWYfMW8oD+PdMPlEwy5HNsR9JutYy6pMeOh61nw==", + "dev": true, + "dependencies": { + "@jest/expect-utils": "^29.7.0", + "jest-get-type": "^29.6.3", + "jest-matcher-utils": "^29.7.0", + "jest-message-util": "^29.7.0", + "jest-util": "^29.7.0" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + } + }, + "node_modules/fast-json-stable-stringify": { + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/fast-json-stable-stringify/-/fast-json-stable-stringify-2.1.0.tgz", + "integrity": "sha512-lhd/wF+Lk98HZoTCtlVraHtfh5XYijIjalXck7saUtuanSDyLMxnHhSXEDJqHxD7msR8D0uCmqlkwjCV8xvwHw==", + "dev": true + }, + "node_modules/fb-watchman": { + "version": "2.0.2", + "resolved": "https://registry.npmjs.org/fb-watchman/-/fb-watchman-2.0.2.tgz", + "integrity": "sha512-p5161BqbuCaSnB8jIbzQHOlpgsPmK5rJVDfDKO91Axs5NC1uu3HRQm6wt9cd9/+GtQQIO53JdGXXoyDpTAsgYA==", + "dev": true, + "dependencies": { + "bser": "2.1.1" + } + }, + "node_modules/filelist": { + "version": "1.0.4", + "resolved": "https://registry.npmjs.org/filelist/-/filelist-1.0.4.tgz", + "integrity": "sha512-w1cEuf3S+DrLCQL7ET6kz+gmlJdbq9J7yXCSjK/OZCPA+qEN1WyF4ZAf0YYJa4/shHJra2t/d/r8SV4Ji+x+8Q==", + "dev": true, + "dependencies": { + "minimatch": "^5.0.1" + } + }, + "node_modules/filelist/node_modules/brace-expansion": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-2.0.1.tgz", + "integrity": "sha512-XnAIvQ8eM+kC6aULx6wuQiwVsnzsi9d3WxzV3FpWTGA19F621kwdbsAcFKXgKUHZWsy+mY6iL1sHTxWEFCytDA==", + "dev": true, + "dependencies": { + "balanced-match": "^1.0.0" + } + }, + "node_modules/filelist/node_modules/minimatch": { + "version": "5.1.6", + "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-5.1.6.tgz", + "integrity": "sha512-lKwV/1brpG6mBUFHtb7NUmtABCb2WZZmm2wNiOA5hAb8VdCS4B3dtMWyvcoViccwAW/COERjXLt0zP1zXUN26g==", + "dev": true, + "dependencies": { + "brace-expansion": "^2.0.1" + }, + "engines": { + "node": ">=10" + } + }, + "node_modules/fill-range": { + "version": "7.1.1", + "resolved": "https://registry.npmjs.org/fill-range/-/fill-range-7.1.1.tgz", + "integrity": "sha512-YsGpe3WHLK8ZYi4tWDg2Jy3ebRz2rXowDxnld4bkQB00cc/1Zw9AWnC0i9ztDJitivtQvaI9KaLyKrc+hBW0yg==", + "dev": true, + "dependencies": { + "to-regex-range": "^5.0.1" + }, + "engines": { + "node": ">=8" + } + }, + "node_modules/find-up": { + "version": "4.1.0", + "resolved": "https://registry.npmjs.org/find-up/-/find-up-4.1.0.tgz", + "integrity": "sha512-PpOwAdQ/YlXQ2vj8a3h8IipDuYRi3wceVQQGYWxNINccq40Anw7BlsEXCMbt1Zt+OLA6Fq9suIpIWD0OsnISlw==", + "dev": true, + "dependencies": { + "locate-path": "^5.0.0", + "path-exists": "^4.0.0" + }, + "engines": { + "node": ">=8" + } + }, + "node_modules/fs.realpath": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/fs.realpath/-/fs.realpath-1.0.0.tgz", + "integrity": "sha512-OO0pH2lK6a0hZnAdau5ItzHPI6pUlvI7jMVnxUQRtw4owF2wk8lOSabtGDCTP4Ggrg2MbGnWO9X8K1t4+fGMDw==", + "dev": true + }, + "node_modules/fsevents": { + "version": "2.3.2", + "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.2.tgz", + "integrity": "sha512-xiqMQR4xAeHTuB9uWm+fFRcIOgKBMiOBP+eXiyT7jsgVCq1bkVygt00oASowB7EdtpOHaaPgKt812P9ab+DDKA==", + "dev": true, + "hasInstallScript": true, + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": "^8.16.0 || ^10.6.0 || >=11.0.0" + } + }, + "node_modules/function-bind": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/function-bind/-/function-bind-1.1.2.tgz", + "integrity": "sha512-7XHNxH7qX9xG5mIwxkhumTox/MIRNcOgDrxWsMt2pAr23WHp6MrRlN7FBSFpCpr+oVO0F744iUgR82nJMfG2SA==", + "dev": true, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/gensync": { + "version": "1.0.0-beta.2", + "resolved": "https://registry.npmjs.org/gensync/-/gensync-1.0.0-beta.2.tgz", + "integrity": "sha512-3hN7NaskYvMDLQY55gnW3NQ+mesEAepTqlg+VEbj7zzqEMBVNhzcGYYeqFo/TlYz6eQiFcp1HcsCZO+nGgS8zg==", + "dev": true, + "engines": { + "node": ">=6.9.0" + } + }, + "node_modules/get-caller-file": { + "version": "2.0.5", + "resolved": "https://registry.npmjs.org/get-caller-file/-/get-caller-file-2.0.5.tgz", + "integrity": "sha512-DyFP3BM/3YHTQOCUL/w0OZHR0lpKeGrxotcHWcqNEdnltqFwXVfhEBQ94eIo34AfQpo0rGki4cyIiftY06h2Fg==", + "dev": true, + "engines": { + "node": "6.* || 8.* || >= 10.*" + } + }, + "node_modules/get-package-type": { + "version": "0.1.0", + "resolved": "https://registry.npmjs.org/get-package-type/-/get-package-type-0.1.0.tgz", + "integrity": "sha512-pjzuKtY64GYfWizNAJ0fr9VqttZkNiK2iS430LtIHzjBEr6bX8Am2zm4sW4Ro5wjWW5cAlRL1qAMTcXbjNAO2Q==", + "dev": true, + "engines": { + "node": ">=8.0.0" + } + }, + "node_modules/get-stream": { + "version": "6.0.1", + "resolved": "https://registry.npmjs.org/get-stream/-/get-stream-6.0.1.tgz", + "integrity": "sha512-ts6Wi+2j3jQjqi70w5AlN8DFnkSwC+MqmxEzdEALB2qXZYV3X/b1CTfgPLGJNMeAWxdPfU8FO1ms3NUfaHCPYg==", + "dev": true, + "engines": { + "node": ">=10" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/glob": { + "version": "7.2.3", + "resolved": "https://registry.npmjs.org/glob/-/glob-7.2.3.tgz", + "integrity": "sha512-nFR0zLpU2YCaRxwoCJvL6UvCH2JFyFVIvwTLsIf21AuHlMskA1hhTdk+LlYJtOlYt9v6dvszD2BGRqBL+iQK9Q==", + "deprecated": "Glob versions prior to v9 are no longer supported", + "dev": true, + "dependencies": { + "fs.realpath": "^1.0.0", + "inflight": "^1.0.4", + "inherits": "2", + "minimatch": "^3.1.1", + "once": "^1.3.0", + "path-is-absolute": "^1.0.0" + }, + "engines": { + "node": "*" + }, + "funding": { + "url": "https://github.com/sponsors/isaacs" + } + }, + "node_modules/globals": { + "version": "11.12.0", + "resolved": "https://registry.npmjs.org/globals/-/globals-11.12.0.tgz", + "integrity": "sha512-WOBp/EEGUiIsJSp7wcv/y6MO+lV9UoncWqxuFfm8eBwzWNgyfBd6Gz+IeKQ9jCmyhoH99g15M3T+QaVHFjizVA==", + "dev": true, + "engines": { + "node": ">=4" + } + }, + "node_modules/graceful-fs": { + "version": "4.2.11", + "resolved": "https://registry.npmjs.org/graceful-fs/-/graceful-fs-4.2.11.tgz", + "integrity": "sha512-RbJ5/jmFcNNCcDV5o9eTnBLJ/HszWV0P73bc+Ff4nS/rJj+YaS6IGyiOL0VoBYX+l1Wrl3k63h/KrH+nhJ0XvQ==", + "dev": true + }, + "node_modules/has-flag": { + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/has-flag/-/has-flag-4.0.0.tgz", + "integrity": "sha512-EykJT/Q1KjTWctppgIAgfSO0tKVuZUjhgMr17kqTumMl6Afv3EISleU7qZUzoXDFTAHTDC4NOoG/ZxU3EvlMPQ==", + "dev": true, + "engines": { + "node": ">=8" + } + }, + "node_modules/hasown": { + "version": "2.0.2", + "resolved": "https://registry.npmjs.org/hasown/-/hasown-2.0.2.tgz", + "integrity": "sha512-0hJU9SCPvmMzIBdZFqNPXWa6dqh7WdH0cII9y+CyS8rG3nL48Bclra9HmKhVVUHyPWNH5Y7xDwAB7bfgSjkUMQ==", + "dev": true, + "dependencies": { + "function-bind": "^1.1.2" + }, + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/html-escaper": { + "version": "2.0.2", + "resolved": "https://registry.npmjs.org/html-escaper/-/html-escaper-2.0.2.tgz", + "integrity": "sha512-H2iMtd0I4Mt5eYiapRdIDjp+XzelXQ0tFE4JS7YFwFevXXMmOp9myNrUvCg0D6ws8iqkRPBfKHgbwig1SmlLfg==", + "dev": true + }, + "node_modules/human-signals": { + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/human-signals/-/human-signals-2.1.0.tgz", + "integrity": "sha512-B4FFZ6q/T2jhhksgkbEW3HBvWIfDW85snkQgawt07S7J5QXTk6BkNV+0yAeZrM5QpMAdYlocGoljn0sJ/WQkFw==", + "dev": true, + "engines": { + "node": ">=10.17.0" + } + }, + "node_modules/import-local": { + "version": "3.2.0", + "resolved": "https://registry.npmjs.org/import-local/-/import-local-3.2.0.tgz", + "integrity": "sha512-2SPlun1JUPWoM6t3F0dw0FkCF/jWY8kttcY4f599GLTSjh2OCuuhdTkJQsEcZzBqbXZGKMK2OqW1oZsjtf/gQA==", + "dev": true, + "dependencies": { + "pkg-dir": "^4.2.0", + "resolve-cwd": "^3.0.0" + }, + "bin": { + "import-local-fixture": "fixtures/cli.js" + }, + "engines": { + "node": ">=8" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/imurmurhash": { + "version": "0.1.4", + "resolved": "https://registry.npmjs.org/imurmurhash/-/imurmurhash-0.1.4.tgz", + "integrity": "sha512-JmXMZ6wuvDmLiHEml9ykzqO6lwFbof0GG4IkcGaENdCRDDmMVnny7s5HsIgHCbaq0w2MyPhDqkhTUgS2LU2PHA==", + "dev": true, + "engines": { + "node": ">=0.8.19" + } + }, + "node_modules/inflight": { + "version": "1.0.6", + "resolved": "https://registry.npmjs.org/inflight/-/inflight-1.0.6.tgz", + "integrity": "sha512-k92I/b08q4wvFscXCLvqfsHCrjrF7yiXsQuIVvVE7N82W3+aqpzuUdBbfhWcy/FZR3/4IgflMgKLOsvPDrGCJA==", + "deprecated": "This module is not supported, and leaks memory. Do not use it. Check out lru-cache if you want a good and tested way to coalesce async requests by a key value, which is much more comprehensive and powerful.", + "dev": true, + "dependencies": { + "once": "^1.3.0", + "wrappy": "1" + } + }, + "node_modules/inherits": { + "version": "2.0.4", + "resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz", + "integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ==", + "dev": true + }, + "node_modules/is-arrayish": { + "version": "0.2.1", + "resolved": "https://registry.npmjs.org/is-arrayish/-/is-arrayish-0.2.1.tgz", + "integrity": "sha512-zz06S8t0ozoDXMG+ube26zeCTNXcKIPJZJi8hBrF4idCLms4CG9QtK7qBl1boi5ODzFpjswb5JPmHCbMpjaYzg==", + "dev": true + }, + "node_modules/is-core-module": { + "version": "2.16.1", + "resolved": "https://registry.npmjs.org/is-core-module/-/is-core-module-2.16.1.tgz", + "integrity": "sha512-UfoeMA6fIJ8wTYFEUjelnaGI67v6+N7qXJEvQuIGa99l4xsCruSYOVSQ0uPANn4dAzm8lkYPaKLrrijLq7x23w==", + "dev": true, + "dependencies": { + "hasown": "^2.0.2" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/is-fullwidth-code-point": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/is-fullwidth-code-point/-/is-fullwidth-code-point-3.0.0.tgz", + "integrity": "sha512-zymm5+u+sCsSWyD9qNaejV3DFvhCKclKdizYaJUuHA83RLjb7nSuGnddCHGv0hk+KY7BMAlsWeK4Ueg6EV6XQg==", + "dev": true, + "engines": { + "node": ">=8" + } + }, + "node_modules/is-generator-fn": { + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/is-generator-fn/-/is-generator-fn-2.1.0.tgz", + "integrity": "sha512-cTIB4yPYL/Grw0EaSzASzg6bBy9gqCofvWN8okThAYIxKJZC+udlRAmGbM0XLeniEJSs8uEgHPGuHSe1XsOLSQ==", + "dev": true, + "engines": { + "node": ">=6" + } + }, + "node_modules/is-number": { + "version": "7.0.0", + "resolved": "https://registry.npmjs.org/is-number/-/is-number-7.0.0.tgz", + "integrity": "sha512-41Cifkg6e8TylSpdtTpeLVMqvSBEVzTttHvERD741+pnZ8ANv0004MRL43QKPDlK9cGvNp6NZWZUBlbGXYxxng==", + "dev": true, + "engines": { + "node": ">=0.12.0" + } + }, + "node_modules/is-stream": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/is-stream/-/is-stream-2.0.1.tgz", + "integrity": "sha512-hFoiJiTl63nn+kstHGBtewWSKnQLpyb155KHheA1l39uvtO9nWIop1p3udqPcUd/xbF1VLMO4n7OI6p7RbngDg==", + "dev": true, + "engines": { + "node": ">=8" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/isexe": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/isexe/-/isexe-2.0.0.tgz", + "integrity": "sha512-RHxMLp9lnKHGHRng9QFhRCMbYAcVpn69smSGcq3f36xjgVVWThj4qqLbTLlq7Ssj8B+fIQ1EuCEGI2lKsyQeIw==", + "dev": true + }, + "node_modules/istanbul-lib-coverage": { + "version": "3.2.2", + "resolved": "https://registry.npmjs.org/istanbul-lib-coverage/-/istanbul-lib-coverage-3.2.2.tgz", + "integrity": "sha512-O8dpsF+r0WV/8MNRKfnmrtCWhuKjxrq2w+jpzBL5UZKTi2LeVWnWOmWRxFlesJONmc+wLAGvKQZEOanko0LFTg==", + "dev": true, + "engines": { + "node": ">=8" + } + }, + "node_modules/istanbul-lib-instrument": { + "version": "6.0.3", + "resolved": "https://registry.npmjs.org/istanbul-lib-instrument/-/istanbul-lib-instrument-6.0.3.tgz", + "integrity": "sha512-Vtgk7L/R2JHyyGW07spoFlB8/lpjiOLTjMdms6AFMraYt3BaJauod/NGrfnVG/y4Ix1JEuMRPDPEj2ua+zz1/Q==", + "dev": true, + "dependencies": { + "@babel/core": "^7.23.9", + "@babel/parser": "^7.23.9", + "@istanbuljs/schema": "^0.1.3", + "istanbul-lib-coverage": "^3.2.0", + "semver": "^7.5.4" + }, + "engines": { + "node": ">=10" + } + }, + "node_modules/istanbul-lib-instrument/node_modules/semver": { + "version": "7.7.1", + "resolved": "https://registry.npmjs.org/semver/-/semver-7.7.1.tgz", + "integrity": "sha512-hlq8tAfn0m/61p4BVRcPzIGr6LKiMwo4VM6dGi6pt4qcRkmNzTcWq6eCEjEh+qXjkMDvPlOFFSGwQjoEa6gyMA==", + "dev": true, + "bin": { + "semver": "bin/semver.js" + }, + "engines": { + "node": ">=10" + } + }, + "node_modules/istanbul-lib-report": { + "version": "3.0.1", + "resolved": "https://registry.npmjs.org/istanbul-lib-report/-/istanbul-lib-report-3.0.1.tgz", + "integrity": "sha512-GCfE1mtsHGOELCU8e/Z7YWzpmybrx/+dSTfLrvY8qRmaY6zXTKWn6WQIjaAFw069icm6GVMNkgu0NzI4iPZUNw==", + "dev": true, + "dependencies": { + "istanbul-lib-coverage": "^3.0.0", + "make-dir": "^4.0.0", + "supports-color": "^7.1.0" + }, + "engines": { + "node": ">=10" + } + }, + "node_modules/istanbul-lib-source-maps": { + "version": "4.0.1", + "resolved": "https://registry.npmjs.org/istanbul-lib-source-maps/-/istanbul-lib-source-maps-4.0.1.tgz", + "integrity": "sha512-n3s8EwkdFIJCG3BPKBYvskgXGoy88ARzvegkitk60NxRdwltLOTaH7CUiMRXvwYorl0Q712iEjcWB+fK/MrWVw==", + "dev": true, + "dependencies": { + "debug": "^4.1.1", + "istanbul-lib-coverage": "^3.0.0", + "source-map": "^0.6.1" + }, + "engines": { + "node": ">=10" + } + }, + "node_modules/istanbul-reports": { + "version": "3.1.7", + "resolved": "https://registry.npmjs.org/istanbul-reports/-/istanbul-reports-3.1.7.tgz", + "integrity": "sha512-BewmUXImeuRk2YY0PVbxgKAysvhRPUQE0h5QRM++nVWyubKGV0l8qQ5op8+B2DOmwSe63Jivj0BjkPQVf8fP5g==", + "dev": true, + "dependencies": { + "html-escaper": "^2.0.0", + "istanbul-lib-report": "^3.0.0" + }, + "engines": { + "node": ">=8" + } + }, + "node_modules/jake": { + "version": "10.9.2", + "resolved": "https://registry.npmjs.org/jake/-/jake-10.9.2.tgz", + "integrity": "sha512-2P4SQ0HrLQ+fw6llpLnOaGAvN2Zu6778SJMrCUwns4fOoG9ayrTiZk3VV8sCPkVZF8ab0zksVpS8FDY5pRCNBA==", + "dev": true, + "dependencies": { + "async": "^3.2.3", + "chalk": "^4.0.2", + "filelist": "^1.0.4", + "minimatch": "^3.1.2" + }, + "bin": { + "jake": "bin/cli.js" + }, + "engines": { + "node": ">=10" + } + }, + "node_modules/jest": { + "version": "29.7.0", + "resolved": "https://registry.npmjs.org/jest/-/jest-29.7.0.tgz", + "integrity": "sha512-NIy3oAFp9shda19hy4HK0HRTWKtPJmGdnvywu01nOqNC2vZg+Z+fvJDxpMQA88eb2I9EcafcdjYgsDthnYTvGw==", + "dev": true, + "dependencies": { + "@jest/core": "^29.7.0", + "@jest/types": "^29.6.3", + "import-local": "^3.0.2", + "jest-cli": "^29.7.0" + }, + "bin": { + "jest": "bin/jest.js" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + }, + "peerDependencies": { + "node-notifier": "^8.0.1 || ^9.0.0 || ^10.0.0" + }, + "peerDependenciesMeta": { + "node-notifier": { + "optional": true + } + } + }, + "node_modules/jest-changed-files": { + "version": "29.7.0", + "resolved": "https://registry.npmjs.org/jest-changed-files/-/jest-changed-files-29.7.0.tgz", + "integrity": "sha512-fEArFiwf1BpQ+4bXSprcDc3/x4HSzL4al2tozwVpDFpsxALjLYdyiIK4e5Vz66GQJIbXJ82+35PtysofptNX2w==", + "dev": true, + "dependencies": { + "execa": "^5.0.0", + "jest-util": "^29.7.0", + "p-limit": "^3.1.0" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + } + }, + "node_modules/jest-circus": { + "version": "29.7.0", + "resolved": "https://registry.npmjs.org/jest-circus/-/jest-circus-29.7.0.tgz", + "integrity": "sha512-3E1nCMgipcTkCocFwM90XXQab9bS+GMsjdpmPrlelaxwD93Ad8iVEjX/vvHPdLPnFf+L40u+5+iutRdA1N9myw==", + "dev": true, + "dependencies": { + "@jest/environment": "^29.7.0", + "@jest/expect": "^29.7.0", + "@jest/test-result": "^29.7.0", + "@jest/types": "^29.6.3", + "@types/node": "*", + "chalk": "^4.0.0", + "co": "^4.6.0", + "dedent": "^1.0.0", + "is-generator-fn": "^2.0.0", + "jest-each": "^29.7.0", + "jest-matcher-utils": "^29.7.0", + "jest-message-util": "^29.7.0", + "jest-runtime": "^29.7.0", + "jest-snapshot": "^29.7.0", + "jest-util": "^29.7.0", + "p-limit": "^3.1.0", + "pretty-format": "^29.7.0", + "pure-rand": "^6.0.0", + "slash": "^3.0.0", + "stack-utils": "^2.0.3" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + } + }, + "node_modules/jest-cli": { + "version": "29.7.0", + "resolved": "https://registry.npmjs.org/jest-cli/-/jest-cli-29.7.0.tgz", + "integrity": "sha512-OVVobw2IubN/GSYsxETi+gOe7Ka59EFMR/twOU3Jb2GnKKeMGJB5SGUUrEz3SFVmJASUdZUzy83sLNNQ2gZslg==", + "dev": true, + "dependencies": { + "@jest/core": "^29.7.0", + "@jest/test-result": "^29.7.0", + "@jest/types": "^29.6.3", + "chalk": "^4.0.0", + "create-jest": "^29.7.0", + "exit": "^0.1.2", + "import-local": "^3.0.2", + "jest-config": "^29.7.0", + "jest-util": "^29.7.0", + "jest-validate": "^29.7.0", + "yargs": "^17.3.1" + }, + "bin": { + "jest": "bin/jest.js" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + }, + "peerDependencies": { + "node-notifier": "^8.0.1 || ^9.0.0 || ^10.0.0" + }, + "peerDependenciesMeta": { + "node-notifier": { + "optional": true + } + } + }, + "node_modules/jest-config": { + "version": "29.7.0", + "resolved": "https://registry.npmjs.org/jest-config/-/jest-config-29.7.0.tgz", + "integrity": "sha512-uXbpfeQ7R6TZBqI3/TxCU4q4ttk3u0PJeC+E0zbfSoSjq6bJ7buBPxzQPL0ifrkY4DNu4JUdk0ImlBUYi840eQ==", + "dev": true, + "dependencies": { + "@babel/core": "^7.11.6", + "@jest/test-sequencer": "^29.7.0", + "@jest/types": "^29.6.3", + "babel-jest": "^29.7.0", + "chalk": "^4.0.0", + "ci-info": "^3.2.0", + "deepmerge": "^4.2.2", + "glob": "^7.1.3", + "graceful-fs": "^4.2.9", + "jest-circus": "^29.7.0", + "jest-environment-node": "^29.7.0", + "jest-get-type": "^29.6.3", + "jest-regex-util": "^29.6.3", + "jest-resolve": "^29.7.0", + "jest-runner": "^29.7.0", + "jest-util": "^29.7.0", + "jest-validate": "^29.7.0", + "micromatch": "^4.0.4", + "parse-json": "^5.2.0", + "pretty-format": "^29.7.0", + "slash": "^3.0.0", + "strip-json-comments": "^3.1.1" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + }, + "peerDependencies": { + "@types/node": "*", + "ts-node": ">=9.0.0" + }, + "peerDependenciesMeta": { + "@types/node": { + "optional": true + }, + "ts-node": { + "optional": true + } + } + }, + "node_modules/jest-diff": { + "version": "29.7.0", + "resolved": "https://registry.npmjs.org/jest-diff/-/jest-diff-29.7.0.tgz", + "integrity": "sha512-LMIgiIrhigmPrs03JHpxUh2yISK3vLFPkAodPeo0+BuF7wA2FoQbkEg1u8gBYBThncu7e1oEDUfIXVuTqLRUjw==", + "dev": true, + "dependencies": { + "chalk": "^4.0.0", + "diff-sequences": "^29.6.3", + "jest-get-type": "^29.6.3", + "pretty-format": "^29.7.0" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + } + }, + "node_modules/jest-docblock": { + "version": "29.7.0", + "resolved": "https://registry.npmjs.org/jest-docblock/-/jest-docblock-29.7.0.tgz", + "integrity": "sha512-q617Auw3A612guyaFgsbFeYpNP5t2aoUNLwBUbc/0kD1R4t9ixDbyFTHd1nok4epoVFpr7PmeWHrhvuV3XaJ4g==", + "dev": true, + "dependencies": { + "detect-newline": "^3.0.0" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + } + }, + "node_modules/jest-each": { + "version": "29.7.0", + "resolved": "https://registry.npmjs.org/jest-each/-/jest-each-29.7.0.tgz", + "integrity": "sha512-gns+Er14+ZrEoC5fhOfYCY1LOHHr0TI+rQUHZS8Ttw2l7gl+80eHc/gFf2Ktkw0+SIACDTeWvpFcv3B04VembQ==", + "dev": true, + "dependencies": { + "@jest/types": "^29.6.3", + "chalk": "^4.0.0", + "jest-get-type": "^29.6.3", + "jest-util": "^29.7.0", + "pretty-format": "^29.7.0" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + } + }, + "node_modules/jest-environment-node": { + "version": "29.7.0", + "resolved": "https://registry.npmjs.org/jest-environment-node/-/jest-environment-node-29.7.0.tgz", + "integrity": "sha512-DOSwCRqXirTOyheM+4d5YZOrWcdu0LNZ87ewUoywbcb2XR4wKgqiG8vNeYwhjFMbEkfju7wx2GYH0P2gevGvFw==", + "dev": true, + "dependencies": { + "@jest/environment": "^29.7.0", + "@jest/fake-timers": "^29.7.0", + "@jest/types": "^29.6.3", + "@types/node": "*", + "jest-mock": "^29.7.0", + "jest-util": "^29.7.0" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + } + }, + "node_modules/jest-get-type": { + "version": "29.6.3", + "resolved": "https://registry.npmjs.org/jest-get-type/-/jest-get-type-29.6.3.tgz", + "integrity": "sha512-zrteXnqYxfQh7l5FHyL38jL39di8H8rHoecLH3JNxH3BwOrBsNeabdap5e0I23lD4HHI8W5VFBZqG4Eaq5LNcw==", + "dev": true, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + } + }, + "node_modules/jest-haste-map": { + "version": "29.7.0", + "resolved": "https://registry.npmjs.org/jest-haste-map/-/jest-haste-map-29.7.0.tgz", + "integrity": "sha512-fP8u2pyfqx0K1rGn1R9pyE0/KTn+G7PxktWidOBTqFPLYX0b9ksaMFkhK5vrS3DVun09pckLdlx90QthlW7AmA==", + "dev": true, + "dependencies": { + "@jest/types": "^29.6.3", + "@types/graceful-fs": "^4.1.3", + "@types/node": "*", + "anymatch": "^3.0.3", + "fb-watchman": "^2.0.0", + "graceful-fs": "^4.2.9", + "jest-regex-util": "^29.6.3", + "jest-util": "^29.7.0", + "jest-worker": "^29.7.0", + "micromatch": "^4.0.4", + "walker": "^1.0.8" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + }, + "optionalDependencies": { + "fsevents": "^2.3.2" + } + }, + "node_modules/jest-leak-detector": { + "version": "29.7.0", + "resolved": "https://registry.npmjs.org/jest-leak-detector/-/jest-leak-detector-29.7.0.tgz", + "integrity": "sha512-kYA8IJcSYtST2BY9I+SMC32nDpBT3J2NvWJx8+JCuCdl/CR1I4EKUJROiP8XtCcxqgTTBGJNdbB1A8XRKbTetw==", + "dev": true, + "dependencies": { + "jest-get-type": "^29.6.3", + "pretty-format": "^29.7.0" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + } + }, + "node_modules/jest-matcher-utils": { + "version": "29.7.0", + "resolved": "https://registry.npmjs.org/jest-matcher-utils/-/jest-matcher-utils-29.7.0.tgz", + "integrity": "sha512-sBkD+Xi9DtcChsI3L3u0+N0opgPYnCRPtGcQYrgXmR+hmt/fYfWAL0xRXYU8eWOdfuLgBe0YCW3AFtnRLagq/g==", + "dev": true, + "dependencies": { + "chalk": "^4.0.0", + "jest-diff": "^29.7.0", + "jest-get-type": "^29.6.3", + "pretty-format": "^29.7.0" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + } + }, + "node_modules/jest-message-util": { + "version": "29.7.0", + "resolved": "https://registry.npmjs.org/jest-message-util/-/jest-message-util-29.7.0.tgz", + "integrity": "sha512-GBEV4GRADeP+qtB2+6u61stea8mGcOT4mCtrYISZwfu9/ISHFJ/5zOMXYbpBE9RsS5+Gb63DW4FgmnKJ79Kf6w==", + "dev": true, + "dependencies": { + "@babel/code-frame": "^7.12.13", + "@jest/types": "^29.6.3", + "@types/stack-utils": "^2.0.0", + "chalk": "^4.0.0", + "graceful-fs": "^4.2.9", + "micromatch": "^4.0.4", + "pretty-format": "^29.7.0", + "slash": "^3.0.0", + "stack-utils": "^2.0.3" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + } + }, + "node_modules/jest-mock": { + "version": "29.7.0", + "resolved": "https://registry.npmjs.org/jest-mock/-/jest-mock-29.7.0.tgz", + "integrity": "sha512-ITOMZn+UkYS4ZFh83xYAOzWStloNzJFO2s8DWrE4lhtGD+AorgnbkiKERe4wQVBydIGPx059g6riW5Btp6Llnw==", + "dev": true, + "dependencies": { + "@jest/types": "^29.6.3", + "@types/node": "*", + "jest-util": "^29.7.0" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + } + }, + "node_modules/jest-pnp-resolver": { + "version": "1.2.3", + "resolved": "https://registry.npmjs.org/jest-pnp-resolver/-/jest-pnp-resolver-1.2.3.tgz", + "integrity": "sha512-+3NpwQEnRoIBtx4fyhblQDPgJI0H1IEIkX7ShLUjPGA7TtUTvI1oiKi3SR4oBR0hQhQR80l4WAe5RrXBwWMA8w==", + "dev": true, + "engines": { + "node": ">=6" + }, + "peerDependencies": { + "jest-resolve": "*" + }, + "peerDependenciesMeta": { + "jest-resolve": { + "optional": true + } + } + }, + "node_modules/jest-regex-util": { + "version": "29.6.3", + "resolved": "https://registry.npmjs.org/jest-regex-util/-/jest-regex-util-29.6.3.tgz", + "integrity": "sha512-KJJBsRCyyLNWCNBOvZyRDnAIfUiRJ8v+hOBQYGn8gDyF3UegwiP4gwRR3/SDa42g1YbVycTidUF3rKjyLFDWbg==", + "dev": true, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + } + }, + "node_modules/jest-resolve": { + "version": "29.7.0", + "resolved": "https://registry.npmjs.org/jest-resolve/-/jest-resolve-29.7.0.tgz", + "integrity": "sha512-IOVhZSrg+UvVAshDSDtHyFCCBUl/Q3AAJv8iZ6ZjnZ74xzvwuzLXid9IIIPgTnY62SJjfuupMKZsZQRsCvxEgA==", + "dev": true, + "dependencies": { + "chalk": "^4.0.0", + "graceful-fs": "^4.2.9", + "jest-haste-map": "^29.7.0", + "jest-pnp-resolver": "^1.2.2", + "jest-util": "^29.7.0", + "jest-validate": "^29.7.0", + "resolve": "^1.20.0", + "resolve.exports": "^2.0.0", + "slash": "^3.0.0" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + } + }, + "node_modules/jest-resolve-dependencies": { + "version": "29.7.0", + "resolved": "https://registry.npmjs.org/jest-resolve-dependencies/-/jest-resolve-dependencies-29.7.0.tgz", + "integrity": "sha512-un0zD/6qxJ+S0et7WxeI3H5XSe9lTBBR7bOHCHXkKR6luG5mwDDlIzVQ0V5cZCuoTgEdcdwzTghYkTWfubi+nA==", + "dev": true, + "dependencies": { + "jest-regex-util": "^29.6.3", + "jest-snapshot": "^29.7.0" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + } + }, + "node_modules/jest-runner": { + "version": "29.7.0", + "resolved": "https://registry.npmjs.org/jest-runner/-/jest-runner-29.7.0.tgz", + "integrity": "sha512-fsc4N6cPCAahybGBfTRcq5wFR6fpLznMg47sY5aDpsoejOcVYFb07AHuSnR0liMcPTgBsA3ZJL6kFOjPdoNipQ==", + "dev": true, + "dependencies": { + "@jest/console": "^29.7.0", + "@jest/environment": "^29.7.0", + "@jest/test-result": "^29.7.0", + "@jest/transform": "^29.7.0", + "@jest/types": "^29.6.3", + "@types/node": "*", + "chalk": "^4.0.0", + "emittery": "^0.13.1", + "graceful-fs": "^4.2.9", + "jest-docblock": "^29.7.0", + "jest-environment-node": "^29.7.0", + "jest-haste-map": "^29.7.0", + "jest-leak-detector": "^29.7.0", + "jest-message-util": "^29.7.0", + "jest-resolve": "^29.7.0", + "jest-runtime": "^29.7.0", + "jest-util": "^29.7.0", + "jest-watcher": "^29.7.0", + "jest-worker": "^29.7.0", + "p-limit": "^3.1.0", + "source-map-support": "0.5.13" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + } + }, + "node_modules/jest-runner/node_modules/source-map-support": { + "version": "0.5.13", + "resolved": "https://registry.npmjs.org/source-map-support/-/source-map-support-0.5.13.tgz", + "integrity": "sha512-SHSKFHadjVA5oR4PPqhtAVdcBWwRYVd6g6cAXnIbRiIwc2EhPrTuKUBdSLvlEKyIP3GCf89fltvcZiP9MMFA1w==", + "dev": true, + "dependencies": { + "buffer-from": "^1.0.0", + "source-map": "^0.6.0" + } + }, + "node_modules/jest-runtime": { + "version": "29.7.0", + "resolved": "https://registry.npmjs.org/jest-runtime/-/jest-runtime-29.7.0.tgz", + "integrity": "sha512-gUnLjgwdGqW7B4LvOIkbKs9WGbn+QLqRQQ9juC6HndeDiezIwhDP+mhMwHWCEcfQ5RUXa6OPnFF8BJh5xegwwQ==", + "dev": true, + "dependencies": { + "@jest/environment": "^29.7.0", + "@jest/fake-timers": "^29.7.0", + "@jest/globals": "^29.7.0", + "@jest/source-map": "^29.6.3", + "@jest/test-result": "^29.7.0", + "@jest/transform": "^29.7.0", + "@jest/types": "^29.6.3", + "@types/node": "*", + "chalk": "^4.0.0", + "cjs-module-lexer": "^1.0.0", + "collect-v8-coverage": "^1.0.0", + "glob": "^7.1.3", + "graceful-fs": "^4.2.9", + "jest-haste-map": "^29.7.0", + "jest-message-util": "^29.7.0", + "jest-mock": "^29.7.0", + "jest-regex-util": "^29.6.3", + "jest-resolve": "^29.7.0", + "jest-snapshot": "^29.7.0", + "jest-util": "^29.7.0", + "slash": "^3.0.0", + "strip-bom": "^4.0.0" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + } + }, + "node_modules/jest-snapshot": { + "version": "29.7.0", + "resolved": "https://registry.npmjs.org/jest-snapshot/-/jest-snapshot-29.7.0.tgz", + "integrity": "sha512-Rm0BMWtxBcioHr1/OX5YCP8Uov4riHvKPknOGs804Zg9JGZgmIBkbtlxJC/7Z4msKYVbIJtfU+tKb8xlYNfdkw==", + "dev": true, + "dependencies": { + "@babel/core": "^7.11.6", + "@babel/generator": "^7.7.2", + "@babel/plugin-syntax-jsx": "^7.7.2", + "@babel/plugin-syntax-typescript": "^7.7.2", + "@babel/types": "^7.3.3", + "@jest/expect-utils": "^29.7.0", + "@jest/transform": "^29.7.0", + "@jest/types": "^29.6.3", + "babel-preset-current-node-syntax": "^1.0.0", + "chalk": "^4.0.0", + "expect": "^29.7.0", + "graceful-fs": "^4.2.9", + "jest-diff": "^29.7.0", + "jest-get-type": "^29.6.3", + "jest-matcher-utils": "^29.7.0", + "jest-message-util": "^29.7.0", + "jest-util": "^29.7.0", + "natural-compare": "^1.4.0", + "pretty-format": "^29.7.0", + "semver": "^7.5.3" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + } + }, + "node_modules/jest-snapshot/node_modules/semver": { + "version": "7.7.1", + "resolved": "https://registry.npmjs.org/semver/-/semver-7.7.1.tgz", + "integrity": "sha512-hlq8tAfn0m/61p4BVRcPzIGr6LKiMwo4VM6dGi6pt4qcRkmNzTcWq6eCEjEh+qXjkMDvPlOFFSGwQjoEa6gyMA==", + "dev": true, + "bin": { + "semver": "bin/semver.js" + }, + "engines": { + "node": ">=10" + } + }, + "node_modules/jest-util": { + "version": "29.7.0", + "resolved": "https://registry.npmjs.org/jest-util/-/jest-util-29.7.0.tgz", + "integrity": "sha512-z6EbKajIpqGKU56y5KBUgy1dt1ihhQJgWzUlZHArA/+X2ad7Cb5iF+AK1EWVL/Bo7Rz9uurpqw6SiBCefUbCGA==", + "dev": true, + "dependencies": { + "@jest/types": "^29.6.3", + "@types/node": "*", + "chalk": "^4.0.0", + "ci-info": "^3.2.0", + "graceful-fs": "^4.2.9", + "picomatch": "^2.2.3" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + } + }, + "node_modules/jest-validate": { + "version": "29.7.0", + "resolved": "https://registry.npmjs.org/jest-validate/-/jest-validate-29.7.0.tgz", + "integrity": "sha512-ZB7wHqaRGVw/9hST/OuFUReG7M8vKeq0/J2egIGLdvjHCmYqGARhzXmtgi+gVeZ5uXFF219aOc3Ls2yLg27tkw==", + "dev": true, + "dependencies": { + "@jest/types": "^29.6.3", + "camelcase": "^6.2.0", + "chalk": "^4.0.0", + "jest-get-type": "^29.6.3", + "leven": "^3.1.0", + "pretty-format": "^29.7.0" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + } + }, + "node_modules/jest-validate/node_modules/camelcase": { + "version": "6.3.0", + "resolved": "https://registry.npmjs.org/camelcase/-/camelcase-6.3.0.tgz", + "integrity": "sha512-Gmy6FhYlCY7uOElZUSbxo2UCDH8owEk996gkbrpsgGtrJLM3J7jGxl9Ic7Qwwj4ivOE5AWZWRMecDdF7hqGjFA==", + "dev": true, + "engines": { + "node": ">=10" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/jest-watcher": { + "version": "29.7.0", + "resolved": "https://registry.npmjs.org/jest-watcher/-/jest-watcher-29.7.0.tgz", + "integrity": "sha512-49Fg7WXkU3Vl2h6LbLtMQ/HyB6rXSIX7SqvBLQmssRBGN9I0PNvPmAmCWSOY6SOvrjhI/F7/bGAv9RtnsPA03g==", + "dev": true, + "dependencies": { + "@jest/test-result": "^29.7.0", + "@jest/types": "^29.6.3", + "@types/node": "*", + "ansi-escapes": "^4.2.1", + "chalk": "^4.0.0", + "emittery": "^0.13.1", + "jest-util": "^29.7.0", + "string-length": "^4.0.1" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + } + }, + "node_modules/jest-worker": { + "version": "29.7.0", + "resolved": "https://registry.npmjs.org/jest-worker/-/jest-worker-29.7.0.tgz", + "integrity": "sha512-eIz2msL/EzL9UFTFFx7jBTkeZfku0yUAyZZZmJ93H2TYEiroIx2PQjEXcwYtYl8zXCxb+PAmA2hLIt/6ZEkPHw==", + "dev": true, + "dependencies": { + "@types/node": "*", + "jest-util": "^29.7.0", + "merge-stream": "^2.0.0", + "supports-color": "^8.0.0" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + } + }, + "node_modules/jest-worker/node_modules/supports-color": { + "version": "8.1.1", + "resolved": "https://registry.npmjs.org/supports-color/-/supports-color-8.1.1.tgz", + "integrity": "sha512-MpUEN2OodtUzxvKQl72cUF7RQ5EiHsGvSsVG0ia9c5RbWGL2CI4C7EpPS8UTBIplnlzZiNuV56w+FuNxy3ty2Q==", + "dev": true, + "dependencies": { + "has-flag": "^4.0.0" + }, + "engines": { + "node": ">=10" + }, + "funding": { + "url": "https://github.com/chalk/supports-color?sponsor=1" + } + }, + "node_modules/js-tokens": { + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/js-tokens/-/js-tokens-4.0.0.tgz", + "integrity": "sha512-RdJUflcE3cUzKiMqQgsCu06FPu9UdIJO0beYbPhHN4k6apgJtifcoCtT9bcxOpYBtpD2kCM6Sbzg4CausW/PKQ==", + "dev": true + }, + "node_modules/js-yaml": { + "version": "3.14.1", + "resolved": "https://registry.npmjs.org/js-yaml/-/js-yaml-3.14.1.tgz", + "integrity": "sha512-okMH7OXXJ7YrN9Ok3/SXrnu4iX9yOk+25nqX4imS2npuvTYDmo/QEZoqwZkYaIDk3jVvBOTOIEgEhaLOynBS9g==", + "dev": true, + "dependencies": { + "argparse": "^1.0.7", + "esprima": "^4.0.0" + }, + "bin": { + "js-yaml": "bin/js-yaml.js" + } + }, + "node_modules/jsesc": { + "version": "3.1.0", + "resolved": "https://registry.npmjs.org/jsesc/-/jsesc-3.1.0.tgz", + "integrity": "sha512-/sM3dO2FOzXjKQhJuo0Q173wf2KOo8t4I8vHy6lF9poUp7bKT0/NHE8fPX23PwfhnykfqnC2xRxOnVw5XuGIaA==", + "dev": true, + "bin": { + "jsesc": "bin/jsesc" + }, + "engines": { + "node": ">=6" + } + }, + "node_modules/json-parse-even-better-errors": { + "version": "2.3.1", + "resolved": "https://registry.npmjs.org/json-parse-even-better-errors/-/json-parse-even-better-errors-2.3.1.tgz", + "integrity": "sha512-xyFwyhro/JEof6Ghe2iz2NcXoj2sloNsWr/XsERDK/oiPCfaNhl5ONfp+jQdAZRQQ0IJWNzH9zIZF7li91kh2w==", + "dev": true + }, + "node_modules/json5": { + "version": "2.2.3", + "resolved": "https://registry.npmjs.org/json5/-/json5-2.2.3.tgz", + "integrity": "sha512-XmOWe7eyHYH14cLdVPoyg+GOH3rYX++KpzrylJwSW98t3Nk+U8XOl8FWKOgwtzdb8lXGf6zYwDUzeHMWfxasyg==", + "dev": true, + "bin": { + "json5": "lib/cli.js" + }, + "engines": { + "node": ">=6" + } + }, + "node_modules/kleur": { + "version": "3.0.3", + "resolved": "https://registry.npmjs.org/kleur/-/kleur-3.0.3.tgz", + "integrity": "sha512-eTIzlVOSUR+JxdDFepEYcBMtZ9Qqdef+rnzWdRZuMbOywu5tO2w2N7rqjoANZ5k9vywhL6Br1VRjUIgTQx4E8w==", + "dev": true, + "engines": { + "node": ">=6" + } + }, + "node_modules/leven": { + "version": "3.1.0", + "resolved": "https://registry.npmjs.org/leven/-/leven-3.1.0.tgz", + "integrity": "sha512-qsda+H8jTaUaN/x5vzW2rzc+8Rw4TAQ/4KjB46IwK5VH+IlVeeeje/EoZRpiXvIqjFgK84QffqPztGI3VBLG1A==", + "dev": true, + "engines": { + "node": ">=6" + } + }, + "node_modules/lines-and-columns": { + "version": "1.2.4", + "resolved": "https://registry.npmjs.org/lines-and-columns/-/lines-and-columns-1.2.4.tgz", + "integrity": "sha512-7ylylesZQ/PV29jhEDl3Ufjo6ZX7gCqJr5F7PKrqc93v7fzSymt1BpwEU8nAUXs8qzzvqhbjhK5QZg6Mt/HkBg==", + "dev": true + }, + "node_modules/locate-path": { + "version": "5.0.0", + "resolved": "https://registry.npmjs.org/locate-path/-/locate-path-5.0.0.tgz", + "integrity": "sha512-t7hw9pI+WvuwNJXwk5zVHpyhIqzg2qTlklJOf0mVxGSbe3Fp2VieZcduNYjaLDoy6p9uGpQEGWG87WpMKlNq8g==", + "dev": true, + "dependencies": { + "p-locate": "^4.1.0" + }, + "engines": { + "node": ">=8" + } + }, + "node_modules/lodash.memoize": { + "version": "4.1.2", + "resolved": "https://registry.npmjs.org/lodash.memoize/-/lodash.memoize-4.1.2.tgz", + "integrity": "sha512-t7j+NzmgnQzTAYXcsHYLgimltOV1MXHtlOWf6GjL9Kj8GK5FInw5JotxvbOs+IvV1/Dzo04/fCGfLVs7aXb4Ag==", + "dev": true + }, + "node_modules/lru-cache": { + "version": "5.1.1", + "resolved": "https://registry.npmjs.org/lru-cache/-/lru-cache-5.1.1.tgz", + "integrity": "sha512-KpNARQA3Iwv+jTA0utUVVbrh+Jlrr1Fv0e56GGzAFOXN7dk/FviaDW8LHmK52DlcH4WP2n6gI8vN1aesBFgo9w==", + "dev": true, + "dependencies": { + "yallist": "^3.0.2" + } + }, + "node_modules/make-dir": { + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/make-dir/-/make-dir-4.0.0.tgz", + "integrity": "sha512-hXdUTZYIVOt1Ex//jAQi+wTZZpUpwBj/0QsOzqegb3rGMMeJiSEu5xLHnYfBrRV4RH2+OCSOO95Is/7x1WJ4bw==", + "dev": true, + "dependencies": { + "semver": "^7.5.3" + }, + "engines": { + "node": ">=10" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/make-dir/node_modules/semver": { + "version": "7.7.1", + "resolved": "https://registry.npmjs.org/semver/-/semver-7.7.1.tgz", + "integrity": "sha512-hlq8tAfn0m/61p4BVRcPzIGr6LKiMwo4VM6dGi6pt4qcRkmNzTcWq6eCEjEh+qXjkMDvPlOFFSGwQjoEa6gyMA==", + "dev": true, + "bin": { + "semver": "bin/semver.js" + }, + "engines": { + "node": ">=10" + } + }, + "node_modules/make-error": { + "version": "1.3.6", + "resolved": "https://registry.npmjs.org/make-error/-/make-error-1.3.6.tgz", + "integrity": "sha512-s8UhlNe7vPKomQhC1qFelMokr/Sc3AgNbso3n74mVPA5LTZwkB9NlXf4XPamLxJE8h0gh73rM94xvwRT2CVInw==", + "dev": true + }, + "node_modules/makeerror": { + "version": "1.0.12", + "resolved": "https://registry.npmjs.org/makeerror/-/makeerror-1.0.12.tgz", + "integrity": "sha512-JmqCvUhmt43madlpFzG4BQzG2Z3m6tvQDNKdClZnO3VbIudJYmxsT0FNJMeiB2+JTSlTQTSbU8QdesVmwJcmLg==", + "dev": true, + "dependencies": { + "tmpl": "1.0.5" + } + }, + "node_modules/merge-stream": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/merge-stream/-/merge-stream-2.0.0.tgz", + "integrity": "sha512-abv/qOcuPfk3URPfDzmZU1LKmuw8kT+0nIHvKrKgFrwifol/doWcdA4ZqsWQ8ENrFKkd67Mfpo/LovbIUsbt3w==", + "dev": true + }, + "node_modules/micromatch": { + "version": "4.0.8", + "resolved": "https://registry.npmjs.org/micromatch/-/micromatch-4.0.8.tgz", + "integrity": "sha512-PXwfBhYu0hBCPw8Dn0E+WDYb7af3dSLVWKi3HGv84IdF4TyFoC0ysxFd0Goxw7nSv4T/PzEJQxsYsEiFCKo2BA==", + "dev": true, + "dependencies": { + "braces": "^3.0.3", + "picomatch": "^2.3.1" + }, + "engines": { + "node": ">=8.6" + } + }, + "node_modules/mimic-fn": { + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/mimic-fn/-/mimic-fn-2.1.0.tgz", + "integrity": "sha512-OqbOk5oEQeAZ8WXWydlu9HJjz9WVdEIvamMCcXmuqUYjTknH/sqsWvhQ3vgwKFRR1HpjvNBKQ37nbJgYzGqGcg==", + "dev": true, + "engines": { + "node": ">=6" + } + }, + "node_modules/minimatch": { + "version": "3.1.2", + "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-3.1.2.tgz", + "integrity": "sha512-J7p63hRiAjw1NDEww1W7i37+ByIrOWO5XQQAzZ3VOcL0PNybwpfmV/N05zFAzwQ9USyEcX6t3UO+K5aqBQOIHw==", + "dev": true, + "dependencies": { + "brace-expansion": "^1.1.7" + }, + "engines": { + "node": "*" + } + }, + "node_modules/ms": { + "version": "2.1.3", + "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz", + "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", + "dev": true + }, + "node_modules/natural-compare": { + "version": "1.4.0", + "resolved": "https://registry.npmjs.org/natural-compare/-/natural-compare-1.4.0.tgz", + "integrity": "sha512-OWND8ei3VtNC9h7V60qff3SVobHr996CTwgxubgyQYEpg290h9J0buyECNNJexkFm5sOajh5G116RYA1c8ZMSw==", + "dev": true + }, + "node_modules/node-int64": { + "version": "0.4.0", + "resolved": "https://registry.npmjs.org/node-int64/-/node-int64-0.4.0.tgz", + "integrity": "sha512-O5lz91xSOeoXP6DulyHfllpq+Eg00MWitZIbtPfoSEvqIHdl5gfcY6hYzDWnj0qD5tz52PI08u9qUvSVeUBeHw==", + "dev": true + }, + "node_modules/node-releases": { + "version": "2.0.19", + "resolved": "https://registry.npmjs.org/node-releases/-/node-releases-2.0.19.tgz", + "integrity": "sha512-xxOWJsBKtzAq7DY0J+DTzuz58K8e7sJbdgwkbMWQe8UYB6ekmsQ45q0M/tJDsGaZmbC+l7n57UV8Hl5tHxO9uw==", + "dev": true + }, + "node_modules/normalize-path": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/normalize-path/-/normalize-path-3.0.0.tgz", + "integrity": "sha512-6eZs5Ls3WtCisHWp9S2GUy8dqkpGi4BVSz3GaqiE6ezub0512ESztXUwUB6C6IKbQkY2Pnb/mD4WYojCRwcwLA==", + "dev": true, + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/npm-run-path": { + "version": "4.0.1", + "resolved": "https://registry.npmjs.org/npm-run-path/-/npm-run-path-4.0.1.tgz", + "integrity": "sha512-S48WzZW777zhNIrn7gxOlISNAqi9ZC/uQFnRdbeIHhZhCA6UqpkOT8T1G7BvfdgP4Er8gF4sUbaS0i7QvIfCWw==", + "dev": true, + "dependencies": { + "path-key": "^3.0.0" + }, + "engines": { + "node": ">=8" + } + }, + "node_modules/once": { + "version": "1.4.0", + "resolved": "https://registry.npmjs.org/once/-/once-1.4.0.tgz", + "integrity": "sha512-lNaJgI+2Q5URQBkccEKHTQOPaXdUxnZZElQTZY0MFUAuaEqe1E+Nyvgdz/aIyNi6Z9MzO5dv1H8n58/GELp3+w==", + "dev": true, + "dependencies": { + "wrappy": "1" + } + }, + "node_modules/onetime": { + "version": "5.1.2", + "resolved": "https://registry.npmjs.org/onetime/-/onetime-5.1.2.tgz", + "integrity": "sha512-kbpaSSGJTWdAY5KPVeMOKXSrPtr8C8C7wodJbcsd51jRnmD+GZu8Y0VoU6Dm5Z4vWr0Ig/1NKuWRKf7j5aaYSg==", + "dev": true, + "dependencies": { + "mimic-fn": "^2.1.0" + }, + "engines": { + "node": ">=6" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/p-limit": { + "version": "3.1.0", + "resolved": "https://registry.npmjs.org/p-limit/-/p-limit-3.1.0.tgz", + "integrity": "sha512-TYOanM3wGwNGsZN2cVTYPArw454xnXj5qmWF1bEoAc4+cU/ol7GVh7odevjp1FNHduHc3KZMcFduxU5Xc6uJRQ==", + "dev": true, + "dependencies": { + "yocto-queue": "^0.1.0" + }, + "engines": { + "node": ">=10" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/p-locate": { + "version": "4.1.0", + "resolved": "https://registry.npmjs.org/p-locate/-/p-locate-4.1.0.tgz", + "integrity": "sha512-R79ZZ/0wAxKGu3oYMlz8jy/kbhsNrS7SKZ7PxEHBgJ5+F2mtFW2fK2cOtBh1cHYkQsbzFV7I+EoRKe6Yt0oK7A==", + "dev": true, + "dependencies": { + "p-limit": "^2.2.0" + }, + "engines": { + "node": ">=8" + } + }, + "node_modules/p-locate/node_modules/p-limit": { + "version": "2.3.0", + "resolved": "https://registry.npmjs.org/p-limit/-/p-limit-2.3.0.tgz", + "integrity": "sha512-//88mFWSJx8lxCzwdAABTJL2MyWB12+eIY7MDL2SqLmAkeKU9qxRvWuSyTjm3FUmpBEMuFfckAIqEaVGUDxb6w==", + "dev": true, + "dependencies": { + "p-try": "^2.0.0" + }, + "engines": { + "node": ">=6" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/p-try": { + "version": "2.2.0", + "resolved": "https://registry.npmjs.org/p-try/-/p-try-2.2.0.tgz", + "integrity": "sha512-R4nPAVTAU0B9D35/Gk3uJf/7XYbQcyohSKdvAxIRSNghFl4e71hVoGnBNQz9cWaXxO2I10KTC+3jMdvvoKw6dQ==", + "dev": true, + "engines": { + "node": ">=6" + } + }, + "node_modules/parse-json": { + "version": "5.2.0", + "resolved": "https://registry.npmjs.org/parse-json/-/parse-json-5.2.0.tgz", + "integrity": "sha512-ayCKvm/phCGxOkYRSCM82iDwct8/EonSEgCSxWxD7ve6jHggsFl4fZVQBPRNgQoKiuV/odhFrGzQXZwbifC8Rg==", + "dev": true, + "dependencies": { + "@babel/code-frame": "^7.0.0", + "error-ex": "^1.3.1", + "json-parse-even-better-errors": "^2.3.0", + "lines-and-columns": "^1.1.6" + }, + "engines": { + "node": ">=8" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/path-exists": { + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/path-exists/-/path-exists-4.0.0.tgz", + "integrity": "sha512-ak9Qy5Q7jYb2Wwcey5Fpvg2KoAc/ZIhLSLOSBmRmygPsGwkVVt0fZa0qrtMz+m6tJTAHfZQ8FnmB4MG4LWy7/w==", + "dev": true, + "engines": { + "node": ">=8" + } + }, + "node_modules/path-is-absolute": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/path-is-absolute/-/path-is-absolute-1.0.1.tgz", + "integrity": "sha512-AVbw3UJ2e9bq64vSaS9Am0fje1Pa8pbGqTTsmXfaIiMpnr5DlDhfJOuLj9Sf95ZPVDAUerDfEk88MPmPe7UCQg==", + "dev": true, + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/path-key": { + "version": "3.1.1", + "resolved": "https://registry.npmjs.org/path-key/-/path-key-3.1.1.tgz", + "integrity": "sha512-ojmeN0qd+y0jszEtoY48r0Peq5dwMEkIlCOu6Q5f41lfkswXuKtYrhgoTpLnyIcHm24Uhqx+5Tqm2InSwLhE6Q==", + "dev": true, + "engines": { + "node": ">=8" + } + }, + "node_modules/path-parse": { + "version": "1.0.7", + "resolved": "https://registry.npmjs.org/path-parse/-/path-parse-1.0.7.tgz", + "integrity": "sha512-LDJzPVEEEPR+y48z93A0Ed0yXb8pAByGWo/k5YYdYgpY2/2EsOsksJrq7lOHxryrVOn1ejG6oAp8ahvOIQD8sw==", + "dev": true + }, + "node_modules/picocolors": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/picocolors/-/picocolors-1.1.1.tgz", + "integrity": "sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA==", + "dev": true + }, + "node_modules/picomatch": { + "version": "2.3.1", + "resolved": "https://registry.npmjs.org/picomatch/-/picomatch-2.3.1.tgz", + "integrity": "sha512-JU3teHTNjmE2VCGFzuY8EXzCDVwEqB2a8fsIvwaStHhAWJEeVd1o1QD80CU6+ZdEXXSLbSsuLwJjkCBWqRQUVA==", + "dev": true, + "engines": { + "node": ">=8.6" + }, + "funding": { + "url": "https://github.com/sponsors/jonschlinkert" + } + }, + "node_modules/pirates": { + "version": "4.0.7", + "resolved": "https://registry.npmjs.org/pirates/-/pirates-4.0.7.tgz", + "integrity": "sha512-TfySrs/5nm8fQJDcBDuUng3VOUKsd7S+zqvbOTiGXHfxX4wK31ard+hoNuvkicM/2YFzlpDgABOevKSsB4G/FA==", + "dev": true, + "engines": { + "node": ">= 6" + } + }, + "node_modules/pkg-dir": { + "version": "4.2.0", + "resolved": "https://registry.npmjs.org/pkg-dir/-/pkg-dir-4.2.0.tgz", + "integrity": "sha512-HRDzbaKjC+AOWVXxAU/x54COGeIv9eb+6CkDSQoNTt4XyWoIJvuPsXizxu/Fr23EiekbtZwmh1IcIG/l/a10GQ==", + "dev": true, + "dependencies": { + "find-up": "^4.0.0" + }, + "engines": { + "node": ">=8" + } + }, + "node_modules/pretty-format": { + "version": "29.7.0", + "resolved": "https://registry.npmjs.org/pretty-format/-/pretty-format-29.7.0.tgz", + "integrity": "sha512-Pdlw/oPxN+aXdmM9R00JVC9WVFoCLTKJvDVLgmJ+qAffBMxsV85l/Lu7sNx4zSzPyoL2euImuEwHhOXdEgNFZQ==", + "dev": true, + "dependencies": { + "@jest/schemas": "^29.6.3", + "ansi-styles": "^5.0.0", + "react-is": "^18.0.0" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || >=18.0.0" + } + }, + "node_modules/pretty-format/node_modules/ansi-styles": { + "version": "5.2.0", + "resolved": "https://registry.npmjs.org/ansi-styles/-/ansi-styles-5.2.0.tgz", + "integrity": "sha512-Cxwpt2SfTzTtXcfOlzGEee8O+c+MmUgGrNiBcXnuWxuFJHe6a5Hz7qwhwe5OgaSYI0IJvkLqWX1ASG+cJOkEiA==", + "dev": true, + "engines": { + "node": ">=10" + }, + "funding": { + "url": "https://github.com/chalk/ansi-styles?sponsor=1" + } + }, + "node_modules/prompts": { + "version": "2.4.2", + "resolved": "https://registry.npmjs.org/prompts/-/prompts-2.4.2.tgz", + "integrity": "sha512-NxNv/kLguCA7p3jE8oL2aEBsrJWgAakBpgmgK6lpPWV+WuOmY6r2/zbAVnP+T8bQlA0nzHXSJSJW0Hq7ylaD2Q==", + "dev": true, + "dependencies": { + "kleur": "^3.0.3", + "sisteransi": "^1.0.5" + }, + "engines": { + "node": ">= 6" + } + }, + "node_modules/pure-rand": { + "version": "6.1.0", + "resolved": "https://registry.npmjs.org/pure-rand/-/pure-rand-6.1.0.tgz", + "integrity": "sha512-bVWawvoZoBYpp6yIoQtQXHZjmz35RSVHnUOTefl8Vcjr8snTPY1wnpSPMWekcFwbxI6gtmT7rSYPFvz71ldiOA==", + "dev": true, + "funding": [ + { + "type": "individual", + "url": "https://github.com/sponsors/dubzzz" + }, + { + "type": "opencollective", + "url": "https://opencollective.com/fast-check" + } + ] + }, + "node_modules/react-is": { + "version": "18.3.1", + "resolved": "https://registry.npmjs.org/react-is/-/react-is-18.3.1.tgz", + "integrity": "sha512-/LLMVyas0ljjAtoYiPqYiL8VWXzUUdThrmU5+n20DZv+a+ClRoevUzw5JxU+Ieh5/c87ytoTBV9G1FiKfNJdmg==", + "dev": true + }, + "node_modules/require-directory": { + "version": "2.1.1", + "resolved": "https://registry.npmjs.org/require-directory/-/require-directory-2.1.1.tgz", + "integrity": "sha512-fGxEI7+wsG9xrvdjsrlmL22OMTTiHRwAMroiEeMgq8gzoLC/PQr7RsRDSTLUg/bZAZtF+TVIkHc6/4RIKrui+Q==", + "dev": true, + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/resolve": { + "version": "1.22.10", + "resolved": "https://registry.npmjs.org/resolve/-/resolve-1.22.10.tgz", + "integrity": "sha512-NPRy+/ncIMeDlTAsuqwKIiferiawhefFJtkNSW0qZJEqMEb+qBt/77B/jGeeek+F0uOeN05CDa6HXbbIgtVX4w==", + "dev": true, + "dependencies": { + "is-core-module": "^2.16.0", + "path-parse": "^1.0.7", + "supports-preserve-symlinks-flag": "^1.0.0" + }, + "bin": { + "resolve": "bin/resolve" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/resolve-cwd": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/resolve-cwd/-/resolve-cwd-3.0.0.tgz", + "integrity": "sha512-OrZaX2Mb+rJCpH/6CpSqt9xFVpN++x01XnN2ie9g6P5/3xelLAkXWVADpdz1IHD/KFfEXyE6V0U01OQ3UO2rEg==", + "dev": true, + "dependencies": { + "resolve-from": "^5.0.0" + }, + "engines": { + "node": ">=8" + } + }, + "node_modules/resolve-from": { + "version": "5.0.0", + "resolved": "https://registry.npmjs.org/resolve-from/-/resolve-from-5.0.0.tgz", + "integrity": "sha512-qYg9KP24dD5qka9J47d0aVky0N+b4fTU89LN9iDnjB5waksiC49rvMB0PrUJQGoTmH50XPiqOvAjDfaijGxYZw==", + "dev": true, + "engines": { + "node": ">=8" + } + }, + "node_modules/resolve.exports": { + "version": "2.0.3", + "resolved": "https://registry.npmjs.org/resolve.exports/-/resolve.exports-2.0.3.tgz", + "integrity": "sha512-OcXjMsGdhL4XnbShKpAcSqPMzQoYkYyhbEaeSko47MjRP9NfEQMhZkXL1DoFlt9LWQn4YttrdnV6X2OiyzBi+A==", + "dev": true, + "engines": { + "node": ">=10" + } + }, + "node_modules/semver": { + "version": "6.3.1", + "resolved": "https://registry.npmjs.org/semver/-/semver-6.3.1.tgz", + "integrity": "sha512-BR7VvDCVHO+q2xBEWskxS6DJE1qRnb7DxzUrogb71CWoSficBxYsiAGd+Kl0mmq/MprG9yArRkyrQxTO6XjMzA==", + "dev": true, + "bin": { + "semver": "bin/semver.js" + } + }, + "node_modules/shebang-command": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/shebang-command/-/shebang-command-2.0.0.tgz", + "integrity": "sha512-kHxr2zZpYtdmrN1qDjrrX/Z1rR1kG8Dx+gkpK1G4eXmvXswmcE1hTWBWYUzlraYw1/yZp6YuDY77YtvbN0dmDA==", + "dev": true, + "dependencies": { + "shebang-regex": "^3.0.0" + }, + "engines": { + "node": ">=8" + } + }, + "node_modules/shebang-regex": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/shebang-regex/-/shebang-regex-3.0.0.tgz", + "integrity": "sha512-7++dFhtcx3353uBaq8DDR4NuxBetBzC7ZQOhmTQInHEd6bSrXdiEyzCvG07Z44UYdLShWUyXt5M/yhz8ekcb1A==", + "dev": true, + "engines": { + "node": ">=8" + } + }, + "node_modules/signal-exit": { + "version": "3.0.7", + "resolved": "https://registry.npmjs.org/signal-exit/-/signal-exit-3.0.7.tgz", + "integrity": "sha512-wnD2ZE+l+SPC/uoS0vXeE9L1+0wuaMqKlfz9AMUo38JsyLSBWSFcHR1Rri62LZc12vLr1gb3jl7iwQhgwpAbGQ==", + "dev": true + }, + "node_modules/sisteransi": { + "version": "1.0.5", + "resolved": "https://registry.npmjs.org/sisteransi/-/sisteransi-1.0.5.tgz", + "integrity": "sha512-bLGGlR1QxBcynn2d5YmDX4MGjlZvy2MRBDRNHLJ8VI6l6+9FUiyTFNJ0IveOSP0bcXgVDPRcfGqA0pjaqUpfVg==", + "dev": true + }, + "node_modules/slash": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/slash/-/slash-3.0.0.tgz", + "integrity": "sha512-g9Q1haeby36OSStwb4ntCGGGaKsaVSjQ68fBxoQcutl5fS1vuY18H3wSt3jFyFtrkx+Kz0V1G85A4MyAdDMi2Q==", + "dev": true, + "engines": { + "node": ">=8" + } + }, + "node_modules/source-map": { + "version": "0.6.1", + "resolved": "https://registry.npmjs.org/source-map/-/source-map-0.6.1.tgz", + "integrity": "sha512-UjgapumWlbMhkBgzT7Ykc5YXUT46F0iKu8SGXq0bcwP5dz/h0Plj6enJqjz1Zbq2l5WaqYnrVbwWOWMyF3F47g==", + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/source-map-support": { + "version": "0.5.21", + "resolved": "https://registry.npmjs.org/source-map-support/-/source-map-support-0.5.21.tgz", + "integrity": "sha512-uBHU3L3czsIyYXKX88fdrGovxdSCoTGDRZ6SYXtSRxLZUzHg5P/66Ht6uoUlHu9EZod+inXhKo3qQgwXUT/y1w==", + "dependencies": { + "buffer-from": "^1.0.0", + "source-map": "^0.6.0" + } + }, + "node_modules/sprintf-js": { + "version": "1.0.3", + "resolved": "https://registry.npmjs.org/sprintf-js/-/sprintf-js-1.0.3.tgz", + "integrity": "sha512-D9cPgkvLlV3t3IzL0D0YLvGA9Ahk4PcvVwUbN0dSGr1aP0Nrt4AEnTUbuGvquEC0mA64Gqt1fzirlRs5ibXx8g==", + "dev": true + }, + "node_modules/stack-utils": { + "version": "2.0.6", + "resolved": "https://registry.npmjs.org/stack-utils/-/stack-utils-2.0.6.tgz", + "integrity": "sha512-XlkWvfIm6RmsWtNJx+uqtKLS8eqFbxUg0ZzLXqY0caEy9l7hruX8IpiDnjsLavoBgqCCR71TqWO8MaXYheJ3RQ==", + "dev": true, + "dependencies": { + "escape-string-regexp": "^2.0.0" + }, + "engines": { + "node": ">=10" + } + }, + "node_modules/string-length": { + "version": "4.0.2", + "resolved": "https://registry.npmjs.org/string-length/-/string-length-4.0.2.tgz", + "integrity": "sha512-+l6rNN5fYHNhZZy41RXsYptCjA2Igmq4EG7kZAYFQI1E1VTXarr6ZPXBg6eq7Y6eK4FEhY6AJlyuFIb/v/S0VQ==", + "dev": true, + "dependencies": { + "char-regex": "^1.0.2", + "strip-ansi": "^6.0.0" + }, + "engines": { + "node": ">=10" + } + }, + "node_modules/string-width": { + "version": "4.2.3", + "resolved": "https://registry.npmjs.org/string-width/-/string-width-4.2.3.tgz", + "integrity": "sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g==", + "dev": true, + "dependencies": { + "emoji-regex": "^8.0.0", + "is-fullwidth-code-point": "^3.0.0", + "strip-ansi": "^6.0.1" + }, + "engines": { + "node": ">=8" + } + }, + "node_modules/strip-ansi": { + "version": "6.0.1", + "resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-6.0.1.tgz", + "integrity": "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A==", + "dev": true, + "dependencies": { + "ansi-regex": "^5.0.1" + }, + "engines": { + "node": ">=8" + } + }, + "node_modules/strip-bom": { + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/strip-bom/-/strip-bom-4.0.0.tgz", + "integrity": "sha512-3xurFv5tEgii33Zi8Jtp55wEIILR9eh34FAW00PZf+JnSsTmV/ioewSgQl97JHvgjoRGwPShsWm+IdrxB35d0w==", + "dev": true, + "engines": { + "node": ">=8" + } + }, + "node_modules/strip-final-newline": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/strip-final-newline/-/strip-final-newline-2.0.0.tgz", + "integrity": "sha512-BrpvfNAE3dcvq7ll3xVumzjKjZQ5tI1sEUIKr3Uoks0XUl45St3FlatVqef9prk4jRDzhW6WZg+3bk93y6pLjA==", + "dev": true, + "engines": { + "node": ">=6" + } + }, + "node_modules/strip-json-comments": { + "version": "3.1.1", + "resolved": "https://registry.npmjs.org/strip-json-comments/-/strip-json-comments-3.1.1.tgz", + "integrity": "sha512-6fPc+R4ihwqP6N/aIv2f1gMH8lOVtWQHoqC4yK6oSDVVocumAsfCqjkXnqiYMhmMwS/mEHLp7Vehlt3ql6lEig==", + "dev": true, + "engines": { + "node": ">=8" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/supports-color": { + "version": "7.2.0", + "resolved": "https://registry.npmjs.org/supports-color/-/supports-color-7.2.0.tgz", + "integrity": "sha512-qpCAvRl9stuOHveKsn7HncJRvv501qIacKzQlO/+Lwxc9+0q2wLyv4Dfvt80/DPn2pqOBsJdDiogXGR9+OvwRw==", + "dev": true, + "dependencies": { + "has-flag": "^4.0.0" + }, + "engines": { + "node": ">=8" + } + }, + "node_modules/supports-preserve-symlinks-flag": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/supports-preserve-symlinks-flag/-/supports-preserve-symlinks-flag-1.0.0.tgz", + "integrity": "sha512-ot0WnXS9fgdkgIcePe6RHNk1WA8+muPa6cSjeR3V8K27q9BB1rTE3R1p7Hv0z1ZyAc8s6Vvv8DIyWf681MAt0w==", + "dev": true, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/test-exclude": { + "version": "6.0.0", + "resolved": "https://registry.npmjs.org/test-exclude/-/test-exclude-6.0.0.tgz", + "integrity": "sha512-cAGWPIyOHU6zlmg88jwm7VRyXnMN7iV68OGAbYDk/Mh/xC/pzVPlQtY6ngoIH/5/tciuhGfvESU8GrHrcxD56w==", + "dev": true, + "dependencies": { + "@istanbuljs/schema": "^0.1.2", + "glob": "^7.1.4", + "minimatch": "^3.0.4" + }, + "engines": { + "node": ">=8" + } + }, + "node_modules/tmpl": { + "version": "1.0.5", + "resolved": "https://registry.npmjs.org/tmpl/-/tmpl-1.0.5.tgz", + "integrity": "sha512-3f0uOEAQwIqGuWW2MVzYg8fV/QNnc/IpuJNG837rLuczAaLVHslWHZQj4IGiEl5Hs3kkbhwL9Ab7Hrsmuj+Smw==", + "dev": true + }, + "node_modules/to-regex-range": { + "version": "5.0.1", + "resolved": "https://registry.npmjs.org/to-regex-range/-/to-regex-range-5.0.1.tgz", + "integrity": "sha512-65P7iz6X5yEr1cwcgvQxbbIw7Uk3gOy5dIdtZ4rDveLqhrdJP+Li/Hx6tyK0NEb+2GCyneCMJiGqrADCSNk8sQ==", + "dev": true, + "dependencies": { + "is-number": "^7.0.0" + }, + "engines": { + "node": ">=8.0" + } + }, + "node_modules/ts-jest": { + "version": "29.3.2", + "resolved": "https://registry.npmjs.org/ts-jest/-/ts-jest-29.3.2.tgz", + "integrity": "sha512-bJJkrWc6PjFVz5g2DGCNUo8z7oFEYaz1xP1NpeDU7KNLMWPpEyV8Chbpkn8xjzgRDpQhnGMyvyldoL7h8JXyug==", + "dev": true, + "dependencies": { + "bs-logger": "^0.2.6", + "ejs": "^3.1.10", + "fast-json-stable-stringify": "^2.1.0", + "jest-util": "^29.0.0", + "json5": "^2.2.3", + "lodash.memoize": "^4.1.2", + "make-error": "^1.3.6", + "semver": "^7.7.1", + "type-fest": "^4.39.1", + "yargs-parser": "^21.1.1" + }, + "bin": { + "ts-jest": "cli.js" + }, + "engines": { + "node": "^14.15.0 || ^16.10.0 || ^18.0.0 || >=20.0.0" + }, + "peerDependencies": { + "@babel/core": ">=7.0.0-beta.0 <8", + "@jest/transform": "^29.0.0", + "@jest/types": "^29.0.0", + "babel-jest": "^29.0.0", + "jest": "^29.0.0", + "typescript": ">=4.3 <6" + }, + "peerDependenciesMeta": { + "@babel/core": { + "optional": true + }, + "@jest/transform": { + "optional": true + }, + "@jest/types": { + "optional": true + }, + "babel-jest": { + "optional": true + }, + "esbuild": { + "optional": true + } + } + }, + "node_modules/ts-jest/node_modules/semver": { + "version": "7.7.1", + "resolved": "https://registry.npmjs.org/semver/-/semver-7.7.1.tgz", + "integrity": "sha512-hlq8tAfn0m/61p4BVRcPzIGr6LKiMwo4VM6dGi6pt4qcRkmNzTcWq6eCEjEh+qXjkMDvPlOFFSGwQjoEa6gyMA==", + "dev": true, + "bin": { + "semver": "bin/semver.js" + }, + "engines": { + "node": ">=10" + } + }, + "node_modules/ts-jest/node_modules/type-fest": { + "version": "4.40.1", + "resolved": "https://registry.npmjs.org/type-fest/-/type-fest-4.40.1.tgz", + "integrity": "sha512-9YvLNnORDpI+vghLU/Nf+zSv0kL47KbVJ1o3sKgoTefl6i+zebxbiDQWoe/oWWqPhIgQdRZRT1KA9sCPL810SA==", + "dev": true, + "engines": { + "node": ">=16" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/ts-node": { + "version": "10.9.2", + "resolved": "https://registry.npmjs.org/ts-node/-/ts-node-10.9.2.tgz", + "integrity": "sha512-f0FFpIdcHgn8zcPSbf1dRevwt047YMnaiJM3u2w2RewrB+fob/zePZcrOyQoLMMO7aBIddLcQIEK5dYjkLnGrQ==", + "dev": true, + "dependencies": { + "@cspotcode/source-map-support": "^0.8.0", + "@tsconfig/node10": "^1.0.7", + "@tsconfig/node12": "^1.0.7", + "@tsconfig/node14": "^1.0.0", + "@tsconfig/node16": "^1.0.2", + "acorn": "^8.4.1", + "acorn-walk": "^8.1.1", + "arg": "^4.1.0", + "create-require": "^1.1.0", + "diff": "^4.0.1", + "make-error": "^1.1.1", + "v8-compile-cache-lib": "^3.0.1", + "yn": "3.1.1" + }, + "bin": { + "ts-node": "dist/bin.js", + "ts-node-cwd": "dist/bin-cwd.js", + "ts-node-esm": "dist/bin-esm.js", + "ts-node-script": "dist/bin-script.js", + "ts-node-transpile-only": "dist/bin-transpile.js", + "ts-script": "dist/bin-script-deprecated.js" + }, + "peerDependencies": { + "@swc/core": ">=1.2.50", + "@swc/wasm": ">=1.2.50", + "@types/node": "*", + "typescript": ">=2.7" + }, + "peerDependenciesMeta": { + "@swc/core": { + "optional": true + }, + "@swc/wasm": { + "optional": true + } + } + }, + "node_modules/type-detect": { + "version": "4.0.8", + "resolved": "https://registry.npmjs.org/type-detect/-/type-detect-4.0.8.tgz", + "integrity": "sha512-0fr/mIH1dlO+x7TlcMy+bIDqKPsw/70tVyeHW787goQjhmqaZe10uwLujubK9q9Lg6Fiho1KUKDYz0Z7k7g5/g==", + "dev": true, + "engines": { + "node": ">=4" + } + }, + "node_modules/type-fest": { + "version": "0.21.3", + "resolved": "https://registry.npmjs.org/type-fest/-/type-fest-0.21.3.tgz", + "integrity": "sha512-t0rzBq87m3fVcduHDUFhKmyyX+9eo6WQjZvf51Ea/M0Q7+T374Jp1aUiyUl0GKxp8M/OETVHSDvmkyPgvX+X2w==", + "dev": true, + "engines": { + "node": ">=10" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/typescript": { + "version": "5.0.4", + "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.0.4.tgz", + "integrity": "sha512-cW9T5W9xY37cc+jfEnaUvX91foxtHkza3Nw3wkoF4sSlKn0MONdkdEndig/qPBWXNkmplh3NzayQzCiHM4/hqw==", + "dev": true, + "bin": { + "tsc": "bin/tsc", + "tsserver": "bin/tsserver" + }, + "engines": { + "node": ">=12.20" + } + }, + "node_modules/update-browserslist-db": { + "version": "1.1.3", + "resolved": "https://registry.npmjs.org/update-browserslist-db/-/update-browserslist-db-1.1.3.tgz", + "integrity": "sha512-UxhIZQ+QInVdunkDAaiazvvT/+fXL5Osr0JZlJulepYu6Jd7qJtDZjlur0emRlT71EN3ScPoE7gvsuIKKNavKw==", + "dev": true, + "funding": [ + { + "type": "opencollective", + "url": "https://opencollective.com/browserslist" + }, + { + "type": "tidelift", + "url": "https://tidelift.com/funding/github/npm/browserslist" + }, + { + "type": "github", + "url": "https://github.com/sponsors/ai" + } + ], + "dependencies": { + "escalade": "^3.2.0", + "picocolors": "^1.1.1" + }, + "bin": { + "update-browserslist-db": "cli.js" + }, + "peerDependencies": { + "browserslist": ">= 4.21.0" + } + }, + "node_modules/v8-compile-cache-lib": { + "version": "3.0.1", + "resolved": "https://registry.npmjs.org/v8-compile-cache-lib/-/v8-compile-cache-lib-3.0.1.tgz", + "integrity": "sha512-wa7YjyUGfNZngI/vtK0UHAN+lgDCxBPCylVXGp0zu59Fz5aiGtNXaq3DhIov063MorB+VfufLh3JlF2KdTK3xg==", + "dev": true + }, + "node_modules/v8-to-istanbul": { + "version": "9.3.0", + "resolved": "https://registry.npmjs.org/v8-to-istanbul/-/v8-to-istanbul-9.3.0.tgz", + "integrity": "sha512-kiGUalWN+rgBJ/1OHZsBtU4rXZOfj/7rKQxULKlIzwzQSvMJUUNgPwJEEh7gU6xEVxC0ahoOBvN2YI8GH6FNgA==", + "dev": true, + "dependencies": { + "@jridgewell/trace-mapping": "^0.3.12", + "@types/istanbul-lib-coverage": "^2.0.1", + "convert-source-map": "^2.0.0" + }, + "engines": { + "node": ">=10.12.0" + } + }, + "node_modules/walker": { + "version": "1.0.8", + "resolved": "https://registry.npmjs.org/walker/-/walker-1.0.8.tgz", + "integrity": "sha512-ts/8E8l5b7kY0vlWLewOkDXMmPdLcVV4GmOQLyxuSswIJsweeFZtAsMF7k1Nszz+TYBQrlYRmzOnr398y1JemQ==", + "dev": true, + "dependencies": { + "makeerror": "1.0.12" + } + }, + "node_modules/which": { + "version": "2.0.2", + "resolved": "https://registry.npmjs.org/which/-/which-2.0.2.tgz", + "integrity": "sha512-BLI3Tl1TW3Pvl70l3yq3Y64i+awpwXqsGBYWkkqMtnbXgrMD+yj7rhW0kuEDxzJaYXGjEW5ogapKNMEKNMjibA==", + "dev": true, + "dependencies": { + "isexe": "^2.0.0" + }, + "bin": { + "node-which": "bin/node-which" + }, + "engines": { + "node": ">= 8" + } + }, + "node_modules/wrap-ansi": { + "version": "7.0.0", + "resolved": "https://registry.npmjs.org/wrap-ansi/-/wrap-ansi-7.0.0.tgz", + "integrity": "sha512-YVGIj2kamLSTxw6NsZjoBxfSwsn0ycdesmc4p+Q21c5zPuZ1pl+NfxVdxPtdHvmNVOQ6XSYG4AUtyt/Fi7D16Q==", + "dev": true, + "dependencies": { + "ansi-styles": "^4.0.0", + "string-width": "^4.1.0", + "strip-ansi": "^6.0.0" + }, + "engines": { + "node": ">=10" + }, + "funding": { + "url": "https://github.com/chalk/wrap-ansi?sponsor=1" + } + }, + "node_modules/wrappy": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz", + "integrity": "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==", + "dev": true + }, + "node_modules/write-file-atomic": { + "version": "4.0.2", + "resolved": "https://registry.npmjs.org/write-file-atomic/-/write-file-atomic-4.0.2.tgz", + "integrity": "sha512-7KxauUdBmSdWnmpaGFg+ppNjKF8uNLry8LyzjauQDOVONfFLNKrKvQOxZ/VuTIcS/gge/YNahf5RIIQWTSarlg==", + "dev": true, + "dependencies": { + "imurmurhash": "^0.1.4", + "signal-exit": "^3.0.7" + }, + "engines": { + "node": "^12.13.0 || ^14.15.0 || >=16.0.0" + } + }, + "node_modules/y18n": { + "version": "5.0.8", + "resolved": "https://registry.npmjs.org/y18n/-/y18n-5.0.8.tgz", + "integrity": "sha512-0pfFzegeDWJHJIAmTLRP2DwHjdF5s7jo9tuztdQxAhINCdvS+3nGINqPd00AphqJR/0LhANUS6/+7SCb98YOfA==", + "dev": true, + "engines": { + "node": ">=10" + } + }, + "node_modules/yallist": { + "version": "3.1.1", + "resolved": "https://registry.npmjs.org/yallist/-/yallist-3.1.1.tgz", + "integrity": "sha512-a4UGQaWPH59mOXUYnAG2ewncQS4i4F43Tv3JoAM+s2VDAmS9NsK8GpDMLrCHPksFT7h3K6TOoUNn2pb7RoXx4g==", + "dev": true + }, + "node_modules/yargs": { + "version": "17.7.2", + "resolved": "https://registry.npmjs.org/yargs/-/yargs-17.7.2.tgz", + "integrity": "sha512-7dSzzRQ++CKnNI/krKnYRV7JKKPUXMEh61soaHKg9mrWEhzFWhFnxPxGl+69cD1Ou63C13NUPCnmIcrvqCuM6w==", + "dev": true, + "dependencies": { + "cliui": "^8.0.1", + "escalade": "^3.1.1", + "get-caller-file": "^2.0.5", + "require-directory": "^2.1.1", + "string-width": "^4.2.3", + "y18n": "^5.0.5", + "yargs-parser": "^21.1.1" + }, + "engines": { + "node": ">=12" + } + }, + "node_modules/yargs-parser": { + "version": "21.1.1", + "resolved": "https://registry.npmjs.org/yargs-parser/-/yargs-parser-21.1.1.tgz", + "integrity": "sha512-tVpsJW7DdjecAiFpbIB1e3qxIQsE6NoPc5/eTdrbbIC4h0LVsWhnoa3g+m2HclBIujHzsxZ4VJVA+GUuc2/LBw==", + "dev": true, + "engines": { + "node": ">=12" + } + }, + "node_modules/yn": { + "version": "3.1.1", + "resolved": "https://registry.npmjs.org/yn/-/yn-3.1.1.tgz", + "integrity": "sha512-Ux4ygGWsu2c7isFWe8Yu1YluJmqVhxqK2cLXNQA5AcC3QfbGNpM7fu0Y8b/z16pXLnFxZYvWhd3fhBY9DLmC6Q==", + "dev": true, + "engines": { + "node": ">=6" + } + }, + "node_modules/yocto-queue": { + "version": "0.1.0", + "resolved": "https://registry.npmjs.org/yocto-queue/-/yocto-queue-0.1.0.tgz", + "integrity": "sha512-rVksvsnNCdJ/ohGc6xgPwyN8eheCxsiLM8mxuE/t/mOVqJewPuO1miLpTHQiRgTKCLexL4MeAFVagts7HmNZ2Q==", + "dev": true, + "engines": { + "node": ">=10" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + } + } +} diff --git a/infrastructure/AutoScaling/cdk/package.json b/infrastructure/AutoScaling/cdk/package.json new file mode 100644 index 00000000..7942d360 --- /dev/null +++ b/infrastructure/AutoScaling/cdk/package.json @@ -0,0 +1,27 @@ +{ + "name": "kda-autoscaling", + "version": "0.1.0", + "bin": { + "kda-autoscaling": "bin/kda-autoscaling.js" + }, + "scripts": { + "build": "tsc", + "watch": "tsc -w", + "test": "jest", + "cdk": "cdk" + }, + "devDependencies": { + "@types/jest": "^29.5.1", + "@types/node": "20.1.7", + "aws-cdk": "^2.1001.0", + "jest": "^29.5.0", + "ts-jest": "^29.1.0", + "ts-node": "^10.9.1", + "typescript": "~5.0.4" + }, + "dependencies": { + "aws-cdk-lib": "^2.192.0", + "constructs": "^10.4.2", + "source-map-support": "^0.5.21" + } +} diff --git a/infrastructure/AutoScaling/cdk/resources/scaling/scaling.py b/infrastructure/AutoScaling/cdk/resources/scaling/scaling.py new file mode 100644 index 00000000..a1fe7e8e --- /dev/null +++ b/infrastructure/AutoScaling/cdk/resources/scaling/scaling.py @@ -0,0 +1,129 @@ +import boto3 +import json +import os + +client_kda = boto3.client('kinesisanalyticsv2') +client_ssm = boto3.client('ssm') +client_cloudwatch = boto3.client('cloudwatch') +client_cloudformation = boto3.client('cloudformation') +client_aas = boto3.client('application-autoscaling') +client_iam = boto3.resource('iam') + +def update_parallelism(context, desiredCapacity, resourceName, appVersionId, currentParallelismPerKPU): + try: + # Compute the new parallelism based on the desired capacity and parallelismPerKPU + newParallelism = desiredCapacity * currentParallelismPerKPU + newParallelismPerKPU = currentParallelismPerKPU # Assume no change for now + + # Call KDA service to update parallelism + response = client_kda.update_application( + ApplicationName=resourceName, + CurrentApplicationVersionId=appVersionId, + ApplicationConfigurationUpdate={ + 'FlinkApplicationConfigurationUpdate': { + 'ParallelismConfigurationUpdate': { + 'ConfigurationTypeUpdate': 'CUSTOM', + 'ParallelismUpdate': int(newParallelism), + 'ParallelismPerKPUUpdate': int(newParallelismPerKPU), + 'AutoScalingEnabledUpdate': False + } + } + } + ) + + print("In update_parallelism; response: ") + print(response) + scalingStatus = "InProgress" + + except Exception as e: + print(e) + scalingStatus = "Failed" + + return scalingStatus + + +def response_function(status_code, response_body): + return_json = { + 'statusCode': status_code, + 'body': response_body, + 'headers': { + 'Content-Type': 'application/json', + }, + } + # log response + print(return_json) + return return_json + + +def handler(event, context): + print(event) + resourceName = os.environ['flinkApplicationName'] + scaleInFactor = int(os.environ['scaleInFactor']) + scaleOutFactor = int(os.environ['scaleOutFactor']) + scaleOperation = os.environ['scaleOperation'] + alarm_status = event['detail']['state']['value'] + alarm_name = event['detail']['alarmName'] + minKPU = int(os.environ['minKPU']) + maxKPU = int(os.environ['maxKPU']) + stop_scale = 0 + + # get details for the KDA app in question + appVersion = event["TaskResult"]["ApplicationDetail"]["ApplicationVersionId"] + applicationStatus = event["TaskResult"]["ApplicationDetail"]["ApplicationStatus"] + parallelism = event["TaskResult"]["ApplicationDetail"]["ApplicationConfigurationDescription"][ + "FlinkApplicationConfigurationDescription"]["ParallelismConfigurationDescription"]["Parallelism"] + parallelismPerKPU = event["TaskResult"]["ApplicationDetail"]["ApplicationConfigurationDescription"]["FlinkApplicationConfigurationDescription"]["ParallelismConfigurationDescription"]["ParallelismPerKPU"] + actualCapacity = (parallelism + parallelismPerKPU - 1) // parallelismPerKPU + print(f"Actual Capacity: {actualCapacity}") + + if applicationStatus == "UPDATING": + scalingStatus = "InProgress" + elif applicationStatus == "RUNNING": + scalingStatus = "Successful" + + # Scaling out scenario (ScaleOut) + if "ScaleOut" in alarm_name and alarm_status == 'ALARM' and applicationStatus == 'RUNNING': + if actualCapacity < maxKPU: + if scaleOperation == 'Multiply/Divide': + desiredCapacity = actualCapacity * scaleOutFactor + print(f"Scaling out, desired capacity: {desiredCapacity}") + else: + desiredCapacity = actualCapacity + scaleOutFactor + print(f"Scaling out, desired capacity: {desiredCapacity}") + if desiredCapacity < maxKPU: + update_parallelism(context, desiredCapacity, resourceName, appVersion, parallelismPerKPU) + else: + update_parallelism(context, maxKPU, resourceName, appVersion, parallelismPerKPU) + print("Application will be set to Max KPU") + stop_scale = 1 + else: + desiredCapacity = actualCapacity + print("Application is already equal or above to Max KPU") + stop_scale = 1 + + # Scaling in scenario (ScaleIn) + elif "ScaleIn" in alarm_name and alarm_status == 'ALARM' and applicationStatus == 'RUNNING': + if actualCapacity > minKPU: + if scaleOperation == 'Multiply/Divide': + desiredCapacity = int(actualCapacity / scaleInFactor) + print(f"Scaling in, desired capacity: {desiredCapacity}") + else: + desiredCapacity = actualCapacity - scaleInFactor + print(f"Scaling in, desired capacity: {desiredCapacity}") + if desiredCapacity > minKPU: + update_parallelism(context, desiredCapacity, resourceName, appVersion, parallelismPerKPU) + else: + update_parallelism(context, minKPU, resourceName, appVersion, parallelismPerKPU) + print("Application will go below Min KPU") + stop_scale = 1 + else: + desiredCapacity = actualCapacity + print("Application is already below or equal to Min KPU") + stop_scale = 1 + + else: + desiredCapacity = actualCapacity + print("Scaling still happening or not required") + stop_scale = 0 + + return response_function(200, stop_scale) diff --git a/infrastructure/AutoScaling/cdk/test/kda-autoscaling.test.d.ts b/infrastructure/AutoScaling/cdk/test/kda-autoscaling.test.d.ts new file mode 100644 index 00000000..e69de29b diff --git a/infrastructure/AutoScaling/cdk/test/kda-autoscaling.test.js b/infrastructure/AutoScaling/cdk/test/kda-autoscaling.test.js new file mode 100644 index 00000000..cbb15bb0 --- /dev/null +++ b/infrastructure/AutoScaling/cdk/test/kda-autoscaling.test.js @@ -0,0 +1,17 @@ +"use strict"; +// import * as cdk from 'aws-cdk-lib'; +// import { Template } from 'aws-cdk-lib/assertions'; +// import * as KdaAutoscaling from '../lib/kda-autoscaling-stack'; +// example test. To run these tests, uncomment this file along with the +// example resource in lib/kda-autoscaling-stack.ts +test('SQS Queue Created', () => { + // const app = new cdk.App(); + // // WHEN + // const stack = new KdaAutoscaling.KdaAutoscalingStack(app, 'MyTestStack'); + // // THEN + // const template = Template.fromStack(stack); + // template.hasResourceProperties('AWS::SQS::Queue', { + // VisibilityTimeout: 300 + // }); +}); +//# sourceMappingURL=data:application/json;base64,eyJ2ZXJzaW9uIjozLCJmaWxlIjoia2RhLWF1dG9zY2FsaW5nLnRlc3QuanMiLCJzb3VyY2VSb290IjoiIiwic291cmNlcyI6WyJrZGEtYXV0b3NjYWxpbmcudGVzdC50cyJdLCJuYW1lcyI6W10sIm1hcHBpbmdzIjoiO0FBQUEsc0NBQXNDO0FBQ3RDLHFEQUFxRDtBQUNyRCxrRUFBa0U7QUFFbEUsdUVBQXVFO0FBQ3ZFLG1EQUFtRDtBQUNuRCxJQUFJLENBQUMsbUJBQW1CLEVBQUUsR0FBRyxFQUFFO0lBQy9CLCtCQUErQjtJQUMvQixjQUFjO0lBQ2QsOEVBQThFO0lBQzlFLGNBQWM7SUFDZCxnREFBZ0Q7SUFFaEQsd0RBQXdEO0lBQ3hELDZCQUE2QjtJQUM3QixRQUFRO0FBQ1IsQ0FBQyxDQUFDLENBQUMiLCJzb3VyY2VzQ29udGVudCI6WyIvLyBpbXBvcnQgKiBhcyBjZGsgZnJvbSAnYXdzLWNkay1saWInO1xuLy8gaW1wb3J0IHsgVGVtcGxhdGUgfSBmcm9tICdhd3MtY2RrLWxpYi9hc3NlcnRpb25zJztcbi8vIGltcG9ydCAqIGFzIEtkYUF1dG9zY2FsaW5nIGZyb20gJy4uL2xpYi9rZGEtYXV0b3NjYWxpbmctc3RhY2snO1xuXG4vLyBleGFtcGxlIHRlc3QuIFRvIHJ1biB0aGVzZSB0ZXN0cywgdW5jb21tZW50IHRoaXMgZmlsZSBhbG9uZyB3aXRoIHRoZVxuLy8gZXhhbXBsZSByZXNvdXJjZSBpbiBsaWIva2RhLWF1dG9zY2FsaW5nLXN0YWNrLnRzXG50ZXN0KCdTUVMgUXVldWUgQ3JlYXRlZCcsICgpID0+IHtcbi8vICAgY29uc3QgYXBwID0gbmV3IGNkay5BcHAoKTtcbi8vICAgICAvLyBXSEVOXG4vLyAgIGNvbnN0IHN0YWNrID0gbmV3IEtkYUF1dG9zY2FsaW5nLktkYUF1dG9zY2FsaW5nU3RhY2soYXBwLCAnTXlUZXN0U3RhY2snKTtcbi8vICAgICAvLyBUSEVOXG4vLyAgIGNvbnN0IHRlbXBsYXRlID0gVGVtcGxhdGUuZnJvbVN0YWNrKHN0YWNrKTtcblxuLy8gICB0ZW1wbGF0ZS5oYXNSZXNvdXJjZVByb3BlcnRpZXMoJ0FXUzo6U1FTOjpRdWV1ZScsIHtcbi8vICAgICBWaXNpYmlsaXR5VGltZW91dDogMzAwXG4vLyAgIH0pO1xufSk7XG4iXX0= \ No newline at end of file diff --git a/infrastructure/AutoScaling/cdk/test/kda-autoscaling.test.ts b/infrastructure/AutoScaling/cdk/test/kda-autoscaling.test.ts new file mode 100644 index 00000000..6c06efa1 --- /dev/null +++ b/infrastructure/AutoScaling/cdk/test/kda-autoscaling.test.ts @@ -0,0 +1,17 @@ +// import * as cdk from 'aws-cdk-lib'; +// import { Template } from 'aws-cdk-lib/assertions'; +// import * as KdaAutoscaling from '../lib/kda-autoscaling-stack'; + +// example test. To run these tests, uncomment this file along with the +// example resource in lib/kda-autoscaling-stack.ts +test('SQS Queue Created', () => { +// const app = new cdk.App(); +// // WHEN +// const stack = new KdaAutoscaling.KdaAutoscalingStack(app, 'MyTestStack'); +// // THEN +// const template = Template.fromStack(stack); + +// template.hasResourceProperties('AWS::SQS::Queue', { +// VisibilityTimeout: 300 +// }); +}); diff --git a/infrastructure/AutoScaling/cdk/tsconfig.json b/infrastructure/AutoScaling/cdk/tsconfig.json new file mode 100644 index 00000000..aaa7dc51 --- /dev/null +++ b/infrastructure/AutoScaling/cdk/tsconfig.json @@ -0,0 +1,31 @@ +{ + "compilerOptions": { + "target": "ES2020", + "module": "commonjs", + "lib": [ + "es2020", + "dom" + ], + "declaration": true, + "strict": true, + "noImplicitAny": true, + "strictNullChecks": true, + "noImplicitThis": true, + "alwaysStrict": true, + "noUnusedLocals": false, + "noUnusedParameters": false, + "noImplicitReturns": true, + "noFallthroughCasesInSwitch": false, + "inlineSourceMap": true, + "inlineSources": true, + "experimentalDecorators": true, + "strictPropertyInitialization": false, + "typeRoots": [ + "./node_modules/@types" + ] + }, + "exclude": [ + "node_modules", + "cdk.out" + ] +} diff --git a/infrastructure/AutoScaling/generateautoscaler.sh b/infrastructure/AutoScaling/generateautoscaler.sh new file mode 100755 index 00000000..93a88767 --- /dev/null +++ b/infrastructure/AutoScaling/generateautoscaler.sh @@ -0,0 +1,68 @@ +#!/bin/bash + +print_help_and_exit() { + echo "Usage: sythautoscaler.sh [type=] metric= stat= " + echo "Parameters:" + echo " type=[KDA|MSK|Kinesis|KinesisEFO]: determines the metric type, default = KDA (AWS/KinesisAnalytics namespace metrics)" + echo " metric=: the name of the metric" + echo " stat=[Average|Sum|Minimum|Maximum]: determines the metric stat" + exit 1 +} + + +# Validate parameters +METRIC_FOUND=false +for ARG in "$@"; do + if [[ ! "$ARG" =~ ^[a-zA-Z0-9]+=[^:]*$ ]]; then + echo "Error: Invalid argument '$ARG'. Must be in the form =" + print_help_and_exit + fi + if [[ "$ARG" =~ ^metric= ]]; then + METRIC_FOUND=true + METRIC_NAME="${ARG#metric=}" + fi + if [[ "$ARG" =~ ^stat= ]]; then + STAT_FOUND=true + STAT_NAME="${ARG#stat=}" + fi +done + +if [ "$METRIC_FOUND" = false ]; then + echo "Missing mandatory 'metric' parameter" + print_help_and_exit +fi +if [ "$STAT_FOUND" = false ]; then + echo "Missing mandatory 'stat' parameter" + print_help_and_exit +fi + +# Move to subdir +CURRDIR=$(pwd) +cd cdk + +# Construct the command +CMD="npx cdk synth" +for ARG in "$@"; do + CMD+=" -c $ARG" +done + +# Install node dependencies if node_modules doesn't exist or package.json has changed +if [ ! -d "node_modules" ] || [ package.json -nt node_modules ]; then + echo "Installing dependencies..." >&2 + npm install --no-fund && echo_success "Dependencies installed" +fi + +# CFN template file name +CFN_TEMPLATE_FILE="Autoscaler-${METRIC_NAME}-${STAT_NAME}.yaml" + +# Execute synth command using local node dependencies +echo "Generating autoscaler CFN template..." +$CMD > "${CURRDIR}/${CFN_TEMPLATE_FILE}" +if [ $? -eq 0 ]; then + echo "Autoscaler CFN template generated: ${CFN_TEMPLATE_FILE}" +else + echo "Error generating autoscaler CFN template" +fi + +# Move back +cd ${CURRDIR} \ No newline at end of file diff --git a/infrastructure/AutoScaling/img/generating-autoscaler.png b/infrastructure/AutoScaling/img/generating-autoscaler.png new file mode 100644 index 00000000..04c5f171 Binary files /dev/null and b/infrastructure/AutoScaling/img/generating-autoscaler.png differ diff --git a/infrastructure/README.md b/infrastructure/README.md index 353833c2..2ede6177 100644 --- a/infrastructure/README.md +++ b/infrastructure/README.md @@ -5,3 +5,4 @@ This folder contains a collection of operational utilities and infrastructure-as * [Custom autoscaling](./AutoScaling/): automatically scale Amazon Managed Service for Apache Flink applications based on any CloudWatch metric. You can also customize metric, threshold and scale up/scale down factor. * [Scheduled scaling](./ScheduledScaling/): scale Amazon Managed Service for Apache Flink applications up and down, based on a daily time schedule. * [CloudWatch Dashboard example](./monitoring/): Example of extended CloudWatch Dashboard to monitor an Amazon Managed Service for Apache Flink application. +* [scripts](./scripts): contains some useful shell script to interact with the Amazon Managed Service for Apache Flink control plane API diff --git a/infrastructure/scripts/README.md b/infrastructure/scripts/README.md new file mode 100644 index 00000000..fdaf9bd4 --- /dev/null +++ b/infrastructure/scripts/README.md @@ -0,0 +1,51 @@ +## Useful shell script + +This folder contains some useful shell script to interact with the Amazon Managed Service for Apache Flink API via AWS CLI. + +All scripts have the following prerequisites: +* [AWS CLI v2](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) +* [jq](https://jqlang.org/) + +All scripts assume the default profile is authenticated to the AWS account hosting the Managed Flink application with sufficient +permissions, and the default region matches the region of the application. +You can modify the script to pass AWS profile or region explicitly. + +These scripts are for demonstration purposes only. + +### Retrieve the status of the tasks + +[`task_status.sh`](task_status.sh) + +This script returns the status of each task in the Flink job. + +This is useful for example to automate operation, to check whether an update has been successfully deployed or is stuck +in a fail-and-restart loop due to some problem at runtime. + +The job is up-and-running and processing data when all the tasks are `RUNNING`. + +When the application is not `RUNNING`, the script always returns `UNKNOWN` + +Example 1: the job has 3 tasks, is healthy and processing data + +```shell +> ./task_status.sh MyApplication +RUNNING +RUNNING +RUNNING +``` + +Example 2: the job has 2 tasks, failing and restarting + +```shell +> ./task_status.sh MyApplication +FAILED +CANCELED +``` + +Example 3: the application is not running + +```shell +> ./task_status.sh MyApplication +UNKNOWN +``` + diff --git a/infrastructure/scripts/task_status.sh b/infrastructure/scripts/task_status.sh new file mode 100755 index 00000000..e5a2f979 --- /dev/null +++ b/infrastructure/scripts/task_status.sh @@ -0,0 +1,47 @@ +#!/bin/bash + +# This script returns the status of all tasks (called "vertices" in the API) of the Flink job. +# The script expects the application name as only parameter. It assumes the AWS default profile +# and Region are correctly set for the Managed Flink application. + +# Validate parameters +if [ $# -eq 0 ]; then + echo "Error: Application name required" + echo "Usage: $0 " + exit 1 +fi + +# Generate the pre-signed URL for the application +output=$(aws kinesisanalyticsv2 create-application-presigned-url \ + --application-name "$1" \ + --url-type FLINK_DASHBOARD_URL 2>&1) + +# Check if the output contains ResourceInUseException. It will happen when the application +# is not RUNNING, and the pre-signed URL is not available +if echo "$output" | grep -q "An error occurred (ResourceInUseException)"; then + echo "UNKNOWN" + exit 0 +fi + +# Parse the pre-signed URL +presigned_url=$(echo "$output" | jq -r '.AuthorizedUrl') +base_url=$(echo "$presigned_url" | grep -o 'https://[^/]*\.amazonaws\.com/flinkdashboard') +auth_token=$(echo "$presigned_url" | grep -o 'authToken=[^&]*' | cut -d'=' -f2) + +# Jobs endpoint URL +jobs_url="${base_url}/jobs?authToken=${auth_token}" + +# GET jobs status. Extract the Job ID assuming a single job is running +jobs_response=$(wget -qO- "${jobs_url}") +job_id=$(echo "$jobs_response" | jq -r '.jobs[0].id') + +# Job detail endpoint URL +job_details_url="${base_url}/jobs/${job_id}?authToken=${auth_token}" + +# GET Job details +job_details=$(wget -qO- "${job_details_url}") + +# Extract statuses of all vertices and join with space +vertices_statuses=$(echo "$job_details" | jq -r '.vertices[].status') + +echo "$vertices_statuses" diff --git a/java/AsyncIO/src/main/resources/flink-application-properties-dev.json b/java/AsyncIO/src/main/resources/flink-application-properties-dev.json index afdb1be9..c32ab034 100644 --- a/java/AsyncIO/src/main/resources/flink-application-properties-dev.json +++ b/java/AsyncIO/src/main/resources/flink-application-properties-dev.json @@ -3,7 +3,7 @@ "PropertyGroupId": "OutputStream0", "PropertyMap": { "aws.region": "us-east-1", - "stream.arn": "ExampleOutputStream-ARN-GOES-HERE", + "stream.arn": "arn:aws:kinesis:us-east-1::stream/ExampleOutputStream", "flink.stream.initpos": "LATEST" } }, @@ -11,8 +11,8 @@ "PropertyGroupId": "EndpointService", "PropertyMap": { "aws.region": "us-east-1", - "api.url": "API-GATEWAY-URL", - "api.key": "API-GATEWAY-KEY" + "api.url": "", + "api.key": "" } } ] diff --git a/java/DynamoDBStreamSource/src/main/resources/flink-application-properties-dev.json b/java/DynamoDBStreamSource/src/main/resources/flink-application-properties-dev.json index 81730be0..90f76d7d 100644 --- a/java/DynamoDBStreamSource/src/main/resources/flink-application-properties-dev.json +++ b/java/DynamoDBStreamSource/src/main/resources/flink-application-properties-dev.json @@ -2,14 +2,14 @@ { "PropertyGroupId": "InputStream0", "PropertyMap": { - "stream.arn": "arn:aws:dynamodb:us-east-1:012345678901:table/my-ddb-table/stream/2024-11-07T17:14:13.766", + "stream.arn": "arn:aws:dynamodb:us-east-1::table/my-ddb-table/stream/2024-11-07T17:14:13.766", "flink.stream.initpos": "TRIM_HORIZON" } }, { "PropertyGroupId": "OutputStream0", "PropertyMap": { - "stream.arn": "arn:aws:kinesis:us-east-1:012345678900:stream/ExampleOutputStream", + "stream.arn": "arn:aws:kinesis:us-east-1::stream/ExampleOutputStream", "aws.region": "us-east-1" } } diff --git a/java/FlinkCDC/FlinkCDCSQLServerSource/README.md b/java/FlinkCDC/FlinkCDCSQLServerSource/README.md new file mode 100644 index 00000000..b1658320 --- /dev/null +++ b/java/FlinkCDC/FlinkCDCSQLServerSource/README.md @@ -0,0 +1,150 @@ +# FlinkCDC SQL Server source example + +This example shows how to capture data from a database (SQL Server in this case) directly from Flink using a Flink CDC source connector. + +* Flink version: 1.20 +* Flink API: SQL +* Language: Java (11) +* Flink connectors: Flink CDC SQL Server source (3.4), JDBC sink + +The job is implemented in SQL embedded in Java. +It uses the [Flink CDC SQL Server source connector](https://nightlies.apache.org/flink/flink-cdc-docs-release-3.4/docs/connectors/flink-sources/sqlserver-cdc/) +to capture changes from a database. +Changes are propagated to the destination database using [JDBC Sink connector](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/connectors/table/jdbc/). + +### Source database + +This example uses Ms SQL Server as source database. To use a different database as CDC source you need to switch to a different Flink CDC Source connector. + +Different Flink CDC Sources require different configurations and support different metadata fields. To switch the source to a different database you need to modify the code. + +See [Flink CDC Sources documentation](https://nightlies.apache.org/flink/flink-cdc-docs-release-3.4/docs/connectors/flink-sources/sqlserver-cdc) for further details. + + +### Destination database + +Note that the JDBC sink is agnostic to the actual destination database technology. +This example is tested with both MySQL and PostgreSQL but can be easily adjusted to different databases. + +The `url` property in the `JdbcSink` configuration group decides the destination database (see [Runtime configuration](#runtime-configuration), below). +The correct JDBC driver must be included in the `pom.xml`. This example includes both MySQL and PostgreSQL drivers. + +### Testing with local databases using Docker Compose + +This example can be run locally using Docker. + +A [Docker Compose file](./docker/docker-compose.yml) is provided to run local SQL Server, MySQL and PostgreSQL databases. +The local databases are initialized by creating users, databases and tables. Some initial records are also inserted into the source table. + +You can run the Flink application inside your IDE following the instructions in [Running in IntelliJ](#running-in-intellij). +The default local configuration connects to the local PostgreSQL db defined in Docker Compose. + +To start the local databases run `docker compose up -d` in the `./docker` folder. + +Use `docker compose down -v` to shut them down, also removing the data volumes. + + +### Database prerequisites + +When running on Amazon Managed Service for Apache Flink and with databases on AWS, you need to set up the databases manually, ensuring you set up all the following: + +> YYou can find the SQL scripts that set up the dockerized databases by checking out the init scripts for +> [SQL Server](docker/sqlserver-init/init.sql), [MySQL](docker/mysql-init/init.sql), +> and [PostgreSQL](docker/postgres-init/init.sql). + +1. **Source database (Ms SQL Server)** + 1. SQL Server Agent must be running + 2. Native (user/password) authentication must be enabled + 3. The login used by Flink CDC (e.g. `flink_cdc`) must be `db_owner` for the database + 4. The source database and table must match the `database.name` and `table.name` you specify in the source configuration (e.g. `SampleDataPlatform` and `Customers`) + 5. The source table must have this schema: + ```sql + CREATE TABLE [dbo].[Customers] + ( + [CustomerID] [int] IDENTITY (1,1) NOT NULL, + [FirstName] [nvarchar](40) NOT NULL, + [MiddleInitial] [nvarchar](40) NULL, + [LastName] [nvarchar](40) NOT NULL, + [mail] [varchar](50) NULL, + CONSTRAINT [CustomerPK] PRIMARY KEY CLUSTERED ([CustomerID] ASC) + ) ON [PRIMARY]; + ``` + 6. CDC must be enabled both on the source database. On Amazon RDS SQL Server use the following stored procedure: + ```sql + exec msdb.dbo.rds_cdc_enable_db 'MyDB' + ``` + On self-managed SQL server you need to call a different procedure, while in the database: + ```sql + USE MyDB; + EXEC sys.sp_cdc_enable_db; + ``` + 7. CDC must also be enabled on the table: + ```sql + EXEC sys.sp_cdc_enable_table + @source_schema = N'dbo', + @source_name = N'Customers', + @role_name = NULL, + @supports_net_changes = 0; + ``` +2. **Destination database (MySQL or PostgreSQL)** + 1. The destination database name must match the `url` configured in the JDBC sink + 2. The destination table must have the following schema + ```sql + CREATE TABLE customers ( + customer_id INT PRIMARY KEY, + first_name VARCHAR(40), + middle_initial VARCHAR(40), + last_name VARCHAR(40), + email VARCHAR(50), + _source_updated_at TIMESTAMP, + _change_processed_at TIMESTAMP + ); + ``` + 3. The destination database user must have SELECT, INSERT, UPDATE and DELETE permissions on the destination table + +### Running in IntelliJ + +You can run this example directly in IntelliJ, without any local Flink cluster or local Flink installation. +Run the databases locally using Docker Compose, as described [above](#testing-with-local-databases-using-docker-compose). + +See [Running examples locally](../../running-examples-locally.md) for details about running the application in the IDE. + + +### Running on Amazon Managed Service for Apache Flink + +To run the application in Amazon Managed Service for Apache Flink make sure the application configuration has the following: +* VPC networking +* The selected Subnets can route traffic to both the source and destination databases +* The Security Group allows traffic from the application to both source and destination databases + + +### Runtime configuration + +When running on Amazon Managed Service for Apache Flink the runtime configuration is read from *Runtime Properties*. + +When running locally, the configuration is read from the [`resources/flink-application-properties-dev.json`](resources/flink-application-properties-dev.json) file located in the resources folder. + +Runtime parameters: + +| Group ID | Key | Description | +|-------------|-----------------|----------------------------------------------------------------------------------------------------------------------------| +| `CDCSource` | `hostname` | Source database DNS hostname or IP | +| `CDCSource` | `port` | Source database port (normally `1433`) | +| `CDCSource` | `username` | Source database username. The user must be `dbo_owner` of the database | +| `CDCSource` | `password` | Source database password | +| `CDCSource` | `database.name` | Source database name | +| `CDCSource` | `table.name` | Source table name. e.g. `dbo.Customers` | +| `JdbcSink` | `url` | Destination database JDBC URL. e.g. `jdbc:postgresql://localhost:5432/targetdb`. Note: the URL includes the database name. | +| `JdbcSink` | `table.name` | Destination table. e.g. `customers` | +| `JdbcSink` | `username` | Destination database user | +| `JdbcSink` | `password` | Destination database password | + +### Known limitations + +Using the SQL interface of Flink CDC Sources greatly simplifies the implementation of a passthrough application. +However, schema changes in the source table are ignored. + +## References + +* [Flink CDC SQL Server documentation](https://nightlies.apache.org/flink/flink-cdc-docs-release-3.4/docs/connectors/flink-sources/sqlserver-cdc) +* [Debezium SQL Server documentation](https://debezium.io/documentation/reference/1.9/connectors/sqlserver.html) \ No newline at end of file diff --git a/java/FlinkCDC/FlinkCDCSQLServerSource/docker/docker-compose.yml b/java/FlinkCDC/FlinkCDCSQLServerSource/docker/docker-compose.yml new file mode 100644 index 00000000..ca6abb46 --- /dev/null +++ b/java/FlinkCDC/FlinkCDCSQLServerSource/docker/docker-compose.yml @@ -0,0 +1,77 @@ +services: + + # Ms SQL Server + init + sqlserver: + image: mcr.microsoft.com/mssql/server:2022-latest + container_name: mssql-server-2022 + environment: + - ACCEPT_EULA=Y + - SA_PASSWORD=YourStrong@Passw0rd + - MSSQL_PID=Developer + - MSSQL_AGENT_ENABLED=true + ports: + - "1433:1433" + volumes: + - sqlserver_data:/var/opt/mssql + - ./sqlserver-init/init.sql:/tmp/init.sql + restart: unless-stopped + healthcheck: + test: ["CMD-SHELL", "/opt/mssql-tools18/bin/sqlcmd -S localhost -U sa -P YourStrong@Passw0rd -Q 'SELECT 1' -C"] + interval: 10s + timeout: 5s + retries: 5 + start_period: 10s + + sqlserver-init: + image: mcr.microsoft.com/mssql/server:2022-latest + depends_on: + sqlserver: + condition: service_healthy + volumes: + - ./sqlserver-init/init.sql:/tmp/init.sql + command: > + bash -c " + echo 'Waiting for SQL Server to be ready...' && + sleep 5 && + echo 'Running initialization script...' && + /opt/mssql-tools18/bin/sqlcmd -S sqlserver -U sa -P YourStrong@Passw0rd -i /tmp/init.sql -C && + echo 'Initialization completed!' + " + + # MySQL database + mysql: + image: mysql:8.0 + container_name: mysql_db + restart: always + environment: + MYSQL_ROOT_PASSWORD: R00tpwd! + MYSQL_DATABASE: targetdb + ports: + - "3306:3306" + volumes: + - mysql_data:/var/lib/mysql + - ./mysql-init:/docker-entrypoint-initdb.d + command: --default-authentication-plugin=mysql_native_password + + # PostgreSQL database + postgres: + image: postgres:15 + container_name: postgres_db + restart: always + environment: + POSTGRES_DB: targetdb + POSTGRES_USER: flinkusr + POSTGRES_PASSWORD: PassW0rd! + ports: + - "5432:5432" + volumes: + - postgres_data:/var/lib/postgresql/data + - ./postgres-init:/docker-entrypoint-initdb.d + +volumes: + sqlserver_data: + driver: local + mysql_data: + driver: local + postgres_data: + driver: local diff --git a/java/FlinkCDC/FlinkCDCSQLServerSource/docker/mysql-init/init.sql b/java/FlinkCDC/FlinkCDCSQLServerSource/docker/mysql-init/init.sql new file mode 100644 index 00000000..211e432d --- /dev/null +++ b/java/FlinkCDC/FlinkCDCSQLServerSource/docker/mysql-init/init.sql @@ -0,0 +1,15 @@ +CREATE USER 'flinkusr'@'%' IDENTIFIED BY 'PassW0rd!'; +GRANT SELECT, INSERT, UPDATE, DELETE, SHOW DATABASES ON *.* TO 'flinkusr'@'%'; + +FLUSH PRIVILEGES; + +-- Create customer table +CREATE TABLE customers ( + customer_id INT PRIMARY KEY, + first_name VARCHAR(40), + middle_initial VARCHAR(40), + last_name VARCHAR(40), + email VARCHAR(50), + _source_updated_at TIMESTAMP, + _change_processed_at TIMESTAMP +); diff --git a/java/FlinkCDC/FlinkCDCSQLServerSource/docker/postgres-init/init.sql b/java/FlinkCDC/FlinkCDCSQLServerSource/docker/postgres-init/init.sql new file mode 100644 index 00000000..05e90511 --- /dev/null +++ b/java/FlinkCDC/FlinkCDCSQLServerSource/docker/postgres-init/init.sql @@ -0,0 +1,10 @@ +-- Create customer table +CREATE TABLE customers ( + customer_id INT PRIMARY KEY, + first_name VARCHAR(40), + middle_initial VARCHAR(40), + last_name VARCHAR(40), + email VARCHAR(50), + _source_updated_at TIMESTAMP, + _change_processed_at TIMESTAMP +); diff --git a/java/FlinkCDC/FlinkCDCSQLServerSource/docker/sqlserver-init/init.sql b/java/FlinkCDC/FlinkCDCSQLServerSource/docker/sqlserver-init/init.sql new file mode 100644 index 00000000..4fb43a98 --- /dev/null +++ b/java/FlinkCDC/FlinkCDCSQLServerSource/docker/sqlserver-init/init.sql @@ -0,0 +1,56 @@ +-- Create SampleDataPlatform database +CREATE DATABASE SampleDataPlatform; +GO + +-- Use the SampleDataPlatform database +USE SampleDataPlatform; +GO + +-- Create login for flink_cdc +CREATE LOGIN flink_cdc WITH PASSWORD = 'FlinkCDC@123'; +GO + +-- Create user in SampleDataPlatform database +CREATE USER flink_cdc FOR LOGIN flink_cdc; +GO + +-- Grant necessary permissions for CDC operations +ALTER ROLE db_owner ADD MEMBER flink_cdc; +GO + +-- Enable CDC on the SampleDataPlatform database +USE SampleDataPlatform; +EXEC sys.sp_cdc_enable_db; +GO + +-- Create Customers table with the specified schema +CREATE TABLE [dbo].[Customers] +( + [CustomerID] [int] IDENTITY (1,1) NOT NULL, + [FirstName] [nvarchar](40) NOT NULL, + [MiddleInitial] [nvarchar](40) NULL, + [LastName] [nvarchar](40) NOT NULL, + [mail] [varchar](50) NULL, + CONSTRAINT [CustomerPK] PRIMARY KEY CLUSTERED ([CustomerID] ASC) + WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, OPTIMIZE_FOR_SEQUENTIAL_KEY = OFF) ON [PRIMARY] +) ON [PRIMARY]; +GO + +-- Enable CDC on the Customers table +EXEC sys.sp_cdc_enable_table + @source_schema = N'dbo', + @source_name = N'Customers', + @role_name = NULL, + @supports_net_changes = 0; +GO + +-- Insert some sample data +INSERT INTO [dbo].[Customers] ([FirstName], [MiddleInitial], [LastName], [mail]) +VALUES ('John', 'A', 'Doe', 'john.doe@example.com'), + ('Jane', NULL, 'Smith', 'jane.smith@example.com'), + ('Bob', 'R', 'Johnson', 'bob.johnson@example.com'), + ('Alice', 'M', 'Williams', 'alice.williams@example.com'), + ('Charlie', NULL, 'Brown', 'charlie.brown@example.com'); +GO + +PRINT 'Database initialization completed successfully!'; diff --git a/java/FlinkCDC/FlinkCDCSQLServerSource/pom.xml b/java/FlinkCDC/FlinkCDCSQLServerSource/pom.xml new file mode 100644 index 00000000..ca8f1123 --- /dev/null +++ b/java/FlinkCDC/FlinkCDCSQLServerSource/pom.xml @@ -0,0 +1,191 @@ + + + 4.0.0 + + com.amazonaws + flink-cdc-sqlserver-sql-source + 1.0 + jar + + + UTF-8 + ${project.basedir}/target + ${project.name}-${project.version} + 11 + ${target.java.version} + ${target.java.version} + 1.20.0 + 3.4.0 + 3.3.0-1.20 + 9.3.0 + 42.7.2 + 1.2.0 + 2.23.1 + + + + + + + + org.apache.flink + flink-streaming-java + ${flink.version} + provided + + + org.apache.flink + flink-clients + ${flink.version} + provided + + + org.apache.flink + flink-runtime-web + ${flink.version} + provided + + + org.apache.flink + flink-json + ${flink.version} + provided + + + org.apache.flink + flink-table-runtime + ${flink.version} + provided + + + org.apache.flink + flink-table-planner-loader + ${flink.version} + provided + + + + + com.amazonaws + aws-kinesisanalytics-runtime + ${kda.runtime.version} + provided + + + + + org.apache.flink + flink-connector-base + ${flink.version} + provided + + + org.apache.flink + flink-connector-sqlserver-cdc + ${flink.cdc.version} + + + org.apache.flink + flink-connector-jdbc + ${flink.jdbc.connector.version} + + + + + + com.mysql + mysql-connector-j + ${mysql.jdbc.driver.version} + + + org.postgresql + postgresql + ${postgresql.jdbc.driver.version} + + + + + + + org.apache.logging.log4j + log4j-slf4j-impl + ${log4j.version} + + + org.apache.logging.log4j + log4j-api + ${log4j.version} + + + org.apache.logging.log4j + log4j-core + ${log4j.version} + + + + + ${buildDirectory} + ${jar.finalName} + + + + + org.apache.maven.plugins + maven-compiler-plugin + 3.8.1 + + ${target.java.version} + ${target.java.version} + + + + + + org.apache.maven.plugins + maven-shade-plugin + 3.2.1 + + + + package + + shade + + + + + org.apache.flink:force-shading + com.google.code.findbugs:jsr305 + org.slf4j:* + log4j:* + + + + + + *:* + + META-INF/*.SF + META-INF/*.DSA + META-INF/*.RSA + + + + + + + com.amazonaws.services.msf.FlinkCDCSqlServer2JdbcJob + + + + + + + + + \ No newline at end of file diff --git a/java/FlinkCDC/FlinkCDCSQLServerSource/src/main/java/com/amazonaws/services/msf/FlinkCDCSqlServer2JdbcJob.java b/java/FlinkCDC/FlinkCDCSQLServerSource/src/main/java/com/amazonaws/services/msf/FlinkCDCSqlServer2JdbcJob.java new file mode 100644 index 00000000..1fbc14fd --- /dev/null +++ b/java/FlinkCDC/FlinkCDCSQLServerSource/src/main/java/com/amazonaws/services/msf/FlinkCDCSqlServer2JdbcJob.java @@ -0,0 +1,157 @@ +package com.amazonaws.services.msf; + +import com.amazonaws.services.kinesisanalytics.runtime.KinesisAnalyticsRuntime; +import org.apache.flink.cdc.common.utils.Preconditions; +import org.apache.flink.streaming.api.environment.LocalStreamEnvironment; +import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; +import org.apache.flink.table.api.EnvironmentSettings; +import org.apache.flink.table.api.bridge.java.StreamStatementSet; +import org.apache.flink.table.api.bridge.java.StreamTableEnvironment; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.util.Map; +import java.util.Properties; + +public class FlinkCDCSqlServer2JdbcJob { + private static final Logger LOG = LoggerFactory.getLogger(FlinkCDCSqlServer2JdbcJob.class); + + // Name of the local JSON resource with the application properties in the same format as they are received from the Amazon Managed Service for Apache Flink runtime + private static final String LOCAL_APPLICATION_PROPERTIES_RESOURCE = "flink-application-properties-dev.json"; + + private static final int DEFAULT_CDC_DB_PORT = 1433; + + private static boolean isLocal(StreamExecutionEnvironment env) { + return env instanceof LocalStreamEnvironment; + } + + /** + * Load application properties from Amazon Managed Service for Apache Flink runtime or from a local resource, when the environment is local + */ + private static Map loadApplicationProperties(StreamExecutionEnvironment env) throws IOException { + if (isLocal(env)) { + LOG.info("Loading application properties from '{}'", LOCAL_APPLICATION_PROPERTIES_RESOURCE); + return KinesisAnalyticsRuntime.getApplicationProperties( + FlinkCDCSqlServer2JdbcJob.class.getClassLoader() + .getResource(LOCAL_APPLICATION_PROPERTIES_RESOURCE).getPath()); + } else { + LOG.info("Loading application properties from Amazon Managed Service for Apache Flink"); + return KinesisAnalyticsRuntime.getApplicationProperties(); + } + } + + public static void main(String[] args) throws Exception { + // set up the streaming execution environment + final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); + final StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env, EnvironmentSettings.newInstance().build()); + + final Map applicationProperties = loadApplicationProperties(env); + LOG.warn("Application properties: {}", applicationProperties); + + // Enable checkpoints and set parallelism when running locally + // On Managed Flink, checkpoints and application parallelism are managed by the service and controlled by the application configuration + if (isLocal(env)) { + env.setParallelism(1); // Ms SQL Server Flink CDC is single-threaded + env.enableCheckpointing(30000); + } + + + // Create CDC source table + Properties cdcSourceProperties = applicationProperties.get("CDCSource"); + tableEnv.executeSql("CREATE TABLE Customers (" + + " CustomerID INT," + + " FirstName STRING," + + " MiddleInitial STRING," + + " LastName STRING," + + " mail STRING," + + // Some additional metadata columns for demonstration purposes + " `_change_processed_at` AS PROCTIME()," + // The time when Flink is processing this record + " `_source_updated_at` TIMESTAMP_LTZ(3) METADATA FROM 'op_ts' VIRTUAL," + // The time when the operation was executed on the db + " `_table_name` STRING METADATA FROM 'table_name' VIRTUAL," + // Name of the table in the source db + " `_schema_name` STRING METADATA FROM 'schema_name' VIRTUAL, " + // Name of the schema in the source db + " `_db_name` STRING METADATA FROM 'database_name' VIRTUAL," + // name of the database + " PRIMARY KEY(CustomerID) NOT ENFORCED" + + ") WITH (" + + " 'connector' = 'sqlserver-cdc'," + + " 'hostname' = '" + Preconditions.checkNotNull(cdcSourceProperties.getProperty("hostname"), "missing CDC source hostname") + "'," + + " 'port' = '" + Preconditions.checkNotNull(cdcSourceProperties.getProperty("port", Integer.toString(DEFAULT_CDC_DB_PORT)), "missing CDC source port") + "'," + + " 'username' = '" + Preconditions.checkNotNull(cdcSourceProperties.getProperty("username"), "missing CDC source username") + "'," + + // For simplicity, we are passing the db password as a runtime configuration unencrypted. This should be avoided in production + " 'password' = '" + Preconditions.checkNotNull(cdcSourceProperties.getProperty("password"), "missing CDC source password") + "'," + + " 'database-name' = '" + Preconditions.checkNotNull(cdcSourceProperties.getProperty("database.name"), "missing CDC source database name") + "'," + + " 'table-name' = '" + Preconditions.checkNotNull(cdcSourceProperties.getProperty("table.name"), "missing CDC source table name") + "'" + + ")"); + + + // Create a JDBC sink table + // Note that the definition of the table is agnostic to the actual destination database (e.g. MySQL or PostgreSQL) + Properties jdbcSinkProperties = applicationProperties.get("JdbcSink"); + tableEnv.executeSql("CREATE TABLE DestinationTable (" + + " customer_id INT," + + " first_name STRING," + + " middle_initial STRING," + + " last_name STRING," + + " email STRING," + + " _source_updated_at TIMESTAMP(3)," + + " _change_processed_at TIMESTAMP(3)," + + " PRIMARY KEY(customer_id) NOT ENFORCED" + + ") WITH (" + + " 'connector' = 'jdbc'," + + " 'url' = '" + Preconditions.checkNotNull(jdbcSinkProperties.getProperty("url"), "missing destination database JDBC URL") + "'," + + " 'table-name' = '" + Preconditions.checkNotNull(jdbcSinkProperties.getProperty("table.name"), "missing destination database table name") + "'," + + " 'username' = '" + Preconditions.checkNotNull(jdbcSinkProperties.getProperty("username"), "missing destination database username") + "'," + + " 'password' = '" + Preconditions.checkNotNull(jdbcSinkProperties.getProperty("password"), "missing destination database password") + "'" + + ")"); + + // When running locally we add a secondary sink to print the output to the console. + // When the job is running on Managed Flink any output to console is not visible and may cause overhead. + // It is recommended not to print any output to the console when running the application on Managed Flink. + if( isLocal(env)) { + tableEnv.executeSql("CREATE TABLE PrintSinkTable (" + + " CustomerID INT," + + " FirstName STRING," + + " MiddleInitial STRING," + + " LastName STRING," + + " mail STRING," + + " `_change_processed_at` TIMESTAMP_LTZ(3)," + + " `_source_updated_at` TIMESTAMP_LTZ(3)," + + " `_table_name` STRING," + + " `_schema_name` STRING," + + " `_db_name` STRING," + + " PRIMARY KEY(CustomerID) NOT ENFORCED" + + ") WITH (" + + " 'connector' = 'print'" + + ")"); + } + + // Note that we use a statement set to add the two "INSERT INTO..." statements. + // When tableEnv.executeSQL(...) is used with INSERT INTO on a job running in Application mode, like on Managed Flink, + // the first statement triggers the job execution, and any code which follows is ignored. + StreamStatementSet statementSet = tableEnv.createStatementSet(); + statementSet.addInsertSql("INSERT INTO DestinationTable (" + + "customer_id, " + + "first_name, " + + "middle_initial, " + + "last_name, " + + "email, " + + "_source_updated_at, " + + "_change_processed_at" + + ") SELECT " + + "CustomerID, " + + "FirstName, " + + "MiddleInitial, " + + "LastName, " + + "mail, " + + "`_source_updated_at`, " + + "`_change_processed_at` " + + "FROM Customers"); + if( isLocal(env)) { + statementSet.addInsertSql("INSERT INTO PrintSinkTable SELECT * FROM Customers"); + } + + + // Execute the two INSERT INTO statements + statementSet.execute(); + } +} diff --git a/java/FlinkCDC/FlinkCDCSQLServerSource/src/main/resources/flink-application-properties-dev.json b/java/FlinkCDC/FlinkCDCSQLServerSource/src/main/resources/flink-application-properties-dev.json new file mode 100644 index 00000000..8974b2f8 --- /dev/null +++ b/java/FlinkCDC/FlinkCDCSQLServerSource/src/main/resources/flink-application-properties-dev.json @@ -0,0 +1,22 @@ +[ + { + "PropertyGroupId": "CDCSource", + "PropertyMap": { + "hostname": "localhost", + "port": "1433", + "username": "flink_cdc", + "password": "FlinkCDC@123", + "database.name": "SampleDataPlatform", + "table.name": "dbo.Customers" + } + }, + { + "PropertyGroupId": "JdbcSink", + "PropertyMap": { + "table.name": "customers", + "url": "jdbc:postgresql://localhost:5432/targetdb", + "username": "flinkusr", + "password": "PassW0rd!" + } + } +] \ No newline at end of file diff --git a/java/FlinkCDC/FlinkCDCSQLServerSource/src/main/resources/log4j2.properties b/java/FlinkCDC/FlinkCDCSQLServerSource/src/main/resources/log4j2.properties new file mode 100644 index 00000000..35466433 --- /dev/null +++ b/java/FlinkCDC/FlinkCDCSQLServerSource/src/main/resources/log4j2.properties @@ -0,0 +1,7 @@ +rootLogger.level = INFO +rootLogger.appenderRef.console.ref = ConsoleAppender + +appender.console.name = ConsoleAppender +appender.console.type = CONSOLE +appender.console.layout.type = PatternLayout +appender.console.layout.pattern = %d{HH:mm:ss,SSS} %-5p %-60c %x - %m%n diff --git a/java/FlinkCDC/README.md b/java/FlinkCDC/README.md new file mode 100644 index 00000000..196b083f --- /dev/null +++ b/java/FlinkCDC/README.md @@ -0,0 +1,9 @@ +# Flink CDC Examples + +Examples demonstrating Change Data Capture (CDC) using [Flink CDC source connectors](https://nightlies.apache.org/flink/flink-cdc-docs-release-3.4/docs/connectors/flink-sources/overview/) +in Amazon Managed Service for Apache Flink. + +## Table of Contents + +### Database Sources +- [**Flink CDC SQL Server Source**](./FlinkCDCSQLServerSource) - Capturing changes from SQL Server database and writing to JDBC sink \ No newline at end of file diff --git a/java/FlinkDataGenerator/README.md b/java/FlinkDataGenerator/README.md new file mode 100644 index 00000000..84376f95 --- /dev/null +++ b/java/FlinkDataGenerator/README.md @@ -0,0 +1,144 @@ +# Flink JSON Data Generator to Kinesis or Kafka + +This example demonstrates how you can use Apache Flink's as a data generator for load testing. + +* Flink version: 1.20 +* Flink API: DataStream API +* Language: Java (11) +* Flink connectors: DataGen, Kafka Sink, Kinesis Sink + +The application generates random stock prices at fixed rate. +Depending on runtime configuration it will send generated records, as JSON, either to a Kinesis Data Stream +or an MSK/Kafka topic (or both). + +The application can easily scale to generate high throughput. For example, with 3 KPU you can generate more than 64,000 records per second. +See [Using the data generator for load testing](#using-the-data-generator-for-load-testing). + +It can be easily modified to generate different type of records, changing the implementation of the record class +[StockPrice](src/main/java/com/amazonaws/services/msf/domain/StockPrice.java), and the function generating data [StockPriceGeneratorFunction](src/main/java/com/amazonaws/services/msf/domain/StockPriceGeneratorFunction.java) + +### Prerequisites + +The data generator application must be able to write to the Kinesis Stream or the Kafka topic +* Kafka/MSK + * The Managed Flink application must have VPC networking. + * Routing and Security must allow the application to reach the Kafka cluster. + * Any Kafka/MSK authentication must be added to the application (this application writes unauthenticated) + * Kafka ACL or IAM must allow the application writing to the topic +* Kinesis Data Stream + * The Managed Flink application IAM Role must have permissions to write to the stream + * Ensure the Kinesis Stream has sufficient capacity for the generated throughput + * If the application has VPC networking, you must also create a VPC Endpoint for Kinesis to be able to write to the Stream + + + +### Runtime configuration + +The application reads the runtime configuration from the Runtime Properties, when running on Amazon Managed Service for Apache Flink, +or, when running locally, from the [`src/main/resources/flink-application-properties-dev.json`](src/main/resources/flink-application-properties-dev.json) file. +All parameters are case-sensitive. + +The presence of the configuration group `KafkaSink` enables the Kafka sink. +Likewise, `KinesisSink` enables the Kinesis sink. + + +| Group ID | Key | Description | +|---------------|-----------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `DataGen` | `records.per.second` | Number of records per second generated across all subtasks. | +| `KinesisSink` | `stream.arn` | ARN of the Kinesis Stream | +| `KinesisSink` | `aws.region` | Region of the Kinesis Stream | +| `KinesisSink` | (any other parameter) | Any other parameters in this group is passed to the Kinesis sink connector as KinesisClientProperties. See [documentation](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/connectors/datastream/kinesis/#kinesis-streams-sink) | +| `KafkaSink` | `bootstrap.servers` | Kafka bootstrap servers. | +| `KafkaSink` | `topic` | Name of the Kafka topic. | +| `KafkaSink` | (any other parameter) | Any other parameters in this group is passed to the Kafka sink connector as KafkaProducerConfig. See [documentation](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/connectors/datastream/kafka/#kafka-sink) | + + +> Renaming `KafkaSink` or `KinesisSink` groups to something different, for example `KinesisSink-DISABLE` prevents +> the generator creating that particular sink. + +### Running in IntelliJ + +You can run this example directly in IntelliJ, without any local Flink cluster or local Flink installation. + +See [Running examples locally](../running-examples-locally.md) for details. + +--- + +## Data Generation + +This example generates random stock price records similar to the following: + +```json +{ + "event_time": "2024-01-15T10:30:45.123", + "ticker": "AAPL", + "price": 150.25 +} +``` + +The data generation can be easily customized to match your specific records, modifying two components: + +* The class [StockPrice](src/main/java/com/amazonaws/services/msf/domain/StockPrice.java) representing the record. + You can use [Jackson annotations](https://github.com/FasterXML/jackson-annotations/wiki/Jackson-Annotations) to customize the generated JSON. +* The class [StockPriceGeneratorFunction](src/main/java/com/amazonaws/services/msf/domain/StockPriceGeneratorFunction.java) + contains the logic for generating each record. + +### Partitioning + +Records published in Kinesis or Kafka are partitioned by the `ticker` field. + +If you customize the data object you also need to modify the `PartitionKeyGenerator` and `SerializationSchema` +extracting the key in the Kinesis and Kafka sink respectively. + +## Using the data generator for load testing + +This application can be used to load test other applications. + +Make sure the data generator application has sufficient resources to generate the desired throughput. + +Also, make sure the Kafka/MSK cluster or the Kinesis Stream have sufficient capacity to ingest the generated throughput. + +⚠️ If the destination system or the data generator Flink application are underprovisioned, you may generate a throughput lower than expected. + +For reference, the following configuration allows generating ~64,000 records/sec to either Kinesis or Kafka: +* `Parallelism = 3`, `Parallelism-per-KPU = 1` (`3 KPU`) +* `DataGen` `records.per.second` = `64000` + +> We recommend to overprovision the data generator to ensure the required throughput can be achieved. +> Use the provided [CloudWatch dashboard](#cloudwatch-dashboard) to monitor the generator. + +### Monitoring the data generator + +The application exposes 3 custom metrics to CloudWatch: +* `generatedRecordCount`: count of generated record, per parallelism +* `generatedRecordRatePerParallelism`: generated records per second, per parallelism +* `taskParallelism`: parallelism of the data generator + +> ⚠️ Flink custom metrics are not global. Each subtask maintains its own metrics. +> Also, for each metric Amazon Managed Service for Apache Flink exports to CloudWatch 4 datapoints per minute, per subtask. +> That considered, to calculate the total generated record and rate, across the entire application, you need to apply +> the following maths: +> - Total generatedRecordCount = `SUM(generatedRecordCount) / 4`, over 1 minute +> - Total generatedRecordsPerSec = `AVG(generatedRecordRatePerParallelism) * AVG(taskParallelism)`, over 1 minute + +#### CloudWatch Dashboard + +The CloudFormation template [dashboard-cfn.yaml](tools/dashboard-cfn.yaml) provided can be used to create a CloudWatch Dashboard +to monitor the data generator + +![Flink Data Generator dashboard](images/dashboard.png). + +When creating the CloudFormation stack you need to provide: +* The name of the Managed Flink application +* The Region +* The name of the Kinesis Stream, if publishing to Kinesis +* The name of the MSK cluster and topic, if publishing to MSK + +> Note: the dashboard assumes an MSK cluster with up to 6 brokers. +> If you have a cluster with more than 6 brokers you need to adjust the *Kafka output* widget + +### Known limitations and possible extensions + +* Only JSON serialization is supported. +* Data generation is stateless. The logic generating each record does not know about other records previously generated. +* Fixed record rate only. No ramp up or ramp down. diff --git a/java/FlinkDataGenerator/images/dashboard.png b/java/FlinkDataGenerator/images/dashboard.png new file mode 100644 index 00000000..248e36cb Binary files /dev/null and b/java/FlinkDataGenerator/images/dashboard.png differ diff --git a/java/FlinkDataGenerator/pom.xml b/java/FlinkDataGenerator/pom.xml new file mode 100644 index 00000000..0ea1d7be --- /dev/null +++ b/java/FlinkDataGenerator/pom.xml @@ -0,0 +1,185 @@ + + + 4.0.0 + + com.amazonaws + flink-data-generator + 1.0 + jar + + + UTF-8 + ${project.basedir}/target + ${project.name}-${project.version} + 11 + ${target.java.version} + ${target.java.version} + 1.20.0 + 5.0.0-1.20 + 3.3.0-1.20 + 1.2.0 + 2.23.1 + + + + + + + org.apache.flink + flink-streaming-java + ${flink.version} + provided + + + org.apache.flink + flink-clients + ${flink.version} + provided + + + org.apache.flink + flink-runtime-web + ${flink.version} + provided + + + org.apache.flink + flink-json + ${flink.version} + provided + + + org.apache.flink + flink-connector-base + ${flink.version} + provided + + + + org.apache.flink + flink-metrics-dropwizard + ${flink.version} + + + + + com.amazonaws + aws-kinesisanalytics-runtime + ${kda.runtime.version} + provided + + + + + org.apache.flink + flink-connector-datagen + ${flink.version} + + + + + org.apache.flink + flink-connector-aws-kinesis-streams + ${aws.connector.version} + + + + + org.apache.flink + flink-connector-kafka + ${kafka.connector.version} + + + + + + + + org.apache.logging.log4j + log4j-slf4j-impl + ${log4j.version} + + + org.apache.logging.log4j + log4j-api + ${log4j.version} + + + org.apache.logging.log4j + log4j-core + ${log4j.version} + + + + + junit + junit + 4.13.2 + test + + + + + ${buildDirectory} + ${jar.finalName} + + + + + org.apache.maven.plugins + maven-compiler-plugin + 3.8.1 + + ${target.java.version} + ${target.java.version} + + + + + + org.apache.maven.plugins + maven-shade-plugin + 3.2.1 + + + + package + + shade + + + + + org.apache.flink:force-shading + com.google.code.findbugs:jsr305 + org.slf4j:* + log4j:* + + + + + + *:* + + META-INF/*.SF + META-INF/*.DSA + META-INF/*.RSA + + + + + + + com.amazonaws.services.msf.DataGeneratorJob + + + + + + + + + diff --git a/java/FlinkDataGenerator/src/main/java/com/amazonaws/services/msf/DataGeneratorJob.java b/java/FlinkDataGenerator/src/main/java/com/amazonaws/services/msf/DataGeneratorJob.java new file mode 100644 index 00000000..d41161f3 --- /dev/null +++ b/java/FlinkDataGenerator/src/main/java/com/amazonaws/services/msf/DataGeneratorJob.java @@ -0,0 +1,225 @@ +package com.amazonaws.services.msf; + +import com.amazonaws.services.kinesisanalytics.runtime.KinesisAnalyticsRuntime; +import com.amazonaws.services.msf.domain.StockPrice; +import com.amazonaws.services.msf.domain.StockPriceGeneratorFunction; +import org.apache.commons.lang3.StringUtils; +import org.apache.flink.api.common.eventtime.WatermarkStrategy; +import org.apache.flink.api.common.serialization.SerializationSchema; +import org.apache.flink.api.common.typeinfo.TypeInformation; +import org.apache.flink.api.connector.source.util.ratelimit.RateLimiterStrategy; +import org.apache.flink.connector.base.DeliveryGuarantee; +import org.apache.flink.connector.datagen.source.DataGeneratorSource; +import org.apache.flink.connector.datagen.source.GeneratorFunction; +import org.apache.flink.connector.kafka.sink.KafkaRecordSerializationSchema; +import org.apache.flink.connector.kafka.sink.KafkaSink; +import org.apache.flink.connector.kinesis.sink.KinesisStreamsSink; +import org.apache.flink.connector.kinesis.sink.PartitionKeyGenerator; +import org.apache.flink.formats.json.JsonSerializationSchema; +import org.apache.flink.streaming.api.datastream.DataStream; +import org.apache.flink.streaming.api.environment.LocalStreamEnvironment; +import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; +import org.apache.flink.util.Preconditions; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.util.Map; +import java.util.Properties; + +/** + * A Flink application that generates random stock data using DataGeneratorSource + * and sends it to Kinesis Data Streams and/or Kafka as JSON based on configuration. + * At least one sink (KinesisSink or KafkaSink) must be configured. + * The generated data matches the schema used by the Python data generator. + */ +public class DataGeneratorJob { + private static final Logger LOG = LoggerFactory.getLogger(DataGeneratorJob.class); + + // Name of the local JSON resource with the application properties in the same format as they are received from the Amazon Managed Service for Apache Flink runtime + private static final String LOCAL_APPLICATION_PROPERTIES_RESOURCE = "flink-application-properties-dev.json"; + + // Default values for configuration + private static final int DEFAULT_RECORDS_PER_SECOND = 10; + + private static boolean isLocal(StreamExecutionEnvironment env) { + return env instanceof LocalStreamEnvironment; + } + + /** + * Load application properties from Amazon Managed Service for Apache Flink runtime or from a local resource, when the environment is local + */ + private static Map loadApplicationProperties(StreamExecutionEnvironment env) throws IOException { + if (isLocal(env)) { + LOG.info("Loading application properties from '{}'", LOCAL_APPLICATION_PROPERTIES_RESOURCE); + return KinesisAnalyticsRuntime.getApplicationProperties( + DataGeneratorJob.class.getClassLoader() + .getResource(LOCAL_APPLICATION_PROPERTIES_RESOURCE).getPath()); + } else { + LOG.info("Loading application properties from Amazon Managed Service for Apache Flink"); + return KinesisAnalyticsRuntime.getApplicationProperties(); + } + } + + /** + * Create a DataGeneratorSource with configurable rate from DataGen properties + * + * @param dataGenProperties Properties from the "DataGen" property group + * @param generatorFunction The generator function to use for data generation + * @param typeInformation Type information for the generated data type + * @param The type of data to generate + * @return Configured DataGeneratorSource + */ + private static DataGeneratorSource createDataGeneratorSource( + Properties dataGenProperties, + GeneratorFunction generatorFunction, + TypeInformation typeInformation) { + + int recordsPerSecond; + if (dataGenProperties != null) { + String recordsPerSecondStr = dataGenProperties.getProperty("records.per.second"); + if (recordsPerSecondStr != null && !recordsPerSecondStr.trim().isEmpty()) { + try { + recordsPerSecond = Integer.parseInt(recordsPerSecondStr.trim()); + } catch (NumberFormatException e) { + LOG.error("Invalid records.per.second value: '{}'. Must be a valid integer. ", recordsPerSecondStr); + throw e; + } + } else { + LOG.info("No records.per.second configured. Using default: {}", DEFAULT_RECORDS_PER_SECOND); + recordsPerSecond = DEFAULT_RECORDS_PER_SECOND; + } + } else { + LOG.info("No DataGen properties found. Using default records per second: {}", DEFAULT_RECORDS_PER_SECOND); + recordsPerSecond = DEFAULT_RECORDS_PER_SECOND; + } + + Preconditions.checkArgument(recordsPerSecond > 0, + "Invalid records.per.second value. Must be positive."); + + + return new DataGeneratorSource( + generatorFunction, + Long.MAX_VALUE, // Generate (practically) unlimited records + RateLimiterStrategy.perSecond(recordsPerSecond), // Configurable rate + typeInformation // Explicit type information + ); + } + + /** + * Create a Kinesis Sink + * + * @param outputProperties Properties from the "KinesisSink" property group + * @param serializationSchema Serialization schema + * @param partitionKeyGenerator Partition key generator + * @param The type of data to sink + * @return an instance of KinesisStreamsSink + */ + private static KinesisStreamsSink createKinesisSink(Properties outputProperties, final SerializationSchema serializationSchema, final PartitionKeyGenerator partitionKeyGenerator + ) { + final String outputStreamArn = outputProperties.getProperty("stream.arn"); + return KinesisStreamsSink.builder() + .setStreamArn(outputStreamArn) + .setKinesisClientProperties(outputProperties) + .setSerializationSchema(serializationSchema) + .setPartitionKeyGenerator(partitionKeyGenerator) + .build(); + } + + /** + * Create a KafkaSink + * + * @param kafkaProperties Properties from the "KafkaSink" property group + * @param recordSerializationSchema Record serialization schema + * @param The type of data to sink + * @return an instance of KafkaSink + */ + private static KafkaSink createKafkaSink(Properties kafkaProperties, KafkaRecordSerializationSchema recordSerializationSchema) { + return KafkaSink.builder() + .setBootstrapServers(kafkaProperties.getProperty("bootstrap.servers")) + .setKafkaProducerConfig(kafkaProperties) + .setRecordSerializer(recordSerializationSchema) + .setDeliveryGuarantee(DeliveryGuarantee.AT_LEAST_ONCE) + .build(); + } + + public static void main(String[] args) throws Exception { + // Set up the streaming execution environment + final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); + + // Allows Flink to reuse objects across forwarded operators, as opposed to do a deep copy + // (this is safe because record objects are never mutated or passed by reference) + env.getConfig().enableObjectReuse(); + + LOG.info("Starting Flink Data Generator Job with conditional sinks"); + + // Load application properties + final Map applicationProperties = loadApplicationProperties(env); + LOG.info("Application properties: {}", applicationProperties); + + // Create a DataGeneratorSource that generates Stock objects using the generic method + DataGeneratorSource source = createDataGeneratorSource( + applicationProperties.get("DataGen"), + new StockPriceGeneratorFunction(), + TypeInformation.of(StockPrice.class) + ); + + // Create the data stream from the source + DataStream stockPricesStream = env.fromSource( + source, + WatermarkStrategy.noWatermarks(), + "Data Generator" + ).uid("data generator"); + + // Add a passthrough operator exposing basic metrics + var outputStream = stockPricesStream.map(new MetricEmitterNoOpMap<>()).uid("metric-emitter"); + + + // Check if at least one sink is configured + Properties kinesisProperties = applicationProperties.get("KinesisSink"); + Properties kafkaProperties = applicationProperties.get("KafkaSink"); + boolean hasKinesisSink = kinesisProperties != null; + boolean hasKafkaSink = kafkaProperties != null; + + if (!hasKinesisSink && !hasKafkaSink) { + throw new IllegalArgumentException( + "At least one sink must be configured. Please provide either 'KinesisSink' or 'KafkaSink' configuration group."); + } + + // Create Kinesis sink with JSON serialization (only if configured) + if (hasKinesisSink) { + PartitionKeyGenerator partitionKeyGenerator = (record) -> String.valueOf(record.getTicker()); + KinesisStreamsSink kinesisSink = createKinesisSink( + kinesisProperties, + // Serialize the Kinesis record as JSON + new JsonSerializationSchema<>(), + // Shard by `ticker` + partitionKeyGenerator + ); + outputStream.sinkTo(kinesisSink).uid("kinesis-sink").disableChaining(); + LOG.info("Kinesis sink configured"); + } + + // Create Kafka sink with JSON serialization (only if configured) + if (hasKafkaSink) { + String kafkaTopic = Preconditions.checkNotNull(StringUtils.trimToNull(kafkaProperties.getProperty("topic")), "Kafka topic not defined"); + SerializationSchema valueSerializationSchema = new JsonSerializationSchema<>(); + SerializationSchema keySerializationSchema = (stockPrice) -> stockPrice.getTicker().getBytes(); + KafkaRecordSerializationSchema kafkaRecordSerializationSchema = + KafkaRecordSerializationSchema.builder() + .setTopic(kafkaTopic) + // Serialize the Kafka record value (payload) as JSON + .setValueSerializationSchema(valueSerializationSchema) + // Partition by `ticker` + .setKeySerializationSchema(keySerializationSchema) + .build(); + + KafkaSink kafkaSink = createKafkaSink(kafkaProperties, kafkaRecordSerializationSchema); + outputStream.sinkTo(kafkaSink).uid("kafka-sink").disableChaining(); + LOG.info("Kafka sink configured"); + } + + // Execute the job + env.execute("Flink Data Generator Job"); + } +} diff --git a/java/FlinkDataGenerator/src/main/java/com/amazonaws/services/msf/MetricEmitterNoOpMap.java b/java/FlinkDataGenerator/src/main/java/com/amazonaws/services/msf/MetricEmitterNoOpMap.java new file mode 100644 index 00000000..c441b3b3 --- /dev/null +++ b/java/FlinkDataGenerator/src/main/java/com/amazonaws/services/msf/MetricEmitterNoOpMap.java @@ -0,0 +1,52 @@ +package com.amazonaws.services.msf; + +import org.apache.flink.api.common.functions.OpenContext; +import org.apache.flink.api.common.functions.RichMapFunction; +import org.apache.flink.dropwizard.metrics.DropwizardMeterWrapper; +import org.apache.flink.metrics.Counter; +import org.apache.flink.metrics.Gauge; +import org.apache.flink.metrics.Meter; + +/** + * No-op Map function exposing 3 custom metrics: generatedRecordCount, generatedRecordRatePerParallelism, and taskParallelism. + * Note each subtask emits its own metrics. + */ +public class MetricEmitterNoOpMap extends RichMapFunction { + private transient Counter recordCounter; + private transient int taskParallelism = 0; + private transient Meter recordMeter; + + @Override + public void open(OpenContext openContext) throws Exception { + this.recordCounter = getRuntimeContext() + .getMetricGroup() + .addGroup("kinesisAnalytics") // Automatically export metric to CloudWatch + .counter("generatedRecordCount"); + + this.recordMeter = getRuntimeContext() + .getMetricGroup() + .addGroup("kinesisAnalytics") // Automatically export metric to CloudWatch + .meter("generatedRecordRatePerParallelism", new DropwizardMeterWrapper(new com.codahale.metrics.Meter())); + + getRuntimeContext() + .getMetricGroup() + .addGroup("kinesisAnalytics") // Automatically export metric to CloudWatch + .gauge("taskParallelism", new Gauge() { + @Override + public Integer getValue() { + return taskParallelism; + } + }); + + // Capture the task parallelism + taskParallelism = getRuntimeContext().getTaskInfo().getNumberOfParallelSubtasks(); + } + + @Override + public T map(T record) throws Exception { + recordCounter.inc(); + recordMeter.markEvent(); + + return record; + } +} diff --git a/java/FlinkDataGenerator/src/main/java/com/amazonaws/services/msf/domain/StockPrice.java b/java/FlinkDataGenerator/src/main/java/com/amazonaws/services/msf/domain/StockPrice.java new file mode 100644 index 00000000..f1590365 --- /dev/null +++ b/java/FlinkDataGenerator/src/main/java/com/amazonaws/services/msf/domain/StockPrice.java @@ -0,0 +1,70 @@ +package com.amazonaws.services.msf.domain; + +import com.fasterxml.jackson.annotation.JsonProperty; + +import java.util.Objects; + +public class StockPrice { + // This annotation as well as the associated jackson2 import is needed to correctly map the JSON input key to the + // appropriate POJO property name to ensure event_time isn't missed in serialization and deserialization + @JsonProperty("event_time") + private String eventTime; + private String ticker; + private float price; + + public StockPrice() {} + + public StockPrice(String eventTime, String ticker, float price) { + this.eventTime = eventTime; + this.ticker = ticker; + this.price = price; + } + + public String getEventTime() { + return eventTime; + } + + public void setEventTime(String eventTime) { + this.eventTime = eventTime; + } + + public String getTicker() { + return ticker; + } + + public void setTicker(String ticker) { + this.ticker = ticker; + } + + public float getPrice() { + return price; + } + + public void setPrice(float price) { + this.price = price; + } + + @Override + public boolean equals(Object o) { + if (this == o) return true; + if (o == null || getClass() != o.getClass()) return false; + StockPrice stock = (StockPrice) o; + return Float.compare(stock.price, price) == 0 && + Objects.equals(eventTime, stock.eventTime) && + Objects.equals(ticker, stock.ticker); + } + + @Override + public int hashCode() { + return Objects.hash(eventTime, ticker, price); + } + + @Override + public String toString() { + return "Stock{" + + "event_time='" + eventTime + '\'' + + ", ticker='" + ticker + '\'' + + ", price=" + price + + '}'; + } +} diff --git a/java/FlinkDataGenerator/src/main/java/com/amazonaws/services/msf/domain/StockPriceGeneratorFunction.java b/java/FlinkDataGenerator/src/main/java/com/amazonaws/services/msf/domain/StockPriceGeneratorFunction.java new file mode 100644 index 00000000..cb32db2f --- /dev/null +++ b/java/FlinkDataGenerator/src/main/java/com/amazonaws/services/msf/domain/StockPriceGeneratorFunction.java @@ -0,0 +1,59 @@ +package com.amazonaws.services.msf.domain; + +import org.apache.flink.connector.datagen.source.GeneratorFunction; + +import java.time.LocalDateTime; +import java.time.format.DateTimeFormatter; +import java.util.Random; + +/** + * Generator function that creates random Stock objects. + * Implements GeneratorFunction to work with DataGeneratorSource. + * + * Modify this class to generate a different record type + */ +public class StockPriceGeneratorFunction implements GeneratorFunction { + // Stock tickers to randomly choose from (same as Python data generator) + private static final String[] TICKERS = { + "AAPL", + "MSFT", + "AMZN", + "GOOGL", + "META", + "NVDA", + "TSLA", + "INTC", + "ADBE", + "NFLX", + "PYPL", + "CSCO", + "PEP", + "AVGO", + "AMD", + "COST", + "QCOM", + "AMGN", + "SBUX", + "BKNG" + }; + + // Random number generator + private static final Random RANDOM = new Random(); + + // Date formatter for ISO format timestamps + private static final DateTimeFormatter ISO_FORMATTER = DateTimeFormatter.ISO_LOCAL_DATE_TIME; + + @Override + public StockPrice map(Long value) throws Exception { + // Generate current timestamp in ISO format + String eventTime = LocalDateTime.now().format(ISO_FORMATTER); + + // Randomly select a ticker + String ticker = TICKERS[RANDOM.nextInt(TICKERS.length)]; + + // Generate random price between 0 and 100, rounded to 2 decimal places + float price = Math.round(RANDOM.nextFloat() * 100 * 100.0f) / 100.0f; + + return new StockPrice(eventTime, ticker, price); + } +} diff --git a/java/FlinkDataGenerator/src/main/resources/flink-application-properties-dev.json b/java/FlinkDataGenerator/src/main/resources/flink-application-properties-dev.json new file mode 100644 index 00000000..03a4ae16 --- /dev/null +++ b/java/FlinkDataGenerator/src/main/resources/flink-application-properties-dev.json @@ -0,0 +1,22 @@ +[ + { + "PropertyGroupId": "DataGen", + "PropertyMap": { + "records.per.second": "1000" + } + }, + { + "PropertyGroupId": "KinesisSink", + "PropertyMap": { + "stream.arn": "arn:aws:kinesis:eu-west-1::stream/FlinkDataGeneratorTestStream", + "aws.region": "eu-west-1" + } + }, + { + "PropertyGroupId": "KafkaSink-DISABLE", + "PropertyMap": { + "bootstrap.servers": "localhost:9092", + "topic": "stock-prices" + } + } +] diff --git a/java/FlinkDataGenerator/src/main/resources/log4j2.properties b/java/FlinkDataGenerator/src/main/resources/log4j2.properties new file mode 100644 index 00000000..95034135 --- /dev/null +++ b/java/FlinkDataGenerator/src/main/resources/log4j2.properties @@ -0,0 +1,16 @@ +# Set to debug or trace if log4j initialization is failing +status = warn + +# Name of the configuration +name = ConsoleLogConfigDemo + +# Console appender configuration +appender.console.type = Console +appender.console.name = consoleLogger +appender.console.layout.type = PatternLayout +appender.console.layout.pattern = %d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n + +# Root logger level +rootLogger.level = info +# Root logger referring to console appender +rootLogger.appenderRef.stdout.ref = consoleLogger diff --git a/java/FlinkDataGenerator/src/test/java/com/amazonaws/services/msf/DataGeneratorJobTest.java b/java/FlinkDataGenerator/src/test/java/com/amazonaws/services/msf/DataGeneratorJobTest.java new file mode 100644 index 00000000..61cd802b --- /dev/null +++ b/java/FlinkDataGenerator/src/test/java/com/amazonaws/services/msf/DataGeneratorJobTest.java @@ -0,0 +1,150 @@ +package com.amazonaws.services.msf; + +import com.amazonaws.services.msf.domain.StockPrice; +import com.amazonaws.services.msf.domain.StockPriceGeneratorFunction; +import org.apache.flink.api.common.typeinfo.TypeInformation; +import org.apache.flink.connector.datagen.source.DataGeneratorSource; +import org.apache.flink.connector.datagen.source.GeneratorFunction; +import org.apache.flink.connector.kafka.sink.KafkaRecordSerializationSchema; +import org.apache.flink.connector.kafka.sink.KafkaSink; +import org.junit.Test; +import static org.junit.Assert.*; +import java.util.Properties; +import java.util.HashMap; +import java.util.Map; +import java.lang.reflect.Method; + +public class DataGeneratorJobTest { + + @Test + public void testCreateDataGeneratorSource() throws Exception { + // Use reflection to test the private createDataGeneratorSource method + Method createDataGeneratorSourceMethod = DataGeneratorJob.class.getDeclaredMethod( + "createDataGeneratorSource", Properties.class, GeneratorFunction.class, TypeInformation.class); + createDataGeneratorSourceMethod.setAccessible(true); + + // Test with valid configuration + Properties dataGenProps = new Properties(); + dataGenProps.setProperty("records.per.second", "15"); + + StockPriceGeneratorFunction generatorFunction = new StockPriceGeneratorFunction(); + TypeInformation typeInfo = TypeInformation.of(StockPrice.class); + + DataGeneratorSource source = (DataGeneratorSource) createDataGeneratorSourceMethod.invoke( + null, dataGenProps, generatorFunction, typeInfo); + + assertNotNull("DataGeneratorSource should not be null", source); + + // Test with null properties (should use default rate) + source = (DataGeneratorSource) createDataGeneratorSourceMethod.invoke( + null, null, generatorFunction, typeInfo); + + assertNotNull("DataGeneratorSource should not be null with null properties", source); + + // Test with empty properties (should use default rate) + Properties emptyProps = new Properties(); + source = (DataGeneratorSource) createDataGeneratorSourceMethod.invoke( + null, emptyProps, generatorFunction, typeInfo); + + assertNotNull("DataGeneratorSource should not be null with empty properties", source); + } + + @Test + public void testCreateKafkaSink() throws Exception { + // Use reflection to test the private createKafkaSink method + Method createKafkaSinkMethod = DataGeneratorJob.class.getDeclaredMethod( + "createKafkaSink", Properties.class, KafkaRecordSerializationSchema.class); + createKafkaSinkMethod.setAccessible(true); + + // Test with valid Kafka properties + Properties kafkaProps = new Properties(); + kafkaProps.setProperty("bootstrap.servers", "localhost:9092"); + kafkaProps.setProperty("topic", "test-topic"); + + // Create a mock KafkaRecordSerializationSchema + KafkaRecordSerializationSchema recordSerializationSchema = + KafkaRecordSerializationSchema.builder() + .setTopic("test-topic") + .setKeySerializationSchema(stock -> stock.getTicker().getBytes()) + .setValueSerializationSchema(new org.apache.flink.formats.json.JsonSerializationSchema<>()) + .build(); + + KafkaSink kafkaSink = (KafkaSink) createKafkaSinkMethod.invoke( + null, kafkaProps, recordSerializationSchema); + + assertNotNull("KafkaSink should not be null", kafkaSink); + } + + @Test + public void testKafkaPartitioningKey() { + // Test that ticker symbol can be used as Kafka partition key + StockPrice stock1 = new StockPrice("2024-01-15T10:30:45", "AAPL", 150.25f); + StockPrice stock2 = new StockPrice("2024-01-15T10:30:46", "MSFT", 200.50f); + + // Test that ticker can be converted to bytes for Kafka key + byte[] key1 = stock1.getTicker().getBytes(); + byte[] key2 = stock2.getTicker().getBytes(); + + assertNotNull("Kafka key should not be null", key1); + assertNotNull("Kafka key should not be null", key2); + assertTrue("Kafka key should not be empty", key1.length > 0); + assertTrue("Kafka key should not be empty", key2.length > 0); + + // Test that different tickers produce different keys + assertFalse("Different tickers should produce different keys", + java.util.Arrays.equals(key1, key2)); + + // Test that same ticker produces same key + StockPrice stock3 = new StockPrice("2024-01-15T10:30:47", "AAPL", 175.50f); + byte[] key3 = stock3.getTicker().getBytes(); + assertTrue("Same ticker should produce same key", + java.util.Arrays.equals(key1, key3)); + } + + @Test + public void testConditionalSinkValidation() { + // Test that the application validates sink configuration properly + Map appProperties = new HashMap<>(); + + // Test with no sinks configured - should be invalid + boolean hasKinesis = appProperties.get("KinesisSink") != null; + boolean hasKafka = appProperties.get("KafkaSink") != null; + assertFalse("Should not have Kinesis sink when not configured", hasKinesis); + assertFalse("Should not have Kafka sink when not configured", hasKafka); + assertTrue("Should require at least one sink", !hasKinesis && !hasKafka); + + // Test with only Kinesis configured - should be valid + Properties kinesisProps = new Properties(); + kinesisProps.setProperty("stream.arn", "test-arn"); + kinesisProps.setProperty("aws.region", "us-east-1"); + appProperties.put("KinesisSink", kinesisProps); + + hasKinesis = appProperties.get("KinesisSink") != null; + hasKafka = appProperties.get("KafkaSink") != null; + assertTrue("Should have Kinesis sink when configured", hasKinesis); + assertFalse("Should not have Kafka sink when not configured", hasKafka); + assertTrue("Should be valid with one sink", hasKinesis || hasKafka); + + // Test with only Kafka configured - should be valid + appProperties.clear(); + Properties kafkaProps = new Properties(); + kafkaProps.setProperty("bootstrap.servers", "localhost:9092"); + kafkaProps.setProperty("topic", "test-topic"); + appProperties.put("KafkaSink", kafkaProps); + + hasKinesis = appProperties.get("KinesisSink") != null; + hasKafka = appProperties.get("KafkaSink") != null; + assertFalse("Should not have Kinesis sink when not configured", hasKinesis); + assertTrue("Should have Kafka sink when configured", hasKafka); + assertTrue("Should be valid with one sink", hasKinesis || hasKafka); + + // Test with both configured - should be valid + appProperties.put("KinesisSink", kinesisProps); + + hasKinesis = appProperties.get("KinesisSink") != null; + hasKafka = appProperties.get("KafkaSink") != null; + assertTrue("Should have Kinesis sink when configured", hasKinesis); + assertTrue("Should have Kafka sink when configured", hasKafka); + assertTrue("Should be valid with both sinks", hasKinesis || hasKafka); + } +} diff --git a/java/FlinkDataGenerator/src/test/java/com/amazonaws/services/msf/domain/StockPriceGeneratorFunctionTest.java b/java/FlinkDataGenerator/src/test/java/com/amazonaws/services/msf/domain/StockPriceGeneratorFunctionTest.java new file mode 100644 index 00000000..a1dc83c5 --- /dev/null +++ b/java/FlinkDataGenerator/src/test/java/com/amazonaws/services/msf/domain/StockPriceGeneratorFunctionTest.java @@ -0,0 +1,77 @@ +package com.amazonaws.services.msf.domain; + +import org.junit.Test; +import static org.junit.Assert.*; +import java.util.Arrays; +import java.util.List; + +public class StockPriceGeneratorFunctionTest { + private static final String[] TICKERS = { + "AAPL", + "MSFT", + "AMZN", + "GOOGL", + "META", + "NVDA", + "TSLA", + "INTC", + "ADBE", + "NFLX", + "PYPL", + "CSCO", + "PEP", + "AVGO", + "AMD", + "COST", + "QCOM", + "AMGN", + "SBUX", + "BKNG" + }; + + @Test + public void testStockGeneratorFunction() throws Exception { + StockPriceGeneratorFunction generator = new StockPriceGeneratorFunction(); + + // Generate a stock record + StockPrice stock = generator.map(1L); + + // Verify the stock is not null + assertNotNull(stock); + + // Verify event_time is not null and not empty + assertNotNull(stock.getEventTime()); + assertFalse(stock.getEventTime().isEmpty()); + + // Verify ticker is one of the expected values + List expectedTickers = Arrays.asList(TICKERS); + assertTrue("Ticker should be one of the expected values", + expectedTickers.contains(stock.getTicker())); + + // Verify price is within expected range (0 to 100) + assertTrue("Price should be >= 0", stock.getPrice() >= 0); + assertTrue("Price should be <= 100", stock.getPrice() <= 100); + + // Verify price has at most 2 decimal places + String priceStr = String.valueOf(stock.getPrice()); + int decimalIndex = priceStr.indexOf('.'); + if (decimalIndex != -1) { + int decimalPlaces = priceStr.length() - decimalIndex - 1; + assertTrue("Price should have at most 2 decimal places", decimalPlaces <= 2); + } + } + + @Test + public void testMultipleGenerations() throws Exception { + StockPriceGeneratorFunction generator = new StockPriceGeneratorFunction(); + + // Generate multiple records to ensure randomness + for (int i = 0; i < 10; i++) { + StockPrice stock = generator.map((long) i); + assertNotNull(stock); + assertNotNull(stock.getEventTime()); + assertNotNull(stock.getTicker()); + assertTrue(stock.getPrice() >= 0 && stock.getPrice() <= 100); + } + } +} diff --git a/java/FlinkDataGenerator/src/test/java/com/amazonaws/services/msf/domain/StockPriceTest.java b/java/FlinkDataGenerator/src/test/java/com/amazonaws/services/msf/domain/StockPriceTest.java new file mode 100644 index 00000000..32f6c133 --- /dev/null +++ b/java/FlinkDataGenerator/src/test/java/com/amazonaws/services/msf/domain/StockPriceTest.java @@ -0,0 +1,64 @@ +package com.amazonaws.services.msf.domain; + +import org.junit.Test; +import static org.junit.Assert.*; + +public class StockPriceTest { + + @Test + public void testStockCreation() { + StockPrice stock = new StockPrice("2024-01-15T10:30:45", "AAPL", 150.25f); + + assertEquals("2024-01-15T10:30:45", stock.getEventTime()); + assertEquals("AAPL", stock.getTicker()); + assertEquals(150.25f, stock.getPrice(), 0.001); + } + + @Test + public void testStockToString() { + StockPrice stock = new StockPrice("2024-01-15T10:30:45", "AAPL", 150.25f); + String expected = "Stock{event_time='2024-01-15T10:30:45', ticker='AAPL', price=150.25}"; + assertEquals(expected, stock.toString()); + } + + @Test + public void testStockSetters() { + StockPrice stock = new StockPrice(); + stock.setEventTime("2024-01-15T10:30:45"); + stock.setTicker("MSFT"); + stock.setPrice(200.50f); + + assertEquals("2024-01-15T10:30:45", stock.getEventTime()); + assertEquals("MSFT", stock.getTicker()); + assertEquals(200.50f, stock.getPrice(), 0.001); + } + + @Test + public void testStockHashCodeForPartitioning() { + // Create test stock objects + StockPrice stock1 = new StockPrice("2024-01-15T10:30:45", "AAPL", 150.25f); + StockPrice stock2 = new StockPrice("2024-01-15T10:30:46", "MSFT", 200.50f); + StockPrice stock3 = new StockPrice("2024-01-15T10:30:45", "AAPL", 150.25f); // Same as stock1 + + // Test that hashCode is consistent for equal objects + assertEquals("Equal stock objects should have same hashCode", + stock1.hashCode(), stock3.hashCode()); + + // Test that equals works correctly + assertEquals("Same stock objects should be equal", stock1, stock3); + assertNotEquals("Different stock objects should not be equal", stock1, stock2); + + // Test that different stocks likely have different hashCodes + assertNotEquals("Different stock objects should likely have different hashCodes", + stock1.hashCode(), stock2.hashCode()); + + // Test that hashCode can be used as partition key (should not throw exception) + String partitionKey1 = String.valueOf(stock1.hashCode()); + String partitionKey2 = String.valueOf(stock2.hashCode()); + + assertNotNull("Partition key should not be null", partitionKey1); + assertNotNull("Partition key should not be null", partitionKey2); + assertFalse("Partition key should not be empty", partitionKey1.isEmpty()); + assertFalse("Partition key should not be empty", partitionKey2.isEmpty()); + } +} diff --git a/java/FlinkDataGenerator/src/test/resources/flink-application-properties-kafka-only.json b/java/FlinkDataGenerator/src/test/resources/flink-application-properties-kafka-only.json new file mode 100644 index 00000000..477b7113 --- /dev/null +++ b/java/FlinkDataGenerator/src/test/resources/flink-application-properties-kafka-only.json @@ -0,0 +1,15 @@ +[ + { + "PropertyGroupId": "DataGen", + "PropertyMap": { + "records.per.second": "5" + } + }, + { + "PropertyGroupId": "KafkaSink", + "PropertyMap": { + "bootstrap.servers": "localhost:9092", + "topic": "test-topic" + } + } +] diff --git a/java/FlinkDataGenerator/src/test/resources/flink-application-properties-kinesis-only.json b/java/FlinkDataGenerator/src/test/resources/flink-application-properties-kinesis-only.json new file mode 100644 index 00000000..7cea8f46 --- /dev/null +++ b/java/FlinkDataGenerator/src/test/resources/flink-application-properties-kinesis-only.json @@ -0,0 +1,15 @@ +[ + { + "PropertyGroupId": "DataGen", + "PropertyMap": { + "records.per.second": "5" + } + }, + { + "PropertyGroupId": "KinesisSink", + "PropertyMap": { + "stream.arn": "arn:aws:kinesis:us-east-1:123456789012:stream/test-stream", + "aws.region": "us-east-1" + } + } +] diff --git a/java/FlinkDataGenerator/src/test/resources/flink-application-properties-no-sinks.json b/java/FlinkDataGenerator/src/test/resources/flink-application-properties-no-sinks.json new file mode 100644 index 00000000..d44e034a --- /dev/null +++ b/java/FlinkDataGenerator/src/test/resources/flink-application-properties-no-sinks.json @@ -0,0 +1,8 @@ +[ + { + "PropertyGroupId": "DataGen", + "PropertyMap": { + "records.per.second": "5" + } + } +] diff --git a/java/FlinkDataGenerator/tools/dashboard-cfn.yaml b/java/FlinkDataGenerator/tools/dashboard-cfn.yaml new file mode 100644 index 00000000..8097940c --- /dev/null +++ b/java/FlinkDataGenerator/tools/dashboard-cfn.yaml @@ -0,0 +1,578 @@ +AWSTemplateFormatVersion: '2010-09-09' +Description: 'CloudWatch Dashboard for Flink Data Generator with conditional Kinesis and Kafka widgets' + +Metadata: + AWS::CloudFormation::Interface: + ParameterGroups: + - Label: + default: "Dashboard Configuration" + Parameters: + - DashboardName + - Application + - Region + - Label: + default: "Kinesis Configuration" + Parameters: + - StreamName + - Label: + default: "MSK Configuration" + Parameters: + - ClusterName + - Topic + ParameterLabels: + DashboardName: + default: "Dashboard Name" + Application: + default: "Application Name" + Region: + default: "AWS Region" + StreamName: + default: "Kinesis Stream Name" + ClusterName: + default: "MSK Cluster Name" + Topic: + default: "Kafka Topic Name" + +Parameters: + DashboardName: + Type: String + Default: FlinkDataGenerator + Description: Name of the CloudWatch Dashboard + + Application: + Type: String + Default: FlinkDataGenerator + Description: Name of the Flink application + + Region: + Type: String + Default: eu-west-1 + Description: AWS region where resources are deployed + AllowedValues: + - us-east-1 + - us-east-2 + - us-west-1 + - us-west-2 + - eu-west-1 + - eu-west-2 + - eu-west-3 + - eu-central-1 + - ap-northeast-1 + - ap-northeast-2 + - ap-southeast-1 + - ap-southeast-2 + - ap-south-1 + - sa-east-1 + + StreamName: + Type: String + Default: '' + Description: Name of the Kinesis Data Stream (leave empty to skip Kinesis widget) + + ClusterName: + Type: String + Default: '' + Description: Name of the MSK (Kafka) cluster (leave empty to skip Kafka widget) + + Topic: + Type: String + Default: '' + Description: Name of the Kafka topic (leave empty to skip Kafka widget) + +Conditions: + HasKafkaConfig: !And + - !Not [!Equals [!Ref ClusterName, '']] + - !Not [!Equals [!Ref Topic, '']] + + HasKinesisConfig: !Not [!Equals [!Ref StreamName, '']] + + HasBothSinks: !And + - !Condition HasKafkaConfig + - !Condition HasKinesisConfig + + HasOnlyKafka: !And + - !Condition HasKafkaConfig + - !Not [!Condition HasKinesisConfig] + + HasOnlyKinesis: !And + - !Condition HasKinesisConfig + - !Not [!Condition HasKafkaConfig] + +Resources: + # Dashboard with both Kafka and Kinesis widgets + DashboardWithBothSinks: + Type: AWS::CloudWatch::Dashboard + Condition: HasBothSinks + Properties: + DashboardName: !Ref DashboardName + DashboardBody: !Sub | + { + "widgets": [ + { + "type": "metric", + "x": 0, + "y": 0, + "width": 8, + "height": 6, + "properties": { + "metrics": [ + [{"expression": "m1 / 4", "label": "globalGeneratedRecordCount", "id": "e1", "region": "${Region}"}], + [{"expression": "m2 * m3", "label": "globalGeneratedRecordsPerSec", "id": "e2", "yAxis": "right", "region": "${Region}"}], + ["AWS/KinesisAnalytics", "generatedRecordCount", "Application", "${Application}", {"stat": "Sum", "id": "m1", "visible": false, "region": "${Region}"}], + [".", "generatedRecordRatePerParallelism", ".", ".", {"id": "m2", "visible": false, "region": "${Region}"}], + [".", "taskParallelism", ".", ".", {"id": "m3", "visible": false, "region": "${Region}"}] + ], + "view": "timeSeries", + "stacked": false, + "region": "${Region}", + "stat": "Average", + "period": 60, + "title": "Data Generator", + "yAxis": { + "left": {"label": "Record count", "showUnits": false}, + "right": {"label": "Record/sec", "showUnits": false} + } + } + }, + { + "type": "metric", + "x": 8, + "y": 0, + "width": 8, + "height": 6, + "properties": { + "metrics": [ + [{"expression": "m1+m2+m3+m4+m5+m6", "label": "Total bytesInPerSec", "id": "e1", "yAxis": "right", "region": "${Region}"}], + ["AWS/Kafka", "BytesInPerSec", "Cluster Name", "${ClusterName}", "Broker ID", "1", "Topic", "${Topic}", {"yAxis": "right", "region": "${Region}", "id": "m1", "visible": false}], + ["...", "2", ".", ".", {"yAxis": "right", "region": "${Region}", "id": "m2", "visible": false}], + ["...", "3", ".", ".", {"yAxis": "right", "region": "${Region}", "id": "m3", "visible": false}], + ["...", "4", ".", ".", {"yAxis": "right", "region": "${Region}", "id": "m4", "visible": false}], + ["...", "5", ".", ".", {"yAxis": "right", "region": "${Region}", "id": "m5", "visible": false}], + ["...", "6", ".", ".", {"yAxis": "right", "region": "${Region}", "id": "m6", "visible": false}], + [{"expression": "m7+m8+m9+m10+m11+m12", "label": "Total MessagesInPerSec", "id": "e2", "region": "${Region}", "color": "#aec7e8"}], + ["AWS/Kafka", "MessagesInPerSec", "Cluster Name", "${ClusterName}", "Broker ID", "1", "Topic", "${Topic}", {"region": "${Region}", "id": "m7", "visible": false}], + ["...", "2", ".", ".", {"region": "${Region}", "id": "m8", "visible": false}], + ["...", "3", ".", ".", {"region": "${Region}", "id": "m9", "visible": false}], + ["...", "4", ".", ".", {"region": "${Region}", "id": "m10", "visible": false}], + ["...", "5", ".", ".", {"region": "${Region}", "id": "m11", "visible": false}], + ["...", "6", ".", ".", {"region": "${Region}", "id": "m12", "visible": false}] + ], + "view": "timeSeries", + "stacked": false, + "region": "${Region}", + "period": 60, + "yAxis": { + "left": {"label": "Messages/Sec", "showUnits": false, "min": 0}, + "right": {"label": "Bytes/Sec", "showUnits": false, "min": 0} + }, + "title": "Kafka output", + "stat": "Average" + } + }, + { + "type": "metric", + "x": 16, + "y": 0, + "width": 8, + "height": 6, + "properties": { + "metrics": [ + [{"expression": "m3 / 60", "label": "PublishedRecordsPerSec", "id": "e1", "yAxis": "right"}], + ["AWS/Kinesis", "PutRecords.ThrottledRecords", "StreamName", "${StreamName}", {"region": "${Region}", "id": "m1"}], + [".", "WriteProvisionedThroughputExceeded", ".", ".", {"id": "m2", "region": "${Region}", "stat": "Average"}], + [".", "IncomingRecords", ".", ".", {"id": "m3", "region": "${Region}", "yAxis": "right", "stat": "Sum", "visible": false}] + ], + "view": "timeSeries", + "stacked": false, + "region": "${Region}", + "stat": "Maximum", + "period": 60, + "yAxis": { + "left": {"label": "Throttling", "showUnits": false, "min": 0}, + "right": {"label": "Records/sec", "showUnits": false, "min": 0} + }, + "title": "Kinesis Data Stream output" + } + }, + { + "type": "metric", + "x": 0, + "y": 6, + "width": 8, + "height": 5, + "properties": { + "metrics": [ + ["AWS/KinesisAnalytics", "containerMemoryUtilization", "Application", "${Application}", {"region": "${Region}", "label": "containerMemoryUtilization", "id": "m1"}], + [".", "containerCPUUtilization", ".", ".", {"yAxis": "right", "region": "${Region}", "label": "containerCPUUtilization", "id": "m2"}] + ], + "view": "timeSeries", + "stacked": false, + "region": "${Region}", + "title": "Generator resource utilization", + "stat": "Average", + "period": 60, + "yAxis": { + "left": {"label": "Mem (%)", "showUnits": false, "min": 0, "max": 100}, + "right": {"label": "CPU (%)", "showUnits": false, "min": 0, "max": 100} + }, + "annotations": { + "horizontal": [{"label": "Threshold", "value": 90}] + } + } + }, + { + "type": "metric", + "x": 8, + "y": 6, + "width": 8, + "height": 5, + "properties": { + "view": "timeSeries", + "stacked": false, + "metrics": [ + ["AWS/KinesisAnalytics", "busyTimeMsPerSecond", "Application", "${Application}", {"region": "${Region}"}], + [".", "backPressuredTimeMsPerSecond", ".", ".", {"region": "${Region}"}], + [".", "idleTimeMsPerSecond", ".", ".", {"region": "${Region}"}] + ], + "region": "${Region}", + "title": "Generator application busy-ness", + "yAxis": { + "left": {"label": "1/1000", "showUnits": false, "min": 0, "max": 1000}, + "right": {"label": ""} + }, + "period": 300, + "liveData": true + } + }, + { + "type": "metric", + "x": 16, + "y": 6, + "width": 8, + "height": 5, + "properties": { + "metrics": [ + ["AWS/KinesisAnalytics", "uptime", "Application", "${Application}", {"region": "${Region}"}], + [".", "fullRestarts", ".", ".", {"yAxis": "right", "region": "${Region}"}] + ], + "view": "timeSeries", + "stacked": false, + "region": "${Region}", + "stat": "Maximum", + "period": 60, + "title": "Generator uptime and restart" + } + } + ] + } + # Dashboard with only Kafka widget + DashboardWithKafkaOnly: + Type: AWS::CloudWatch::Dashboard + Condition: HasOnlyKafka + Properties: + DashboardName: !Ref DashboardName + DashboardBody: !Sub | + { + "widgets": [ + { + "type": "metric", + "x": 0, + "y": 0, + "width": 12, + "height": 6, + "properties": { + "metrics": [ + [{"expression": "m1 / 4", "label": "globalGeneratedRecordCount", "id": "e1", "region": "${Region}"}], + [{"expression": "m2 * m3", "label": "globalGeneratedRecordsPerSec", "id": "e2", "yAxis": "right", "region": "${Region}"}], + ["AWS/KinesisAnalytics", "generatedRecordCount", "Application", "${Application}", {"stat": "Sum", "id": "m1", "visible": false, "region": "${Region}"}], + [".", "generatedRecordRatePerParallelism", ".", ".", {"id": "m2", "visible": false, "region": "${Region}"}], + [".", "taskParallelism", ".", ".", {"id": "m3", "visible": false, "region": "${Region}"}] + ], + "view": "timeSeries", + "stacked": false, + "region": "${Region}", + "stat": "Average", + "period": 60, + "title": "Data Generator", + "yAxis": { + "left": {"label": "Record count", "showUnits": false}, + "right": {"label": "Record/sec", "showUnits": false} + } + } + }, + { + "type": "metric", + "x": 12, + "y": 0, + "width": 12, + "height": 6, + "properties": { + "metrics": [ + [{"expression": "m1+m2+m3+m4+m5+m6", "label": "Total bytesInPerSec", "id": "e1", "yAxis": "right", "region": "${Region}"}], + ["AWS/Kafka", "BytesInPerSec", "Cluster Name", "${ClusterName}", "Broker ID", "1", "Topic", "${Topic}", {"yAxis": "right", "region": "${Region}", "id": "m1", "visible": false}], + ["...", "2", ".", ".", {"yAxis": "right", "region": "${Region}", "id": "m2", "visible": false}], + ["...", "3", ".", ".", {"yAxis": "right", "region": "${Region}", "id": "m3", "visible": false}], + ["...", "4", ".", ".", {"yAxis": "right", "region": "${Region}", "id": "m4", "visible": false}], + ["...", "5", ".", ".", {"yAxis": "right", "region": "${Region}", "id": "m5", "visible": false}], + ["...", "6", ".", ".", {"yAxis": "right", "region": "${Region}", "id": "m6", "visible": false}], + [{"expression": "m7+m8+m9+m10+m11+m12", "label": "Total MessagesInPerSec", "id": "e2", "region": "${Region}", "color": "#aec7e8"}], + ["AWS/Kafka", "MessagesInPerSec", "Cluster Name", "${ClusterName}", "Broker ID", "1", "Topic", "${Topic}", {"region": "${Region}", "id": "m7", "visible": false}], + ["...", "2", ".", ".", {"region": "${Region}", "id": "m8", "visible": false}], + ["...", "3", ".", ".", {"region": "${Region}", "id": "m9", "visible": false}], + ["...", "4", ".", ".", {"region": "${Region}", "id": "m10", "visible": false}], + ["...", "5", ".", ".", {"region": "${Region}", "id": "m11", "visible": false}], + ["...", "6", ".", ".", {"region": "${Region}", "id": "m12", "visible": false}] + ], + "view": "timeSeries", + "stacked": false, + "region": "${Region}", + "period": 60, + "yAxis": { + "left": {"label": "Messages/Sec", "showUnits": false, "min": 0}, + "right": {"label": "Bytes/Sec", "showUnits": false, "min": 0} + }, + "title": "Kafka output", + "stat": "Average" + } + }, + { + "type": "metric", + "x": 0, + "y": 6, + "width": 8, + "height": 5, + "properties": { + "metrics": [ + ["AWS/KinesisAnalytics", "containerMemoryUtilization", "Application", "${Application}", {"region": "${Region}", "label": "containerMemoryUtilization", "id": "m1"}], + [".", "containerCPUUtilization", ".", ".", {"yAxis": "right", "region": "${Region}", "label": "containerCPUUtilization", "id": "m2"}] + ], + "view": "timeSeries", + "stacked": false, + "region": "${Region}", + "title": "Generator resource utilization", + "stat": "Average", + "period": 60, + "yAxis": { + "left": {"label": "Mem (%)", "showUnits": false, "min": 0, "max": 100}, + "right": {"label": "CPU (%)", "showUnits": false, "min": 0, "max": 100} + }, + "annotations": { + "horizontal": [{"label": "Threshold", "value": 90}] + } + } + }, + { + "type": "metric", + "x": 8, + "y": 6, + "width": 8, + "height": 5, + "properties": { + "view": "timeSeries", + "stacked": false, + "metrics": [ + ["AWS/KinesisAnalytics", "busyTimeMsPerSecond", "Application", "${Application}", {"region": "${Region}"}], + [".", "backPressuredTimeMsPerSecond", ".", ".", {"region": "${Region}"}], + [".", "idleTimeMsPerSecond", ".", ".", {"region": "${Region}"}] + ], + "region": "${Region}", + "title": "Generator application busy-ness", + "yAxis": { + "left": {"label": "1/1000", "showUnits": false, "min": 0, "max": 1000}, + "right": {"label": ""} + }, + "period": 300, + "liveData": true + } + }, + { + "type": "metric", + "x": 16, + "y": 6, + "width": 8, + "height": 5, + "properties": { + "metrics": [ + ["AWS/KinesisAnalytics", "uptime", "Application", "${Application}", {"region": "${Region}"}], + [".", "fullRestarts", ".", ".", {"yAxis": "right", "region": "${Region}"}] + ], + "view": "timeSeries", + "stacked": false, + "region": "${Region}", + "stat": "Maximum", + "period": 60, + "title": "Generator uptime and restart" + } + } + ] + } + # Dashboard with only Kinesis widget + DashboardWithKinesisOnly: + Type: AWS::CloudWatch::Dashboard + Condition: HasOnlyKinesis + Properties: + DashboardName: !Ref DashboardName + DashboardBody: !Sub | + { + "widgets": [ + { + "type": "metric", + "x": 0, + "y": 0, + "width": 12, + "height": 6, + "properties": { + "metrics": [ + [{"expression": "m1 / 4", "label": "globalGeneratedRecordCount", "id": "e1", "region": "${Region}"}], + [{"expression": "m2 * m3", "label": "globalGeneratedRecordsPerSec", "id": "e2", "yAxis": "right", "region": "${Region}"}], + ["AWS/KinesisAnalytics", "generatedRecordCount", "Application", "${Application}", {"stat": "Sum", "id": "m1", "visible": false, "region": "${Region}"}], + [".", "generatedRecordRatePerParallelism", ".", ".", {"id": "m2", "visible": false, "region": "${Region}"}], + [".", "taskParallelism", ".", ".", {"id": "m3", "visible": false, "region": "${Region}"}] + ], + "view": "timeSeries", + "stacked": false, + "region": "${Region}", + "stat": "Average", + "period": 60, + "title": "Data Generator", + "yAxis": { + "left": {"label": "Record count", "showUnits": false}, + "right": {"label": "Record/sec", "showUnits": false} + } + } + }, + { + "type": "metric", + "x": 12, + "y": 0, + "width": 12, + "height": 6, + "properties": { + "metrics": [ + [{"expression": "m3 / 60", "label": "PublishedRecordsPerSec", "id": "e1", "yAxis": "right"}], + ["AWS/Kinesis", "PutRecords.ThrottledRecords", "StreamName", "${StreamName}", {"region": "${Region}", "id": "m1"}], + [".", "WriteProvisionedThroughputExceeded", ".", ".", {"id": "m2", "region": "${Region}", "stat": "Average"}], + [".", "IncomingRecords", ".", ".", {"id": "m3", "region": "${Region}", "yAxis": "right", "stat": "Sum", "visible": false}] + ], + "view": "timeSeries", + "stacked": false, + "region": "${Region}", + "stat": "Maximum", + "period": 60, + "yAxis": { + "left": {"label": "Throttling", "showUnits": false, "min": 0}, + "right": {"label": "Records/sec", "showUnits": false, "min": 0} + }, + "title": "Kinesis Data Stream output" + } + }, + { + "type": "metric", + "x": 0, + "y": 6, + "width": 8, + "height": 5, + "properties": { + "metrics": [ + ["AWS/KinesisAnalytics", "containerMemoryUtilization", "Application", "${Application}", {"region": "${Region}", "label": "containerMemoryUtilization", "id": "m1"}], + [".", "containerCPUUtilization", ".", ".", {"yAxis": "right", "region": "${Region}", "label": "containerCPUUtilization", "id": "m2"}] + ], + "view": "timeSeries", + "stacked": false, + "region": "${Region}", + "title": "Generator resource utilization", + "stat": "Average", + "period": 60, + "yAxis": { + "left": {"label": "Mem (%)", "showUnits": false, "min": 0, "max": 100}, + "right": {"label": "CPU (%)", "showUnits": false, "min": 0, "max": 100} + }, + "annotations": { + "horizontal": [{"label": "Threshold", "value": 90}] + } + } + }, + { + "type": "metric", + "x": 8, + "y": 6, + "width": 8, + "height": 5, + "properties": { + "view": "timeSeries", + "stacked": false, + "metrics": [ + ["AWS/KinesisAnalytics", "busyTimeMsPerSecond", "Application", "${Application}", {"region": "${Region}"}], + [".", "backPressuredTimeMsPerSecond", ".", ".", {"region": "${Region}"}], + [".", "idleTimeMsPerSecond", ".", ".", {"region": "${Region}"}] + ], + "region": "${Region}", + "title": "Generator application busy-ness", + "yAxis": { + "left": {"label": "1/1000", "showUnits": false, "min": 0, "max": 1000}, + "right": {"label": ""} + }, + "period": 300, + "liveData": true + } + }, + { + "type": "metric", + "x": 16, + "y": 6, + "width": 8, + "height": 5, + "properties": { + "metrics": [ + ["AWS/KinesisAnalytics", "uptime", "Application", "${Application}", {"region": "${Region}"}], + [".", "fullRestarts", ".", ".", {"yAxis": "right", "region": "${Region}"}] + ], + "view": "timeSeries", + "stacked": false, + "region": "${Region}", + "stat": "Maximum", + "period": 60, + "title": "Generator uptime and restart" + } + } + ] + } +Outputs: + DashboardName: + Description: Name of the created CloudWatch Dashboard + Value: !If + - HasBothSinks + - !Ref DashboardWithBothSinks + - !If + - HasOnlyKafka + - !Ref DashboardWithKafkaOnly + - !If + - HasOnlyKinesis + - !Ref DashboardWithKinesisOnly + - 'No dashboard created - missing required parameters' + + DashboardURL: + Description: URL to access the CloudWatch Dashboard + Value: !If + - HasBothSinks + - !Sub 'https://${Region}.console.aws.amazon.com/cloudwatch/home?region=${Region}#dashboards:name=${DashboardName}' + - !If + - HasOnlyKafka + - !Sub 'https://${Region}.console.aws.amazon.com/cloudwatch/home?region=${Region}#dashboards:name=${DashboardName}' + - !If + - HasOnlyKinesis + - !Sub 'https://${Region}.console.aws.amazon.com/cloudwatch/home?region=${Region}#dashboards:name=${DashboardName}' + - 'No dashboard URL - missing required parameters' + + ConfigurationSummary: + Description: Summary of the dashboard configuration + Value: !If + - HasBothSinks + - 'Dashboard created with both Kafka and Kinesis widgets' + - !If + - HasOnlyKafka + - 'Dashboard created with Kafka widget only' + - !If + - HasOnlyKinesis + - 'Dashboard created with Kinesis widget only' + - 'No dashboard created - ClusterName+Topic or StreamName required' diff --git a/java/Iceberg/IcebergDataStreamSink/README.md b/java/Iceberg/IcebergDataStreamSink/README.md index 7da74d0d..e1b0f630 100644 --- a/java/Iceberg/IcebergDataStreamSink/README.md +++ b/java/Iceberg/IcebergDataStreamSink/README.md @@ -2,7 +2,7 @@ * Flink version: 1.20.0 * Flink API: DataStream API -* Iceberg 1.6.1 +* Iceberg 1.8.1 * Language: Java (11) * Flink connectors: [DataGen](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/connectors/datastream/datagen/) and [Iceberg](https://iceberg.apache.org/docs/latest/flink/) @@ -37,16 +37,15 @@ When running locally, the configuration is read from the Runtime parameters: -| Group ID | Key | Default | Description | -|-----------|--------------------------|-------------------|---------------------------------------------------------------------------------------------------------------------| -| `DataGen` | `records.per.sec` | `10.0` | Records per second generated. | -| `Iceberg` | `bucket.prefix` | (mandatory) | S3 bucket prefix, for example `s3://my-bucket/iceberg`. | -| `Iceberg` | `catalog.db` | `default` | Name of the Glue Data Catalog database. | -| `Iceberg` | `catalog.table` | `prices_iceberg` | Name of the Glue Data Catalog table. | -| `Iceberg` | `partition.fields` | `symbol` | Comma separated list of partition fields. | -| `Iceberg` | `sort.field` | `timestamp` | Sort field. | -| `Iceberg` | `operation` | `updsert` | Iceberg operation. One of `upsert`, `append` or `overwrite`. | -| `Iceberg` | `upsert.equality.fields` | `symbol` | Comma separated list of fields used for upsert. It must match partition fields. Required if `operation` = `upsert`. | +| Group ID | Key | Default | Description | +|-----------|--------------------------|------------------|---------------------------------------------------------------------------------------------------------------------| +| `DataGen` | `records.per.sec` | `10.0` | Records per second generated. | +| `Iceberg` | `bucket.prefix` | (mandatory) | S3 bucket prefix, for example `s3://my-bucket/iceberg`. | +| `Iceberg` | `catalog.db` | `default` | Name of the Glue Data Catalog database. | +| `Iceberg` | `catalog.table` | `prices_iceberg` | Name of the Glue Data Catalog table. | +| `Iceberg` | `partition.fields` | `symbol` | Comma separated list of partition fields. | +| `Iceberg` | `operation` | `upsert` | Iceberg operation. One of `upsert`, `append` or `overwrite`. | +| `Iceberg` | `upsert.equality.fields` | `symbol` | Comma separated list of fields used for upsert. It must match partition fields. Required if `operation` = `upsert`. | ### Checkpoints diff --git a/java/Iceberg/IcebergDataStreamSink/pom.xml b/java/Iceberg/IcebergDataStreamSink/pom.xml index 0a1b7e60..215cff5a 100644 --- a/java/Iceberg/IcebergDataStreamSink/pom.xml +++ b/java/Iceberg/IcebergDataStreamSink/pom.xml @@ -18,7 +18,7 @@ 1.20.0 1.11.3 3.4.0 - 1.6.1 + 1.8.1 1.2.0 2.23.1 5.8.1 @@ -93,7 +93,7 @@ org.apache.iceberg - iceberg-flink-1.19 + iceberg-flink-1.20 ${iceberg.version} diff --git a/java/Iceberg/IcebergDataStreamSink/src/main/java/com/amazonaws/services/msf/iceberg/IcebergSinkBuilder.java b/java/Iceberg/IcebergDataStreamSink/src/main/java/com/amazonaws/services/msf/iceberg/IcebergSinkBuilder.java index 8dbbcdc2..c851a763 100644 --- a/java/Iceberg/IcebergDataStreamSink/src/main/java/com/amazonaws/services/msf/iceberg/IcebergSinkBuilder.java +++ b/java/Iceberg/IcebergDataStreamSink/src/main/java/com/amazonaws/services/msf/iceberg/IcebergSinkBuilder.java @@ -35,7 +35,6 @@ public class IcebergSinkBuilder { private static final String DEFAULT_GLUE_DB = "default"; private static final String DEFAULT_ICEBERG_TABLE_NAME = "prices_iceberg"; - private static final String DEFAULT_ICEBERG_SORT_ORDER_FIELD = "accountNr"; private static final String DEFAULT_ICEBERG_PARTITION_FIELDS = "symbol"; private static final String DEFAULT_ICEBERG_OPERATION = "upsert"; private static final String DEFAULT_ICEBERG_UPSERT_FIELDS = "symbol"; @@ -45,14 +44,10 @@ public class IcebergSinkBuilder { * If Iceberg Table has not been previously created, we will create it using the Partition Fields specified in the * Properties, as well as add a Sort Field to improve query performance */ - private static void createTable(Catalog catalog, TableIdentifier outputTable, org.apache.iceberg.Schema icebergSchema, PartitionSpec partitionSpec, String sortField) { + private static void createTable(Catalog catalog, TableIdentifier outputTable, org.apache.iceberg.Schema icebergSchema, PartitionSpec partitionSpec) { // If table has been previously created, we do not do any operation or modification if (!catalog.tableExists(outputTable)) { Table icebergTable = catalog.createTable(outputTable, icebergSchema, partitionSpec); - // Modifying newly created iceberg table to have a sort field - icebergTable.replaceSortOrder() - .asc(sortField, NullOrder.NULLS_LAST) - .commit(); // The catalog.create table creates an Iceberg V1 table. If we want to perform upserts, we need to upgrade the table version to 2. TableOperations tableOperations = ((BaseTable) icebergTable).operations(); TableMetadata appendTableMetadata = tableOperations.current(); @@ -83,8 +78,6 @@ public static FlinkSink.Builder createBuilder(Properties icebergProperties, Data String partitionFields = icebergProperties.getProperty("partition.fields", DEFAULT_ICEBERG_PARTITION_FIELDS); List partitionFieldList = Arrays.asList(partitionFields.split("\\s*,\\s*")); - String sortField = icebergProperties.getProperty("sort.field", DEFAULT_ICEBERG_SORT_ORDER_FIELD); - // Iceberg you can perform Appends, Upserts and Overwrites. String icebergOperation = icebergProperties.getProperty("operation", DEFAULT_ICEBERG_OPERATION); Preconditions.checkArgument(icebergOperation.equals("append") || icebergOperation.equals("upsert") || icebergOperation.equals("overwrite"), "Invalid Iceberg Operation"); @@ -123,7 +116,7 @@ public static FlinkSink.Builder createBuilder(Properties icebergProperties, Data // Based on how many fields we want to partition, we create the Partition Spec PartitionSpec partitionSpec = getPartitionSpec(icebergSchema, partitionFieldList); // We create the Iceberg Table, using the Iceberg Catalog, Table Identifier, Schema parsed in Iceberg Schema Format and the partition spec - createTable(catalog, outputTable, icebergSchema, partitionSpec, sortField); + createTable(catalog, outputTable, icebergSchema, partitionSpec); // Once the table has been created in the job or before, we load it TableLoader tableLoader = TableLoader.fromCatalog(glueCatalogLoader, outputTable); // Get RowType Schema from Iceberg Schema diff --git a/java/Iceberg/IcebergDataStreamSink/src/main/resources/flink-application-properties-dev.json b/java/Iceberg/IcebergDataStreamSink/src/main/resources/flink-application-properties-dev.json index 2bd46f0b..30a6ac03 100644 --- a/java/Iceberg/IcebergDataStreamSink/src/main/resources/flink-application-properties-dev.json +++ b/java/Iceberg/IcebergDataStreamSink/src/main/resources/flink-application-properties-dev.json @@ -12,8 +12,7 @@ "catalog.db": "default", "catalog.table": "prices_iceberg", "partition.fields": "symbol", - "sort.field": "timestamp", - "operation": "upsert", + "operation": "append", "upsert.equality.fields": "symbol" } } diff --git a/java/Iceberg/IcebergDataStreamSource/README.md b/java/Iceberg/IcebergDataStreamSource/README.md index e0d1d7bd..f96711f2 100644 --- a/java/Iceberg/IcebergDataStreamSource/README.md +++ b/java/Iceberg/IcebergDataStreamSource/README.md @@ -2,7 +2,7 @@ * Flink version: 1.20.0 * Flink API: DataStream API -* Iceberg 1.6.1 +* Iceberg 1.8.1 * Language: Java (11) * Flink connectors: [Iceberg](https://iceberg.apache.org/docs/latest/flink/) diff --git a/java/Iceberg/IcebergDataStreamSource/pom.xml b/java/Iceberg/IcebergDataStreamSource/pom.xml index 2b97e4d7..3dab28bb 100644 --- a/java/Iceberg/IcebergDataStreamSource/pom.xml +++ b/java/Iceberg/IcebergDataStreamSource/pom.xml @@ -18,7 +18,7 @@ 1.20.0 1.11.3 3.4.0 - 1.6.1 + 1.8.1 1.2.0 2.23.1 5.8.1 @@ -88,7 +88,7 @@ org.apache.iceberg - iceberg-flink-1.19 + iceberg-flink-1.20 ${iceberg.version} diff --git a/java/Iceberg/IcebergDataStreamSource/src/main/resources/flink-application-properties-dev.json b/java/Iceberg/IcebergDataStreamSource/src/main/resources/flink-application-properties-dev.json index e367d8f9..e5b2dba5 100644 --- a/java/Iceberg/IcebergDataStreamSource/src/main/resources/flink-application-properties-dev.json +++ b/java/Iceberg/IcebergDataStreamSource/src/main/resources/flink-application-properties-dev.json @@ -3,7 +3,7 @@ "PropertyGroupId": "Iceberg", "PropertyMap": { "bucket.prefix": "s3:///iceberg", - "catalog.db": "iceberg", + "catalog.db": "default", "catalog.table": "prices_iceberg" } } diff --git a/java/Iceberg/IcebergSQLSink/README.md b/java/Iceberg/IcebergSQLSink/README.md new file mode 100644 index 00000000..f0a656fc --- /dev/null +++ b/java/Iceberg/IcebergSQLSink/README.md @@ -0,0 +1,109 @@ +## Iceberg Sink (Glue Data Catalog) using SQL + +* Flink version: 1.20.0 +* Flink API: SQL API +* Iceberg 1.9.1 +* Language: Java (11) +* Flink connectors: [DataGen](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/connectors/datastream/datagen/) + and [Iceberg](https://iceberg.apache.org/docs/latest/flink/) sink + +This example demonstrates how to use +[Flink SQL API with Iceberg](https://iceberg.apache.org/docs/latest/flink-writes/) and the Glue Data Catalog. + +For simplicity, the application generates synthetic data, random stock prices, internally. +Data is generated as POJO objects, simulating a real source, for example a Kafka Source, that receives records +that can be converted to table format for SQL operations. + +### Prerequisites + +The application expects the following resources: +* A Glue Data Catalog database in the current AWS region. The database name is configurable (default: "default"). + The application creates the Table, but the Catalog must exist already. +* An S3 bucket to write the Iceberg table. + +#### IAM Permissions + +The application must have IAM permissions to: +* Show and alter Glue Data Catalog databases, show and create Glue Data Catalog tables. + See [Glue Data Catalog permissions](https://docs.aws.amazon.com/athena/latest/ug/fine-grained-access-to-glue-resources.html). +* Read and Write from the S3 bucket. + +### Runtime configuration + +When running on Amazon Managed Service for Apache Flink the runtime configuration is read from Runtime Properties. + +When running locally, the configuration is read from the +[resources/flink-application-properties-dev.json](./src/main/resources/flink-application-properties-dev.json) file. + +Runtime parameters: + +| Group ID | Key | Default | Description | +|-----------|--------------------------|-------------------|--------------------------------------------------------------------------------------------| +| `DataGen` | `records.per.sec` | `10.0` | Records per second generated. | +| `Iceberg` | `bucket.prefix` | (mandatory) | S3 bucket and path URL prefix, starting with `s3://`. For example `s3://mybucket/iceberg`. | +| `Iceberg` | `catalog.db` | `default` | Name of the Glue Data Catalog database. | +| `Iceberg` | `catalog.table` | `prices_iceberg` | Name of the Glue Data Catalog table. | + +### Running locally, in IntelliJ + +You can run this example directly in IntelliJ, without any local Flink cluster or local Flink installation. + +See [Running examples locally](https://github.com/nicusX/amazon-managed-service-for-apache-flink-examples/blob/main/java/running-examples-locally.md) for details. + +### Checkpoints + +Checkpointing must be enabled. Iceberg commits writes on checkpoint. + +When running locally, the application enables checkpoints programmatically, every 30 seconds. +When deployed to Managed Service for Apache Flink, checkpointing is controlled by the application configuration. + +### Sample Data Schema + +The application uses a predefined schema for the stock price data with the following fields: +* `timestamp`: STRING - ISO timestamp of the record +* `symbol`: STRING - Stock symbol (e.g., AAPL, AMZN) +* `price`: FLOAT - Stock price (0-10 range) +* `volumes`: INT - Trade volumes (0-1000000 range) + +### Known limitations of the Flink Iceberg sink + +At the moment there are current limitations concerning Flink Iceberg integration: +* Doesn't support Iceberg Table with hidden partitioning +* Doesn't support adding columns, removing columns, renaming columns or changing columns. + +--- + +### Known Flink issue: Hadoop library clash + +When integrating Flink with Iceberg, there's a common issue affecting most Flink setups + +When using Flink SQL's `CREATE CATALOG` statements, Hadoop libraries must be available on the system classpath. +However, standard Flink distributions use shaded dependencies that can create class loading conflicts with Hadoop's +expectations. +Flink default classloading, when running in Application mode, prevents from using some Hadoop classes even if +included in the application uber-jar. + +#### Solution + +This example shows a simple workaround to prevent the Hadoop class clashing: +1. Include a modified version of the Flink class `org.apache.flink.runtime.util.HadoopUtils` +2. Use Maven Shade Plugin to prevent class conflicts + +The modified [`org.apache.flink.runtime.util.HadoopUtils`](src/main/java/org/apache/flink/runtime/util/HadoopUtils.java) +class is included in the source code of this project. You can include it as-is in your project, using the same package name. + +The shading is configured in the [`pom.xml`](pom.xml). In your project you can copy the `...` configuration +into the `maven-shade-plugin` configuration. + +```xml + + + org.apache.hadoop.conf + shaded.org.apache.hadoop.conf + + + org.apache.flink.runtime.util.HadoopUtils + shadow.org.apache.flink.runtime.util.HadoopUtils + + +``` \ No newline at end of file diff --git a/java/Iceberg/IcebergSQLSink/pom.xml b/java/Iceberg/IcebergSQLSink/pom.xml new file mode 100644 index 00000000..a79b73ec --- /dev/null +++ b/java/Iceberg/IcebergSQLSink/pom.xml @@ -0,0 +1,193 @@ + + + 4.0.0 + + com.amazonaws + iceberg-sql-sink + 1.0 + jar + + + UTF-8 + 11 + ${target.java.version} + ${target.java.version} + 1.20 + 1.20.0 + 2.12 + 1.9.1 + 1.2.0 + 2.23.1 + 5.8.1 + + + + + + + + + + + + + + + software.amazon.awssdk + bom + 2.28.29 + pom + import + + + + + + + org.apache.flink + flink-streaming-java + ${flink.version} + provided + + + org.apache.flink + flink-table-api-java-bridge + ${flink.version} + provided + + + + org.apache.flink + flink-table-planner_${scala.version} + ${flink.version} + provided + + + org.apache.flink + flink-clients + ${flink.version} + provided + + + + org.apache.flink + flink-connector-datagen + ${flink.version} + provided + + + + com.amazonaws + aws-kinesisanalytics-runtime + ${kda.runtime.version} + provided + + + + org.apache.iceberg + iceberg-flink-runtime-${flink.major.version} + ${iceberg.version} + + + org.apache.iceberg + iceberg-aws-bundle + ${iceberg.version} + + + + + org.apache.flink + flink-s3-fs-hadoop + ${flink.version} + + + + + org.apache.logging.log4j + log4j-slf4j-impl + ${log4j.version} + + + org.apache.logging.log4j + log4j-api + ${log4j.version} + + + org.apache.logging.log4j + log4j-core + ${log4j.version} + + + + + + + + org.apache.maven.plugins + maven-compiler-plugin + 3.8.1 + + ${target.java.version} + ${target.java.version} + + + + + + org.apache.maven.plugins + maven-shade-plugin + 3.6.0 + + + package + + shade + + + + + org.apache.flink:force-shading + com.google.code.findbugs:jsr305 + org.slf4j:* + log4j:* + + + + + *:* + + META-INF/*.SF + META-INF/*.DSA + META-INF/*.RSA + + + + + + + com.amazonaws.services.msf.IcebergSQLSinkJob + + + + + + org.apache.hadoop.conf + shaded.org.apache.hadoop.conf + + + org.apache.flink.runtime.util.HadoopUtils + shadow.org.apache.flink.runtime.util.HadoopUtils + + + + + + + + + diff --git a/java/Iceberg/IcebergSQLSink/src/main/java/com/amazonaws/services/msf/IcebergSQLSinkJob.java b/java/Iceberg/IcebergSQLSink/src/main/java/com/amazonaws/services/msf/IcebergSQLSinkJob.java new file mode 100644 index 00000000..c2969260 --- /dev/null +++ b/java/Iceberg/IcebergSQLSink/src/main/java/com/amazonaws/services/msf/IcebergSQLSinkJob.java @@ -0,0 +1,179 @@ +package com.amazonaws.services.msf;/* + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * SPDX-License-Identifier: MIT-0 + * + * Permission is hereby granted, free of charge, to any person obtaining a copy of this + * software and associated documentation files (the "Software"), to deal in the Software + * without restriction, including without limitation the rights to use, copy, modify, + * merge, publish, distribute, sublicense, and/or sell copies of the Software, and to + * permit persons to whom the Software is furnished to do so. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, + * INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A + * PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION + * OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE + * SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + */ + +import com.amazonaws.services.kinesisanalytics.runtime.KinesisAnalyticsRuntime; +import com.amazonaws.services.msf.pojo.StockPrice; +import com.amazonaws.services.msf.source.StockPriceGeneratorFunction; +import org.apache.flink.api.common.eventtime.WatermarkStrategy; +import org.apache.flink.api.common.typeinfo.TypeInformation; +import org.apache.flink.api.connector.source.util.ratelimit.RateLimiterStrategy; +import org.apache.flink.connector.datagen.source.DataGeneratorSource; +import org.apache.flink.streaming.api.datastream.DataStream; +import org.apache.flink.streaming.api.environment.LocalStreamEnvironment; +import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; +import org.apache.flink.table.api.Table; +import org.apache.flink.table.api.TableResult; +import org.apache.flink.table.api.bridge.java.StreamTableEnvironment; +import org.apache.flink.util.Preconditions; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.util.Map; +import java.util.Objects; +import java.util.Properties; + +public class IcebergSQLSinkJob { + private static final Logger LOG = LoggerFactory.getLogger(IcebergSQLSinkJob.class); + + // Constants + private static final String CATALOG_NAME = "glue"; + private static final String LOCAL_APPLICATION_PROPERTIES_RESOURCE = "flink-application-properties-dev.json"; + public static final String DEFAULT_DATABASE = "default"; + public static final String DEFAULT_TABLE = "prices_iceberg"; + + private static boolean isLocal(StreamExecutionEnvironment env) { + return env instanceof LocalStreamEnvironment; + } + + private static void validateURI(String uri) { + String s3UriPattern = "^s3://([a-z0-9.-]+)(/[a-z0-9-_/]+/?)$"; + Preconditions.checkArgument(uri != null && uri.matches(s3UriPattern), + "Invalid S3 URI format: %s. URI must match pattern: s3://bucket-name/path/", uri); + } + + private static Map loadApplicationProperties(StreamExecutionEnvironment env) throws IOException { + if (isLocal(env)) { + LOG.info("Loading application properties from '{}'", LOCAL_APPLICATION_PROPERTIES_RESOURCE); + return KinesisAnalyticsRuntime.getApplicationProperties( + Objects.requireNonNull(IcebergSQLSinkJob.class.getClassLoader() + .getResource(LOCAL_APPLICATION_PROPERTIES_RESOURCE)).getPath()); + } else { + LOG.info("Loading application properties from Amazon Managed Service for Apache Flink"); + return KinesisAnalyticsRuntime.getApplicationProperties(); + } + } + + private static DataGeneratorSource createDataGenerator(Properties dataGeneratorProperties) { + double recordsPerSecond = Double.parseDouble(dataGeneratorProperties.getProperty("records.per.sec", "10.0")); + Preconditions.checkArgument(recordsPerSecond > 0, "Generator records per sec must be > 0"); + + LOG.info("Data generator: {} record/sec", recordsPerSecond); + return new DataGeneratorSource(new StockPriceGeneratorFunction(), + Long.MAX_VALUE, + RateLimiterStrategy.perSecond(recordsPerSecond), + TypeInformation.of(StockPrice.class)); + } + + private static String createCatalogStatement(String s3BucketPrefix) { + return "CREATE CATALOG " + CATALOG_NAME + " WITH (" + + "'type' = 'iceberg', " + + "'catalog-impl' = 'org.apache.iceberg.aws.glue.GlueCatalog'," + + "'io-impl' = 'org.apache.iceberg.aws.s3.S3FileIO'," + + "'warehouse' = '" + s3BucketPrefix + "')"; + } + + private static String createTableStatement(String sinkTableName) { + return "CREATE TABLE IF NOT EXISTS " + sinkTableName + " (" + + "`timestamp` STRING, " + + "symbol STRING," + + "price FLOAT," + + "volumes INT" + + ") PARTITIONED BY (symbol) "; + } + + private static IcebergConfig setupIcebergProperties(Properties icebergProperties) { + String s3BucketPrefix = icebergProperties.getProperty("bucket.prefix"); + String glueDatabase = icebergProperties.getProperty("catalog.db", DEFAULT_DATABASE); + String glueTable = icebergProperties.getProperty("catalog.table", DEFAULT_TABLE); + + Preconditions.checkNotNull(s3BucketPrefix, "You must supply an s3 bucket prefix for the warehouse."); + Preconditions.checkNotNull(glueDatabase, "You must supply a database name"); + Preconditions.checkNotNull(glueTable, "You must supply a table name"); + + // Validate S3 URI format + validateURI(s3BucketPrefix); + + LOG.info("Iceberg configuration: bucket={}, database={}, table={}", + s3BucketPrefix, glueDatabase, glueTable); + + return new IcebergConfig(s3BucketPrefix, glueDatabase, glueTable); + } + + private static class IcebergConfig { + final String s3BucketPrefix; + final String glueDatabase; + final String glueTable; + + IcebergConfig(String s3BucketPrefix, String glueDatabase, String glueTable) { + this.s3BucketPrefix = s3BucketPrefix; + this.glueDatabase = glueDatabase; + this.glueTable = glueTable; + } + } + + public static void main(String[] args) throws Exception { + // 1. Initialize environments + final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); + final StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env); + + // 2. If running local, we need to enable Checkpoints. Iceberg commits data with every checkpoint + if (isLocal(env)) { + // For development, we are checkpointing every 30 second to have data commited faster. + env.enableCheckpointing(30000); + } + + // 3. Parse and validate the configuration for the Iceberg sink + Map applicationProperties = loadApplicationProperties(env); + Properties icebergProperties = applicationProperties.get("Iceberg"); + IcebergConfig config = setupIcebergProperties(icebergProperties); + + // 4. Create data generator source, using DataStream API + Properties dataGenProperties = applicationProperties.get("DataGen"); + DataStream stockPriceDataStream = env.fromSource( + createDataGenerator(dataGenProperties), + WatermarkStrategy.noWatermarks(), + "DataGen"); + + // 5. Convert DataStream to a Table and create view + Table stockPriceTable = tableEnv.fromDataStream(stockPriceDataStream); + tableEnv.createTemporaryView("stockPriceTable", stockPriceTable); + + String sinkTableName = CATALOG_NAME + "." + config.glueDatabase + "." + config.glueTable; + + // Create catalog and configure it + tableEnv.executeSql(createCatalogStatement(config.s3BucketPrefix)); + tableEnv.executeSql("USE CATALOG " + CATALOG_NAME); + tableEnv.executeSql("CREATE DATABASE IF NOT EXISTS " + config.glueDatabase); + tableEnv.executeSql("USE " + config.glueDatabase); + + // Create table + String createTableStatement = createTableStatement(sinkTableName); + LOG.info("Creating table with statement: {}", createTableStatement); + tableEnv.executeSql(createTableStatement); + + // 7. Execute SQL operations - Insert data from stock price stream + String insertQuery = "INSERT INTO " + sinkTableName + " " + + "SELECT `timestamp`, symbol, price, volumes " + + "FROM default_catalog.default_database.stockPriceTable"; + TableResult insertResult = tableEnv.executeSql(insertQuery); + + // Keep the job running to continuously insert data + LOG.info("Application started successfully. Inserting data into Iceberg table: {}", sinkTableName); + } +} diff --git a/java/Iceberg/IcebergSQLSink/src/main/java/com/amazonaws/services/msf/pojo/StockPrice.java b/java/Iceberg/IcebergSQLSink/src/main/java/com/amazonaws/services/msf/pojo/StockPrice.java new file mode 100644 index 00000000..3b4f9235 --- /dev/null +++ b/java/Iceberg/IcebergSQLSink/src/main/java/com/amazonaws/services/msf/pojo/StockPrice.java @@ -0,0 +1,60 @@ +package com.amazonaws.services.msf.pojo; + +public class StockPrice { + private String timestamp; + private String symbol; + private Float price; + private Integer volumes; + + public StockPrice() { + } + + public StockPrice(String timestamp, String symbol, Float price, Integer volumes) { + this.timestamp = timestamp; + this.symbol = symbol; + this.price = price; + this.volumes = volumes; + } + + public String getTimestamp() { + return timestamp; + } + + public void setTimestamp(String timestamp) { + this.timestamp = timestamp; + } + + public String getSymbol() { + return symbol; + } + + public void setSymbol(String symbol) { + this.symbol = symbol; + } + + public Float getPrice() { + return price; + } + + public void setPrice(Float price) { + this.price = price; + } + + public Integer getVolumes() { + return volumes; + } + + public void setVolumes(Integer volumes) { + this.volumes = volumes; + } + + @Override + public String toString() { + return "com.amazonaws.services.msf.pojo.StockPrice{" + + "timestamp='" + timestamp + '\'' + + ", symbol='" + symbol + '\'' + + ", price=" + price + + ", volumes=" + volumes + + '}'; + } +} diff --git a/java/Iceberg/IcebergSQLSink/src/main/java/com/amazonaws/services/msf/source/StockPriceGeneratorFunction.java b/java/Iceberg/IcebergSQLSink/src/main/java/com/amazonaws/services/msf/source/StockPriceGeneratorFunction.java new file mode 100644 index 00000000..1cf55542 --- /dev/null +++ b/java/Iceberg/IcebergSQLSink/src/main/java/com/amazonaws/services/msf/source/StockPriceGeneratorFunction.java @@ -0,0 +1,27 @@ +package com.amazonaws.services.msf.source; + +import com.amazonaws.services.msf.pojo.StockPrice; +import org.apache.commons.lang3.RandomUtils; +import org.apache.flink.connector.datagen.source.GeneratorFunction; +import java.time.Instant; + +/** + * Function used by DataGen source to generate random records as com.amazonaws.services.msf.pojo.StockPrice POJOs. + * + * The generator mimics the behavior of AvroGenericStockTradeGeneratorFunction + * from the IcebergDataStreamSink example. + */ +public class StockPriceGeneratorFunction implements GeneratorFunction { + + private static final String[] SYMBOLS = {"AAPL", "AMZN", "MSFT", "INTC", "TBV"}; + + @Override + public StockPrice map(Long sequence) throws Exception { + String symbol = SYMBOLS[RandomUtils.nextInt(0, SYMBOLS.length)]; + float price = RandomUtils.nextFloat(0, 10); + int volumes = RandomUtils.nextInt(0, 1000000); + String timestamp = Instant.now().toString(); + + return new StockPrice(timestamp, symbol, price, volumes); + } +} diff --git a/java/Iceberg/IcebergSQLSink/src/main/java/org/apache/flink/runtime/util/HadoopUtils.java b/java/Iceberg/IcebergSQLSink/src/main/java/org/apache/flink/runtime/util/HadoopUtils.java new file mode 100644 index 00000000..b177d06b --- /dev/null +++ b/java/Iceberg/IcebergSQLSink/src/main/java/org/apache/flink/runtime/util/HadoopUtils.java @@ -0,0 +1,120 @@ +package org.apache.flink.runtime.util; + +import org.apache.flink.api.java.tuple.Tuple2; +import org.apache.flink.util.FlinkRuntimeException; +import org.apache.flink.util.Preconditions; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.io.Text; +import org.apache.hadoop.security.UserGroupInformation; +import org.apache.hadoop.security.token.Token; +import org.apache.hadoop.security.token.TokenIdentifier; +import org.apache.hadoop.util.VersionInfo; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.Collection; + + +/** + * This class is a copy of org.apache.flink.runtime.util.HadoopUtils with the getHadoopConfiguration() method replaced to + * return an org.apache.hadoop.conf.Configuration instead of org.apache.hadoop.hdfs.HdfsConfiguration. + * + * This class is then shaded, along with org.apache.hadoop.conf.*, to avoid conflicts with the same classes provided by + * org.apache.flink:flink-s3-fs-hadoop, which is normally installed as plugin in Flink when S3. + * + * Other methods are copied from the original class. + */ +public class HadoopUtils { + private static final Logger LOG = LoggerFactory.getLogger(HadoopUtils.class); + + static final Text HDFS_DELEGATION_TOKEN_KIND = new Text("HDFS_DELEGATION_TOKEN"); + + /** + * This method has been re-implemented to always return a org.apache.hadoop.conf.Configuration + */ + public static Configuration getHadoopConfiguration( + org.apache.flink.configuration.Configuration flinkConfiguration) { + return new Configuration(false); + } + + public static boolean isKerberosSecurityEnabled(UserGroupInformation ugi) { + return UserGroupInformation.isSecurityEnabled() + && ugi.getAuthenticationMethod() + == UserGroupInformation.AuthenticationMethod.KERBEROS; + } + + + public static boolean areKerberosCredentialsValid( + UserGroupInformation ugi, boolean useTicketCache) { + Preconditions.checkState(isKerberosSecurityEnabled(ugi)); + + // note: UGI::hasKerberosCredentials inaccurately reports false + // for logins based on a keytab (fixed in Hadoop 2.6.1, see HADOOP-10786), + // so we check only in ticket cache scenario. + if (useTicketCache && !ugi.hasKerberosCredentials()) { + if (hasHDFSDelegationToken(ugi)) { + LOG.warn( + "Hadoop security is enabled but current login user does not have Kerberos credentials, " + + "use delegation token instead. Flink application will terminate after token expires."); + return true; + } else { + LOG.error( + "Hadoop security is enabled, but current login user has neither Kerberos credentials " + + "nor delegation tokens!"); + return false; + } + } + + return true; + } + + /** + * Indicates whether the user has an HDFS delegation token. + */ + public static boolean hasHDFSDelegationToken(UserGroupInformation ugi) { + Collection> usrTok = ugi.getTokens(); + for (Token token : usrTok) { + if (token.getKind().equals(HDFS_DELEGATION_TOKEN_KIND)) { + return true; + } + } + return false; + } + + /** + * Checks if the Hadoop dependency is at least the given version. + */ + public static boolean isMinHadoopVersion(int major, int minor) throws FlinkRuntimeException { + final Tuple2 hadoopVersion = getMajorMinorBundledHadoopVersion(); + int maj = hadoopVersion.f0; + int min = hadoopVersion.f1; + + return maj > major || (maj == major && min >= minor); + } + + /** + * Checks if the Hadoop dependency is at most the given version. + */ + public static boolean isMaxHadoopVersion(int major, int minor) throws FlinkRuntimeException { + final Tuple2 hadoopVersion = getMajorMinorBundledHadoopVersion(); + int maj = hadoopVersion.f0; + int min = hadoopVersion.f1; + + return maj < major || (maj == major && min < minor); + } + + private static Tuple2 getMajorMinorBundledHadoopVersion() { + String versionString = VersionInfo.getVersion(); + String[] versionParts = versionString.split("\\."); + + if (versionParts.length < 2) { + throw new FlinkRuntimeException( + "Cannot determine version of Hadoop, unexpected version string: " + + versionString); + } + + int maj = Integer.parseInt(versionParts[0]); + int min = Integer.parseInt(versionParts[1]); + return Tuple2.of(maj, min); + } +} \ No newline at end of file diff --git a/java/Iceberg/IcebergSQLSink/src/main/resources/flink-application-properties-dev.json b/java/Iceberg/IcebergSQLSink/src/main/resources/flink-application-properties-dev.json new file mode 100644 index 00000000..cbbe0b0b --- /dev/null +++ b/java/Iceberg/IcebergSQLSink/src/main/resources/flink-application-properties-dev.json @@ -0,0 +1,16 @@ +[ + { + "PropertyGroupId": "DataGen", + "PropertyMap": { + "records.per.sec": 10.0 + } + }, + { + "PropertyGroupId": "Iceberg", + "PropertyMap": { + "bucket.prefix": "s3:///iceberg", + "catalog.db": "iceberg", + "catalog.table": "sqlsink_prices" + } + } +] \ No newline at end of file diff --git a/java/Iceberg/IcebergSQLSink/src/main/resources/log4j2.properties b/java/Iceberg/IcebergSQLSink/src/main/resources/log4j2.properties new file mode 100644 index 00000000..97c21b9d --- /dev/null +++ b/java/Iceberg/IcebergSQLSink/src/main/resources/log4j2.properties @@ -0,0 +1,13 @@ +# Log4j2 configuration +status = warn +name = PropertiesConfig + +# Console appender configuration +appender.console.type = Console +appender.console.name = ConsoleAppender +appender.console.layout.type = PatternLayout +appender.console.layout.pattern = %d{HH:mm:ss,SSS} %-5p %-60c %x - %m%n + +# Root logger configuration +rootLogger.level = INFO +rootLogger.appenderRef.console.ref = ConsoleAppender \ No newline at end of file diff --git a/java/Iceberg/README.md b/java/Iceberg/README.md new file mode 100644 index 00000000..640a3820 --- /dev/null +++ b/java/Iceberg/README.md @@ -0,0 +1,12 @@ +# Apache Iceberg Examples + +Examples demonstrating how to work with Apache Iceberg tables in Amazon Managed Service for Apache Flink using the DataStream API. + +## Table of Contents + +### Iceberg Sinks +- [**Iceberg DataStream Sink**](./IcebergDataStreamSink) - Writing data to Iceberg tables using AWS Glue Data Catalog +- [**S3 Table Sink**](./S3TableSink) - Writing data to Iceberg tables stored directly in S3 + +### Iceberg Sources +- [**Iceberg DataStream Source**](./IcebergDataStreamSource) - Reading data from Iceberg tables using AWS Glue Data Catalog diff --git a/java/Iceberg/S3TableSink/README.md b/java/Iceberg/S3TableSink/README.md index 7b24105a..07ef1e32 100644 --- a/java/Iceberg/S3TableSink/README.md +++ b/java/Iceberg/S3TableSink/README.md @@ -2,7 +2,7 @@ * Flink version: 1.19.0 * Flink API: DataStream API -* Iceberg 1.6.1 +* Iceberg 1.8.1 * Language: Java (11) * Flink connectors: [DataGen](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/connectors/datastream/datagen/) and [Iceberg](https://iceberg.apache.org/docs/latest/flink/) @@ -16,7 +16,7 @@ serialized with AVRO. ### Prerequisites -### Create a Table Bucket +#### Create a Table Bucket The sample application expects the S3 Table Bucket to exist and to have the ARN in the local environment: ```bash aws s3tables create-table-bucket --name flink-example @@ -34,6 +34,13 @@ aws s3tables list-table-buckets This will show you the list of table buckets. Select the one you wish to write to and paste it into the config file in this project. +#### Create a Namespace in the Table Bucket (Database) +The sample application expects the Namespace in the Table Bucket to exist +```bash +aws s3tables create-namespace \ + --table-bucket-arn arn:aws:s3tables:us-east-1:111122223333:bucket/flink-example \ + --namespace default +``` #### IAM Permissions @@ -49,16 +56,15 @@ When running locally, the configuration is read from the Runtime parameters: -| Group ID | Key | Default | Description | -|-----------|--------------------------|-------------------|---------------------------------------------------------------------------------------------------------------------| -| `DataGen` | `records.per.sec` | `100.0` | Records per second generated. | -| `Iceberg` | `table.bucket.arn` | (mandatory) | ARN of the S3 bucket, e.g., `arn:aws:s3tables:region:account-id:bucket/bucket-name` | -| `Iceberg` | `catalog.db` | `test_from_flink` | Name of the S3 table database. | -| `Iceberg` | `catalog.table` | `test_table` | Name of the S3 table. | -| `Iceberg` | `partition.fields` | `symbol` | Comma separated list of partition fields. | -| `Iceberg` | `sort.field` | `timestamp` | Sort field. | -| `Iceberg` | `operation` | `upsert` | Iceberg operation. One of `upsert`, `append` or `overwrite`. | -| `Iceberg` | `upsert.equality.fields` | `symbol` | Comma separated list of fields used for upsert. It must match partition fields. Required if `operation` = `upsert`. | +| Group ID | Key | Default | Description | +|-----------|--------------------------|------------------|---------------------------------------------------------------------------------------------------------------------| +| `DataGen` | `records.per.sec` | `100.0` | Records per second generated. | +| `Iceberg` | `table.bucket.arn` | (mandatory) | ARN of the S3 bucket, e.g., `arn:aws:s3tables:region:account-id:bucket/bucket-name` | +| `Iceberg` | `catalog.db` | `default` | Name of the S3 table database. | +| `Iceberg` | `catalog.table` | `prices_s3table` | Name of the S3 table. | +| `Iceberg` | `partition.fields` | `symbol` | Comma separated list of partition fields. | +| `Iceberg` | `operation` | `append` | Iceberg operation. One of `upsert`, `append` or `overwrite`. | +| `Iceberg` | `upsert.equality.fields` | `symbol` | Comma separated list of fields used for upsert. It must match partition fields. Required if `operation` = `upsert`. | ### Checkpoints diff --git a/java/Iceberg/S3TableSink/src/main/java/com/amazonaws/services/msf/StreamingJob.java b/java/Iceberg/S3TableSink/src/main/java/com/amazonaws/services/msf/StreamingJob.java index d8e77c4c..ff7a9602 100644 --- a/java/Iceberg/S3TableSink/src/main/java/com/amazonaws/services/msf/StreamingJob.java +++ b/java/Iceberg/S3TableSink/src/main/java/com/amazonaws/services/msf/StreamingJob.java @@ -68,7 +68,7 @@ private static DataGeneratorSource createDataGenerator(Properties } public static void main(String[] args) throws Exception { - StreamExecutionEnvironment env = StreamExecutionEnvironment.createLocalEnvironmentWithWebUI(new Configuration()); + StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env); // Local dev specific settings @@ -81,11 +81,6 @@ public static void main(String[] args) throws Exception { Map applicationProperties = loadApplicationProperties(env); icebergProperties = applicationProperties.get("Iceberg"); - Catalog s3 = createCatalog(tableEnv); - s3.createDatabase(icebergProperties.getProperty("catalog.db"), - new CatalogDatabaseImpl(Map.of(), - "Sample Database"), true); - // Get AVRO Schema from the definition bundled with the application // Note that the application must "knows" the AVRO schema upfront, i.e. the schema must be either embedded // with the application or fetched at start time. @@ -109,21 +104,6 @@ public static void main(String[] args) throws Exception { env.execute("Flink S3 Table Sink"); } - private static Catalog createCatalog(StreamTableEnvironment tableEnv) { - - Configuration conf = new Configuration(); - conf.setString("warehouse", icebergProperties.getProperty("table.bucket.arn")); - conf.setString("catalog-impl", "software.amazon.s3tables.iceberg.S3TablesCatalog"); - conf.setString("type", "iceberg"); - conf.setString("io-impl", "org.apache.iceberg.aws.s3.S3FileIO"); - - final String catalogName = "s3"; - CatalogDescriptor descriptor = CatalogDescriptor.of(catalogName, conf); - tableEnv.createCatalog(catalogName, descriptor); - return tableEnv.getCatalog(catalogName).get(); - } - - private static DataStream createDataStream(StreamExecutionEnvironment env, Map applicationProperties, Schema avroSchema) { Properties dataGeneratorProperties = applicationProperties.get("DataGen"); return env.fromSource( diff --git a/java/Iceberg/S3TableSink/src/main/java/com/amazonaws/services/msf/iceberg/IcebergSinkBuilder.java b/java/Iceberg/S3TableSink/src/main/java/com/amazonaws/services/msf/iceberg/IcebergSinkBuilder.java index 91ccac0f..923cab47 100644 --- a/java/Iceberg/S3TableSink/src/main/java/com/amazonaws/services/msf/iceberg/IcebergSinkBuilder.java +++ b/java/Iceberg/S3TableSink/src/main/java/com/amazonaws/services/msf/iceberg/IcebergSinkBuilder.java @@ -24,7 +24,6 @@ public class IcebergSinkBuilder { private static final String DEFAULT_S3_CATALOG_DB = "default"; private static final String DEFAULT_ICEBERG_TABLE_NAME = "prices_iceberg"; - private static final String DEFAULT_ICEBERG_SORT_ORDER_FIELD = "accountNr"; private static final String DEFAULT_ICEBERG_PARTITION_FIELDS = "symbol"; private static final String DEFAULT_ICEBERG_OPERATION = "upsert"; private static final String DEFAULT_ICEBERG_UPSERT_FIELDS = "symbol"; @@ -33,14 +32,11 @@ public class IcebergSinkBuilder { * If Iceberg Table has not been previously created, we will create it using the Partition Fields specified in the * Properties, as well as add a Sort Field to improve query performance */ - private static void createTable(Catalog catalog, TableIdentifier outputTable, org.apache.iceberg.Schema icebergSchema, PartitionSpec partitionSpec, String sortField) { + private static void createTable(Catalog catalog, TableIdentifier outputTable, org.apache.iceberg.Schema icebergSchema, PartitionSpec partitionSpec) { // If table has been previously created, we do not do any operation or modification if (!catalog.tableExists(outputTable)) { Table icebergTable = catalog.createTable(outputTable, icebergSchema, partitionSpec); - // Modifying newly created iceberg table to have a sort field - icebergTable.replaceSortOrder() - .asc(sortField, NullOrder.NULLS_LAST) - .commit(); + // The catalog.create table creates an Iceberg V1 table. If we want to perform upserts, we need to upgrade the table version to 2. TableOperations tableOperations = ((BaseTable) icebergTable).operations(); TableMetadata appendTableMetadata = tableOperations.current(); @@ -78,8 +74,6 @@ public static FlinkSink.Builder createBuilder(Properties icebergProperties, Data String partitionFields = icebergProperties.getProperty("partition.fields", DEFAULT_ICEBERG_PARTITION_FIELDS); List partitionFieldList = Arrays.asList(partitionFields.split("\\s*,\\s*")); - String sortField = icebergProperties.getProperty("sort.field", DEFAULT_ICEBERG_SORT_ORDER_FIELD); - // Iceberg you can perform Appends, Upserts and Overwrites. String icebergOperation = icebergProperties.getProperty("operation", DEFAULT_ICEBERG_OPERATION); Preconditions.checkArgument(icebergOperation.equals("append") || icebergOperation.equals("upsert") || icebergOperation.equals("overwrite"), "Invalid Iceberg Operation"); @@ -116,7 +110,7 @@ public static FlinkSink.Builder createBuilder(Properties icebergProperties, Data // Based on how many fields we want to partition, we create the Partition Spec PartitionSpec partitionSpec = getPartitionSpec(icebergSchema, partitionFieldList); // We create the Iceberg Table, using the Iceberg Catalog, Table Identifier, Schema parsed in Iceberg Schema Format and the partition spec - createTable(catalog, outputTable, icebergSchema, partitionSpec, sortField); + createTable(catalog, outputTable, icebergSchema, partitionSpec); // Once the table has been created in the job or before, we load it TableLoader tableLoader = TableLoader.fromCatalog(icebergCatalogLoader, outputTable); // Get RowType Schema from Iceberg Schema diff --git a/java/Iceberg/S3TableSink/src/main/resources/flink-application-properties-dev.json b/java/Iceberg/S3TableSink/src/main/resources/flink-application-properties-dev.json index fd94ccbb..b941e5ad 100644 --- a/java/Iceberg/S3TableSink/src/main/resources/flink-application-properties-dev.json +++ b/java/Iceberg/S3TableSink/src/main/resources/flink-application-properties-dev.json @@ -10,11 +10,11 @@ "PropertyMap": { "table.bucket.arn": "<>", - "catalog.db": "test_from_flink", - "catalog.table": "test_table", + "catalog.db": "default", + "catalog.table": "prices_s3table", "partition.fields": "symbol", "sort.field": "timestamp", - "operation": "upsert", + "operation": "append", "upsert.equality.fields": "symbol" } } diff --git a/java/KafkaConfigProviders/README.md b/java/KafkaConfigProviders/README.md index 8e32abf0..db4a2bfd 100644 --- a/java/KafkaConfigProviders/README.md +++ b/java/KafkaConfigProviders/README.md @@ -1,10 +1,15 @@ -## Configuring Kafka connectors secrets at runtime, using Config Providers +# Kafka Config Providers Examples -This directory includes example that shows how to configure secrets for Kafka connector authentication -scheme at runtime, using [MSK Config Providers](https://github.com/aws-samples/msk-config-providers). +Examples demonstrating secure configuration management for Kafka connectors using MSK Config Providers in Amazon Managed Service for Apache Flink. -Using Config Providers, secrets and files (TrustStore and KeyStore) required to set up the Kafka authentication -and SSL, can be fetched at runtime and not embedded in the application JAR. +These examples show how to configure secrets and certificates for Kafka connector authentication at runtime, +without embedding sensitive information in the application JAR, leveraging [MSK Config Providers](https://github.com/aws-samples/msk-config-providers). -* [Configuring mTLS TrustStore and KeyStore using Config Providers](./Kafka-mTLS-KeystoreWithConfigProviders) -* [Configuring SASL/SCRAM (SASL_SSL) TrustStore and credentials using Config Providers](./Kafka-SASL_SSL-WithConfigProviders) +## Table of Contents + +### mTLS Authentication +- [**Kafka mTLS with DataStream API**](./Kafka-mTLS-Keystore-ConfigProviders) - Using Config Providers to fetch KeyStore and passwords for mTLS authentication with DataStream API +- [**Kafka mTLS with Table API & SQL**](./Kafka-mTLS-Keystore-Sql-ConfigProviders) - Using Config Providers to fetch KeyStore and passwords for mTLS authentication with Table API & SQL + +### SASL Authentication +- [**Kafka SASL/SCRAM**](./Kafka-SASL_SSL-ConfigProviders) - Using Config Providers to fetch SASL/SCRAM credentials from AWS Secrets Manager diff --git a/java/KinesisConnectors/src/main/resources/flink-application-properties-dev.json b/java/KinesisConnectors/src/main/resources/flink-application-properties-dev.json index 7e4281ad..89167403 100644 --- a/java/KinesisConnectors/src/main/resources/flink-application-properties-dev.json +++ b/java/KinesisConnectors/src/main/resources/flink-application-properties-dev.json @@ -2,7 +2,7 @@ { "PropertyGroupId": "InputStream0", "PropertyMap": { - "stream.arn": "arn:aws:kinesis:us-east-1:012345678900:stream/ExampleInputStream", + "stream.arn": "arn:aws:kinesis:us-east-1::stream/ExampleInputStream", "source.init.position": "LATEST", "aws.region": "us-east-1" } @@ -10,7 +10,7 @@ { "PropertyGroupId": "OutputStream0", "PropertyMap": { - "stream.arn": "arn:aws:kinesis:us-east-1:012345678900:stream/ExampleOutputStream", + "stream.arn": "arn:aws:kinesis:us-east-1::stream/ExampleOutputStream", "aws.region": "us-east-1" } } diff --git a/java/KinesisSourceDeaggregation/README.md b/java/KinesisSourceDeaggregation/README.md new file mode 100644 index 00000000..14d1e08b --- /dev/null +++ b/java/KinesisSourceDeaggregation/README.md @@ -0,0 +1,31 @@ +## KinesisStreamsSource de-aggregation + +* Flink version: 1.20 +* Flink API: DataStream API +* Language: Java (11) +* Flink connectors: Kinesis Source and Sink + +This example demonstrates how to consume records published using KPL aggregation using `KinesisStreamsSource`. + +This folder contains two separate modules: +1. [kpl-producer](kpl-producer): a simple command line Java application to produce the JSON record to a Kinesis Stream, using KPL aggregation +2. [flink-app](flink-app): the Flink application demonstrating how to consume the aggregated stream, and publishing the de-aggregated records to another stream. + +Look at the instructions in the subfolders to run the KPL Producer (data generator) and the Flink application. + +### Background and motivation + +As of version `5.0.0`, `KinesisStreamsSource` does not support de-aggregation yet. + +If the connector is used to consume a stream produced with KPL aggregation, the source is not able to deserialize the records out of the box. + +This example shows how to implement de-aggregation in the deserialization schema. + +In particular, this example uses a wrapper which can be used to add de-aggregation to potentially any implementation +of `org.apache.flink.api.common.serialization.DeserializationSchema`. + +Implementation: +[KinesisDeaggregatingDeserializationSchemaWrapper.java](flink-app/src/main/java/com/amazonaws/services/msf/deaggregation/KinesisDeaggregatingDeserializationSchemaWrapper.java) + +> *IMPORTANT*: This implementation of de-aggregation is for demonstration purposes only. +> The code is not meant for production use and is not optimized in terms of performance. diff --git a/java/KinesisSourceDeaggregation/flink-app/README.md b/java/KinesisSourceDeaggregation/flink-app/README.md new file mode 100644 index 00000000..692ee95d --- /dev/null +++ b/java/KinesisSourceDeaggregation/flink-app/README.md @@ -0,0 +1,55 @@ +## Flink job consuming an aggregated stream + +This Flink job consumes a Kinesis stream with aggregated JSON records, and publish them to another stream. + +### Prerequisites + +The application expects two Kinesis Streams: +* Input stream containing the aggregated records +* Output stream where the de-aggregated records are published + +The application must have sufficient permissions to read and write the streams. + +### Runtime configuration + +The application reads the runtime configuration from the Runtime Properties, when running on Amazon Managed Service for Apache Flink, +or, when running locally, from the [`resources/flink-application-properties-dev.json`](resources/flink-application-properties-dev.json) file located in the resources folder. + +All parameters are case-sensitive. + +| Group ID | Key | Description | +|------------------|---------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `InputStream0` | `stream.arn` | ARN of the input stream. | +| `InputStream0` | `aws.region` | Region of the input stream. | +| `InputStream0` | `source.init.position` | (optional) Starting position when the application starts with no state. Default is `LATEST` | +| `InputStream0` | `source.reader.type` | (optional) Choose between standard (`POLLING`) and Enhanced Fan-Out (`EFO`) consumer. Default is `POLLING`. | +| `InputStream0` | `source.efo.consumer.name` | (optional, for EFO consumer mode only) Name of the EFO consumer. Only used if `source.reader.type=EFO`. | +| `InputStream0` | `source.efo.consumer.lifecycle` | (optional, for EFO consumer mode only) Lifecycle management mode of EFO consumer. Choose between `JOB_MANAGED` and `SELF_MANAGED`. Default is `JOB_MANAGED`. | +| `OutputStream0` | `stream.arn` | ARN of the output stream. | +| `OutputStream0` | `aws.region` | Region of the output stream. | + +Every parameter in the `InputStream0` group is passed to the Kinesis consumer, and every parameter in the `OutputStream0` is passed to the Kinesis client of the sink. + +See Flink Kinesis connector docs](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/connectors/datastream/kinesis/) for details about configuring the Kinesis connector. + +To configure the application on Managed Service for Apache Flink, set up these parameter in the *Runtime properties*. + +To configure the application for running locally, edit the [json file](resources/flink-application-properties-dev.json). + +### Running in IntelliJ + +You can run this example directly in IntelliJ, without any local Flink cluster or local Flink installation. + +See [Running examples locally](../../running-examples-locally.md) for details. + + +### Data Generator + +Use the [KPL Producer](../kpl-producer) to generate aggregates StockPrices to the Kinesis stream + + +### Data example + +``` +{'event_time': '2024-05-28T19:53:17.497201', 'ticker': 'AMZN', 'price': 42.88} +``` \ No newline at end of file diff --git a/java/KinesisSourceDeaggregation/flink-app/pom.xml b/java/KinesisSourceDeaggregation/flink-app/pom.xml new file mode 100644 index 00000000..ef4a81d8 --- /dev/null +++ b/java/KinesisSourceDeaggregation/flink-app/pom.xml @@ -0,0 +1,199 @@ + + + 4.0.0 + + com.amazonaws + kinesis-source-deaggregation + 1.0 + + + UTF-8 + ${project.basedir}/target + ${project.name}-${project.version} + 11 + ${target.java.version} + ${target.java.version} + 1.20.0 + 5.0.0-1.20 + 2.7.0 + 1.2.0 + 2.23.1 + + + + + + com.amazonaws + aws-java-sdk-bom + + 1.12.677 + pom + import + + + software.amazon.awssdk + bom + 2.31.28 + pom + import + + + + + + + + + org.apache.flink + flink-streaming-java + ${flink.version} + provided + + + org.apache.flink + flink-clients + ${flink.version} + provided + + + org.apache.flink + flink-runtime-web + ${flink.version} + provided + + + org.apache.flink + flink-json + ${flink.version} + provided + + + + + com.amazonaws + aws-kinesisanalytics-runtime + ${kda.runtime.version} + provided + + + + software.amazon.kinesis + amazon-kinesis-client + ${kinesis.client.version} + + + + software.amazon.awssdk + * + + + software.amazon.glue + * + + + io.reactivex.rxjava3 + rxjava + + + com.google.guava + * + + + + + + + org.apache.flink + flink-connector-base + ${flink.version} + provided + + + org.apache.flink + flink-connector-aws-kinesis-streams + ${aws.connector.version} + + + + + + org.apache.logging.log4j + log4j-slf4j-impl + ${log4j.version} + + + org.apache.logging.log4j + log4j-api + ${log4j.version} + + + org.apache.logging.log4j + log4j-core + ${log4j.version} + + + + + ${buildDirectory} + ${jar.finalName} + + + + + org.apache.maven.plugins + maven-compiler-plugin + 3.8.1 + + ${target.java.version} + ${target.java.version} + + + + + + org.apache.maven.plugins + maven-shade-plugin + 3.2.1 + + + + package + + shade + + + + + org.apache.flink:force-shading + com.google.code.findbugs:jsr305 + org.slf4j:* + log4j:* + + + + + + *:* + + META-INF/*.SF + META-INF/*.DSA + META-INF/*.RSA + + + + + + + com.amazonaws.services.msf.StreamingJob + + + + + + + + + \ No newline at end of file diff --git a/java/KinesisSourceDeaggregation/flink-app/src/main/java/com/amazonaws/services/msf/StreamingJob.java b/java/KinesisSourceDeaggregation/flink-app/src/main/java/com/amazonaws/services/msf/StreamingJob.java new file mode 100644 index 00000000..be5c986c --- /dev/null +++ b/java/KinesisSourceDeaggregation/flink-app/src/main/java/com/amazonaws/services/msf/StreamingJob.java @@ -0,0 +1,99 @@ +package com.amazonaws.services.msf; + +import com.amazonaws.services.kinesisanalytics.runtime.KinesisAnalyticsRuntime; +import com.amazonaws.services.msf.deaggregation.KinesisDeaggregatingDeserializationSchemaWrapper; +import com.amazonaws.services.msf.model.StockPrice; +import org.apache.flink.api.common.eventtime.WatermarkStrategy; +import org.apache.flink.api.common.serialization.SerializationSchema; +import org.apache.flink.api.common.typeinfo.TypeInformation; +import org.apache.flink.configuration.Configuration; +import org.apache.flink.connector.kinesis.sink.KinesisStreamsSink; +import org.apache.flink.connector.kinesis.source.KinesisStreamsSource; +import org.apache.flink.connector.kinesis.source.serialization.KinesisDeserializationSchema; +import org.apache.flink.formats.json.JsonDeserializationSchema; +import org.apache.flink.formats.json.JsonSerializationSchema; +import org.apache.flink.shaded.guava31.com.google.common.collect.Maps; +import org.apache.flink.streaming.api.datastream.DataStream; +import org.apache.flink.streaming.api.environment.LocalStreamEnvironment; +import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.util.Map; +import java.util.Properties; + + +public class StreamingJob { + private static final Logger LOG = LoggerFactory.getLogger(StreamingJob.class); + + // Name of the local JSON resource with the application properties in the same format as they are received from the Amazon Managed Service for Apache Flink runtime + private static final String LOCAL_APPLICATION_PROPERTIES_RESOURCE = "flink-application-properties-dev.json"; + + private static boolean isLocal(StreamExecutionEnvironment env) { + return env instanceof LocalStreamEnvironment; + } + + /** + * Load application properties from Amazon Managed Service for Apache Flink runtime or from a local resource, when the environment is local + */ + private static Map loadApplicationProperties(StreamExecutionEnvironment env) throws IOException { + if (isLocal(env)) { + LOG.info("Loading application properties from '{}'", LOCAL_APPLICATION_PROPERTIES_RESOURCE); + return KinesisAnalyticsRuntime.getApplicationProperties( + StreamingJob.class.getClassLoader() + .getResource(LOCAL_APPLICATION_PROPERTIES_RESOURCE).getPath()); + } else { + LOG.info("Loading application properties from Amazon Managed Service for Apache Flink"); + return KinesisAnalyticsRuntime.getApplicationProperties(); + } + } + + // Create a source using a KinesisDeserializationSchema + private static KinesisStreamsSource createKinesisSource(Properties inputProperties, final KinesisDeserializationSchema kinesisDeserializationSchema) { + final String inputStreamArn = inputProperties.getProperty("stream.arn"); + return KinesisStreamsSource.builder() + .setStreamArn(inputStreamArn) + .setSourceConfig(Configuration.fromMap(Maps.fromProperties(inputProperties))) + .setDeserializationSchema(kinesisDeserializationSchema) + .build(); + } + + // Create a sink + private static KinesisStreamsSink createKinesisSink(Properties outputProperties, final SerializationSchema serializationSchema) { + final String outputStreamArn = outputProperties.getProperty("stream.arn"); + return KinesisStreamsSink.builder() + .setStreamArn(outputStreamArn) + .setKinesisClientProperties(outputProperties) + .setSerializationSchema(serializationSchema) + .setPartitionKeyGenerator(element -> String.valueOf(element.hashCode())) + .build(); + } + + public static void main(String[] args) throws Exception { + final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); + final Map applicationProperties = loadApplicationProperties(env); + LOG.warn("Application properties: {}", applicationProperties); + + // Wrap the deserialization schema + KinesisDeserializationSchema deaggregatingKinesisDeserializationSchema = + // Wrapper which takes care of deaggregation + new KinesisDeaggregatingDeserializationSchemaWrapper<>( + new JsonDeserializationSchema<>(StockPrice.class) // DeserializationSchema to deserialize each deaggregated record + ); + KinesisStreamsSource source = createKinesisSource(applicationProperties.get("InputStream0"), deaggregatingKinesisDeserializationSchema); + + // Set up the source + DataStream input = env.fromSource(source, + WatermarkStrategy.noWatermarks(), + "Kinesis source", + TypeInformation.of(StockPrice.class)); + + // Send the deaggregated records to the sink + KinesisStreamsSink sink = createKinesisSink(applicationProperties.get("OutputStream0"), new JsonSerializationSchema<>()); + + input.sinkTo(sink); + + env.execute("Kinesis de-aggregating Source"); + } +} diff --git a/java/KinesisSourceDeaggregation/flink-app/src/main/java/com/amazonaws/services/msf/deaggregation/KinesisDeaggregatingDeserializationSchemaWrapper.java b/java/KinesisSourceDeaggregation/flink-app/src/main/java/com/amazonaws/services/msf/deaggregation/KinesisDeaggregatingDeserializationSchemaWrapper.java new file mode 100644 index 00000000..91518394 --- /dev/null +++ b/java/KinesisSourceDeaggregation/flink-app/src/main/java/com/amazonaws/services/msf/deaggregation/KinesisDeaggregatingDeserializationSchemaWrapper.java @@ -0,0 +1,105 @@ +package com.amazonaws.services.msf.deaggregation; + +import org.apache.flink.api.common.serialization.DeserializationSchema; +import org.apache.flink.api.common.typeinfo.TypeInformation; +import org.apache.flink.connector.kinesis.source.serialization.KinesisDeserializationSchema; +import org.apache.flink.util.Collector; +import software.amazon.awssdk.services.kinesis.model.Record; +import software.amazon.kinesis.retrieval.AggregatorUtil; +import software.amazon.kinesis.retrieval.KinesisClientRecord; + +import java.io.Serializable; +import java.nio.ByteBuffer; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; + +/** + * Wrapper adding de-aggregation to any DeserializationSchema + * + * @param type returned by the DeserializationSchema + */ +public class KinesisDeaggregatingDeserializationSchemaWrapper implements KinesisDeserializationSchema { + private static final long serialVersionUID = 1L; + + private final DeserializationSchema deserializationSchema; + private final RecordDeaggregator recordDeaggregator = new RecordDeaggregator(); + + public KinesisDeaggregatingDeserializationSchemaWrapper(DeserializationSchema deserializationSchema) { + this.deserializationSchema = deserializationSchema; + } + + @Override + public void open(DeserializationSchema.InitializationContext context) throws Exception { + this.deserializationSchema.open(context); + } + + @Override + public void deserialize(Record record, String stream, String shardId, Collector output) { + try { + // Deaggregate the record read from the stream, if required + List deaggregatedRecords = recordDeaggregator.deaggregate(record); + // Deserialize each deaggregated record, independently + for (KinesisClientRecord deaggregatedRecord : deaggregatedRecords) { + T deserializedRecord = deserializationSchema.deserialize(toByteArray(deaggregatedRecord.data())); + output.collect(deserializedRecord); + } + } catch (Exception e) { + throw new RuntimeException("Error while deaggregating Kinesis record from stream " + stream + ", shardId " + shardId, e); + } + } + + @Override + public TypeInformation getProducedType() { + return deserializationSchema.getProducedType(); + } + + private static byte[] toByteArray(ByteBuffer buffer) { + byte[] data = new byte[buffer.remaining()]; + buffer.get(data); + return data; + } + + /** + * De-aggregate software.amazon.awssdk.services.kinesis.model.Record into a collection + * of software.amazon.kinesis.retrieval.KinesisClientRecord. + *

+ * The code of this class is copied from + * https://github.com/awslabs/kinesis-aggregation/blob/master/java/KinesisDeaggregatorV2/src/main/java/com/amazonaws/kinesis/deagg/RecordDeaggregator.java + * which is not available in Maven Central. + * See https://github.com/awslabs/kinesis-aggregation/issues/120 + */ + private static class RecordDeaggregator implements Serializable { + + /** + * Method to deaggregate a single Kinesis Record into a List of UserRecords + * + * @param inputRecord The Kinesis Record provided by AWS Lambda or Kinesis SDK + * @return A list of Kinesis UserRecord objects obtained by deaggregating the + * input list of KinesisEventRecords + */ + public List deaggregate(T inputRecord) throws Exception { + return new AggregatorUtil().deaggregate(convertType(Arrays.asList(inputRecord))); + } + + @SuppressWarnings("unchecked") + private List convertType(List inputRecords) throws Exception { + List records = null; + + if (!inputRecords.isEmpty() && inputRecords.get(0) instanceof Record) { + records = new ArrayList<>(); + for (Record rec : (List) inputRecords) { + records.add(KinesisClientRecord.fromRecord(rec)); + } + } else { + if (inputRecords.isEmpty()) { + return new ArrayList<>(); + } else { + throw new Exception("Input Types must be a Model Records"); + } + } + + return records; + } + } +} diff --git a/java/KinesisSourceDeaggregation/flink-app/src/main/java/com/amazonaws/services/msf/model/StockPrice.java b/java/KinesisSourceDeaggregation/flink-app/src/main/java/com/amazonaws/services/msf/model/StockPrice.java new file mode 100644 index 00000000..4e423d7f --- /dev/null +++ b/java/KinesisSourceDeaggregation/flink-app/src/main/java/com/amazonaws/services/msf/model/StockPrice.java @@ -0,0 +1,47 @@ +package com.amazonaws.services.msf.model; + +import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.annotation.JsonProperty; + +public class StockPrice { + // This annotation as well as the associated jackson2 import is needed to correctly map the JSON input key to the + // appropriate POJO property name to ensure event_time isn't missed in serialization and deserialization + @JsonProperty("event_time") + private String eventTime; + private String ticker; + private float price; + + public StockPrice() {} + + public String getEventTime() { + return eventTime; + } + + public void setEventTime(String eventTime) { + this.eventTime = eventTime; + } + + public String getTicker() { + return ticker; + } + + public void setTicker(String ticker) { + this.ticker = ticker; + } + + public float getPrice() { + return price; + } + + public void setPrice(float price) { + this.price = price; + } + + @Override + public String toString() { + return "StockPrice{" + + "event_time='" + eventTime + '\'' + + ", ticker='" + ticker + '\'' + + ", price=" + price + + '}'; + } +} diff --git a/java/KinesisSourceDeaggregation/flink-app/src/main/resources/flink-application-properties-dev.json b/java/KinesisSourceDeaggregation/flink-app/src/main/resources/flink-application-properties-dev.json new file mode 100644 index 00000000..880a58f9 --- /dev/null +++ b/java/KinesisSourceDeaggregation/flink-app/src/main/resources/flink-application-properties-dev.json @@ -0,0 +1,17 @@ +[ + { + "PropertyGroupId": "InputStream0", + "PropertyMap": { + "stream.arn": "arn:aws:kinesis:us-east-1::stream/ExampleInputStream", + "source.init.position": "LATEST", + "aws.region": "us-east-1" + } + }, + { + "PropertyGroupId": "OutputStream0", + "PropertyMap": { + "stream.arn": "arn:aws:kinesis:us-east-1::stream/ExampleOutputStream", + "aws.region": "us-east-1" + } + } +] diff --git a/java/KinesisSourceDeaggregation/flink-app/src/main/resources/log4j2.properties b/java/KinesisSourceDeaggregation/flink-app/src/main/resources/log4j2.properties new file mode 100644 index 00000000..b4d75f24 --- /dev/null +++ b/java/KinesisSourceDeaggregation/flink-app/src/main/resources/log4j2.properties @@ -0,0 +1,8 @@ +rootLogger.level = INFO +rootLogger.appenderRef.console.ref = ConsoleAppender + +appender.console.name = ConsoleAppender +appender.console.type = CONSOLE +appender.console.layout.type = PatternLayout +appender.console.layout.pattern = %d{HH:mm:ss,SSS} %-5p %-60c %x - %m%n + diff --git a/java/KinesisSourceDeaggregation/kpl-producer/README.md b/java/KinesisSourceDeaggregation/kpl-producer/README.md new file mode 100644 index 00000000..d906df16 --- /dev/null +++ b/java/KinesisSourceDeaggregation/kpl-producer/README.md @@ -0,0 +1,20 @@ +## Simple KPL aggregating data generator + +This module contains a simple random data generator which publishes stock prices records to a Kinesis Data Streams +using KPL aggregation. + +1. Compile: `mvn package` +2. Run: `java -jar target/kpl-producer-1.0.jar --streamName --streamRegion --sleep 10` + +Runtime parameters (all optional) + +* `streamName`: default `ExampleInputStream` +* `stramRegion`: default `us-east-1` +* `sleep`: default 10 (milliseconds between records) + + +### Data example + +``` +{'event_time': '2024-05-28T19:53:17.497201', 'ticker': 'AMZN', 'price': 42.88} +``` \ No newline at end of file diff --git a/java/KinesisSourceDeaggregation/kpl-producer/pom.xml b/java/KinesisSourceDeaggregation/kpl-producer/pom.xml new file mode 100644 index 00000000..e5fc5045 --- /dev/null +++ b/java/KinesisSourceDeaggregation/kpl-producer/pom.xml @@ -0,0 +1,79 @@ + + 4.0.0 + com.amazonaws + kpl-producer + 1.0 + + + 11 + ${target.java.version} + ${target.java.version} + 1.0.0 + 2.0.17 + 2.23.1 + + + + + software.amazon.kinesis + amazon-kinesis-producer + ${kpl.version} + + + + commons-cli + commons-cli + 1.9.0 + + + + + org.slf4j + slf4j-api + ${slf4j.version} + + + org.slf4j + slf4j-simple + ${slf4j.version} + + + + + + + + org.apache.maven.plugins + maven-compiler-plugin + 3.8.1 + + ${target.java.version} + ${target.java.version} + + + + org.apache.maven.plugins + maven-shade-plugin + 3.2.1 + + + + package + + shade + + + + + com.amazonaws.services.kds.producer.KplAggregatingProducer + + + + + + + + + + diff --git a/java/KinesisSourceDeaggregation/kpl-producer/src/main/java/com/amazonaws/services/kds/producer/KplAggregatingProducer.java b/java/KinesisSourceDeaggregation/kpl-producer/src/main/java/com/amazonaws/services/kds/producer/KplAggregatingProducer.java new file mode 100644 index 00000000..8db24583 --- /dev/null +++ b/java/KinesisSourceDeaggregation/kpl-producer/src/main/java/com/amazonaws/services/kds/producer/KplAggregatingProducer.java @@ -0,0 +1,182 @@ +package com.amazonaws.services.kds.producer; + +import com.amazonaws.services.kds.producer.model.StockPrice; +import com.google.common.util.concurrent.FutureCallback; +import com.google.common.util.concurrent.Futures; +import com.google.common.util.concurrent.ListenableFuture; +import org.apache.commons.cli.*; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import software.amazon.kinesis.producer.KinesisProducer; +import software.amazon.kinesis.producer.KinesisProducerConfiguration; +import software.amazon.kinesis.producer.UserRecordResult; + +import java.nio.ByteBuffer; +import java.time.LocalDateTime; +import java.time.ZoneOffset; +import java.time.format.DateTimeFormatter; +import java.util.Random; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicLong; + +/** + * Simple KPL producer publishing random StockRecords as JSON to a Kinesis stream + */ +public class KplAggregatingProducer { + private static final Logger LOG = LoggerFactory.getLogger(KplAggregatingProducer.class); + + private static final String[] TICKERS = {"AAPL", "AMZN", "MSFT", "INTC", "TBV"}; + private static final Random RANDOM = new Random(); + private static final DateTimeFormatter ISO_FORMATTER = + DateTimeFormatter.ofPattern("yyyy-MM-dd'T'HH:mm:ss.SSSSSS") + .withZone(ZoneOffset.UTC); + + private final String streamName; + private final String streamRegion; + private final long sleepTimeBetweenRecords; + + private final AtomicLong queuedRecordCount = new AtomicLong(); + private final AtomicLong sentRecordCount = new AtomicLong(); + + private static final String DEFAULT_STREAM_REGION = "us-east-1"; + private static final String DEFAULT_STREAM_NAME = "ExampleInputStream"; + private static final long DEFAULT_SLEEP_TIME_BETWEEN_RECORDS = 50L; + + public static void main(String[] args) throws Exception { + Options options = new Options() + .addOption(Option.builder() + .longOpt("streamName") + .hasArg() + .desc("Stream name (default: " + DEFAULT_STREAM_NAME + ")") + .build()) + .addOption(Option.builder() + .longOpt("streamRegion") + .hasArg() + .desc("Stream AWS region (default: " + DEFAULT_STREAM_REGION + ")") + .build()) + .addOption(Option.builder() + .longOpt("sleep") + .hasArg() + .desc("Sleep duration in seconds (default: " + DEFAULT_SLEEP_TIME_BETWEEN_RECORDS + ")") + .build()); + + CommandLineParser parser = new DefaultParser(); + HelpFormatter formatter = new HelpFormatter(); + + try { + CommandLine cmd = parser.parse(options, args); + String streamNameValue = cmd.getOptionValue("streamName", DEFAULT_STREAM_NAME); + String streamRegionValue = cmd.getOptionValue("region", DEFAULT_STREAM_REGION); + long sleepTimeBetweenRecordsMillis = Long.parseLong(cmd.getOptionValue("sleep", String.valueOf(DEFAULT_SLEEP_TIME_BETWEEN_RECORDS))); + + LOG.info("StreamName: {}, region: {}", streamNameValue, streamNameValue); + LOG.info("SleepTimeBetweenRecords: {} ms", sleepTimeBetweenRecordsMillis); + + KplAggregatingProducer instance = new KplAggregatingProducer(streamNameValue, streamRegionValue, sleepTimeBetweenRecordsMillis); + instance.produce(); + + } catch (ParseException e) { + System.err.println("Error parsing command line arguments: " + e.getMessage()); + formatter.printHelp("KplAggregatingProducer", options); + System.exit(1); + } catch (NumberFormatException e) { + System.err.println("Error: sleep parameter must be a valid integer"); + formatter.printHelp("KplAggregatingProducer", options); + System.exit(1); + } + } + + public KplAggregatingProducer(String streamName, String streamRegion, long sleepTimeBetweenRecords) { + this.streamName = streamName; + this.streamRegion = streamRegion; + this.sleepTimeBetweenRecords = sleepTimeBetweenRecords; + } + + public void produce() { + KinesisProducerConfiguration config = new KinesisProducerConfiguration() + .setRegion(streamRegion) + .setAggregationEnabled(true) + .setRecordMaxBufferedTime(100) + .setMaxConnections(4) + .setRequestTimeout(60000); + + KinesisProducer producer = new KinesisProducer(config); + ExecutorService callbackThreadPool = Executors.newCachedThreadPool(); + + // Setup shutdown hook for cleanup + Runtime.getRuntime().addShutdownHook(new Thread(() -> { + LOG.info("Shutting down..."); + producer.flushSync(); + producer.destroy(); + callbackThreadPool.shutdown(); + try { + callbackThreadPool.awaitTermination(10, TimeUnit.SECONDS); + } catch (InterruptedException e) { + LOG.warn("Thread pool shutdown interrupted", e); + } + })); + + try { + while (true) { + StockPrice stockPrice = generateRandomStockPrice(); + sendStockPriceToKinesis(stockPrice, streamName, producer, callbackThreadPool); + Thread.sleep(sleepTimeBetweenRecords); // Avoid flooding the stream + } + } catch (InterruptedException e) { + LOG.warn("Producer interrupted" + e); + } + } + + private StockPrice generateRandomStockPrice() { + String ticker = TICKERS[RANDOM.nextInt(TICKERS.length)]; + double price = RANDOM.nextDouble() * 100.0; + String timestamp = LocalDateTime.now().format(ISO_FORMATTER); + + return new StockPrice(timestamp, ticker, price); + } + + private void sendStockPriceToKinesis(StockPrice stockPrice, String streamName, KinesisProducer producer, + ExecutorService callbackThreadPool) { + try { + String jsonPayload = String.format( + "{\"event_time\": \"%s\", \"ticker\": \"%s\", \"price\": %.2f}", + stockPrice.getEventTime(), + stockPrice.getTicker(), + stockPrice.getPrice() + ); + + byte[] bytes = jsonPayload.getBytes(); + LOG.trace("Sending stock price: {}", jsonPayload); + + + ListenableFuture future = producer.addUserRecord( + streamName, + stockPrice.getTicker(), // Use ticker as partition key + ByteBuffer.wrap(bytes)); + long queued = queuedRecordCount.incrementAndGet(); + if (queued % 1000 == 0) { + long sent = sentRecordCount.get(); + LOG.info("Queued {} records. Sent {} records", queued, sent); + } + + + // Handle success/failure asynchronously + Futures.addCallback(future, new FutureCallback() { + @Override + public void onSuccess(UserRecordResult result) { + sentRecordCount.incrementAndGet(); + LOG.trace("Successfully sent record: {}", result.getSequenceNumber()); + } + + @Override + public void onFailure(Throwable t) { + LOG.error("Failed to send record" + t); + } + }, callbackThreadPool); + } catch (Exception e) { + LOG.error("Error serializing or sending stock price", e); + } + } +} diff --git a/java/KinesisSourceDeaggregation/kpl-producer/src/main/java/com/amazonaws/services/kds/producer/model/StockPrice.java b/java/KinesisSourceDeaggregation/kpl-producer/src/main/java/com/amazonaws/services/kds/producer/model/StockPrice.java new file mode 100644 index 00000000..7fc7daba --- /dev/null +++ b/java/KinesisSourceDeaggregation/kpl-producer/src/main/java/com/amazonaws/services/kds/producer/model/StockPrice.java @@ -0,0 +1,37 @@ +package com.amazonaws.services.kds.producer.model; + +public class StockPrice { + private String eventTime; + private String ticker; + private double price; + + public StockPrice(String eventTime, String ticker, double price) { + this.eventTime = eventTime; + this.ticker = ticker; + this.price = price; + } + + public String getEventTime() { + return eventTime; + } + + public void setEventTime(String eventTime) { + this.eventTime = eventTime; + } + + public String getTicker() { + return ticker; + } + + public void setTicker(String ticker) { + this.ticker = ticker; + } + + public double getPrice() { + return price; + } + + public void setPrice(double price) { + this.price = price; + } +} diff --git a/java/KinesisSourceDeaggregation/kpl-producer/src/main/resources/simplelogger.properties b/java/KinesisSourceDeaggregation/kpl-producer/src/main/resources/simplelogger.properties new file mode 100644 index 00000000..7d6c7102 --- /dev/null +++ b/java/KinesisSourceDeaggregation/kpl-producer/src/main/resources/simplelogger.properties @@ -0,0 +1,8 @@ +# Configuration for SLF4J Simple Logger + +org.slf4j.simpleLogger.defaultLogLevel=info +org.slf4j.simpleLogger.showDateTime=false +org.slf4j.simpleLogger.showLogName=true +org.slf4j.simpleLogger.showShortLogName=true +org.slf4j.simpleLogger.showThreadName=true +org.slf4j.simpleLogger.levelInBrackets=true diff --git a/java/KinesisSourceDeaggregation/pom.xml b/java/KinesisSourceDeaggregation/pom.xml new file mode 100644 index 00000000..080082d7 --- /dev/null +++ b/java/KinesisSourceDeaggregation/pom.xml @@ -0,0 +1,16 @@ + + + 4.0.0 + + com.amazonaws + kinesis-source-deaggregation-example + 1.0 + pom + + kpl-producer + flink-app + + + \ No newline at end of file diff --git a/java/S3AvroSink/README.md b/java/S3AvroSink/README.md new file mode 100644 index 00000000..85947d21 --- /dev/null +++ b/java/S3AvroSink/README.md @@ -0,0 +1,54 @@ +# S3 Avro Sink + +* Flink version: 1.20 +* Flink API: DataStream API +* Language Java (11) +* Connectors: FileSystem Sink (and DataGen connector) + +This example demonstrates how to write AVRO files to S3. + +The example generates random stock price data using the DataGen connector and writes to S3 as AVRO files with +a bucketing in the format `year=yyyy/month=MM/day=dd/hour=HH/` and rotating files on checkpoint. + +Note that FileSystem sink commit the writes on checkpoint. For this reason, checkpoint are programmatically enabled when running locally. +When running on Managed Flink checkpoints are controlled by the application configuration and enabled by default. + +This application can be used in combination with the [S3AvroSource](../S3AvroSource) example application which read AVRO data with the same schema from S3. + +## Prerequisites + +* An S3 bucket for writing data. The application IAM Role must allow writing to the bucket + + +## Runtime Configuration + +The application reads the runtime configuration from the Runtime Properties, when running on Amazon Managed Service for Apache Flink, +or, when running locally, from the [`resources/flink-application-properties-dev.json`](resources/flink-application-properties-dev.json) file located in the resources folder. + +All parameters are case-sensitive. + +| Group ID | Key | Description | +|----------------|---------------|------------------------------------| +| `OutputBucket` | `bucket.name` | Name of the destination S3 bucket. | +| `OutputBucket` | `bucket.path` | Base path withing the bucket. | + +To configure the application on Managed Service for Apache Flink, set up these parameter in the *Runtime properties*. + +To configure the application for running locally, edit the [json file](resources/flink-application-properties-dev.json). + +### Running in IntelliJ + +You can run this example directly in IntelliJ, without any local Flink cluster or local Flink installation. + +See [Running examples locally](../running-examples-locally.md) for details. + +## AVRO Specific Record usage + +The AVRO schema definition (`avdl` file) is included as part of the source code in the `./src/main/resources/avro` folder. +The AVRO Maven Plugin is used to generate the Java object `StockPrice` at compile time. + +If IntelliJ cannot find the `StockPrice` class when you import the project: +1. Run `mvn generate-sources` manually once +2. Ensure that the IntelliJ module configuration, in Project settings, also includes `target/generated-sources/avro` as Sources. If IntelliJ does not auto-detect it, add the path manually + +These operations are only needed once. diff --git a/java/S3AvroSink/pom.xml b/java/S3AvroSink/pom.xml new file mode 100644 index 00000000..92e20b2c --- /dev/null +++ b/java/S3AvroSink/pom.xml @@ -0,0 +1,212 @@ + + + 4.0.0 + + com.amazonaws + s3-avro-sink + 1.0 + + + UTF-8 + ${project.basedir}/target + ${project.name}-${project.version} + UTF-8 + 11 + ${target.java.version} + ${target.java.version} + + 1.20.0 + 1.2.0 + 1.11.3 + 1.15.1 + + 2.23.1 + 5.8.1 + + + + + + com.amazonaws + aws-java-sdk-bom + + 1.12.782 + pom + import + + + + + + + + + org.apache.flink + flink-streaming-java + ${flink.version} + provided + + + org.apache.flink + flink-clients + ${flink.version} + provided + + + org.apache.flink + flink-runtime-web + ${flink.version} + provided + + + + + com.amazonaws + aws-kinesisanalytics-runtime + ${kda.runtime.version} + provided + + + + + org.apache.flink + flink-connector-files + ${flink.version} + provided + + + org.apache.flink + flink-s3-fs-hadoop + ${flink.version} + provided + + + + + org.apache.flink + flink-avro + ${flink.version} + + + org.apache.avro + avro + ${avro.version} + + + + + org.junit.jupiter + junit-jupiter + ${junit5.version} + test + + + + + + org.apache.logging.log4j + log4j-slf4j-impl + ${log4j.version} + + + org.apache.logging.log4j + log4j-api + ${log4j.version} + + + org.apache.logging.log4j + log4j-core + ${log4j.version} + runtime + + + + + + + + + org.apache.avro + avro-maven-plugin + ${avro.version} + + + generate-sources + + idl-protocol + + + ${project.basedir}/src/main/resources/avro + ${project.basedir}/src/test/resources/avro + private + String + true + + + + + + + + org.apache.maven.plugins + maven-compiler-plugin + 3.8.1 + + ${target.java.version} + ${target.java.version} + true + + + + + + org.apache.maven.plugins + maven-shade-plugin + 3.2.1 + + + + package + + shade + + + + + org.apache.flink:force-shading + com.google.code.findbugs:jsr305 + org.slf4j:* + org.apache.logging.log4j:* + + + + + + *:* + + META-INF/*.SF + META-INF/*.DSA + META-INF/*.RSA + + + + + + + com.amazonaws.services.msf.StreamingJob + + + + + + + + + + + \ No newline at end of file diff --git a/java/S3AvroSink/src/main/java/com/amazonaws/services/msf/StreamingJob.java b/java/S3AvroSink/src/main/java/com/amazonaws/services/msf/StreamingJob.java new file mode 100644 index 00000000..7a123655 --- /dev/null +++ b/java/S3AvroSink/src/main/java/com/amazonaws/services/msf/StreamingJob.java @@ -0,0 +1,112 @@ +package com.amazonaws.services.msf; + +import com.amazonaws.services.kinesisanalytics.runtime.KinesisAnalyticsRuntime; +import com.amazonaws.services.msf.avro.StockPrice; +import com.amazonaws.services.msf.datagen.StockPriceGeneratorFunction; +import org.apache.flink.api.common.eventtime.WatermarkStrategy; +import org.apache.flink.api.common.typeinfo.TypeInformation; +import org.apache.flink.api.connector.source.util.ratelimit.RateLimiterStrategy; +import org.apache.flink.connector.datagen.source.DataGeneratorSource; +import org.apache.flink.connector.file.sink.FileSink; +import org.apache.flink.core.fs.Path; +import org.apache.flink.formats.avro.AvroWriters; +import org.apache.flink.streaming.api.datastream.DataStream; +import org.apache.flink.streaming.api.environment.LocalStreamEnvironment; +import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; +import org.apache.flink.streaming.api.functions.sink.filesystem.OutputFileConfig; +import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.DateTimeBucketAssigner; +import org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.OnCheckpointRollingPolicy; +import org.apache.flink.util.Preconditions; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.util.Map; +import java.util.Properties; + +public class StreamingJob { + private static final Logger LOGGER = LoggerFactory.getLogger(StreamingJob.class); + + // Name of the local JSON resource with the application properties in the same format as they are received from the Amazon Managed Service for Apache Flink runtime + private static final String LOCAL_APPLICATION_PROPERTIES_RESOURCE = "flink-application-properties-dev.json"; + + private static boolean isLocal(StreamExecutionEnvironment env) { + return env instanceof LocalStreamEnvironment; + } + + /** + * Load application properties from Amazon Managed Service for Apache Flink runtime or from a local resource, when the environment is local + */ + private static Map loadApplicationProperties(StreamExecutionEnvironment env) throws IOException { + if (isLocal(env)) { + LOGGER.info("Loading application properties from '{}'", LOCAL_APPLICATION_PROPERTIES_RESOURCE); + return KinesisAnalyticsRuntime.getApplicationProperties( + StreamingJob.class.getClassLoader() + .getResource(LOCAL_APPLICATION_PROPERTIES_RESOURCE).getPath()); + } else { + LOGGER.info("Loading application properties from Amazon Managed Service for Apache Flink"); + return KinesisAnalyticsRuntime.getApplicationProperties(); + } + } + + private static DataGeneratorSource getStockPriceDataGeneratorSource() { + long recordPerSecond = 100; + return new DataGeneratorSource<>( + new StockPriceGeneratorFunction(), + Long.MAX_VALUE, + RateLimiterStrategy.perSecond(recordPerSecond), + TypeInformation.of(StockPrice.class)); + } + + private static FileSink getParquetS3Sink(String s3UrlPath) { + return FileSink + .forBulkFormat(new Path(s3UrlPath), AvroWriters.forSpecificRecord(StockPrice.class)) + // Bucketing + .withBucketAssigner(new DateTimeBucketAssigner<>("'year='yyyy'/month='MM'/day='dd'/hour='HH/")) + // Part file rolling - this is actually the default, rolling on checkpoint + .withRollingPolicy(OnCheckpointRollingPolicy.build()) + .withOutputFileConfig(OutputFileConfig.builder() + .withPartSuffix(".avro") + .build()) + .build(); + } + + public static void main(String[] args) throws Exception { + // set up the streaming execution environment + StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); + + // Local dev specific settings + if (isLocal(env)) { + // Checkpointing and parallelism are set by Amazon Managed Service for Apache Flink when running on AWS + env.enableCheckpointing(30000); + env.setParallelism(2); + } + + // Application configuration + Properties applicationProperties = loadApplicationProperties(env).get("OutputBucket"); + String bucketName = Preconditions.checkNotNull(applicationProperties.getProperty("bucket.name"), "Bucket for S3 not defined"); + String bucketPath = Preconditions.checkNotNull(applicationProperties.getProperty("bucket.path"), "Path in S3 not defined"); + + // Build S3 URL. Strip any initial fwd slash from bucket path + String s3UrlPath = String.format("s3a://%s/%s", bucketName.trim(), bucketPath.trim().replaceFirst("^/+", "")); + LOGGER.info("Output URL: {}", s3UrlPath); + + // Source (data generator) + DataGeneratorSource source = getStockPriceDataGeneratorSource(); + + // DataStream from source + DataStream stockPrices = env.fromSource( + source, WatermarkStrategy.noWatermarks(), "data-generator").setParallelism(1); + + // Sink (Parquet files to S3) + FileSink sink = getParquetS3Sink(s3UrlPath); + + stockPrices.sinkTo(sink).name("avro-s3-sink"); + + // Also print the output + // (This is for illustration purposes only and used when running locally. No output is printed when running on Managed Flink) + stockPrices.print(); + + env.execute("Sink Avro to S3"); + } +} diff --git a/java/S3AvroSink/src/main/java/com/amazonaws/services/msf/datagen/StockPriceGeneratorFunction.java b/java/S3AvroSink/src/main/java/com/amazonaws/services/msf/datagen/StockPriceGeneratorFunction.java new file mode 100644 index 00000000..21669697 --- /dev/null +++ b/java/S3AvroSink/src/main/java/com/amazonaws/services/msf/datagen/StockPriceGeneratorFunction.java @@ -0,0 +1,20 @@ +package com.amazonaws.services.msf.datagen; + +import com.amazonaws.services.msf.avro.StockPrice; +import org.apache.commons.lang3.RandomUtils; +import org.apache.flink.connector.datagen.source.GeneratorFunction; + +import java.time.Instant; + +public class StockPriceGeneratorFunction implements GeneratorFunction { + private static final String[] TICKERS = {"AAPL", "AMZN", "MSFT", "INTC", "TBV"}; + + @Override + public StockPrice map(Long aLong) { + return new StockPrice( + TICKERS[RandomUtils.nextInt(0, TICKERS.length)], + RandomUtils.nextFloat(10, 100), + RandomUtils.nextInt(1, 10000), + Instant.now().toEpochMilli()); + } +} \ No newline at end of file diff --git a/java/S3AvroSink/src/main/resources/avro/stockprice.avdl b/java/S3AvroSink/src/main/resources/avro/stockprice.avdl new file mode 100644 index 00000000..3292400d --- /dev/null +++ b/java/S3AvroSink/src/main/resources/avro/stockprice.avdl @@ -0,0 +1,9 @@ +@namespace("com.amazonaws.services.msf.avro") +protocol In { + record StockPrice { + string symbol; + float price; + int volume; + long timestamp; + } +} \ No newline at end of file diff --git a/java/S3AvroSink/src/main/resources/flink-application-properties-dev.json b/java/S3AvroSink/src/main/resources/flink-application-properties-dev.json new file mode 100644 index 00000000..e159802d --- /dev/null +++ b/java/S3AvroSink/src/main/resources/flink-application-properties-dev.json @@ -0,0 +1,9 @@ +[ + { + "PropertyGroupId": "OutputBucket", + "PropertyMap": { + "bucket.name": "", + "bucket.path": "avrostockprices" + } + } +] \ No newline at end of file diff --git a/java/S3AvroSink/src/main/resources/log4j2.properties b/java/S3AvroSink/src/main/resources/log4j2.properties new file mode 100644 index 00000000..b7e6ea52 --- /dev/null +++ b/java/S3AvroSink/src/main/resources/log4j2.properties @@ -0,0 +1,14 @@ +appender.console.name = ConsoleAppender +appender.console.type = CONSOLE +appender.console.layout.type = PatternLayout +#appender.console.layout.pattern = %d{yyyy-MM-dd HH:mm:ss} %-5p %c{1} - %m%n +appender.console.layout.pattern = %d{HH:mm:ss.SSS} %-5level %logger{36} - %msg%n + + +rootLogger.level = INFO +rootLogger.appenderRef.console.ref = ConsoleAppender + +#logger.verbose.name = org.apache.flink.connector.file +#logger.verbose.level = DEBUG +#logger.verbose.additivity = false +#logger.verbose.appenderRef.console.ref = ConsoleAppender \ No newline at end of file diff --git a/java/S3AvroSource/README.md b/java/S3AvroSource/README.md new file mode 100644 index 00000000..c4ed97a0 --- /dev/null +++ b/java/S3AvroSource/README.md @@ -0,0 +1,57 @@ +# S3 Avro Source + + +* Flink API: DataStream API +* Language Java (11) +* Connectors: FileSystem Source and Kinesis Sink + +This example demonstrates how to read AVRO files from S3. + +The application reads AVRO records written by the [S3AvroSink](../S3AvroSink) example application, from an S3 bucket and publish them to Kinesis as JSON. + +## Prerequisites + +* An S3 bucket containing the data. The application IAM Role must allow reading from the bucket +* A Kinesis Data Stream to output the data. The application IAM Role must allow publishing to the stream + +## Runtime Configuration + +The application reads the runtime configuration from the Runtime Properties, when running on Amazon Managed Service for Apache Flink, +or, when running locally, from the [`resources/flink-application-properties-dev.json`](resources/flink-application-properties-dev.json) file located in the resources folder. + +All parameters are case-sensitive. + +| Group ID | Key | Description | +|-----------------|----------------|-------------------------------| +| `InputBucket` | `bucket.name` | Name of the source S3 bucket. | +| `InputBucket` | `bucket.path` | Base path within the bucket. | +| `OutputStream0` | `stream.arn` | ARN of the output stream. | +| `OutputStream0` | `aws.region` | Region of the output stream. | + +Every parameter in the `OutputStream0` is passed to the Kinesis client of the sink. + +To configure the application on Managed Service for Apache Flink, set up these parameter in the *Runtime properties*. + +To configure the application for running locally, edit the [json file](resources/flink-application-properties-dev.json). + +### Running in IntelliJ + +You can run this example directly in IntelliJ, without any local Flink cluster or local Flink installation. + +See [Running examples locally](../running-examples-locally.md) for details. + +### Generating data + +You can use the [S3AvroSink](../S3AvroSink) example application to write AVRO data into an S3 bucket. +Both examples use the same AVRO schema. + +## AVRO Specific Record usage + +The AVRO reader's schema definition (`avdl` file) is included as part of the source code in the `./src/main/resources/avro` folder. +The AVRO Maven Plugin is used to generate the Java object `StockPrice` at compile time. + +If IntelliJ cannot find the `StockPrice` class when you import the project: +1. Run `mvn generate-sources` manually once +2. Ensure that the IntelliJ module configuration, in Project settings, also includes `target/generated-sources/avro` as Sources. If IntelliJ does not auto-detect it, add the path manually + +These operations are only needed once. diff --git a/java/S3AvroSource/pom.xml b/java/S3AvroSource/pom.xml new file mode 100644 index 00000000..576f4a26 --- /dev/null +++ b/java/S3AvroSource/pom.xml @@ -0,0 +1,224 @@ + + + 4.0.0 + + com.amazonaws + s3-avro-source + 1.0 + + + UTF-8 + ${project.basedir}/target + ${project.name}-${project.version} + UTF-8 + 11 + ${target.java.version} + ${target.java.version} + + 1.20.0 + 1.2.0 + 1.11.3 + 1.15.1 + 5.0.0-1.20 + + 2.23.1 + 5.8.1 + + + + + + com.amazonaws + aws-java-sdk-bom + + 1.12.782 + pom + import + + + + + + + + + org.apache.flink + flink-streaming-java + ${flink.version} + provided + + + org.apache.flink + flink-clients + ${flink.version} + provided + + + org.apache.flink + flink-runtime-web + ${flink.version} + provided + + + org.apache.flink + flink-json + ${flink.version} + provided + + + + + com.amazonaws + aws-kinesisanalytics-runtime + ${kda.runtime.version} + provided + + + + + org.apache.flink + flink-connector-files + ${flink.version} + provided + + + org.apache.flink + flink-s3-fs-hadoop + ${flink.version} + provided + + + org.apache.flink + flink-connector-aws-kinesis-streams + ${aws.connector.version} + + + + + org.apache.flink + flink-avro + ${flink.version} + + + org.apache.avro + avro + ${avro.version} + + + + + org.junit.jupiter + junit-jupiter + ${junit5.version} + test + + + + + + org.apache.logging.log4j + log4j-slf4j-impl + ${log4j.version} + + + org.apache.logging.log4j + log4j-api + ${log4j.version} + + + org.apache.logging.log4j + log4j-core + ${log4j.version} + runtime + + + + + + + + + org.apache.avro + avro-maven-plugin + ${avro.version} + + + generate-sources + + idl-protocol + + + ${project.basedir}/src/main/resources/avro + ${project.basedir}/src/test/resources/avro + private + String + true + + + + + + + + org.apache.maven.plugins + maven-compiler-plugin + 3.8.1 + + ${target.java.version} + ${target.java.version} + true + + + + + + org.apache.maven.plugins + maven-shade-plugin + 3.2.1 + + + + package + + shade + + + + + org.apache.flink:force-shading + com.google.code.findbugs:jsr305 + org.slf4j:* + org.apache.logging.log4j:* + + + + + + *:* + + META-INF/*.SF + META-INF/*.DSA + META-INF/*.RSA + + + + + + + com.amazonaws.services.msf.StreamingJob + + + + + + + + + + + \ No newline at end of file diff --git a/java/S3AvroSource/src/main/java/com/amazonaws/services/msf/AvroSpecificRecordBulkFormat.java b/java/S3AvroSource/src/main/java/com/amazonaws/services/msf/AvroSpecificRecordBulkFormat.java new file mode 100644 index 00000000..357f4846 --- /dev/null +++ b/java/S3AvroSource/src/main/java/com/amazonaws/services/msf/AvroSpecificRecordBulkFormat.java @@ -0,0 +1,44 @@ +package com.amazonaws.services.msf; + +import org.apache.avro.generic.GenericData; +import org.apache.avro.generic.GenericRecord; +import org.apache.avro.specific.SpecificData; +import org.apache.flink.api.common.typeinfo.TypeInformation; +import org.apache.flink.connector.file.src.FileSourceSplit; +import org.apache.flink.formats.avro.AbstractAvroBulkFormat; + +import java.util.function.Function; + +/** + * Simple BulkFormat to read AVRO files into SpecificRecord + * @param + */ +public class AvroSpecificRecordBulkFormat + extends AbstractAvroBulkFormat { + + private final TypeInformation producedTypeInfo; + private final org.apache.avro.Schema avroReaderSchema; + + + public AvroSpecificRecordBulkFormat(Class recordClass, org.apache.avro.Schema avroSchema) { + super(avroSchema); + avroReaderSchema = avroSchema; + producedTypeInfo = TypeInformation.of(recordClass); + } + + @Override + protected GenericRecord createReusedAvroRecord() { + return new GenericData.Record(avroReaderSchema); + } + + @Override + protected Function createConverter() { + return genericRecord -> (O) SpecificData.get().deepCopy(avroReaderSchema, genericRecord); + } + + @Override + public TypeInformation getProducedType() { + return producedTypeInfo; + } + +} diff --git a/java/S3AvroSource/src/main/java/com/amazonaws/services/msf/JsonConverter.java b/java/S3AvroSource/src/main/java/com/amazonaws/services/msf/JsonConverter.java new file mode 100644 index 00000000..d8eda3e1 --- /dev/null +++ b/java/S3AvroSource/src/main/java/com/amazonaws/services/msf/JsonConverter.java @@ -0,0 +1,37 @@ +package com.amazonaws.services.msf; + +import org.apache.avro.Schema; +import org.apache.avro.io.EncoderFactory; +import org.apache.avro.io.JsonEncoder; +import org.apache.avro.specific.SpecificDatumWriter; +import org.apache.flink.api.common.functions.OpenContext; +import org.apache.flink.api.common.functions.RichMapFunction; + +import java.io.ByteArrayOutputStream; + +/** + * Simple converter for Avro records to JSON. Not designed for production. + */ +public class JsonConverter extends RichMapFunction { + + private final Schema avroSchema; + private transient SpecificDatumWriter writer; + + public JsonConverter(Schema avroSchema) { + this.avroSchema = avroSchema; + } + + @Override + public void open(OpenContext openContext) throws Exception { + this.writer = new SpecificDatumWriter<>(avroSchema); + } + + @Override + public String map(T record) throws Exception { + ByteArrayOutputStream outputStream = new ByteArrayOutputStream(); + JsonEncoder encoder = EncoderFactory.get().jsonEncoder(record.getSchema(), outputStream); + writer.write(record, encoder); + encoder.flush(); + return outputStream.toString(); + } +} diff --git a/java/S3AvroSource/src/main/java/com/amazonaws/services/msf/StreamingJob.java b/java/S3AvroSource/src/main/java/com/amazonaws/services/msf/StreamingJob.java new file mode 100644 index 00000000..5ef854f6 --- /dev/null +++ b/java/S3AvroSource/src/main/java/com/amazonaws/services/msf/StreamingJob.java @@ -0,0 +1,109 @@ +package com.amazonaws.services.msf; + +import com.amazonaws.services.kinesisanalytics.runtime.KinesisAnalyticsRuntime; +import com.amazonaws.services.msf.avro.StockPrice; +import org.apache.avro.Schema; +import org.apache.avro.specific.SpecificRecord; +import org.apache.flink.api.common.eventtime.WatermarkStrategy; +import org.apache.flink.api.common.serialization.SerializationSchema; +import org.apache.flink.api.common.serialization.SimpleStringSchema; +import org.apache.flink.api.common.typeinfo.TypeInformation; +import org.apache.flink.connector.file.src.FileSource; +import org.apache.flink.connector.kinesis.sink.KinesisStreamsSink; +import org.apache.flink.core.fs.Path; +import org.apache.flink.formats.json.JsonSerializationSchema; +import org.apache.flink.streaming.api.datastream.DataStream; +import org.apache.flink.streaming.api.environment.LocalStreamEnvironment; +import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; +import org.apache.flink.util.Preconditions; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.time.Duration; +import java.util.Map; +import java.util.Properties; + +public class StreamingJob { + private static final Logger LOGGER = LoggerFactory.getLogger(StreamingJob.class); + + // Name of the local JSON resource with the application properties in the same format as they are received from the Amazon Managed Service for Apache Flink runtime + private static final String LOCAL_APPLICATION_PROPERTIES_RESOURCE = "flink-application-properties-dev.json"; + + private static boolean isLocal(StreamExecutionEnvironment env) { + return env instanceof LocalStreamEnvironment; + } + + /** + * Load application properties from Amazon Managed Service for Apache Flink runtime or from a local resource, when the environment is local + */ + private static Map loadApplicationProperties(StreamExecutionEnvironment env) throws IOException { + if (isLocal(env)) { + LOGGER.info("Loading application properties from '{}'", LOCAL_APPLICATION_PROPERTIES_RESOURCE); + return KinesisAnalyticsRuntime.getApplicationProperties( + StreamingJob.class.getClassLoader() + .getResource(LOCAL_APPLICATION_PROPERTIES_RESOURCE).getPath()); + } else { + LOGGER.info("Loading application properties from Amazon Managed Service for Apache Flink"); + return KinesisAnalyticsRuntime.getApplicationProperties(); + } + } + + + private static KinesisStreamsSink createKinesisSink(Properties outputProperties, final SerializationSchema serializationSchema) { + final String outputStreamArn = outputProperties.getProperty("stream.arn"); + return KinesisStreamsSink.builder() + .setStreamArn(outputStreamArn) + .setKinesisClientProperties(outputProperties) + .setSerializationSchema(serializationSchema) + .setPartitionKeyGenerator(element -> String.valueOf(element.hashCode())) + .build(); + } + + private static FileSource createAvroFileSource(Properties sourceProperties, Class avroRecordClass, Schema avroSchema) { + String bucketName = Preconditions.checkNotNull(sourceProperties.getProperty("bucket.name"), "Bucket for S3 not defined"); + String bucketPath = Preconditions.checkNotNull(sourceProperties.getProperty("bucket.path"), "Path in S3 not defined"); + + // Build S3 URL. Strip any initial fwd slash from bucket path + String s3UrlPath = String.format("s3a://%s/%s", bucketName.trim(), bucketPath.trim().replaceFirst("^/+", "")); + LOGGER.info("Input URL: {}", s3UrlPath); + + // A custom BulkFormat is required to read AVRO files + AvroSpecificRecordBulkFormat bulkFormat = new AvroSpecificRecordBulkFormat<>(avroRecordClass, avroSchema); + + return FileSource.forBulkFileFormat(bulkFormat, new Path(s3UrlPath)) + .monitorContinuously(Duration.ofSeconds(10)) + .build(); + } + + public static void main(String[] args) throws Exception { + // set up the streaming execution environment + StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); + + // Application configuration + Map applicationPropertiesMap = loadApplicationProperties(env); + + // Source reading AVRO files into a SpecificRecord + FileSource avroFileSource = createAvroFileSource(applicationPropertiesMap.get("InputBucket"), StockPrice.class, StockPrice.SCHEMA$); + + // DataStream from source + DataStream stockPrices = env.fromSource( + avroFileSource, WatermarkStrategy.noWatermarks(), "avro-source", TypeInformation.of(StockPrice.class)).setParallelism(1); + + // Convert the AVRO record into a String containing JSON + DataStream jsonStockPrices = stockPrices.map(new JsonConverter<>(StockPrice.SCHEMA$)); + + // Output the Strings to Kinesis + // (You cannot use JsonSerializationSchema to convert AVRO specific records into JSON, directly) + KinesisStreamsSink kinesisSink = createKinesisSink(applicationPropertiesMap.get("OutputStream0"), new SimpleStringSchema()); + + // Attach the sink + jsonStockPrices.sinkTo(kinesisSink); + + // Also print the output + // (This is for illustration purposes only and used when running locally. No output is printed when running on Managed Flink) + jsonStockPrices.print(); + + env.execute("Source Avro from S3"); + } +} diff --git a/java/S3AvroSource/src/main/resources/avro/stockprice.avdl b/java/S3AvroSource/src/main/resources/avro/stockprice.avdl new file mode 100644 index 00000000..3292400d --- /dev/null +++ b/java/S3AvroSource/src/main/resources/avro/stockprice.avdl @@ -0,0 +1,9 @@ +@namespace("com.amazonaws.services.msf.avro") +protocol In { + record StockPrice { + string symbol; + float price; + int volume; + long timestamp; + } +} \ No newline at end of file diff --git a/java/S3AvroSource/src/main/resources/flink-application-properties-dev.json b/java/S3AvroSource/src/main/resources/flink-application-properties-dev.json new file mode 100644 index 00000000..a70379c7 --- /dev/null +++ b/java/S3AvroSource/src/main/resources/flink-application-properties-dev.json @@ -0,0 +1,16 @@ +[ + { + "PropertyGroupId": "InputBucket", + "PropertyMap": { + "bucket.name": "", + "bucket.path": "avrostockprices" + } + }, + { + "PropertyGroupId": "OutputStream0", + "PropertyMap": { + "stream.arn": "arn:aws:kinesis:us-east-1::stream/", + "aws.region": "us-east-1" + } + } +] \ No newline at end of file diff --git a/java/S3AvroSource/src/main/resources/log4j2.properties b/java/S3AvroSource/src/main/resources/log4j2.properties new file mode 100644 index 00000000..b7e6ea52 --- /dev/null +++ b/java/S3AvroSource/src/main/resources/log4j2.properties @@ -0,0 +1,14 @@ +appender.console.name = ConsoleAppender +appender.console.type = CONSOLE +appender.console.layout.type = PatternLayout +#appender.console.layout.pattern = %d{yyyy-MM-dd HH:mm:ss} %-5p %c{1} - %m%n +appender.console.layout.pattern = %d{HH:mm:ss.SSS} %-5level %logger{36} - %msg%n + + +rootLogger.level = INFO +rootLogger.appenderRef.console.ref = ConsoleAppender + +#logger.verbose.name = org.apache.flink.connector.file +#logger.verbose.level = DEBUG +#logger.verbose.additivity = false +#logger.verbose.appenderRef.console.ref = ConsoleAppender \ No newline at end of file diff --git a/java/S3ParquetSink/README.md b/java/S3ParquetSink/README.md index 21027f61..24aa367e 100644 --- a/java/S3ParquetSink/README.md +++ b/java/S3ParquetSink/README.md @@ -6,6 +6,7 @@ * Connectors: FileSystem Sink (and DataGen connector) This example demonstrates how to write Parquet files to S3. +See the [S3 Parquet Source](../S3ParquetSource) example to read Parquet files from S3. The example generates random stock price data using the DataGen connector and writes to S3 as Parquet files with a bucketing in the format `year=yyyy/month=MM/day=dd/hour=HH/` and rotating files on checkpoint. @@ -13,6 +14,7 @@ a bucketing in the format `year=yyyy/month=MM/day=dd/hour=HH/` and rotating file Note that FileSystem sink commit the writes on checkpoint. For this reason, checkpoint are programmatically enabled when running locally. When running on Managed Flink checkpoints are controlled by the application configuration and enabled by default. + ## Prerequisites * An S3 bucket for writing data. The application IAM Role must allow writing to the bucket diff --git a/java/S3ParquetSource/README.md b/java/S3ParquetSource/README.md new file mode 100644 index 00000000..b9f721d4 --- /dev/null +++ b/java/S3ParquetSource/README.md @@ -0,0 +1,76 @@ +# S3 Parquet Source + +* Flink API: DataStream API +* Language Java (11) +* Connectors: FileSystem Source and Kinesis Sink + +This example demonstrates how to read Parquet files from S3. + +The application reads records written by the [S3ParquetSink](../S3ParquetSink) example application, from an S3 bucket and publish them to Kinesis as JSON. + +The records read from Parquet are deserialized into AVRO specific objects. + +## Important note about reading Parquet + +Flink relies on Hadoop libraries to read Parquet files from S3. +Because of the [Flink classloading system](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/ops/debugging/debugging_classloading/) introduced after Flink 1.5, +and because Flink S3 File System is installed in the cluster to support checkpointing, reading Parquet files normally causes a class not found exception in some Hadoop classes, +even if you include these classes in the uber-jar. + +This examples demonstrates a workaround to this issue. The `org.apache.flink.runtime.util.HadoopUtils` class is replaced +by a custom implementation, and some Hadoop classes are shaded (remapped) by the `maven-shade-plugin` used to build the uber-jar. + +Note that this workaround works in this specific case and may not help in other cases of Hadoop class conflict you may encounter. + +## Prerequisites + +* An S3 bucket containing the data. The application IAM Role must allow reading from the bucket +* A Kinesis Data Stream to output the data. The application IAM Role must allow publishing to the stream + +## Runtime Configuration + +The application reads the runtime configuration from the Runtime Properties, when running on Amazon Managed Service for Apache Flink, +or, when running locally, from the [`resources/flink-application-properties-dev.json`](resources/flink-application-properties-dev.json) file located in the resources folder. + +All parameters are case-sensitive. + +| Group ID | Key | Description | +|-----------------|--------------------------------|---------------------------------------------------------------------------------| +| `InputBucket` | `bucket.name` | Name of the destination S3 bucket. | +| `InputBucket` | `bucket.path` | Base path within the bucket. | +| `InputBucket` | `discovery.interval.seconds` | Inteval the bucket path is scanned for new files, in seconds (default = 30 sec) | +| `OutputStream0` | `stream.arn` | ARN of the output stream. | +| `OutputStream0` | `aws.region` | Region of the output stream. | + +Every parameter in the `OutputStream0` is passed to the Kinesis client of the sink. + +To configure the application on Managed Service for Apache Flink, set up these parameter in the *Runtime properties*. + +To configure the application for running locally, edit the [json file](resources/flink-application-properties-dev.json). + +### Running in IntelliJ + +You can run this example directly in IntelliJ, without any local Flink cluster or local Flink installation. + +See [Running examples locally](../running-examples-locally.md) for details. + +Note: when running locally, the application also prints the records to the console. +Records starts appearing about 30 seconds after the application is fully initialized and starts reading from S3. + +### Generating data + +You can use the [S3ParquetSink](../S3ParquetSink) example application to write Parquet data into an S3 bucket. +Both examples use the same AVRO schema. + +## AVRO Specific Record usage + +The records read from Parquet are deserialized into AVRO specific objects. + +The AVRO reader's schema definition (`avdl` file) is included as part of the source code in the `./src/main/resources/avro` folder. +The AVRO Maven Plugin is used to generate the Java object `StockPrice` at compile time. + +If IntelliJ cannot find the `StockPrice` class when you import the project: +1. Run `mvn generate-sources` manually once +2. Ensure that the IntelliJ module configuration, in Project settings, also includes `target/generated-sources/avro` as Sources. If IntelliJ does not auto-detect it, add the path manually + +These operations are only needed once. diff --git a/java/S3ParquetSource/pom.xml b/java/S3ParquetSource/pom.xml new file mode 100644 index 00000000..f8eb5aed --- /dev/null +++ b/java/S3ParquetSource/pom.xml @@ -0,0 +1,400 @@ + + + 4.0.0 + + com.amazonaws + s3-parquet-source + 1.0 + + + UTF-8 + ${project.basedir}/target + ${project.name}-${project.version} + UTF-8 + 11 + ${target.java.version} + ${target.java.version} + + 1.20.0 + 1.2.0 + 1.11.3 + 1.12.3 + 3.3.4 + 5.0.0-1.20 + 2.23.1 + 5.8.1 + + + + + + com.amazonaws + aws-java-sdk-bom + + 1.12.782 + pom + import + + + + + + + + + org.apache.flink + flink-streaming-java + ${flink.version} + provided + + + org.apache.flink + flink-clients + ${flink.version} + provided + + + org.apache.flink + flink-runtime-web + ${flink.version} + provided + + + org.apache.flink + flink-json + ${flink.version} + provided + + + + + com.amazonaws + aws-kinesisanalytics-runtime + ${kda.runtime.version} + provided + + + + + org.apache.flink + flink-connector-aws-kinesis-streams + ${aws.connector.version} + + + org.apache.flink + flink-connector-files + ${flink.version} + provided + + + + + org.apache.flink + flink-s3-fs-hadoop + ${flink.version} + provided + + + + + org.apache.hadoop + hadoop-client + ${hadoop.version} + + + + + org.apache.hadoop + hadoop-yarn-api + + + org.apache.hadoop + hadoop-yarn-client + + + org.apache.hadoop + hadoop-mapreduce-client-jobclient + + + jdk.tools + jdk.tools + + + com.jcraft + jsch + + + com.sun.jersey + jersey-core + + + com.sun.jersey + jersey-servlet + + + com.sun.jersey + jersey-json + + + com.sun.jersey + jersey-server + + + org.apache.avro + avro + + + org.eclipse.jetty + jetty-server + + + org.eclipse.jetty + jetty-util + + + org.eclipse.jetty + jetty-servlet + + + org.eclipse.jetty + jetty-webapp + + + javax.servlet + javax.servlet-api + + + javax.servlet.jsp + jsp-api + + + org.apache.kerby + kerb-simplekdc + + + org.apache.curator + curator-client + + + org.apache.curator + curator-framework + + + org.apache.curator + curator-recipes + + + org.apache.zookeeper + zookeeper + + + commons-net + commons-net + + + commons-cli + commons-cli + + + commons-codec + commons-codec + + + com.google.protobuf + protobuf-java + + + com.google.code.gson + gson + + + org.apache.httpcomponents + httpclient + + + org.apache.commons + commons-math3 + + + com.nimbusds + nimbus-jose-jwt + + + net.minidev + json-smart + + + + + ch.qos.reload4j + reload4j + + + org.slf4j + * + + + org.slf4j + slf4j-log4j12 + + + log4j + log4j + + + + + + + org.apache.flink + flink-parquet + ${flink.version} + + + org.apache.parquet + parquet-avro + ${parquet.avro.version} + + + + + org.apache.flink + flink-avro + ${flink.version} + + + org.apache.avro + avro + ${avro.version} + + + + + org.junit.jupiter + junit-jupiter + ${junit5.version} + test + + + + + + org.apache.logging.log4j + log4j-slf4j-impl + ${log4j.version} + + + org.apache.logging.log4j + log4j-api + ${log4j.version} + + + org.apache.logging.log4j + log4j-core + ${log4j.version} + runtime + + + + + ${jar.finalName} + + + + + org.apache.avro + avro-maven-plugin + ${avro.version} + + + generate-sources + + idl-protocol + + + ${project.basedir}/src/main/resources/avro + ${project.basedir}/src/test/resources/avro + private + String + true + + + + + + + + org.apache.maven.plugins + maven-compiler-plugin + 3.8.1 + + ${target.java.version} + ${target.java.version} + true + + + + + + org.apache.maven.plugins + maven-shade-plugin + 3.2.1 + + + + package + + shade + + + + + org.apache.flink:force-shading + com.google.code.findbugs:jsr305 + org.slf4j:* + org.apache.logging.log4j:* + + + + + *:* + + + META-INF/*.SF + META-INF/*.DSA + META-INF/*.RSA + + + + + + + com.amazonaws.services.msf.S3ParquetToKinesisJob + + + + + + + org.apache.hadoop.conf + shaded.org.apache.hadoop.conf + + + org.apache.flink.runtime.util.HadoopUtils + shadow.org.apache.flink.runtime.util.HadoopUtils + + + + + + + + + + + \ No newline at end of file diff --git a/java/S3ParquetSource/src/main/java/com/amazonaws/services/msf/JsonConverter.java b/java/S3ParquetSource/src/main/java/com/amazonaws/services/msf/JsonConverter.java new file mode 100644 index 00000000..d8eda3e1 --- /dev/null +++ b/java/S3ParquetSource/src/main/java/com/amazonaws/services/msf/JsonConverter.java @@ -0,0 +1,37 @@ +package com.amazonaws.services.msf; + +import org.apache.avro.Schema; +import org.apache.avro.io.EncoderFactory; +import org.apache.avro.io.JsonEncoder; +import org.apache.avro.specific.SpecificDatumWriter; +import org.apache.flink.api.common.functions.OpenContext; +import org.apache.flink.api.common.functions.RichMapFunction; + +import java.io.ByteArrayOutputStream; + +/** + * Simple converter for Avro records to JSON. Not designed for production. + */ +public class JsonConverter extends RichMapFunction { + + private final Schema avroSchema; + private transient SpecificDatumWriter writer; + + public JsonConverter(Schema avroSchema) { + this.avroSchema = avroSchema; + } + + @Override + public void open(OpenContext openContext) throws Exception { + this.writer = new SpecificDatumWriter<>(avroSchema); + } + + @Override + public String map(T record) throws Exception { + ByteArrayOutputStream outputStream = new ByteArrayOutputStream(); + JsonEncoder encoder = EncoderFactory.get().jsonEncoder(record.getSchema(), outputStream); + writer.write(record, encoder); + encoder.flush(); + return outputStream.toString(); + } +} diff --git a/java/S3ParquetSource/src/main/java/com/amazonaws/services/msf/S3ParquetToKinesisJob.java b/java/S3ParquetSource/src/main/java/com/amazonaws/services/msf/S3ParquetToKinesisJob.java new file mode 100644 index 00000000..5bd00c64 --- /dev/null +++ b/java/S3ParquetSource/src/main/java/com/amazonaws/services/msf/S3ParquetToKinesisJob.java @@ -0,0 +1,108 @@ +package com.amazonaws.services.msf; + +import com.amazonaws.services.kinesisanalytics.runtime.KinesisAnalyticsRuntime; +import com.amazonaws.services.msf.avro.StockPrice; +import org.apache.avro.specific.SpecificRecordBase; +import org.apache.flink.api.common.eventtime.WatermarkStrategy; +import org.apache.flink.api.common.serialization.SerializationSchema; +import org.apache.flink.api.common.serialization.SimpleStringSchema; +import org.apache.flink.connector.file.src.FileSource; +import org.apache.flink.connector.file.src.reader.StreamFormat; +import org.apache.flink.connector.kinesis.sink.KinesisStreamsSink; +import org.apache.flink.core.fs.Path; +import org.apache.flink.formats.json.JsonSerializationSchema; +import org.apache.flink.formats.parquet.avro.AvroParquetReaders; +import org.apache.flink.streaming.api.datastream.DataStream; +import org.apache.flink.streaming.api.environment.LocalStreamEnvironment; +import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; +import org.apache.flink.util.Preconditions; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.time.Duration; +import java.util.Map; +import java.util.Properties; + +public class S3ParquetToKinesisJob { + private static final Logger LOGGER = LoggerFactory.getLogger(S3ParquetToKinesisJob.class); + + // Name of the local JSON resource with the application properties in the same format as they are received from the Amazon Managed Service for Apache Flink runtime + private static final String LOCAL_APPLICATION_PROPERTIES_RESOURCE = "flink-application-properties-dev.json"; + + private static boolean isLocal(StreamExecutionEnvironment env) { + return env instanceof LocalStreamEnvironment; + } + + /** + * Load application properties from Amazon Managed Service for Apache Flink runtime or from a local resource, when the environment is local + */ + private static Map loadApplicationProperties(StreamExecutionEnvironment env) throws IOException { + if (isLocal(env)) { + LOGGER.info("Loading application properties from '{}'", LOCAL_APPLICATION_PROPERTIES_RESOURCE); + return KinesisAnalyticsRuntime.getApplicationProperties( + S3ParquetToKinesisJob.class.getClassLoader() + .getResource(LOCAL_APPLICATION_PROPERTIES_RESOURCE).getPath()); + } else { + LOGGER.info("Loading application properties from Amazon Managed Service for Apache Flink"); + return KinesisAnalyticsRuntime.getApplicationProperties(); + } + } + + private static FileSource createParquetS3Source(Properties applicationProperties, final Class clazz) { + String bucketName = Preconditions.checkNotNull(applicationProperties.getProperty("bucket.name"), "Bucket for S3 not defined"); + String bucketPath = Preconditions.checkNotNull(applicationProperties.getProperty("bucket.path"), "Path in S3 not defined"); + int discoveryIntervalSec = Integer.parseInt(applicationProperties.getProperty("bucket.discovery.interval.sec", "30")); + + // Build S3 URL. Strip any initial fwd slash from bucket path + String s3UrlPath = String.format("s3a://%s/%s", bucketName.trim(), bucketPath.trim().replaceFirst("^/+", "")); + LOGGER.info("Input URL: {}", s3UrlPath); + LOGGER.info("Discovery interval: {} sec", discoveryIntervalSec); + + return FileSource + .forRecordStreamFormat(AvroParquetReaders.forSpecificRecord(clazz), new Path(s3UrlPath)) + .monitorContinuously(Duration.ofSeconds(discoveryIntervalSec)) + .build(); + } + + private static KinesisStreamsSink createKinesisSink(Properties outputProperties, final SerializationSchema serializationSchema) { + final String outputStreamArn = outputProperties.getProperty("stream.arn"); + return KinesisStreamsSink.builder() + .setStreamArn(outputStreamArn) + .setKinesisClientProperties(outputProperties) + .setSerializationSchema(serializationSchema) + .setPartitionKeyGenerator(element -> String.valueOf(element.hashCode())) + .build(); + } + + public static void main(String[] args) throws Exception { + // set up the streaming execution environment + StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); + + final Map applicationProperties = loadApplicationProperties(env); + LOGGER.info("Application properties: {}", applicationProperties); + + FileSource source = createParquetS3Source(applicationProperties.get("InputBucket"), StockPrice.class); + KinesisStreamsSink sink = createKinesisSink(applicationProperties.get("OutputStream0"),new SimpleStringSchema()); + + // DataStream from source + DataStream stockPrices = env.fromSource( + source, WatermarkStrategy.noWatermarks(), "parquet-source").setParallelism(1); + + // Convert to JSON + // (We cannot use JsonSerializationSchema on the sink with an AVRO specific object) + DataStream jsonPrices = stockPrices + .map(new JsonConverter<>(StockPrice.getClassSchema())).uid("json-converter"); + + // Sink JSON to Kinesis + jsonPrices.sinkTo(sink).name("kinesis-sink"); + + // Also print to stdout for local testing + // (Do not print records to stdout in a production application. This adds overhead and is not visible when deployed on Managed Flink) + jsonPrices.print().name("stdout-sink"); + + env.execute("Source Parquet from S3"); + } + + +} diff --git a/java/S3ParquetSource/src/main/java/org/apache/flink/runtime/util/HadoopUtils.java b/java/S3ParquetSource/src/main/java/org/apache/flink/runtime/util/HadoopUtils.java new file mode 100644 index 00000000..bc8cd8c9 --- /dev/null +++ b/java/S3ParquetSource/src/main/java/org/apache/flink/runtime/util/HadoopUtils.java @@ -0,0 +1,120 @@ +package org.apache.flink.runtime.util; + +import org.apache.flink.api.java.tuple.Tuple2; +import org.apache.flink.util.FlinkRuntimeException; +import org.apache.flink.util.Preconditions; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.io.Text; +import org.apache.hadoop.security.UserGroupInformation; +import org.apache.hadoop.security.token.Token; +import org.apache.hadoop.security.token.TokenIdentifier; +import org.apache.hadoop.util.VersionInfo; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.Collection; + + +/** + * This class is a copy of org.apache.flink.runtime.util.HadoopUtils with the getHadoopConfiguration() method replaced to + * return an org.apache.hadoop.conf.Configuration instead of org.apache.hadoop.hdfs.HdfsConfiguration. + * + * This class is then shaded, along with org.apache.hadoop.conf.*, to avoid conflicts with the same classes provided by + * org.apache.flink:flink-s3-fs-hadoop, which is normally installed as plugin in Flink when S3. + * + * Other methods are copied from the original class. + */ +public class HadoopUtils { + private static final Logger LOG = LoggerFactory.getLogger(HadoopUtils.class); + + static final Text HDFS_DELEGATION_TOKEN_KIND = new Text("HDFS_DELEGATION_TOKEN"); + + /** + * This method has been re-implemented to always return a org.apache.hadoop.conf.Configuration + */ + public static Configuration getHadoopConfiguration( + org.apache.flink.configuration.Configuration flinkConfiguration) { + return new Configuration(false); + } + + public static boolean isKerberosSecurityEnabled(UserGroupInformation ugi) { + return UserGroupInformation.isSecurityEnabled() + && ugi.getAuthenticationMethod() + == UserGroupInformation.AuthenticationMethod.KERBEROS; + } + + + public static boolean areKerberosCredentialsValid( + UserGroupInformation ugi, boolean useTicketCache) { + Preconditions.checkState(isKerberosSecurityEnabled(ugi)); + + // note: UGI::hasKerberosCredentials inaccurately reports false + // for logins based on a keytab (fixed in Hadoop 2.6.1, see HADOOP-10786), + // so we check only in ticket cache scenario. + if (useTicketCache && !ugi.hasKerberosCredentials()) { + if (hasHDFSDelegationToken(ugi)) { + LOG.warn( + "Hadoop security is enabled but current login user does not have Kerberos credentials, " + + "use delegation token instead. Flink application will terminate after token expires."); + return true; + } else { + LOG.error( + "Hadoop security is enabled, but current login user has neither Kerberos credentials " + + "nor delegation tokens!"); + return false; + } + } + + return true; + } + + /** + * Indicates whether the user has an HDFS delegation token. + */ + public static boolean hasHDFSDelegationToken(UserGroupInformation ugi) { + Collection> usrTok = ugi.getTokens(); + for (Token token : usrTok) { + if (token.getKind().equals(HDFS_DELEGATION_TOKEN_KIND)) { + return true; + } + } + return false; + } + + /** + * Checks if the Hadoop dependency is at least the given version. + */ + public static boolean isMinHadoopVersion(int major, int minor) throws FlinkRuntimeException { + final Tuple2 hadoopVersion = getMajorMinorBundledHadoopVersion(); + int maj = hadoopVersion.f0; + int min = hadoopVersion.f1; + + return maj > major || (maj == major && min >= minor); + } + + /** + * Checks if the Hadoop dependency is at most the given version. + */ + public static boolean isMaxHadoopVersion(int major, int minor) throws FlinkRuntimeException { + final Tuple2 hadoopVersion = getMajorMinorBundledHadoopVersion(); + int maj = hadoopVersion.f0; + int min = hadoopVersion.f1; + + return maj < major || (maj == major && min < minor); + } + + private static Tuple2 getMajorMinorBundledHadoopVersion() { + String versionString = VersionInfo.getVersion(); + String[] versionParts = versionString.split("\\."); + + if (versionParts.length < 2) { + throw new FlinkRuntimeException( + "Cannot determine version of Hadoop, unexpected version string: " + + versionString); + } + + int maj = Integer.parseInt(versionParts[0]); + int min = Integer.parseInt(versionParts[1]); + return Tuple2.of(maj, min); + } +} diff --git a/java/S3ParquetSource/src/main/resources/avro/stockprice.avdl b/java/S3ParquetSource/src/main/resources/avro/stockprice.avdl new file mode 100644 index 00000000..3292400d --- /dev/null +++ b/java/S3ParquetSource/src/main/resources/avro/stockprice.avdl @@ -0,0 +1,9 @@ +@namespace("com.amazonaws.services.msf.avro") +protocol In { + record StockPrice { + string symbol; + float price; + int volume; + long timestamp; + } +} \ No newline at end of file diff --git a/java/S3ParquetSource/src/main/resources/flink-application-properties-dev.json b/java/S3ParquetSource/src/main/resources/flink-application-properties-dev.json new file mode 100644 index 00000000..b4b8fefe --- /dev/null +++ b/java/S3ParquetSource/src/main/resources/flink-application-properties-dev.json @@ -0,0 +1,17 @@ +[ + { + "PropertyGroupId": "InputBucket", + "PropertyMap": { + "bucket.name": "", + "bucket.path": "parquetstockprices", + "discovery.interval.seconds": "30" + } + }, + { + "PropertyGroupId": "OutputStream0", + "PropertyMap": { + "stream.arn": "arn:aws:kinesis:us-east-1::stream/", + "aws.region": "us-east-1" + } + } +] \ No newline at end of file diff --git a/java/S3ParquetSource/src/main/resources/log4j2.properties b/java/S3ParquetSource/src/main/resources/log4j2.properties new file mode 100644 index 00000000..2faa1c83 --- /dev/null +++ b/java/S3ParquetSource/src/main/resources/log4j2.properties @@ -0,0 +1,14 @@ +appender.console.name = ConsoleAppender +appender.console.type = CONSOLE +appender.console.layout.type = PatternLayout +#appender.console.layout.pattern = %d{yyyy-MM-dd HH:mm:ss} %-5p %c{1} - %m%n +appender.console.layout.pattern = %d{HH:mm:ss.SSS} %-5level %logger{36} - %msg%n + + +rootLogger.level = INFO +rootLogger.appenderRef.console.ref = ConsoleAppender + +logger.verbose.name = org.apache.flink.connector.file +logger.verbose.level = DEBUG +logger.verbose.additivity = false +logger.verbose.appenderRef.console.ref = ConsoleAppender \ No newline at end of file diff --git a/java/SQSSink/src/main/resources/flink-application-properties-dev.json b/java/SQSSink/src/main/resources/flink-application-properties-dev.json index 287ed366..4fd412da 100644 --- a/java/SQSSink/src/main/resources/flink-application-properties-dev.json +++ b/java/SQSSink/src/main/resources/flink-application-properties-dev.json @@ -2,7 +2,7 @@ { "PropertyGroupId": "OutputQueue0", "PropertyMap": { - "sqs-url": "https://sqs.us-east-1.amazonaws.com/012345678901/MyTestQueue", + "sqs-url": "https://sqs.us-east-1.amazonaws.com//MyTestQueue", "aws.region": "us-east-1" } } diff --git a/java/Serialization/README.md b/java/Serialization/README.md new file mode 100644 index 00000000..dfa39ac4 --- /dev/null +++ b/java/Serialization/README.md @@ -0,0 +1,8 @@ +# Serialization Examples + +Examples demonstrating data serialization patterns and custom type handling in Amazon Managed Service for Apache Flink. + +## Table of Contents + +### Custom Serialization +- [**Custom TypeInfo**](./CustomTypeInfo) - Using custom TypeInformation to avoid Kryo serialization fallback diff --git a/java/SideOutputs/src/main/resources/flink-application-properties-dev.json b/java/SideOutputs/src/main/resources/flink-application-properties-dev.json index 098cd569..174a8d1f 100644 --- a/java/SideOutputs/src/main/resources/flink-application-properties-dev.json +++ b/java/SideOutputs/src/main/resources/flink-application-properties-dev.json @@ -3,7 +3,7 @@ "PropertyGroupId": "ProcessedOutputStream", "PropertyMap": { "aws.region": "us-east-1", - "stream.arn": "ExampleOutputStream-ARN-GOES-HERE", + "stream.arn": "arn:aws:kinesis:us-east-1::stream/ExampleOutputStream", "flink.stream.initpos": "LATEST" } }, @@ -11,7 +11,7 @@ "PropertyGroupId": "DLQOutputStream", "PropertyMap": { "aws.region": "us-east-1", - "stream.arn": "DLQStream-ARN-GOES-HERE", + "stream.arn": "arn:aws:kinesis:us-east-1::stream/DLQStream", "flink.stream.initpos": "LATEST" } } diff --git a/java/pom.xml b/java/pom.xml index bc44dbeb..08543f2f 100644 --- a/java/pom.xml +++ b/java/pom.xml @@ -23,20 +23,27 @@ GettingStartedTable Iceberg/IcebergDataStreamSink Iceberg/IcebergDataStreamSource + Iceberg/IcebergSQLSink Iceberg/S3TableSink KafkaConfigProviders/Kafka-mTLS-Keystore-ConfigProviders KafkaConfigProviders/Kafka-SASL_SSL-ConfigProviders KafkaConfigProviders/Kafka-mTLS-Keystore-Sql-ConfigProviders KafkaConnectors KinesisConnectors + KinesisSourceDeaggregation DynamoDBStreamSource KinesisFirehoseSink S3ParquetSink + S3ParquetSource S3Sink Windowing Serialization/CustomTypeInfo SideOutputs PrometheusSink SQSSink + S3AvroSink + S3AvroSource + FlinkCDC/FlinkCDCSQLServerSource + FlinkDataGenerator \ No newline at end of file diff --git a/python/GettingStarted/README.md b/python/GettingStarted/README.md index 65e2a812..ec126191 100644 --- a/python/GettingStarted/README.md +++ b/python/GettingStarted/README.md @@ -8,6 +8,15 @@ Sample PyFlink application reading from and writing to Kinesis Data Stream. * Language: Python This example provides the basic skeleton for a PyFlink application. +It shows how to correctly package JAR dependencies such as Flink connectors. + + +> This example does not include external Python dependencies. +> To learn **how to package Pythion dependencies** check out these two examples: +> 1. [PythonDependencies](../PythonDependencies): how to download Python dependencies at runtime using `requirements.txt` +> 2. [PackagedPythonDependencies](../PackagedPythonDependencies): how to package dependencies with the application artifact + +--- The application is written in Python, but operators are defined using SQL. This is a popular way of defining applications in PyFlink, but not the only one. You could attain the same results diff --git a/python/GettingStarted/pom.xml b/python/GettingStarted/pom.xml index 322d1d34..4bea6305 100644 --- a/python/GettingStarted/pom.xml +++ b/python/GettingStarted/pom.xml @@ -4,7 +4,7 @@ 4.0.0 com.amazonaws - managed-flink-pyfink-getting-started + managed-flink-pyflink-getting-started 1.0.0 diff --git a/python/IcebergSink/README.md b/python/IcebergSink/README.md new file mode 100644 index 00000000..720b09c7 --- /dev/null +++ b/python/IcebergSink/README.md @@ -0,0 +1,249 @@ +## Example of writing to Apache Iceberg with PyFlink + +Example showing a PyFlink application writing to Iceberg table in S3. + +* Flink version: 1.20 +* Flink API: Table API & SQL +* Flink Connectors: Apache Iceberg & Flink S3 +* Language: Python + +This application demonstrates settings up Apache Iceberg table as sink. + +The application is written in Python, but operators are defined using SQL. This is a popular way of defining applications in PyFlink, but not the only one. You could attain the same results using Table API ar DataStream API, in Python. + +The job can run both on Amazon Managed Service for Apache Flink, and locally for development. +--- + +### Dependency Shading + +This project uses Maven Shade Plugin to handle dependency conflicts: + +```xml + + + org.apache.hadoop.conf + shaded.org.apache.hadoop.conf + + + org.apache.flink.runtime.util.HadoopUtils + shadow.org.apache.flink.runtime.util.HadoopUtils + + +``` + +#### Why this matters: + +* Prevents classpath conflicts with Hadoop/Flink internals +* Ensures our bundled dependencies don't clash with AWS Managed Flink's runtime +* Required for stable operation with Iceberg connector + +--- + +## Excluded Java Versions in maven-shade-plugin + +Apache Flink 1.20 [only supports Java 11 as non-experimental](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/deployment/java_compatibility/). +Amazon Managed Service for Apache Flink currently runs on Java 11. +Any Java code must be compiled with a target java version 11. + +We have excluded certain Java versions to avoid errors caused by [multi-release JARs](https://openjdk.org/jeps/238) where the META-INF/versions/XX directories contain **Java version-specific class files**. + +Example: A dependency might include optimized code for Java 21+ in META-INF/versions/21/.... + +### Why Exclude Specific Versions + +* Avoid Compatibility Issues - If you include JAR dependencies compiled for Java 21/22 you may face errors like *java.lang.UnsupportedClassVersionError: Unsupported major.minor version 65* +* Prevent Conflicts - Some libraries include multi-release JARs that conflict with Flink’s dependencies when merged into a fat JAR. + +### Requirements + +#### Development and build environment requirements + +* Python 3.11 +* PyFlink library: `apache-flink==1.20.0` +* Java JDK 11+ and Maven + +> ⚠️ As of 2024-06-27, the Flink Python library 1.19.x may fail installing on Python 3.12. +> We recommend using Python 3.11 for development, the same Python version used by Amazon Managed Service for Apache Flink +> runtime 1.20. + +> JDK and Maven are used to download and package any required Flink dependencies, e.g. connectors, and + to package the application as `.zip` file, for deployment to Amazon Managed Service for Apache Flink. + +> This code has been tested and compiled using OpenJDK 11.0.22 and Maven 3.9.6 + +#### External dependencies + +The application requires Amazon S3 bucket to write the Iceberg table data. The application also needs appropriate permissions for Amazon S3 and AWS Glue as [discussed below](#iam-permissions). + +The target Iceberg table parameters are defined in the configuration (see [below](#runtime-configuration)). + +#### IAM permissions + +The application must have sufficient permissions to read and write to AWS Glue Data Catalog where catalog details will be stored and Amazon S3 bucket where Iceberg table data and metadata will be stored. + +When running locally, you need active valid AWS credentials that allow reading and writing catalog to Glue and data to S3 bucket. + +### Runtime configuration + +* **Local development**: uses the local file [application_properties.json](./application_properties.json) - **Edit the file with your Iceberg table details which includes catalog name, warehouse location, database name, table name, and AWS region** +* **On Amazon Managed Service for Apache Fink**: define Runtime Properties, using Group ID and property names based on the content of [application_properties.json](./application_properties.json) + +For this application, the configuration properties to specify are: + +Runtime parameters: + +| Group ID | Key | Mandatory | Example Value (default for local) | Notes | +|-----------------|-----------------|-----------|-----------------------------------|--------------------------------------------------| +| `IcebergTable0` | `catalog.name` | Y | `glue_catalog` | Catalog name to defined | +| `IcebergTable0` | `warehouse.path`| Y | `s3://my_bucket/my_warehouse` | Warehouse path for catalog | +| `IcebergTable0` | `database.name` | Y | `my_database` | Database name for Iceberg table | +| `IcebergTable0` | `table.name` | Y | `my_table` | Table name to write the data | +| `IcebergTable0` | `aws.region` | Y | `us-east-1` | Region for the output Iceberg table and catalog. | + + +In addition to these configuration properties, when running a PyFlink application in Managed Flink you need to set two +[Additional configuring for PyFink application on Managed Flink](#additional-configuring-for-pyfink-application-on-managed-flink). + +> If you forget to edit the local `application_properties.json` configuration to point your Iceberg table and warehouse path, the application will fail to start locally. + +#### Additional configuring for PyFink application on Managed Flink + +To tell Managed Flink what Python script to run and the fat-jar containing all dependencies, you need to specific some +additional Runtime Properties, as part of the application configuration: + +| Group ID | Key | Mandatory | Value | Notes | +|---------------------------------------|-----------|-----------|--------------------------------|---------------------------------------------------------------------------| +| `kinesis.analytics.flink.run.options` | `python` | Y | `main.py` | The Python script containing the main() method to start the job. | +| `kinesis.analytics.flink.run.options` | `jarfile` | Y | `lib/pyflink-dependencies.jar` | Location (inside the zip) of the fat-jar containing all jar dependencies. | + +> ⚠️ If you forget adding these parameters to the Runtime properties, the application will not start. +--- + +### How to run and build the application + +#### Local development - in the IDE + +1. Make sure you have created the Kinesis Streams and you have a valid AWS session that allows you to publish to the Streams (the way of doing it depends on your setup) +2. Run `mvn package` once, from this directory. This step is required to download the jar dependencies - the Kinesis connector in this case +3. Set the environment variable `IS_LOCAL=true`. You can do from the prompt or in the run profile of the IDE +4. Run `main.py` + +You can also run the python script directly from the command line, like `python main.py`. This still require running `mvn package` before. + +If you are using Virtual Environments, make sure the to select the venv as a runtime in your IDE. + +If you forget the set the environment variable `IS_LOCAL=true` or forget to run `mvn package` the application fails on start. + +> 🚨 The application does not log or print anything. +> If you do not see any output in the console, it does not mean the application is not running. +> The output is sent to the Iceberg table. You can inspect the content of the table using Amazon Athena or Trino/Spark on EMR. + +Note: if you modify the Python code, you do not need to re-run `mvn package` before running the application locally. + +##### Troubleshooting the application when running locally + +By default, the PyFlink application running locally does not send logs to the console. +Any exception thrown by the Flink runtime (i.e. not due to Python error) will not appear in the console. +The application may appear to be running, but actually continuously failing and restarting. + +To see any error messages, you need to inspect the Flink logs. +By default, PyFlink will send logs to the directory where the PyFlink module is installed (Flink home). +Use this command to find the directory: + +``` +$ python -c "import pyflink;import os;print(os.path.dirname(os.path.abspath(pyflink.__file__))+'/log')" +``` + + +#### Deploy and run on Amazon Managed Service for Apache Flink + +1. Make sure you have the S3 bucket location for Iceberg warehouse path +2. Create a Managed Flink application +3. Modify the application IAM role to allow writing to Glue Data catalog and S3 location for the Iceberg table +4. Package the application: run `mvn clean package` from this directory +5. Upload to an S3 bucket the zip file that the previous creates in the [`./target`](./target) subdirectory +6. Configure the Managed Flink application: set Application Code Location to the bucket and zip file you just uploaded +7. Configure the Runtime Properties of the application, creating the Group ID, Keys and Values as defined in the [application_properties.json](./application_properties.json) +8. Start the application +9. When the application transitions to "Ready" you can open the Flink Dashboard to verify the job is running, and you can inspect the data published to the Iceberg table from Athena or Trino/Spark on EMR. + +##### Troubleshooting Python errors when the application runs on Amazon Managed Service for Apache Flink + +Amazon Managed Service for Apache Flink sends all logs to CloudWatch Logs. +You can find the name of the Log Group and Log Stream in the configuration of the application, in the console. + +Errors caused by the Flink engine are usually logged as `ERROR` and easy to find. However, errors reported by the Python +runtime are **not** logged as `ERROR`. + +Apache Flink logs any entry reported by the Python runtime using a logger named `org.apache.flink.client.python.PythonDriver`. + +The easiest way to find errors reported by Python is using CloudWatch Insight, and run the following query:] + +``` +fields @timestamp, message +| sort @timestamp asc +| filter logger like /PythonDriver/ +| limit 1000 +``` + +> 🚨 If the Flink jobs fails to start due to an error reported by Python, for example a missing expected configuration +> parameters, the Amazon Managed Service for Apache Flink may report as *Running* but the job fails to start. +> You can check whether the job is actually running using the Apache Flink Dashboard. If the job is not listed in the +> Running Job List, it means it probably failed to start due to an error. +> +> In CloudWatch Logs you may find an `ERROR` entry with not very explanatory message "Run python process failed". +> To find the actual cause of the problem, run the CloudWatch Insight above, to see the actual error reported by +> the Python runtime. + + +#### Publishing code changes to Amazon Managed Service for Apache Flink + +Follow this process to make changes to the Python code + +1. Modify the code locally (test/run locally, as required) +2. Re-run `mvn clean package` - **if you skip this step, the zipfile is not updated**, and contains the old Python script. +3. Upload the new zip file to the same location on S3 (overwriting the previous zip file) +4. In the Managed Flink application console, enter *Configure*, scroll down and press *Save Changes* + * If your application was running when you published the change, Managed Flink stops the application and restarts it with the new code + * If the application was not running (in Ready state) you need to click *Run* to restart it with the new code + +> 🚨 by design, Managed Flink does not detect the new zip file automatically. +> You control when you want to restart the application with the code changes. This is done saving a new configuration from the +> console or using the [*UpdateApplication*](https://docs.aws.amazon.com/managed-flink/latest/apiv2/API_UpdateApplication.html) +> API. + +--- + +### Application structure + +The application generates synthetic data using the [DataGen](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/connectors/table/datagen/) connector. +No external data generator is required. + +Generated records are written to a destination Iceberg table which cataloged in AWS Glue and data is written to S3. + +Data format is Parquet, and partitioning is by the `sensor_id`. + +Note that the Iceberg connector writes to S3 on checkpoint. For this reason, when running locally checkpoint is set up programmatically by the application every minute. When deployed on Amazon Managed Service for Apache Flink, the checkpoint +configuration is configured as part of the Managed Flink application configuration. By default, it's every minute. + +If you disable checkpoints (or forget to set it up when running locally) the application runs but never writes any data to S3. + +--- + +### Application packaging and dependencies + +This examples also demonstrate how to include jar dependencies - e.g. connectors - in a PyFlink application, and how to +package it, for deploying on Amazon Managed Service for Apache Flink. + +Any jar dependencies must be added to the `` block in the [pom.xml](pom.xml) file. +In this case, you can see we have included `iceberg-flink-runtime`, `iceberg-aws-bundle`, `flink-s3-fs-hadoop`, `flink-hadoop-fs`, and AWS SDK for Glue & S3 for Iceberg sink to work with Flink. + +Executing `mvn package` takes care of downloading any defined dependencies and create a single "fat-jar" containing all of them. +This file, is generated in the `./target` subdirectory and is called `pyflink-dependencies.jar` + +> The `./target` directory and any generated files are not supposed to be committed to git. + +When running locally, for example in your IDE, PyFlink will look for this jar file in `./target`. + +When you are happy with your Python code and you are ready to deploy the application to Amazon Managed Service for Apache Flink, +run `mvn package` **again**. The zip file you find in `./target` is the artifact to upload to S3, containing both jar dependencies and your Python code. \ No newline at end of file diff --git a/python/IcebergSink/application_properties.json b/python/IcebergSink/application_properties.json new file mode 100644 index 00000000..15adccc8 --- /dev/null +++ b/python/IcebergSink/application_properties.json @@ -0,0 +1,19 @@ +[ + { + "PropertyGroupId": "kinesis.analytics.flink.run.options", + "PropertyMap": { + "python": "main.py", + "jarfile": "lib/pyflink-dependencies.jar" + } + }, + { + "PropertyGroupId": "IcebergTable0", + "PropertyMap": { + "catalog.name": "glue_catalog", + "warehouse.path": "s3:///my_warehouse", + "database.name": "my_database", + "table.name": "my_table", + "aws.region": "us-east-1" + } + } +] \ No newline at end of file diff --git a/python/IcebergSink/assembly/assembly.xml b/python/IcebergSink/assembly/assembly.xml new file mode 100644 index 00000000..12665c1a --- /dev/null +++ b/python/IcebergSink/assembly/assembly.xml @@ -0,0 +1,25 @@ + + my-assembly + + zip + + false + + + ${project.basedir} + / + + main.py + + + + ${project.build.directory} + lib + + *.jar + + + + \ No newline at end of file diff --git a/python/IcebergSink/main.py b/python/IcebergSink/main.py new file mode 100644 index 00000000..4473f4ed --- /dev/null +++ b/python/IcebergSink/main.py @@ -0,0 +1,237 @@ +""" +Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + +Permission is hereby granted, free of charge, to any person obtaining a copy of this +software and associated documentation files (the "Software"), to deal in the Software +without restriction, including without limitation the rights to use, copy, modify, +merge, publish, distribute, sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, +INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A +PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION +OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE +SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +""" +# -*- coding: utf-8 -*- +""" +main.py +~~~~~~~~~~~~~~~~~~~ +This module: +FIXME + 1. Creates a execution environment + 2. Set any special configuration for local mode (e.g. when running in the IDE) + 3. Retrieve the runtime configuration + 4. Creates a source table to generate data using DataGen connector + 5. Create a catalog in AWS Glue data catalog + 6. Create a sink table writing to an Apache Iceberg table on Amazon S3 + 7. Insert into the sink table (Iceberg S3) +""" + +import os +import json +import pyflink +from pyflink.table import EnvironmentSettings, TableEnvironment + +####################################### +# 1. Creates the execution environment +####################################### + +env_settings = EnvironmentSettings.in_streaming_mode() +table_env = TableEnvironment.create(env_settings) + +table_env.get_config().get_configuration().set_string( + "execution.checkpointing.mode", "EXACTLY_ONCE" +) + +table_env.get_config().get_configuration().set_string( + "execution.checkpointing.interval", "1 min" +) + +# Location of the configuration file when running on Managed Flink. +# NOTE: this is not the file included in the project, but a file generated by Managed Flink, based on the +# application configuration. +APPLICATION_PROPERTIES_FILE_PATH = "/etc/flink/application_properties.json" + +# Set the environment variable IS_LOCAL=true in your local development environment, +# or in the run profile of your IDE: the application relies on this variable to run in local mode (as a standalone +# Python application, as opposed to running in a Flink cluster). +# Differently from Java Flink, PyFlink cannot automatically detect when running in local mode +is_local = ( + True if os.environ.get("IS_LOCAL") else False +) + +############################################## +# 2. Set special configuration for local mode +############################################## + +if is_local: + # Load the configuration from the json file included in the project + APPLICATION_PROPERTIES_FILE_PATH = "application_properties.json" + + # Point to the fat-jar generated by Maven, containing all jar dependencies (e.g. connectors) + CURRENT_DIR = os.path.dirname(os.path.realpath(__file__)) + table_env.get_config().get_configuration().set_string( + "pipeline.jars", + # For local development (only): use the fat-jar containing all dependencies, generated by `mvn package` + # located in the target/ subdirectory + "file:///" + CURRENT_DIR + "/target/pyflink-dependencies.jar", + ) + + # Show the PyFlink home directory and the directory where logs will be written, when running locally + print("PyFlink home: " + os.path.dirname(os.path.abspath(pyflink.__file__))) + print("Logging directory: " + os.path.dirname(os.path.abspath(pyflink.__file__)) + '/log') + +# Utility method, extracting properties from the runtime configuration file +def get_application_properties(): + if os.path.isfile(APPLICATION_PROPERTIES_FILE_PATH): + with open(APPLICATION_PROPERTIES_FILE_PATH, "r") as file: + contents = file.read() + properties = json.loads(contents) + return properties + else: + print('A file at "{}" was not found'.format(APPLICATION_PROPERTIES_FILE_PATH)) + +# Utility method, extracting a property from a property group +def property_map(props, property_group_id): + for prop in props: + if prop["PropertyGroupId"] == property_group_id: + return prop["PropertyMap"] + +def main(): + + ##################################### + # Default configs + ##################################### + + # Default catalog, database, and input/output tables + catalog = "default_catalog" + database = "default_database" + input_table = f"{catalog}.{database}.sensor_readings" + print_output_table = f"{catalog}.{database}.sensor_output" + + + ##################################### + # 3. Retrieve runtime configuration + ##################################### + + props = get_application_properties() + + # Iceberg table configuration + iceberg_table_properties = property_map(props, "IcebergTable0") + iceberg_catalog_name = iceberg_table_properties["catalog.name"] + iceberg_warehouse_path = iceberg_table_properties["warehouse.path"] + iceberg_database_name = iceberg_table_properties["database.name"] + iceberg_table_name = iceberg_table_properties["table.name"] + iceberg_table_region = iceberg_table_properties["aws.region"] + + ################################################# + # 4. Define input table using datagen connector + ################################################# + + # In a real application, this table will probably be connected to a source stream, using for example the 'kinesis' + # connector. + table_env.execute_sql(f""" + CREATE TABLE {input_table} ( + sensor_id INT, + temperature NUMERIC(6,2), + measurement_time TIMESTAMP(3) + ) + PARTITIONED BY (sensor_id) + WITH ( + 'connector' = 'datagen', + 'fields.sensor_id.min' = '10', + 'fields.sensor_id.max' = '20', + 'fields.temperature.min' = '0', + 'fields.temperature.max' = '100' + ) + """) + + ################################################# + # 5. Define catalog for iceberg table + ################################################# + + table_env.execute_sql(f""" + CREATE CATALOG {iceberg_catalog_name} WITH ( + 'type' = 'iceberg', + 'property-version' = '1', + 'catalog-impl' = 'org.apache.iceberg.aws.glue.GlueCatalog', + 'io-impl' = 'org.apache.iceberg.aws.s3.S3FileIO', + 'warehouse' = '{iceberg_warehouse_path}', + 'aws.region' = '{iceberg_table_region}' + ) + """) + + ################################################# + # 6. Use the catalog and create database + ################################################# + + # Start by using the catalog + table_env.execute_sql(f"USE CATALOG `{iceberg_catalog_name}`;") + + # Create database if not exists + table_env.execute_sql(f"CREATE DATABASE IF NOT EXISTS `{iceberg_database_name}`;") + + # Use database + table_env.execute_sql(f"USE `{iceberg_database_name}`;") + + ################################################# + # 7. Define sink table for Iceberg table + ################################################# + + table_env.execute_sql(f""" + CREATE TABLE IF NOT EXISTS `{iceberg_catalog_name}`.`{iceberg_database_name}`.`{iceberg_table_name}` ( + sensor_id INT NOT NULL, + temperature NUMERIC(6,2) NOT NULL, + `time` TIMESTAMP_LTZ(3) NOT NULL + ) + PARTITIONED BY (sensor_id) + WITH ( + 'type' = 'iceberg', + 'write.format.default' = 'parquet', + 'write.parquet.compression-codec' = 'snappy', + 'format-version' = '2' + ) + """) + + # table_env.execute_sql(f""" + # CREATE TABLE {print_output_table}( + # sensor_id INT NOT NULL, + # temperature NUMERIC(6,2) NOT NULL, + # `time` TIMESTAMP_LTZ(3) NOT NULL + # ) + # PARTITIONED BY (sensor_id) + # WITH ( + # 'connector' = 'print' + # ) + # """) + + # In a real application we would probably have some transformations between the input and the output. + # For simplicity, we will send the source table directly to the sink table. + + ########################################################################################## + # 8. Insert into the sink table + ########################################################################################## + + table_result = table_env.execute_sql(f""" + INSERT INTO `{iceberg_catalog_name}`.`{iceberg_database_name}`.`{iceberg_table_name}` + SELECT sensor_id, temperature, measurement_time as `time` + FROM {input_table}""") + + ## Uncomment below when using "print" + # table_result = table_env.execute_sql(f""" + # INSERT INTO sensors_output + # SELECT sensor_id, temperature, measurement_time as `time` + # FROM {input_table}""") + + + # When running locally, as a standalone Python application, you must instruct Python not to exit at the end of the + # main() method, otherwise the job will stop immediately. + # When running the job deployed in a Flink cluster or in Amazon Managed Service for Apache Flink, the main() method + # must end once the flow has been defined and handed over to the Flink framework to run. + if is_local: + table_result.wait() + +if __name__ == "__main__": + main() \ No newline at end of file diff --git a/python/IcebergSink/pom.xml b/python/IcebergSink/pom.xml new file mode 100644 index 00000000..f793604f --- /dev/null +++ b/python/IcebergSink/pom.xml @@ -0,0 +1,189 @@ + + 4.0.0 + + com.amazonaws + managed-flink-pyfink-iceberg-sink-example + 1.0.0 + + + UTF-8 + ${project.basedir}/target + ${project.name}-${project.version} + pyflink-dependencies + + 11 + ${target.java.version} + ${target.java.version} + + 1.20.0 + 1.20 + 1.9.0 + 2.31.48 + 1.2.0 + + + + + + software.amazon.awssdk + bom + ${awssdk.version} + pom + import + + + + + + + + + software.amazon.awssdk + glue + + + + software.amazon.awssdk + s3 + + + org.apache.iceberg + iceberg-flink-runtime-${flink.major.version} + ${iceberg.version} + + + org.apache.iceberg + iceberg-aws-bundle + ${iceberg.version} + + + org.apache.flink + flink-s3-fs-hadoop + ${flink.version} + + + org.apache.flink + flink-hadoop-fs + ${flink.version} + + + + + ${buildDirectory} + + + + org.apache.maven.plugins + maven-compiler-plugin + 3.8.1 + + ${target.java.version} + ${target.java.version} + + + + + + org.apache.maven.plugins + maven-shade-plugin + 3.6.0 + + + + package + + shade + + + + ${project.build.outputDirectory} + + + + org.apache.flink:force-shading + com.google.code.findbugs:jsr305 + org.slf4j:* + log4j:* + + + + + + *:* + + META-INF/*.SF + META-INF/*.DSA + META-INF/*.RSA + + META-INF/versions/21/** + META-INF/versions/22/** + + + + + + + + + + + + org.apache.hadoop.conf + shaded.org.apache.hadoop.conf + + + org.apache.flink.runtime.util.HadoopUtils + shadow.org.apache.flink.runtime.util.HadoopUtils + + + + + + + + + + maven-assembly-plugin + 3.3.0 + + + assembly/assembly.xml + + ${zip.finalName} + ${buildDirectory} + false + + + + make-assembly + package + + single + + + + + + + + org.apache.maven.plugins + maven-jar-plugin + 3.2.0 + + ${jar.finalName} + + + + + \ No newline at end of file diff --git a/python/IcebergSink/src/main/java/org/apache/flink/runtime/util/HadoopUtils.java b/python/IcebergSink/src/main/java/org/apache/flink/runtime/util/HadoopUtils.java new file mode 100644 index 00000000..9ffaae18 --- /dev/null +++ b/python/IcebergSink/src/main/java/org/apache/flink/runtime/util/HadoopUtils.java @@ -0,0 +1,14 @@ +// This utility is used by Flink to dynamically load Hadoop configurations at runtime. +// Why Needed: Required for Flink to interact with Hadoop-compatible systems (e.g., S3 via s3a:// or s3:// paths). +// Returns an empty Hadoop Configuration object (new Configuration(false)). + +package org.apache.flink.runtime.util; + +import org.apache.hadoop.conf.Configuration; + +public class HadoopUtils { + public static Configuration getHadoopConfiguration( + org.apache.flink.configuration.Configuration flinkConfiguration) { + return new Configuration(false); + } + } \ No newline at end of file diff --git a/python/PackagedPythonDependencies/.gitignore b/python/PackagedPythonDependencies/.gitignore new file mode 100644 index 00000000..f36bcce8 --- /dev/null +++ b/python/PackagedPythonDependencies/.gitignore @@ -0,0 +1 @@ +/dep/ diff --git a/python/PackagedPythonDependencies/README.md b/python/PackagedPythonDependencies/README.md new file mode 100644 index 00000000..ecb9c694 --- /dev/null +++ b/python/PackagedPythonDependencies/README.md @@ -0,0 +1,196 @@ +## Packaging Python dependencies with the ZIP + +Example showing how you can package Python dependencies with the ZIP file you upload to S3. + +* Flink version: 1.20 +* Flink API: Table API & SQL +* Flink Connectors: Kinesis Connector +* Language: Python + +This example shows how you can package Python dependencies within the ZIP and make them available to the application. + +> This method is alternative to what illustrated in the [Python Dependencies](../PythonDependencies) example which relies on the +`requirements.txt` file for installing the dependencies at runtime. + +The approach shown in this example has the following benefits: + +* It works with any number of Python libraries +* It supports Python libraries which include **native dependencies**, such as SciPy or Pydantic specific to the CPU architecture (note that Pandas, NumPy, and PyArrow also have native dependencies, but are already available as transitive dependencies +of `apache-flink` and should not be added as additional dependencies). +* It allows to run the application locally, in your machine, and in Managed Service for Apache Flink, with no code changes. +* Dependencies are available both during job initialization, in the `main()` method, and for data processing, for example in a User Defined Function (UDF). + +Drawbacks: +* You need to use a virtual environment for the Python dependencies when running locally, because the CPU architecture of your machine may differ from the architecture used by Managed Service for Apache Flink +* Python dependencies are included in the ZIP file slowing down a bit operations + +For more details about how packaging dependencies works, see [Packaging application and dependencies](#packaging-application-and-dependencies), below. + + + +The application includes [SciPy](https://scipy.org/) used in a UDF. The actual use is not important. +It also shows how the same library can be used during the job initialization. + +The application generates random data using [DataGen](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/connectors/table/datagen/) +and send the output to a Kinesis Data Streams. + +--- + +### How to run and build the application + +#### Prerequisites + +We recommend to use a Virtual environment (venv) on the development machine. + +Development and build environment requirements: +* Python 3.11 +* PyFlink library: `apache-flink==1.20.0` +* Java JDK 11+ and Maven + + +#### Setting up the local environment + +To run the application locally, from the command line or in your IDE, you need to install the following Python dependencies: + +* `apache-flink==1.20.0` +* Any additional Python dependency is defined in `requirements.txt` + +Assuming you use virtualenv: + +1. Create the Virtual Environment in the project directory: `virtualenv venv` +2. Activate the Virtual Environment you just created: `source venv/bin/activate` +3. Install PyFlink library: `pip install apache-flink==1.20.0` +4. Define the additional dependencies + * Add JAR-dependencies in the `pom.xml` file + * Add Python dependencies in the `requirements.txt` (do not include any Flink dependency!) +5. Install Python from `requirements.txt` into the venv, for local development: `pip install -r requirements.txt` +6. Download the Python dependencies for the target architecture: `pip install -r requirements.txt --target=dep/ --platform=manylinux2014_x86_64 --only-binary=:all:` +7. Run `mvn package` to download and package the JAR dependencies and build the ZIP artifact + +> ⚠️ The Flink Python library 1.20.0 may fail installing on Python 3.12. We recommend using Python 3.11 for development, the same Python version used by Amazon Managed Service for Apache Flink runtime 1.20. + +> JDK and Maven are uses to download and package any required Flink dependencies, e.g. connectors, and to package the application as .zip file, for deployment to Amazon Managed Service for Apache Flink. + +### Runtime configuration + +* **Local development**: uses the local file [application_properties.json](./application_properties.json) +* **On Amazon Managed Service for Apache Fink**: define Runtime Properties, using Group ID and property names based on the content of [application_properties.json](./application_properties.json) + +For this application, the configuration properties to specify are: + + +| Group ID | Key | Mandatory | Example Value (default for local) | Notes | +|------------------|---------------|-----------|-----------------------------------|-------------------------------| +| `OutputStream0` | `stream.name` | Y | `ExampleOutputStream` | Output stream name. | +| `OutputStream0` | `aws.region` | Y | `us-east-1` | Region for the output stream. | + + + +To tell Managed Flink what Python script to run, the fat-jar containing all dependencies, and the Python dependencies, you need to specific some +additional Runtime Properties, as part of the application configuration: + +| Group ID | Key | Mandatory | Value | Notes | +|---------------------------------------|-----------|-----------|--------------------------------|------------------------------------------------------------------------------------| +| `kinesis.analytics.flink.run.options` | `python` | Y | `main.py` | The Python script containing the main() method to start the job. | +| `kinesis.analytics.flink.run.options` | `jarfile` | Y | `lib/pyflink-dependencies.jar` | Location (inside the zip) of the fat-jar containing all jar dependencies. | +| `kinesis.analytics.flink.run.options` | `pyFiles` | Y | `dep/` | Relative path of the subdirectory (inside the zip) containing Python dependencies. | + +Note that these properties are ignored when running locally. + +--- + +### Local development - in the IDE + +1. Make sure you have created the Kinesis Streams and you have a valid AWS session that allows you to publish to the Streams (the way of doing it depends on your setup) +2. Make sure your IDE uses the venv you created. Follow the documentations of your IDE (PyCharm, Visual Studio Code) +3. Run `mvn package` once, from this directory. This step is required to download the jar dependencies - the Kinesis connector in this case +4. Set the environment variable `IS_LOCAL=true`. You can do from the prompt or in the run profile of the IDE +5. Run `main.py` + +You can also run the python script directly from the command line, like python main.py. This still require running mvn package before. + +If you forget the set the environment variable `IS_LOCAL=true` or forget to run `mvn package` the application fails on start. + +> 🚨 The application does not log or print anything. If you do not see any output in the console, it does not mean the application is not running. The output is sent to the Kinesis streams. You can inspect the content of the streams using the Data Viewer in the Kinesis console + + + +### Deploy and run on Amazon Managed Service for Apache Flink + +1. Make sure you have the required Kinesis Streams +2. Create a Managed Flink application +3. Modify the application IAM role to allow writing to the Kinesis Stream +4. If you haven't done already, download the Python dependencies for the target architecture: `pip install -r requirements.txt --target=dep/ --platform=manylinux2014_x86_64 --only-binary=:all:` +5. Package the application: run `mvn clean package` from this directory +6. Upload to an S3 bucket the zip file that the previous creates in the ./target subdirectory +7. Configure the Managed Flink application: set Application Code Location to the bucket and zip file you just uploaded +8. Configure the Runtime Properties of the application, creating the Group ID, Keys and Values as defined in the [`application_properties.json`](application_properties.json) (a) +9. Start the application +10. When the application transitions to "RUNNING" you can open the Flink Dashboard to verify the job is running, and you can inspect the data published to the Kinesis Streams, using the Data Viewer in the Kinesis console. + + + +### Publishing code changes to Amazon Managed Service for Apache Flink + +Follow this process to make changes to the Python code or the dependencies + +1. Modify the code locally (test/run locally, as required) +2. Re-run `mvn clean package` - if you skip this step, the zipfile is not updated, and contains the old Python script. +3. Upload the new zip file to the same location on S3 (overwriting the previous zip file) +4. In the Managed Flink application console, enter Configure, scroll down and press Save Changes + * If your application was running when you published the change, Managed Flink stops the application and restarts it with the new code + * If the application was not running (in Ready state) you need to click Run to restart it with the new code + + +> 🚨 by design, Managed Flink does not detect the new zip file automatically. You control when you want to restart the application with the code changes. This is done saving a new configuration from the console or using the UpdateApplication API. + +--- + +### Packaging application and dependencies + + +This example also demonstrates how to include both jar dependencies - e.g. connectors - and Python libraries in a PyFlink application. It demonstrates how to package it for deploying on Amazon Managed Service for Apache Flink. + +The [`assembly/assembly.xml`](assembly/assembly.xml) file instructs Maven for including the correct files in the ZIP-file. + +#### Jar dependencies + +Any jar dependencies must be added to the `` block in the [pom.xml](pom.xml) file. +In this case, you can see we have included `flink-sql-connector-kinesis` + +Executing `mvn package` takes care of downloading any defined dependencies and create a single "fat-jar" containing all of them. +This file, is generated in the `./target` subdirectory and is called `pyflink-dependencies.jar` + +> The `./target` directory and any generated files are not supposed to be committed to git. + +When running locally, for example in your IDE, PyFlink will look for this jar file in `./target`. + +When you are happy with your Python code and you are ready to deploy the application to Amazon Managed Service for Apache Flink, +run `mvn package` **again**. The zip file you find in `./target` is the artifact to upload to S3, containing +both jar dependencies and your Python code. + +#### Python 3rd-party libraries + +Any additional 3rd-party Python library (i.e. Python libraries not provided by PyFlink directly) must also be available +when the application runs. + +There are different approaches for including these libraries in an application deployed on Managed Service for Apache Flink. +The approach demonstrated in this example is the following: + +1. Define a `requirements.txt` with all additional Python dependencies - **DO NOT include any PyFlink dependency** +2. Download the dependencies for the target architecture (`manylinux2014_x86_64`) into the `dep/` sub-folder +3. Package the `dep/` sub-folder in the ZIP file +4. At runtime, register the dependency folder. There are two **alternative** methods (use one of the following, not both): + 1. (recommended) Use the Managed Flink application configuration + * Group ID: `kinesis.analytics.flink.run.options` + * Key: `pyFiles` + * Value: `dep/` + 2. Alternatively, you can programmatically register the directory but only when not running locally + ```python + if not is_local: + python_source_dir = str(pathlib.Path(__file__).parent) + table_env.add_python_file(file_path="file:///" + python_source_dir + "/dep") + ``` + +> This approach differs from what shown in the [Python Dependencies](../PythonDependencies) example because the Python dependencies +> are packaged within the ZIP. The `requirements.txt` file is NOT used to download the dependencies at runtime. diff --git a/python/PackagedPythonDependencies/application_properties.json b/python/PackagedPythonDependencies/application_properties.json new file mode 100644 index 00000000..9a673b9b --- /dev/null +++ b/python/PackagedPythonDependencies/application_properties.json @@ -0,0 +1,17 @@ +[ + { + "PropertyGroupId": "OutputStream0", + "PropertyMap": { + "stream.name": "ExampleOutputStream", + "aws.region": "us-east-1" + } + }, + { + "PropertyGroupId": "kinesis.analytics.flink.run.options", + "PropertyMap": { + "python": "main.py", + "jarfile": "lib/pyflink-dependencies.jar", + "pyFiles": "dep/" + } + } +] \ No newline at end of file diff --git a/python/PackagedPythonDependencies/assembly/assembly.xml b/python/PackagedPythonDependencies/assembly/assembly.xml new file mode 100644 index 00000000..f3d64861 --- /dev/null +++ b/python/PackagedPythonDependencies/assembly/assembly.xml @@ -0,0 +1,44 @@ + + my-assembly + + zip + + false + + + + ${project.basedir} + / + + **/*.py + + + + dep/** + + requirements.txt + + + + + + ${project.basedir}/dep + dep + + ** + + + + + + ${project.build.directory} + lib + + *.jar + + + + + \ No newline at end of file diff --git a/python/PackagedPythonDependencies/main.py b/python/PackagedPythonDependencies/main.py new file mode 100644 index 00000000..d87b7210 --- /dev/null +++ b/python/PackagedPythonDependencies/main.py @@ -0,0 +1,263 @@ +""" +Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + +Permission is hereby granted, free of charge, to any person obtaining a copy of this +software and associated documentation files (the "Software"), to deal in the Software +without restriction, including without limitation the rights to use, copy, modify, +merge, publish, distribute, sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, +INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A +PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION +OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE +SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +""" + +""" +main.py +~~~~~~~~~~~~~~~~~~~ +This module: + 1. Creates the execution environment and specify 3rd-party Python dependencies + 2. Sets any special configuration for local mode (e.g. when running in the IDE) + 3. (Optional) Register the Python dependencies + 4. Retrieves the runtime configuration + 5. Defines and register a UDF that use the python library + 6. Creates a source table to generate data using DataGen connector + 7. Creates a view from a query that uses the UDF + 8. Creates a sink table to Kinesis Data Streams and inserts into the sink table from the view +""" + +from pyflink.table import EnvironmentSettings, TableEnvironment, DataTypes +from pyflink.table.udf import udf +import os +import json +import logging +import pyflink +import pathlib + +################################################################################# +# 1. Creates the execution environment and specify 3rd-party Python dependencies +################################################################################# + +env_settings = EnvironmentSettings.in_streaming_mode() +table_env = TableEnvironment.create(env_settings) + +############################################## +# 2. Set special configuration for local mode +############################################## + +# Location of the configuration file when running on Managed Flink. +# NOTE: this is not the file included in the project, but a file generated by Managed Flink, based on the +# application configuration. +APPLICATION_PROPERTIES_FILE_PATH = "/etc/flink/application_properties.json" + +# Set the environment variable IS_LOCAL=true in your local development environment, +# or in the run profile of your IDE: the application relies on this variable to run in local mode (as a standalone +# Python application, as opposed to running in a Flink cluster). +# Differently from Java Flink, PyFlink cannot automatically detect when running in local mode +is_local = ( + True if os.environ.get("IS_LOCAL") else False +) + +if is_local: + # Load the configuration from the json file included in the project + APPLICATION_PROPERTIES_FILE_PATH = "application_properties.json" + + # Point to the fat-jar generated by Maven, containing all jar dependencies (e.g. connectors) + CURRENT_DIR = os.path.dirname(os.path.realpath(__file__)) + table_env.get_config().get_configuration().set_string( + "pipeline.jars", + # For local development (only): use the fat-jar containing all dependencies, generated by `mvn package` + "file:///" + CURRENT_DIR + "/target/pyflink-dependencies.jar", + ) + + # Show the PyFlink home directory and the directory where logs will be written, when running locally + print("PyFlink home: " + os.path.dirname(os.path.abspath(pyflink.__file__))) + print("Logging directory: " + os.path.dirname(os.path.abspath(pyflink.__file__)) + '/log') + + +########################################################################################## +# 3. (Optional) Register the additional Python dependencies - alt. to specifying pyFiles +########################################################################################## + +# Alternatively to specifying the runtime property kinesis.analytics.flink.run.options : pyFiles, you can +# programmatically register the sub-folder containing the Python dependencies. +# IMPORTANT: you must either specify pyFiles OR registering the dependencies programmatically. NOT both. +# Also, important: when running locally for development you should install the Python dependencies in a venv and not +# register them programmatically. The reason is that the dependencies downloaded in the dep/ subdirectory must match +# the target architecture used by Managed Flink (linux x86_64), which can differ from the architecture of the machine +# you are using for development. + +# Uncomment the following code as alternative to specifying +# the runtime property kinesis.analytics.flink.run.options : pyFiles = dep/ +# DO NOT use both runtime property and programmatic registration. +# if not is_local: +# # Only register the Python dependencies when running locally +# python_source_dir = str(pathlib.Path(__file__).parent) +# table_env.add_python_file(file_path="file:///" + python_source_dir + "/dep") + + +# Utility method, extracting properties from the runtime configuration file +def get_application_properties(): + if os.path.isfile(APPLICATION_PROPERTIES_FILE_PATH): + with open(APPLICATION_PROPERTIES_FILE_PATH, "r") as file: + contents = file.read() + properties = json.loads(contents) + return properties + else: + print('A file at "{}" was not found'.format(APPLICATION_PROPERTIES_FILE_PATH)) + + +# Utility method, extracting a property from a property group +def property_map(props, property_group_id): + for prop in props: + if prop["PropertyGroupId"] == property_group_id: + return prop["PropertyMap"] + + +##################################### +# 4. Retrieve runtime configuration +##################################### + +props = get_application_properties() + +# Get name and region of the Kinesis stream from application configuration +output_stream_name = property_map(props, "OutputStream0")["stream.name"] +output_stream_region = property_map(props, "OutputStream0")["aws.region"] +logging.info(f"Output stream: {output_stream_name}, region: {output_stream_region}") + + +############################################################# +# 5. Defines and register a UDF that uses the Python library +############################################################# + + +@udf(input_types=[DataTypes.FLOAT(), DataTypes.FLOAT(), DataTypes.FLOAT(), DataTypes.FLOAT()], + result_type=DataTypes.FLOAT()) +def determinant(element1, element2, element3, element4): + import numpy as np + from scipy import linalg + a = np.array([[element1, element2], [element3, element4]]) + det = linalg.det(a) + return det + + +# Register the UDF +table_env.create_temporary_system_function("determinant", determinant) + + +def main(): + # Demonstrate the Python dependency is also available in the main() method + # This piece of code is not doing anything useful. The goal is just to shows that the registered dependencies + # are also available job initialization, in the main() method. + # A more realistic case would be, for example, using boto3 to fetch some resources you need to initialize the job. + import numpy as np + from scipy import linalg + matrix = np.array([[42, 43], [44, 43]]) + det = linalg.det(matrix) + print(f"Check dependency in main(): determinant({matrix}) = {det}") + + + ################################################# + # 6. Define input table using datagen connector + ################################################# + + # In a real application, this table will probably be connected to a source stream, using for example the 'kinesis' + # connector. + + table_env.execute_sql(""" + CREATE TABLE random_numbers ( + seed_time TIMESTAMP(3), + element1 FLOAT, + element2 FLOAT, + element3 FLOAT, + element4 FLOAT + ) + PARTITIONED BY (seed_time) + WITH ( + 'connector' = 'datagen', + 'rows-per-second' = '1', + 'fields.element1.min' = '0', + 'fields.element1.max' = '100', + 'fields.element2.min' = '0', + 'fields.element2.max' = '100', + 'fields.element3.min' = '0', + 'fields.element3.max' = '100', + 'fields.element4.min' = '0', + 'fields.element4.max' = '100' + ) + """) + + ################################################### + # 7. Creates a view from a query that uses the UDF + ################################################### + + table_env.execute_sql(""" + CREATE TEMPORARY VIEW determinants + AS + SELECT seed_time, + element1, element2, element3, element4, + determinant(element1, element2, element3, element4) AS determinant + FROM random_numbers + """) + + ################################################# + # 8. Define sink table using kinesis connector + ################################################# + + table_env.execute_sql(f""" + CREATE TABLE output ( + seed_time TIMESTAMP(3), + element1 FLOAT, + element2 FLOAT, + element3 FLOAT, + element4 FLOAT, + determinant FLOAT + ) + WITH ( + 'connector' = 'kinesis', + 'stream' = '{output_stream_name}', + 'aws.region' = '{output_stream_region}', + 'sink.partitioner-field-delimiter' = ';', + 'sink.batch.max-size' = '5', + 'format' = 'json', + 'json.timestamp-format.standard' = 'ISO-8601' + ) + """) + + # For local development purposes, you might want to print the output to the console, instead of sending it to a + # Kinesis Stream. To do that, you can replace the sink table using the 'kinesis' connector, above, with a sink table + # using the 'print' connector. Comment the statement immediately above and uncomment the one immediately below. + # table_env.execute_sql(""" + # CREATE TABLE output ( + # seed_time TIMESTAMP(3), + # element1 FLOAT, + # element2 FLOAT, + # element3 FLOAT, + # element4 FLOAT, + # determinant FLOAT + # ) + # WITH ( + # 'connector' = 'print' + # ) + # """) + + # Executing an INSERT INTO statement will trigger the job + table_result = table_env.execute_sql(""" + INSERT INTO output + SELECT seed_time, element1, element2, element3, element4, determinant + FROM determinants + """) + + # When running locally, as a standalone Python application, you must instruct Python not to exit at the end of the + # main() method, otherwise the job will stop immediately. + # When running the job deployed in a Flink cluster or in Amazon Managed Service for Apache Flink, the main() method + # must end once the flow has been defined and handed over to the Flink framework to run. + if is_local: + table_result.wait() + + +if __name__ == "__main__": + main() diff --git a/python/PackagedPythonDependencies/pom.xml b/python/PackagedPythonDependencies/pom.xml new file mode 100644 index 00000000..9203b5c0 --- /dev/null +++ b/python/PackagedPythonDependencies/pom.xml @@ -0,0 +1,115 @@ + + 4.0.0 + + com.amazonaws + managed-flink-pyflink-packaged-dependencies-example + 1.0.0 + + + UTF-8 + ${project.basedir}/target + ${project.name}-${project.version} + pyflink-dependencies + 1.19.1 + 4.3.0-1.19 + 1.2.0 + + + + + + org.apache.flink + flink-sql-connector-kinesis + ${aws.connector.version} + + + + + ${buildDirectory} + + + + + org.apache.maven.plugins + maven-shade-plugin + 3.2.1 + + + + package + + shade + + + + ${project.build.outputDirectory} + + + + org.apache.flink:force-shading + com.google.code.findbugs:jsr305 + org.slf4j:* + log4j:* + + + + + + *:* + + META-INF/*.SF + META-INF/*.DSA + META-INF/*.RSA + + + + + + + + + + + + + + + + maven-assembly-plugin + 3.3.0 + + + assembly/assembly.xml + + ${zip.finalName} + ${buildDirectory} + false + + + + make-assembly + package + + single + + + + + + + + org.apache.maven.plugins + maven-jar-plugin + 3.2.0 + + ${jar.finalName} + + + + + \ No newline at end of file diff --git a/python/PackagedPythonDependencies/requirements.txt b/python/PackagedPythonDependencies/requirements.txt new file mode 100644 index 00000000..9c61c736 --- /dev/null +++ b/python/PackagedPythonDependencies/requirements.txt @@ -0,0 +1 @@ +scipy \ No newline at end of file diff --git a/python/PythonDependencies/README.md b/python/PythonDependencies/README.md index 7819bc11..9a8ec272 100644 --- a/python/PythonDependencies/README.md +++ b/python/PythonDependencies/README.md @@ -1,25 +1,34 @@ -## Packaging Python dependencies +## Downloading Python dependencies at runtime -Examples showing how to include Python libraries in your PyFlink application. +Examples showing how to include Python libraries in your PyFlink application, downloading them at runtime. * Flink version: 1.20 * Flink API: Table API & SQL * Flink Connectors: Kinesis Connector * Language: Python -This example demonstrate how to include in your PyFlink application additional Python libraries. +This example demonstrate how to include in your PyFlink application additional Python libraries, using `requirements.txt` +to have Managed Service for Apache Flink downloading the dependencies at runtime -There are multiple ways of adding Python dependencies to an application deployed on Amazon Managed Service for Apache Flink. -The approach demonstrated in this example has several benefits: +> This approach is alternative to what illustrated by the [Packaging Python dependencies with the ZIP](../PackagedPythonDependencies) +> which includes the Python dependencies in the ZIP file, as opposed to downloading them at runtime. + + +The approach shown in this example has the following benefits: * It works with any number of Python libraries +* It keeps the ZIP artifact small * It allows to run the application locally, in your machine, and in Managed Service for Apache Flink, with no code changes * It supports Python libraries not purely written in Python, like PyArrow for example, that are specific to a CPU architecture. Including these libraries may be challenging because they architecture of your development machine may be different from the architecture of Managed Service for Apache Flink. -As many other examples, this example also packages any JAR dependencies required for the application. In this case the Kinesis -connector. +Drawbacks: +* The dependencies downloaded at runtime are only available for data processing, for example in UDF. They are NOT available during the job initialization, in the `main()` method. + If you need to usa a Python dependency during job initialization, you must use the approach illustrated by the [Packaging Python dependencies with the ZIP](../PackagedPythonDependencies) + +For more details about how packaging dependencies works, see [Packaging application and dependencies](#packaging-application-and-dependencies), below. + The application is very simple, and uses _botocore_ and _boto3_ Python libraries. These libraries are used to invoke Amazon Bedrock to get a fun fact about a random number. @@ -82,8 +91,18 @@ For this application, the configuration properties to specify are: | `OutputStream0` | `stream.name` | Y | `ExampleOutputStream` | Output stream name. | | `OutputStream0` | `aws.region` | Y | `us-east-1` | Region for the output stream. | -In addition to these configuration properties, when running a PyFlink application in Managed Flink you need to set two -[Additional configuring for PyFink application on Managed Flink](#additional-configuring-for-pyfink-application-on-managed-flink). + +#### Additional configuring for PyFink application on Managed Flink + +To tell Managed Flink what Python script to run and the fat-jar containing all dependencies, you need to specific some +additional Runtime Properties, as part of the application configuration: + +| Group ID | Key | Mandatory | Value | Notes | +|---------------------------------------|-----------|-----------|--------------------------------|---------------------------------------------------------------------------| +| `kinesis.analytics.flink.run.options` | `python` | Y | `main.py` | The Python script containing the main() method to start the job. | +| `kinesis.analytics.flink.run.options` | `jarfile` | Y | `lib/pyflink-dependencies.jar` | Location (inside the zip) of the fat-jar containing all jar dependencies. | + +These parameters are ignored when running locally. --- @@ -169,11 +188,15 @@ libraries, boto3 and botocore in this case. --- -### Application packaging and dependencies +### Packaging application and dependencies This example also demonstrates how to include both jar dependencies - e.g. connectors - and Python libraries in a PyFlink application. It demonstrates how to package it for deploying on Amazon Managed Service for Apache Flink. + +The [`assembly/assembly.xml`](assembly/assembly.xml) file instructs Maven for including the correct files in the ZIP-file. + + #### Jar dependencies Any jar dependencies must be added to the `` block in the [pom.xml](pom.xml) file. @@ -209,13 +232,3 @@ The approach demonstrated in this example is the following: With this approach the Python library are **not packaged** with the application artifact you deploy to Managed Service for Apache Flink. They are installed by the runtime on the cluster, when the application starts. -#### Additional configuring for PyFink application on Managed Flink - -To tell Managed Flink what Python script to run and the fat-jar containing all dependencies, you need to specific some -additional Runtime Properties, as part of the application configuration: - -| Group ID | Key | Mandatory | Value | Notes | -|---------------------------------------|-----------|-----------|--------------------------------|---------------------------------------------------------------------------| -| `kinesis.analytics.flink.run.options` | `python` | Y | `main.py` | The Python script containing the main() method to start the job. | -| `kinesis.analytics.flink.run.options` | `jarfile` | Y | `lib/pyflink-dependencies.jar` | Location (inside the zip) of the fat-jar containing all jar dependencies. | - diff --git a/python/README.md b/python/README.md index 8ce35b48..fcaf57cc 100644 --- a/python/README.md +++ b/python/README.md @@ -1,39 +1,43 @@ -## Flink Python examples +## Flink Python Examples This folder contains examples of Flink applications written in Python. --- -### Packaging dependencies for running in Amazon Managed Service for Apache Flink +### Packaging Dependencies for Amazon Managed Service for Apache Flink -There multiple ways of packaging a PyFlink application with multiple dependencies, Python libraries or JAR dependencies, like connectors. [Amazon Managed Service for Apache Flink](https://aws.amazon.com/managed-service-apache-flink/) expects a specific packaging and runtime configuration for PyFlink applications. +There are multiple ways to package a PyFlink application with dependencies, including Python libraries and JAR dependencies like connectors. [Amazon Managed Service for Apache Flink](https://aws.amazon.com/managed-service-apache-flink/) expects specific packaging and runtime configuration for PyFlink applications. -#### JAR dependencies +#### JAR Dependencies Amazon Managed Service for Apache Flink expects **a single JAR file** containing all JAR dependencies of a PyFlink application. These dependencies include any Flink connector and any other Java library your PyFlink application requires. All these dependencies must be packaged in a single *uber-jar* using [Maven](https://maven.apache.org/) or other similar tools. -The [Getting Started](./GettingStarted/) example, and most of the other example in this directory, show a project set up that allows you to add any number of JAR dependencies to your PyFlink project. It requires Java JDK and [Maven](https://maven.apache.org/) to develop and to package the PyFlink application, and uses Maven to build the *uber-jar* and to package the PyFlink application in the `zip` file for deploymenbt on Managed Service for Apache Flink. This set up also allows you to run your application in your IDE, for debugging and development, and in Managed Service for Apache Flink **without any code changes**. +The [Getting Started](./GettingStarted) example, and most of the other examples in this directory, show a project setup that allows you to add any number of JAR dependencies to your PyFlink project. +It requires Java JDK and [Maven](https://maven.apache.org/) to develop and package the PyFlink application, and uses Maven to build the *uber-jar* and package the PyFlink application in the `zip` file for deployment on Managed Service for Apache Flink. +This setup also allows you to run your application in your IDE for debugging and development, and in Managed Service for Apache Flink **without any code changes**. +**No local Flink cluster is required** for development. +#### Python Dependencies -#### Python dependencies +Apache Flink supports multiple ways of adding Python libraries to your PyFlink application. +Check out these two examples to learn how to correctly package dependencies: -In Apache Flink supports multiple ways of adding Python libraries to your PyFlink application. +1. [PythonDependencies](./PythonDependencies): How to download Python dependencies at runtime using `requirements.txt` +2. [PackagedPythonDependencies](./PackagedPythonDependencies): How to package dependencies with the application artifact + +> The patterns shown in these examples allow you to run the application locally and on Amazon Managed Service +> for Apache Flink **without any code changes**, regardless of the machine architecture you are developing with. -Thre [Python Dependencies](./PythonDependencies/) example shows the most general way of adding any number of Python libraries to your application, using the `requriement.txt` file. This method works with any type of Python library, and does not require packaging these dependencies into the `zip` file deployed on Managed Service for Apache Flink. This also allows you to run the application in your IDE, for debugging and development, and in Managed Service for Apache Flink **without any code changes**. --- -### Python and Flink versions for local development +### Python and Flink Versions for Local Development -There are some known issues with some specific Python and PyFlink versions, for local development +There are some known issues with specific Python and PyFlink versions for local development: -* We recommend using **Python 3.11** to develop Python Flink 1.20.0 applications. +* We recommend using **Python 3.11** to develop PyFlink 1.20.0 applications. This is also the runtime used by Amazon Managed Service for Apache Flink. - Installation of the Python Flink 1.19 library on Python 3.12 may fail. -* Installation of the Python Flink **1.15** library on machines based on **Apple Silicon** fail. - We recommend upgrading to the Flink 1.20, 1.19 or 1.18. Versions 1.18+ work correctly also on Apple Silicon machines. - If you need to maintain a Flink 1.15 application using a machine based on Apple Silicon, you can follow [the guide to develop Flink 1.15 on Apple Silicon](LocalDevelopmentOnAppleSilicon). - - -> None of these issues affects Python Flink applications running on Amazon Managed Service for Apache Flink. -> The managed service uses Python versions compatible with the Flink runtime version you choose. + Installation of the PyFlink 1.19 library on Python 3.12 may fail. +* If you are using Flink **1.15 or earlier** and developing on a machine based on **Apple Silicon**, installing PyFlink locally may fail. + This is a know issue not affecting the application deployed on Amazon Managed Service for Apache Flink. + To develop on Apple Silicon follow [the guide to develop Flink 1.15 on Apple Silicon](LocalDevelopmentOnAppleSilicon).