Augment deployment

[p14n]

Summary by Sourcery

Introduce AWS deployment infrastructure using Terraform and adapt the application for cloud deployment, including TLS encryption and build process updates.

Enhancements:

• Enable TLS for internal gRPC communication between services.
• Improve catchup and unprocessed event submission logic with batching and ensure sequential execution using OneAtATimeBehaviour.
• Add a database index to optimize finding unprocessed events.
• Implement automatic cleanup of inactive PostgreSQL replication slots on startup.
• Configure Debezium to drop its replication slot upon stopping.
• Introduce a fixed thread pool option for async execution and utilize it for gRPC client network operations.

Build:

• Replace custom uber JAR task with the Gradle Shadow plugin.
• Add postgresql and grpc-netty-shaded dependencies.

Deployment:

• Add Terraform configuration for deploying the application on AWS ECS Fargate with RDS PostgreSQL, Application Load Balancers (ALB), and supporting resources (IAM, VPC Endpoints, ACM).
• Configure ALBs for gRPC over HTTPS using self-signed certificates managed by ACM.
• Add a script (make-image.sh) for building the application uber JAR and pushing the corresponding Docker image to ECR.

Tests:

• Disable flaky deterministic consumer tests (DeterministicConsumerTest).
• Add tests for database replication slot cleanup.
• Update test configurations to align with application changes (e.g., executors, batch sizes).

Chores:

• Update application startup to use a random affinity ID. random affinity ID generation for affinity ID. affinity generation to use random UUIDs.

[sourcery-ai]

Reviewer's Guide by Sourcery

This pull request augments the application for deployment by introducing Terraform configurations for AWS infrastructure, updating the application code for cloud readiness, enhancing event processing logic with batching and "one-at-a-time" execution, improving database replication slot management, and updating the build process to use the shadowJar plugin.

No diagrams generated as the changes look simple and do not need a visual representation.

File-Level Changes

Change	Details	Files
Introduction of Terraform configurations for AWS infrastructure.	Added Terraform files for defining ECS cluster, task definitions, and services. Added Terraform files for configuring IAM roles for ECS tasks and execution. Added Terraform files for setting up RDS instances with logical replication enabled. Added Terraform files for creating Application Load Balancers with HTTPS/gRPC listeners. Added Terraform files for defining VPC endpoints for AWS services. Added Terraform files for managing SSL certificates using ACM. Added Terraform files for defining security groups and variables.	app/infra/tf/ecs.tf app/infra/tf/iam.tf app/infra/tf/main.tf app/infra/tf/rds.tf app/infra/tf/vpc_endpoints.tf app/infra/tf/acm.tf app/infra/tf/variables.tf app/infra/tf/security.tf app/infra/tf/outputs.tf
Code modifications to support cloud deployment and secure communication.	Changed affinity generation to use a random UUID substring. Added methods to build gRPC client channels with SSL/TLS. Updated runConsumerClient to use SSL-enabled gRPC channels. Commented out continuous writing logic for remote topic hosts.	app/src/main/java/com/p14n/postevent/App.java
Enhanced event processing logic with batching and "one-at-a-time" execution.	Added OneAtATimeBehaviour interface for controlling concurrent execution. Updated CatchupService and UnprocessedSubmitter to implement OneAtATimeBehaviour. Modified UnprocessedSubmitter to find and resubmit unprocessed events in batches. Added logic to trigger subsequent unprocessed checks if a batch is fully processed. Added findUnprocessedEventsWithLimit method to UnprocessedEventFinder.	library/src/main/java/com/p14n/postevent/catchup/CatchupService.java library/src/main/java/com/p14n/postevent/catchup/UnprocessedSubmitter.java library/src/main/java/com/p14n/postevent/data/UnprocessedEventFinder.java library/src/main/java/com/p14n/postevent/catchup/OneAtATimeBehaviour.java
Added functionality to manage database replication slots and indexes.	Added clearOldSlots method to DatabaseSetup to drop inactive replication slots. Added a new index (idx_messages_status_time) to the messages table. Configured Debezium connector to drop the replication slot on stop. Added a test for clearing old slots.	library/src/main/java/com/p14n/postevent/db/DatabaseSetup.java library/src/main/java/com/p14n/postevent/debezium/DebeziumServer.java library/src/test/java/com/p14n/postevent/DatabaseSetupTest.java library/src/test/java/com/p14n/postevent/PostgresDebeziumConnectorTest.java
Updated the build process to use the shadowJar plugin and added new dependencies.	Added shadow Gradle plugin for creating uber JARs. Added grpc-netty-shaded and postgresql dependencies. Configured the shadowJar task for the application. Updated the fastTest task configuration in library build.gradle.	app/build.gradle library/build.gradle
Introduced flexible executor configuration for better resource control.	Added new constructor to DefaultExecutor to configure fixed-size executor. Added protected methods for creating different types of executors. Updated ConsumerClient and LocalPersistentConsumer constructors to accept batch size and use the new executor configuration. Updated MessageBrokerGrpcClient and TransactionalBroker to accept AsyncExecutor.	library/src/main/java/com/p14n/postevent/broker/DefaultExecutor.java library/src/main/java/com/p14n/postevent/ConsumerClient.java library/src/main/java/com/p14n/postevent/LocalPersistentConsumer.java library/src/main/java/com/p14n/postevent/broker/grpc/MessageBrokerGrpcClient.java library/src/main/java/com/p14n/postevent/broker/TransactionalBroker.java library/src/test/java/com/p14n/postevent/dst/DeterministicConsumerTest.java library/src/test/java/com/p14n/postevent/example/LocalPersistentConsumerExample.java library/src/test/java/com/p14n/postevent/example/RemoteConsumerExample.java
Added Dockerfile and build script for creating and pushing the application image.	Added Dockerfile to build an application image containing the uber JAR. Added make-image.sh script to build the uber JAR, build the Docker image, tag it, and push it to ECR.	app/infra/image/Dockerfile make-image.sh
Adjusted application logging level for more detailed output.	Changed the logger level for com.p14n.postevent package from INFO to DEBUG.	app/src/main/resources/logback.xml

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Hey @p14n - I've reviewed your changes - here's some feedback:

Overall Comments:

• This PR mixes application code changes with significant infrastructure additions (Terraform); consider splitting these into separate PRs for easier review.
• The gRPC client now uses TLS but skips certificate verification; ensure this is acceptable for the target deployment environment security requirements.
• The main application loop in App.java appears to have changed significantly with the commenting out of writeContinuously; ensure this reflects the intended runtime behavior.

Here's what I looked at during the review

• 🟡 General issues: 1 issue found
• 🟢 Security: all looks good
• 🟡 Testing: 1 issue found
• 🟢 Complexity: all looks good
• 🟢 Documentation: all looks good

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[sourcery-ai]

issue (bug_risk): Revisit shared state in OneAtATimeBehaviour.

Defining final AtomicInteger and AtomicBoolean in the interface means these fields are shared across all implementers. This could lead to unexpected interactions when multiple instances use the oneAtATime behavior concurrently. Consider moving these to instance variables in an abstract class or as part of the concrete implementations.

[sourcery-ai]

suggestion (testing): Test for clearOldSlots should verify behavior more thoroughly.

While it's good that a test was added for the new clearOldSlots method, this test only calls the method without asserting its effects. Consider enhancing this test to:

1. Create one or two inactive replication slots with the expected naming pattern (postevent%) before calling clearOldSlots.
2. Verify that these specific slots are deleted after the call.
3. Verify that active slots or slots with different names are not deleted.
4. Test the case where no matching inactive slots exist.