v1.1.0 - Automated Data Processing with Spring Batch and CI/CD Pipeline on Google Cloud Run

Release Notes - v1.1.0

Release Date: 11-12-2024
Version: v1.1.0

Overview:

This release introduces key updates to the Spring Batch application and CI/CD pipeline. The changes include enhancements to handle larger files, improved error handling in job execution, and optimizations to the Cloud Run job configuration. Additionally, the deployment process has been streamlined for better scalability and efficiency.

Key Features:

1. Enhanced Spring Batch Application:

• Updated the Spring Batch application to improve handling of larger datasets, ensuring that the data processing tasks scale more efficiently.
• Added better logging and error handling for improved job traceability.

2. CI/CD Pipeline Enhancements:

• Optimized the CI/CD pipeline for faster and more reliable deployments to Google Cloud Run.
• Automated Error Handling: Improved job status monitoring with more granular failure detection and reporting.
• Job Timeout Adjustments: Increased job timeouts to ensure larger files can be processed without timeouts in Cloud Run.

3. Cloud Run Job Optimizations:

• CPU and Memory Configuration: Increased CPU and memory allocations for Cloud Run jobs to handle large-scale data processing.
• Improved Error Handling: Enhanced the Cloud Run job configuration for better resilience against failure scenarios.
• Scalable Processing: Optimized the Cloud Run setup for better scalability with larger workloads.

Benefits:

• Improved Scalability: The updates ensure the application can efficiently handle larger files, offering better scalability.
• Better Job Monitoring: Enhanced error handling and job monitoring for more reliable and predictable job execution.
• Optimized Resource Usage: Increased resource allocations for Cloud Run jobs to prevent timeouts during large-scale data processing.
• Streamlined Deployment: Improved deployment speed and reliability for faster and more predictable releases.

Deployment and Configuration:

1. Deployment Process:

   • The pipeline triggers automatically on code changes, builds the Docker image, and deploys to Cloud Run.
   • The Cloud Run job is now optimized for large datasets with increased timeouts, CPU, and memory settings.

2. Environment Variables:

   • The following environment variables are set for the Cloud Run job:
     • DB_URL: Database connection URL.
     • DB_USERNAME: Username for database authentication.
     • DB_PASSWORD: Password for database authentication.
     • OUTPUT_PATH: Path for storing the processed file on GCS.

3. Job Monitoring:

   • Enhanced job monitoring with automatic notifications for failures and errors, ensuring immediate action can be taken if needed.

Testing and Validation:

• Unit Testing: All unit tests for Spring Batch logic were executed and passed successfully.
• Integration Testing: Full pipeline testing was conducted in staging environments to validate the job's performance with larger files.
• CI/CD Pipeline Testing: The pipeline was tested end-to-end to ensure job timeouts, error handling, and scaling optimizations worked as expected.

Known Issues:

• No known issues at this time. All tests and workflows have passed successfully.

Technology Stack:

• Spring Batch for batch data processing.
• Docker for containerization.
• GitHub Actions for CI/CD automation.
• Google Cloud Run for serverless job execution.
• Google Cloud Storage (GCS) for data storage.

What's Changed

• [FEATURE] Enhance Data Processing Capabilities for Larger Datasets by @IMISSHER99 in #15
• [RELEASE] Cloud Run Job Optimizations for Large File Handling by @IMISSHER99 in #16
• [RELEASE] Improve CI/CD Pipeline for Faster Deployments by @IMISSHER99 in #17
• [RELEASE] Increase Task Timeout and Resource Allocation for Cloud Run Job Execution in CI/CD

Full Changelog: https://github.com/DataArize/recommendations-data-preprocessing-batch/commits/v1.1.0

v1.0.0 - Automated Data Processing with Spring Batch and CI/CD Pipeline on Google Cloud Run

Release Notes - v1.0.0

Release Date: 11-11-2024
Version: v1.0.0

Overview:

This release introduces a Spring Batch application designed for data processing, along with a fully automated CI/CD pipeline for deployment to Google Cloud Run. This setup ensures data is read from Google Cloud Storage (GCS), processed, and then written back to GCS with minimal manual intervention, following enterprise best practices for continuous delivery.

Key Features:

1. Spring Batch Application:

• Developed a Spring Batch application to perform data processing tasks.
• Reads raw data from Google Cloud Storage (GCS), processes it, and generates a pre-processed output file.
• Automatically transfers the processed data back to GCS, enabling seamless data flow within the organization’s cloud architecture.

2. CI/CD Pipeline:

• Established a CI/CD pipeline using GitHub Actions for seamless automation.
• Key pipeline steps:
  • Docker Image Build: Automatically builds the Docker image for the Spring Batch application.
  • Push to Artifactory: The Docker image is securely pushed to Artifactory, maintaining version control and enabling rollbacks if needed.
  • Cloud Run Job Deployment: Deploys the Docker image to Google Cloud Run, a fully managed serverless execution environment.

3. Automated Cloud Run Job Execution:

• Created a GitHub Action to trigger the Cloud Run job.
• Ensures the job runs automatically in a controlled environment after every deployment, improving operational efficiency.
• Includes error handling and job monitoring to guarantee successful execution.

Benefits:

• Fully Automated Workflow: Reduces manual intervention by automating the build, deployment, and execution pipeline.
• Scalable and Serverless: Leverages Google Cloud Run for scalable, serverless execution of batch jobs.
• Seamless Data Integration: Integrates directly with Google Cloud Storage for reading and writing data, ensuring smooth data processing pipelines.
• Version Control: Using Artifactory for Docker image management, ensuring secure versioning and traceability.

Deployment and Configuration:

1. Deployment Process:

   • The pipeline automatically triggers on code change, builds the Docker image, pushes it to Artifactory, and deploys it to Cloud Run.
   • Once the Cloud Run job is deployed, the GitHub Action triggers job execution with the specified parameters (e.g., environment variables, GCS paths).

2. Environment Variables:

   • The following environment variables are set for the Cloud Run job:
     • DB_URL: Database connection URL.
     • DB_USERNAME: Username for database authentication.
     • DB_PASSWORD: Password for database authentication.
     • OUTPUT_PATH: Path for storing the processed file on GCS.

3. Rollbacks and Versioning:

   • Rollbacks to previous versions of the Docker image are possible through Artifactory, ensuring quick restoration of previous working states if necessary.

Testing and Validation:

• Unit Testing: Unit tests were implemented for core Spring Batch logic to ensure data processing accuracy.
• Integration Testing: The full pipeline was tested on staging environments with sample data to ensure integration with Google Cloud Storage and Cloud Run.
• CI/CD Pipeline Testing: The pipeline was tested end-to-end to ensure the Docker image builds correctly, deploys to Cloud Run, and the Cloud Run job executes successfully.

Known Issues:

• No known issues at this time. All tests and workflows have passed successfully.

Technology Stack:

• Spring Batch for batch data processing.
• Docker for containerization.
• GitHub Actions for CI/CD automation.
• Google Cloud Run for serverless job execution.
• Google Cloud Storage (GCS) for data storage.
• Artifactory for Docker image storage.

What's Changed

• [FEATURE] Implement Data Preprocessing and File Transfer for Recommendation Engine by @IMISSHER99 in #2
• [RELEASE] Merge Data Preprocessing and Cloud Run Deployment for Recommendation Engine by @IMISSHER99 in #9
• [RELEASE] Deploy GitHub Action Workflow for Cloud Run Job Execution by @IMISSHER99 in #12

Full Changelog: https://github.com/DataArize/recommendations-data-preprocessing-batch/commits/v1.0.0