v1.0.0 - Automated Data Processing with Spring Batch and CI/CD Pipeline on Google Cloud Run

Release Notes - v1.0.0

Release Date: 11-11-2024
Version: v1.0.0

Overview:

This release introduces a Spring Batch application designed for data processing, along with a fully automated CI/CD pipeline for deployment to Google Cloud Run. This setup ensures data is read from Google Cloud Storage (GCS), processed, and then written back to GCS with minimal manual intervention, following enterprise best practices for continuous delivery.

Key Features:

1. Spring Batch Application:

• Developed a Spring Batch application to perform data processing tasks.
• Reads raw data from Google Cloud Storage (GCS), processes it, and generates a pre-processed output file.
• Automatically transfers the processed data back to GCS, enabling seamless data flow within the organization’s cloud architecture.

2. CI/CD Pipeline:

• Established a CI/CD pipeline using GitHub Actions for seamless automation.
• Key pipeline steps:
  • Docker Image Build: Automatically builds the Docker image for the Spring Batch application.
  • Push to Artifactory: The Docker image is securely pushed to Artifactory, maintaining version control and enabling rollbacks if needed.
  • Cloud Run Job Deployment: Deploys the Docker image to Google Cloud Run, a fully managed serverless execution environment.

3. Automated Cloud Run Job Execution:

• Created a GitHub Action to trigger the Cloud Run job.
• Ensures the job runs automatically in a controlled environment after every deployment, improving operational efficiency.
• Includes error handling and job monitoring to guarantee successful execution.

Benefits:

• Fully Automated Workflow: Reduces manual intervention by automating the build, deployment, and execution pipeline.
• Scalable and Serverless: Leverages Google Cloud Run for scalable, serverless execution of batch jobs.
• Seamless Data Integration: Integrates directly with Google Cloud Storage for reading and writing data, ensuring smooth data processing pipelines.
• Version Control: Using Artifactory for Docker image management, ensuring secure versioning and traceability.

Deployment and Configuration:

1. Deployment Process:

   • The pipeline automatically triggers on code change, builds the Docker image, pushes it to Artifactory, and deploys it to Cloud Run.
   • Once the Cloud Run job is deployed, the GitHub Action triggers job execution with the specified parameters (e.g., environment variables, GCS paths).

2. Environment Variables:

   • The following environment variables are set for the Cloud Run job:
     • DB_URL: Database connection URL.
     • DB_USERNAME: Username for database authentication.
     • DB_PASSWORD: Password for database authentication.
     • OUTPUT_PATH: Path for storing the processed file on GCS.

3. Rollbacks and Versioning:

   • Rollbacks to previous versions of the Docker image are possible through Artifactory, ensuring quick restoration of previous working states if necessary.

Testing and Validation:

• Unit Testing: Unit tests were implemented for core Spring Batch logic to ensure data processing accuracy.
• Integration Testing: The full pipeline was tested on staging environments with sample data to ensure integration with Google Cloud Storage and Cloud Run.
• CI/CD Pipeline Testing: The pipeline was tested end-to-end to ensure the Docker image builds correctly, deploys to Cloud Run, and the Cloud Run job executes successfully.

Known Issues:

• No known issues at this time. All tests and workflows have passed successfully.

Technology Stack:

• Spring Batch for batch data processing.
• Docker for containerization.
• GitHub Actions for CI/CD automation.
• Google Cloud Run for serverless job execution.
• Google Cloud Storage (GCS) for data storage.
• Artifactory for Docker image storage.

What's Changed

• [FEATURE] Implement Data Preprocessing and File Transfer for Recommendation Engine by @IMISSHER99 in #2
• [RELEASE] Merge Data Preprocessing and Cloud Run Deployment for Recommendation Engine by @IMISSHER99 in #9
• [RELEASE] Deploy GitHub Action Workflow for Cloud Run Job Execution by @IMISSHER99 in #12

Full Changelog: https://github.com/DataArize/recommendations-data-preprocessing-batch/commits/v1.0.0