VED-1103: Refactor Batch Processor (recordprocessor) deployment to a dedicated pipeline stage#1279
Merged
Thomas-Boyle merged 88 commits intomasterfrom Mar 27, 2026
Merged
Conversation
Contributor
Thomas-Boyle
commented
Mar 10, 2026
Thomas-Boyle
commented
- Added a new GitHub Actions workflow for deploying the Recordprocessor batch processor.
- Updated the continuous deployment workflow to include a dependency on the new Recordprocessor deployment.
- Introduced a new variable for the Recordprocessor image tag in Terraform configuration.
- Refactored ECS task definition to use the new image tag variable for the Recordprocessor container.
…deployment dependencies - Added a new GitHub Actions workflow for deploying the Recordprocessor batch processor. - Updated the continuous deployment workflow to include a dependency on the new Recordprocessor deployment. - Introduced a new variable for the Recordprocessor image tag in Terraform configuration. - Refactored ECS task definition to use the new image tag variable for the Recordprocessor container.
- Explicitly set the AWS_INFRA_ROLE_ARN secret in the continuous deployment workflow. - Upgrade aws-actions/configure-aws-credentials and aws-actions/amazon-ecr-login to specific versions for improved stability.
Thomas-Boyle
temporarily deployed
to
internal-dev
GitHub Actions
Inactive
Thomas-Boyle
temporarily deployed
to
internal-dev-sandbox
GitHub Actions
Inactive
Thomas-Boyle
temporarily deployed
to
internal-dev-sandbox
GitHub Actions
Inactive
Thomas-Boyle
temporarily deployed
to
internal-dev-sandbox
GitHub Actions
Inactive
Thomas-Boyle
had a problem deploying
to
internal-dev
GitHub Actions
Failure
Thomas-Boyle
temporarily deployed
to
internal-dev
GitHub Actions
Inactive
- Removed the separate Recordprocessor deployment workflow and integrated its functionality into the backend deployment workflow. - Added a new input parameter `enable_recordprocessor` to conditionally build and push the Recordprocessor Docker image. - Updated dependencies in the backend deployment to ensure proper execution order and image tagging for the Recordprocessor. - Enhanced the continuous deployment workflow by simplifying dependencies and improving clarity.
edhall-nhs
previously requested changes
Mar 10, 2026
Contributor
edhall-nhs
left a comment
edhall-nhs
left a comment
There was a problem hiding this comment.
Discussed on a call but to summarise:
- Integrate in deploy-backend workflow, not separate
- Add manual check that ECR repo exists first
- Pass through new image tag
… repository - Changed the behavior of the ECR repository check to fail the build-and-push-recordprocessor job if the repository does not exist, instead of skipping the image build and push. This ensures that the deployment process is more robust and provides clearer feedback on repository availability.
Thomas-Boyle
temporarily deployed
to
internal-dev
GitHub Actions
Inactive
Thomas-Boyle
temporarily deployed
to
internal-dev-sandbox
GitHub Actions
Inactive
Thomas-Boyle
temporarily deployed
to
internal-dev-sandbox
GitHub Actions
Inactive
Thomas-Boyle
had a problem deploying
to
internal-dev
GitHub Actions
Failure
Thomas-Boyle
temporarily deployed
to
internal-dev-sandbox
GitHub Actions
Inactive
Thomas-Boyle
temporarily deployed
to
internal-dev
GitHub Actions
Inactive
Thomas-Boyle
had a problem deploying
to
internal-dev
GitHub Actions
Failure
Thomas-Boyle
temporarily deployed
to
internal-dev
GitHub Actions
Inactive
- Modified the backend deployment workflow to include a new step for resolving the image tag based on the `enable_recordprocessor` input. This change ensures that the image tag is set to the output of the build-and-push step if the Recordprocessor is enabled, otherwise defaults to "latest". This enhances the clarity and reliability of the image tagging process during deployment.
- Replaced string concatenation for `diff_paths` with an array to manage paths more effectively. - Updated condition checks for deployment mode to utilize the new array structure, enhancing clarity and maintainability.
…nt workflow to streamline the process and improve efficiency.
… before building and pushing Docker images. This prevents unnecessary builds and optimizes the deployment process by reusing existing images when available.
…0, optimizing image management and storage.
avshetty1980
requested changes
Mar 24, 2026
Contributor
avshetty1980
left a comment
avshetty1980
left a comment
There was a problem hiding this comment.
just a few more changes before we can handoff to the next PR I think. Nearly there 👍🏾
… prefix reuse. Introduced a new input parameter to allow resolving the latest tag when the image version is not provided, improving flexibility in deployment configurations. Additionally, updated the manifest generation to include more contextual information such as build timestamp and Git metadata.
…ude new parameters for image processing. Added support for building the record processor image and specifying the image version based on the Git SHA, enhancing deployment flexibility and traceability.
| uses: ./.github/workflows/deploy-backend.yml | ||
| with: | ||
| apigee_environment: ${{ matrix.sub_environment_name }} | ||
| recordprocessor_image_version: internal-dev-git-${{ github.sha }} |
Contributor
There was a problem hiding this comment.
I know that you only update the deploy-lambda-artifact.yml file but I want to capture this incase it is missed.
Higher dev environments does not consume the actual output from the producer job: The build workflow creates the recordprocessor GIT_TAG from the first 12 characters of the commit SHA.
Instead they synthesize their own selector as internal-dev-git-${{ github.sha }}, which uses the full SHA.
The reuse path resolves an image digest from the exact requested image version. So if internal-dev built internal-dev-git-abcdef123456, but higher environments request internal-dev-git-abcdef1234567890..., the lookup will fail.
…age version. Adjusted dependencies in the continuous deployment workflow to ensure proper image versioning based on previous job outputs, enhancing deployment accuracy.
shivani-rana-kainos
previously approved these changes
Mar 25, 2026
avshetty1980
previously approved these changes
Mar 25, 2026
Contributor
avshetty1980
left a comment
avshetty1980
left a comment
There was a problem hiding this comment.
Nice, Great start to moving to consistent and reusable deployment for recordprocessor and lambdas 👍🏾
…n continuous deployment. Adjusted output variable for image version in backend deployment workflow and refined implicit tag prefix reuse logic. Added AWS connection step in Lambda artifact deployment workflow, enhancing deployment efficiency and flexibility.
… image URI. Added logic to set a placeholder value if the image output is unavailable, ensuring variable validation during destruction processes remains functional.
…yment and introduce conditional deployment mode for Lambda artifacts. This change enhances flexibility by allowing image reuse when a version is specified, streamlining the deployment process.
avshetty1980
approved these changes
Mar 26, 2026
Contributor
avshetty1980
left a comment
avshetty1980
left a comment
There was a problem hiding this comment.
Looks really good, thanks Thomas 👍🏾
|
Akol125
approved these changes
Mar 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.