WG2026.01 - Tazama Helm-based deployments

[Justus-at-Tazama]

Objectives/Outcomes: To be defined
Members: To be defined
Lead: To be nominated at kick-off
Kick-off: Date to be determined

[Sandy-at-Tazama]

Here is the link to the kick off meeting on 26 May 2026

[Justus-at-Tazama]

To include in our discussion tomorrow, the future use of this app:

[Justus-at-Tazama]

Working Group Scope and Kick-off

Attendees:

@Justus-at-Tazama
@Sandy-at-Tazama
@johanfol
@dom-apollotech
@omeco4christ

CC:

Festus Were
@techknowmage

Problem statement:

Everything is stale and needs to be updated (or be made new and fresh).

Primary Objectives:

1. Refresh or new?
2. Consolidation of all deployment repos into a single generic and universal approach/repo
3. Establish the process for delivering a cluster for testing purposes (at least)
4. Develop a deployment approach for Tazama on a Kubernetes substrate
   1. With Jenkins (building your own images)
   2. Without Jenkins (using the published DockerHub images)
5. Catalog the decisions that need to be made to achieve scale (for different components, overall, etc.)
6. Include all Tazama functional component groups (Core, Extensions, BIAR)

Secondary objectives / specific tasks

1. Identify, evaluate, and select appropriate tools.
2. External/independent secrets management.
3. Mechanisms for authentication to facilitate deployments via Helm charts and assess the impact on the Helm Charts development.
4. The point of integration with the cloud-specific Kubernetes service with authentication.

Constraints/Assumptions/Risks/Dependencies

1. Limited access to physical/cloud infrastructure (AWS only)

Out of Scope

1. Rule processors deployed out of the Tazama Rule Studio
2. Creating a reference infrastructure deployment for Tazama (5. Consider a reference infrastructure deployment as an example of successful infrastructure provisioning as a dependency for a successful Tazama implementation)
3. Phase 2: Ensure the helm-charts stay up to date and relevant with ongoing system releases by including their testing through deployment into the release CI/CD processes

Actions for next session

1. Set up a follow-up meeting (and recurring) for Tuesdays 16:00 SAST.
2. Consider the objective of including a documented, reproducible reference prodution-grade implementation of Tazama on AWS infrastructure capable of operating evaluations at 100 evaluations per second.

[Justus-at-Tazama]

For scope consideration: should we implement something like Hashicorp Vault to centrally manage environment variables and secrets during a Tazama Kubernetes deployment?

[Justus-at-Tazama]

Tooling

Current

Tool	Link	Description
Jenkins	https://www.jenkins.io/	The leading open source automation server, Jenkins provides hundreds of plugins to support building, deploying and automating any project.
Helm	https://helm.sh/	Helm helps you manage Kubernetes applications — Helm Charts help you define, install, and upgrade even the most complex Kubernetes application. Helm files can be parameterised with a values file.
Docker	https://docker.com/	Dockerfiles define the build process for building a container from source.

Future consideration

Tool	Link	Description
ArgoCD	https://github.com/argoproj/argo-cd	Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes.
Helmfile	https://github.com/helmfile/helmfile	Helmfile is a declarative spec for deploying helm charts. It lets you... Keep a directory of chart value files and maintain changes in version control. Apply CI/CD to configuration changes. Periodically sync to avoid skew in environments.
OpenTofu	https://opentofu.org/	OpenTofu is a reliable, flexible, community-driven infrastructure as code tool under the Linux Foundation's stewardship. It serves as a drop-in replacement for Terraform, preserving your existing workflows and configurations.

[dom-apollotech]

Hi, all, thanks for the invite to the working group meeting. It was good to meet you all.

Putting myself in the position of DevOps tasked with instantiating Tazama on Kubernetes, with the initial purpose of getting the deployment operational as quickly as possible while still supporting future growth and scale, I would suggest the following approach.

Separate image build and CI concerns from Kubernetes deployment concerns. The Kubernetes deployment should assume that all required container images already exist in a registry, whether sourced from DockerHub or from project-managed build pipelines (this is in Justus' document).

Treat foundational infrastructure components as prerequisites managed independently, either through their upstream Helm charts or through managed services:

• NATS
• PostgreSQL
• Valkey/Redis
• Keycloak

This allows the deployment effort to focus specifically on the core Tazama runtime components using Helm:

• tms-service
• event-director
• event-sidecar (subject to architectural validation)
• typology-processor
• transaction-aggregation-decisioning-processor (TADProc)
• scalable rule-worker deployments driven by rule configuration stored in PostgreSQL

The deployment approach should also include a standardized mechanism for:

• database schema migration
• seed configuration initialization
• loading/updating network maps
• loading/updating rule and typology configuration

The following components can initially be considered optional, integration-specific, or deferred until later phases:

• cms-service
• lumberjack
• nats-utilities
• elasticsearch

[Justus-at-Tazama]

Working Group Scope and Kick-off - part II

Attendees:

@Justus-at-Tazama
@Sandy-at-Tazama
@johanfol
@dom-apollotech
@techknowmage
@elnyry-sam-k
Brian Ouma

CC:

Festus Were
@omeco4christ

Actions/Decisions:

1. @johanfol to lead the working group
2. @elnyry-sam-k volunteered to test the final deliverable out of the working group in a Mojaloop deployment
3. Proposals and recommendations should ideally be submitted as discussions for asynchronous collaboration between formal meetings so that the meeting time can be spent on decisions and actions and not extended debate.
4. We'll start our discussion at the next session with an overview of tooling - if anyone has any tools to propose, comment under WG2026.01 - Tazama Helm-based deployments #21 (comment)

[Justus-at-Tazama]

Working Group - Discussions on tooling

Attendees:

@Justus-at-Tazama
@johanfol
@dom-apollotech
@techknowmage
Brian Ouma
@omeco4christ

CC:

@Sandy-at-Tazama
@elnyry-sam-k
Festus Were

Principles:

1. Manual steps in the deployment are to be avoided.

Discussion

The following diagram describes the workflows for CI/CD as it is implemented in Tazama for deployment (light) and deployment (production-grade).

[Tazama] dev branch merge / RC tag
        |
        v
[Tazama] GitHub Actions builds service images on merge (Docker Images)
        |
        v
[Tazama] Push immutable images to Docker Hub / GHCR
        |
        v
Available for Docker-based


[Tazama] Develop Jenkins scripts (build containers) for SI's to use? (now)
[Tazama] Develop Helm charts (deploy containers) for SI's to use? (now)
[Tazama] Develop ArgoCD documents for SI's to use? (Later)


[SI] dev branch merge / RC tag
        |
        v
[SI] Jenkins builds service images on merge (Containers)
        |
        v
[SI] Push immutable images to Container registry
        |
        v
[SI] Update Helm values with image tags (e.g. rc)
        |
        v
[SI] Helm chart deploys Tazama to Kubernetes
        |
        v
[SI] Argo CD syncs cluster from Git
        |
        v
[SI] Smoke tests / Postman / health checks
        |
        v
[SI] Promote same image set to staging, prod, or latest

Actions:

1. If no alternatives are forthcoming for Jenkins and Helm, we'll stick with these for now.
2. ArgoCD investigation and analysis will be deferred to a future phase.

[dom-apollotech]

A self-contained helm chart for deploying a simple Tazama cluster

I've put together a helm chart that deploys a minimal but functional Tazama installation on Kubernetes. This is essentially a translation of Full-Stack-Docker-Tazama using Gemini AI to do most of the heavy lifting.

The helm chart is self-contained. It deploys its own infrastructure services (NATS, Postgres, and Valkey) as well as Tazama core services (admin service, transaction monitoring service, event director, etc.) and Tazama rules (Rule 901 and Rule 902). For the images, it uses the precompiled versions from Docker Hub.

However, it doesn't currently use authentication services (nor Keycloak) or relay services.

I've tested this on our company's internal Kubernetes cluster, where it deploys the containers correctly.

Note: Don't forget to set up kubectl proxy forwarders for the exposed services.

Configuring

After deploying the helm chart, it still needs to be configured with the rules, typologies, and network map. For this, see the Postgres configuration script. Again, this is based off Full-Stack-Docker-Tazama.

Testing

It's possible to test some simple transactions against Tazama deployment. See my Rule 901 and Rule 902 test scripts.

Notes

How this turned out was it became an integration test for different existing Tazama components. We just assume that the container images are already correct. This meets the step [SI] Helm chart deploys Tazama to Kubernetes discussed previously.

Can we have canonical steps to configure the Tazama instance and test it afterwards, in such a way that it can be automated? (I tested sending API requests to the admin service, but it missed key parameters; hence I had to revert to inserting configurations directly into the database.)

I tested this for both the '3.0.0' and 'rc' versions on Docker Hub. Both are able to deploy correctly, the only change needed being the version number on the configmap. However, the 'rc' version seems to require a new column on the accounts table.

For the most part, the difficulty I found was setting the correct environment variables for the core services and the rules; otherwise the containers would crash. This needed a lot of trial and error. This could be emphasized more in the documentation.

[Justus-at-Tazama]

Working Group - Getting into the deployment

Attendees:

@Justus-at-Tazama
@johanfol
@dom-apollotech
@techknowmage
Brian Ouma
@Sandy-at-Tazama

CC:

@omeco4christ
@elnyry-sam-k
Festus Were

Notes:

1. Dom walked us through his own exploration of the system deployment via Helm, constructed from the full-stack as background.
2. Add as an deliverable: A guide to completing and testing the configuration of the system after all components have been deployed.

• Post implementation migration of data

3. The PostgreSQL databases and tables were created by embedding the SQL creation script from a yaml file as part of the proces.
4. Foley was able to get the Helm charts to execute the database scripts using the "entry-point" feature.
5. Helm accommodates startup, liveliness and readiness probes into the Kubernetes container to check if the database is ready to be set up.
6. Proposed: Seperate setup (DB tables) from configuration (e.g. via Admin Services).

• Agreed
• Constraint: The admin-services is the trust boundary (authentication/validation/audit)
• Apply the configuration via the admin-service API
• Current: configs are in Postman, but posted and stored in JSON
• Dependency: Authenticated User - default tazama-api-client user from Keycloak with its credentials
• Prefabbed shell script running curl commands to feed three JSON files into three endpoints. (Set the networtk map active upfront).

7. Automation requires the "entryPoint" - cannot use curl to seed configuration through the API.
8. Setting up Keycloak: How?

• Ship a default Realm file with default roles and default users (with pre-set passwords/parameterised passwords from a central setup file?).

Next session:

• Review the current helm deployment, compile requirements for updating then and roadmap the update.

[Justus-at-Tazama]

Working Group - Getting into the deployment

Attendees:

@Justus-at-Tazama
@dom-apollotech
@elnyry-sam-k
Brian Ouma
@Sandy-at-Tazama

CC:

@techknowmage
@johanfol
@omeco4christ
Festus Were

Notes:

1. Curl is a fair substitute for Postman for API-based configuration of the system through the Admin-Service API
2. Not all of the components would be deployed in all of the change management environments: the TCS and TRS are specifically development-focused tools. It would make sense for the implementer to be able to choose which components they want to deploy, or what configuration of components they want to deploy.

[masterxavierfox]

Hello Team here are some of my thoughts I think we should define success criteria for the Helm workstream as a supported deployment workflow.

Right now the discussion already surfaces the main gaps:

• post-deploy configuration is not yet canonical
• direct database setup is still being used where the intended trust boundary is the admin-service
• Keycloak bootstrap is still unclear
• different environments do not need the same component set
• contributors still rely too much on trial and error for required values and environment variables

I suggest we treat Helm support as a deployment standardization effort with explicit acceptance criteria.

Proposed definition of done:

• a clean Kubernetes install succeeds with documented prerequisites
• required dependencies start in the correct order and probes behave predictably
• authentication is bootstrapped through a default Keycloak realm/client pattern
• Tazama configuration is applied through the admin-service API, not primarily through direct DB writes
• sample rules, typologies, and network map can be loaded through a documented bootstrap path
• a smoke test can submit sample transactions and confirm expected outcomes
• the deployment supports upgrade and rollback validation
• implementers can choose deployment profiles, for example minimal runtime, development, and full stack

I also think we should produce a canonical automation path:

• values examples for each deployment profile
• a bootstrap script or job for auth and configuration
• a smoke test harness for post-install validation
• a conformance checklist so beta testers can report deployment issues consistently

That would give the community something much more valuable than a working chart in one environment. It would give us a repeatable, testable, supportable Kubernetes deployment path that can be validated during the 4.0.0 beta and maintained after it.

Ill work on samething i can share if we are agreeable to this , id want to work on a "validation kit" and building upon the work done by @dom-apollotech great work btw, loving the momentum we are making.

Please share your thoughts

This is Festus BTW :-)

[Justus-at-Tazama]

Hi Festus! Thank you for the comments! Wow, this is comprehensive. We're meeting again in two weeks on the 14th - I look forward to you joining, if you can!

[Justus-at-Tazama]

Working Group - Getting into the deployment

Attendees:

@Justus-at-Tazama
@dom-apollotech
Brian Ouma
@rtkay123
@Sandy-at-Tazama

CC:

@elnyry-sam-k
@techknowmage
@johanfol
@omeco4christ
@masterxavierfox

The focus of today's session was to delve into the process of a deployment. The team defined the follows as a starting point and anchor for the specific deliverables required to facilitate a complete deployment via Helm.

Notes:

1. Action: review the existing Helm charts to understand how they function so that we can understand how they need to be modified for Tazama release 4.
2. Rodney pointed out that each Tazama repository contains a deployment.yaml file that is intended to be part of the process, e.g. https://github.com/tazama-lf/tms-service/blob/main/deployment.yaml
3. There are three options for producing the images that are to be deployed (DockerHub, Jenkins, GitHub Actions). We prioritized these for this project with DockerHub (easiest MVP), followed by Jenkins (familiar and expected), and finally the GitHub Actions (green fields, backlogged).
4. Rodney also raised an open question about whether we should pre-publish our Helm Charts as Helm Packages on a registry somewhere. Open for discussion and consideration - Justus's view is that if the Helm Packages is only a compiled Helm Chart, we need to compare the options to understand what the specific value of the additional step would be.

[Justus-at-Tazama]

Working Group - Definition of Done

Attendees:

@Justus-at-Tazama
@dom-apollotech
Brian Ouma
@rtkay123
@Sandy-at-Tazama

CC:

@elnyry-sam-k
@techknowmage
@johanfol
@omeco4christ
@masterxavierfox

Decision: Packaging and hosting on a registry: By general agreement, let's do it. Ref: https://helm.sh/docs/howto/chart_releaser_action/ - ArtifactHub

Definition of Done:

• Deployment readiness check / process
  • Ingress Controller (e.g. nginx / Kubernetes API Gateway)
  • Kubernetes API access available
• Helm charts updated / refreshed
  • Core
    • Priority Phase 1 (Default) = Tazama rules
    • Phase 2 = TCS custom data rails
    • Phase 2 = TRS custom rules
    • Phase 3 = Custom TMS API
    • Phase 3 = Custom rules
  • Dev Tools
  • Extensions
  • BIAR
  • Optional components should be optional
  • As few charts as possible
• Values / parameters capabilities
  • "Open" services, e.g. https://github.com/tazama-lf/Full-Stack-Docker-Tazama/blob/dev/infra/aws/aws-deployment-instructions.md#architecture-summary
• Comprehensive documentation
  • Processes
  • Values
• Automation of a package publishing process
• Publishing of the package on ArtifactHub
• Test the deployment through Helm for real
• Configure the deployed instance
• Smoke test of the running platform

Notes:

• The Data Lakehouse needs a clustered PostGres replica for DLH replication to preserve performance.
• No accommodation of infra components swap-outs (ValKey, Redis, NATS).

[Justus-at-Tazama]

@tazama-lf/tsc-wg-helm - Hi everyone! I've created the tazama-helm repo and set this team up as "Maintainers" and (temporary) code owners until we've matured this repo enough to cut a release out of the dev branch.

[Justus-at-Tazama]

Working Group - Definition of Done

Attendees:

@Justus-at-Tazama
@elnyry-sam-k
Brian Ouma
@rtkay123
@Sandy-at-Tazama

CC:

@dom-apollotech
@techknowmage
@johanfol
@omeco4christ
@masterxavierfox

Notes:

1. Discussion around connection pooling (strong yes) and also a real-time DB replica of the Core PostgresDB to stage data for the DLH (strong yes) should be part of the Helm deployment process.
2. Brian demoed the work he has done to deploy Tazama via Helm.
3. Add Helm tests to the definition of done
4. Create the starting point for iteration on the repo and iterate through issues and collaboration

Actions:

Justus to compose the outcomes of the WG into a spec for delivery.
Justus to create a new tazama-helm repo
Justus to create a team for the working group members and grant repo access
Brian to commit his WIP Helm charts to the new repo as a starting point for delivery