This article will be an end-to-end guide for installing Tazama to any cluster but only once your EKS infrastructure is setup.
- EKS Detailed Installation Guide
- Instructions
- Step 1 - Helm charts
- Step 2 : Configuration
- Step 3: Post-Installation Configuration
- Setting up TLS for Ingress
- Configuring Ingress Domain Names
- Vault Configuration
- Logstash Configuration
- APM Configuration
- Jenkins Configuration
- Adding Credentials in Jenkins
- Adding Managed Files for NPM Configuration in Jenkins
- Jenkins Node.js Configuration
- Jenkins Docker Installation Configuration
- Building Jenkin Agent Locally
- Setting up a Jenkins cloud agent that will interact with your Kubernetes cluster
- Steps to Configure Jenkins Global Variables
- Adding Jenkins Jobs
- Step 4 :Running Jenkins Jobs to Install Processors
- Common Errors**
- Conclusion: Finalizing Tazama System Installation
Read through the infrastructure spec before starting with the deployment guide.
Infrastructure Spec for Tazama Sandbox
Infrastructure Spec for Tazama
Important: Access to the Tazama GIT Repository is required to proceed. If you do not currently have this access, or if you are unsure about your access level, please reach out to the Tazama Team to request the necessary permissions. It's crucial to ensure that you have the appropriate credentials to access the repository for seamless integration and workflow management.
Our repository list includes a variety of components, each representing specific microservices and tools within our ecosystem.
Repository list:
Default release version: rel-1-0-0
rule-001-<release version>-<envName variable set in jenkins>rule-002-<release version>-<envName variable set in jenkins>rule-003-<release version>-<envName variable set in jenkins>rule-004-<release version>-<envName variable set in jenkins>rule-006-<release version>-<envName variable set in jenkins>rule-007-<release version>-<envName variable set in jenkins>rule-008-<release version>-<envName variable set in jenkins>rule-010-<release version>-<envName variable set in jenkins>rule-011-<release version>-<envName variable set in jenkins>rule-016-<release version>-<envName variable set in jenkins>rule-017-<release version>-<envName variable set in jenkins>rule-018-<release version>-<envName variable set in jenkins>rule-021-<release version>-<envName variable set in jenkins>rule-024-<release version>-<envName variable set in jenkins>rule-025-<release version>-<envName variable set in jenkins>rule-026-<release version>-<envName variable set in jenkins>rule-027-<release version>-<envName variable set in jenkins>rule-028-<release version>-<envName variable set in jenkins>rule-030-<release version>-<envName variable set in jenkins>rule-044-<release version>-<envName variable set in jenkins>rule-045-<release version>-<envName variable set in jenkins>rule-048-<release version>-<envName variable set in jenkins>rule-054-<release version>-<envName variable set in jenkins>rule-063-<release version>-<envName variable set in jenkins>rule-074-<release version>-<envName variable set in jenkins>rule-075-<release version>-<envName variable set in jenkins>rule-076-<release version>-<envName variable set in jenkins>rule-078-<release version>-<envName variable set in jenkins>rule-083-<release version>-<envName variable set in jenkins>rule-084-<release version>-<envName variable set in jenkins>rule-090-<release version>-<envName variable set in jenkins>rule-091-<release version>-<envName variable set in jenkins>jenkins-inbound-agentchannel-router-setup-processor-<release version>-<envName variable set in jenkins>event-sidecar-<release version>lumberjack-<envName variable set in jenkins>tms-service-<release version>-<envName variable set in jenkins>transaction-aggregation-decisioning-processor-<release version>-<envName variable set in jenkins>typology-processor-<release version>-<envName variable set in jenkins>
This guide will walk you through the setup of the Tazama (Real-time Antifraud and Money Laundering Monitoring System) on a Kubernetes cluster using Helm charts. Helm charts simplify the deployment and management of applications on Kubernetes clusters. We will deploy various services, ingresses, pods, replica sets, and more.
- A Kubernetes cluster up and running.
- Helm installed on your local machine or wherever you plan to run the commands from.
- Basic understanding of Kubernetes concepts like namespaces, pods, and ingress.
The installation of our system requires the creation of specific namespaces within your cluster. These namespaces will be automatically created by the infra-chart Helm chart. Ensure these namespaces exist before proceeding:
cicddevelopmentingress-nginxprocessorecks-ns
If they are not created automatically, you can manually add them using the following command for each namespace:
kubectl create namespace <namespace-name>First, add the Tazama Helm repository to enable the installation of charts:
The list below are the different helm charts:
- NATS
- ElasticSearch
- ArangoDB (single deployment)
- ArangoDb Ingress Proxy
- Jenkins
- Redis-Cluster
- Nginx ingress
- APM (Elasticsearch)
- Logstash (Elasticsearch)
- Kibana (Elasticsearch)
- Infra-chart
- aws-ecr-credential
- Grafana - Optional
- Prometheus - Optional
- Vault - Optional
- KeyCloak - Optional
Optional - Please note that these are additional features; while not required, they can enhance the platform's capabilities. Implementing them is optional and will not hinder the basic operation or the end-to-end functionality of the platform.
ie: Another HELM chart exists for the clustered version of ArangoDB. However, the single deployment version is preferred over the clustered one because it includes functionality that is absent or required in the enterprise option.
https://github.com/frmscoe/EKS-helm
First, add the Tazama Helm repository to enable the installation of charts:
helm repo add Tazama https://frmscoe.github.io/EKS-helm/
helm repo updateTo confirm the FRMS repo has been successfully added:
helm search repo TazamaTo expose services outside your cluster, enable ingress on necessary charts:
- Kibana
- ArangoDb
- Jenkins
- TMS
helm install kibana Tazama/kibana --namespace=development --set ingress.enabled=true
...If you prefer not to configure an ingress controller, you can simply use port forwarding to access the front-end interfaces of your applications. This approach will not impact the end-to-end functionality of your system, as it is designed to utilize fully qualified domain names (FQDNs) for internal cluster communication.
The Tazama system is composed of multiple Helm charts for various services and components. These need to be installed in a specific order due to dependencies.
- Infra-chart - Sets up necessary namespaces and storage classes.
helm install infra-chart Tazama/infra-chart- Follow with the installation of other charts as listed, specifying the namespace as required:
helm install nginx-ingress-controller Tazama/ingress-nginx --namespace=ingress-nginx
helm install elasticsearch Tazama/elasticsearch --namespace=development
helm install kibana Tazama/kibana --namespace=development
helm install apm Tazama/apm-server --namespace=development
helm install logstash Tazama/logstash --namespace=development
helm install arangodb-ingress-proxy Tazama/arangodb-ingress-proxy --namespace=development
helm install arango Tazama/arangodb --namespace=development
helm install redis-cluster Tazama/redis-cluster --namespace=development
helm install jenkins Tazama/jenkins --namespace=cicd
helm install nats Tazama/nats --namespace=developmentSetup Notes for Deploying AWS ECR Credentials with Helm
To deploy the AWS ECR credentials using our Helm chart, you will need to provide the ECR registry URL, your access key ID, and your secret access key. These are sensitive credentials that allow Kubernetes to pull your container images from AWS ECR. This installation will create the frmpullsecrets that will be used to pull images from the ECR.
https://docs.aws.amazon.com/AmazonECR/latest/userguide/registry_auth.html
- Obtain AWS ECR Registry URL:
- Navigate to the Amazon ECR console.
- In the navigation pane, choose 'Repositories'.
- Select the desired repository.
- Find the 'URI' for your repository. It will look something like
123456789012.dkr.ecr.region.amazonaws.com/my-repository.
- Obtain AWS Access Key ID and Secret Access Key:
- Go to the IAM console.
- Create a new IAM user or use an existing one with permissions to access the ECR registry.
- Generate a new access key pair under the user's 'Security credentials' tab.
- Configure Helm Command:
- Use the
helm installcommand to deploy your chart, substituting in your ECR registry URL and the base64-encoded AWS credentials. For example:
helm install aws-ecr Tazama/aws-ecr-credential \\
--set aws.ecrRegistry="123456789012.dkr.ecr.region.amazonaws.com/my-repository" \\
--set aws.accessKeyId="AKIAIOSFODNN7EXAMPLE" \\
--set aws.secretAccessKey="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"Caution: Never expose your AWS access key ID or secret access key in version control or in a public setting.
For optional components like Grafana, Prometheus, Vault, and KeyCloak, use similar commands if you decide to implement these features.
Extra Information: https://helm.sh/docs/helm/helm_install/
If you need to remove the Tazama deployment:
helm uninstall Tazama
For a system utilizing a variety of Helm charts, optimizing performance, storage, and configuration can significantly impact its efficiency and scalability. Below are details on how to configure and optimize each component for your FRMS system:
- Configuration: Adjust NATS cluster size, memory, and CPU limits according to your workload.
- Storage: For persistent storage, configure the
nats-streamingstateful set with appropriate volume sizes. - Performance: Tune the
max_connections,max_payload, andwrite_deadlinesettings for better performance. - Documentation: NATS Documentation
- Configuration: Increase the number of nodes for redundancy and scalability. Adjust JVM settings for better performance.
- Storage: Utilize persistent volumes with dynamic provisioning for data storage. Consider SSDs for faster I/O.
- Performance: Optimize index settings, such as
number_of_shardsandnumber_of_replicas, based on your use case. - Documentation: Elasticsearch Documentation
- Configuration: Choose the right deployment mode (Single, Active Failover, or Cluster) based on your needs.
- Storage: Ensure that persistent volume claims (PVCs) are adequately sized for your database growth.
- Performance: Configure resource limits and requests to ensure that ArangoDB has enough CPU and memory.
- Documentation: ArangoDB Documentation
- Configuration: Set up the ingress rules to properly route traffic to the ArangoDB service.
- Security: Implement TLS termination at the ingress level for secure database connections.
- Documentation: The documentation would typically be part of the ArangoDB Kubernetes Operator guide or the specific ingress controller documentation used in your setup.
- Configuration: Utilize configuration as code (JCasC) for easier management and scalability of Jenkins settings.
- Storage: Configure persistent storage for Jenkins home to ensure data is retained across restarts.
- Performance: Adjust executors and resource limits based on your CI/CD pipeline requirements.
- Documentation: Jenkins Documentation
- Configuration: Set up Redis in cluster mode if high availability and scalability are required.
- Storage: Use persistent storage for data durability. Configure memory limits appropriately.
- Performance: Tune
maxmemorypolicies and replication settings for optimal performance. - Documentation: Redis Documentation
- Configuration: Adjust rate limiting, body size limits, and SSL/TLS configurations to meet your security and usability requirements.
- Performance: Enable HTTP/2, configure SSL ciphers, and tuning worker processes for better performance.
- Documentation: NGINX Ingress Controller Documentation
- Configuration: Link these components together for a cohesive monitoring and logging solution. Configure Kibana dashboards for easy visualization.
- Storage: Ensure Elasticsearch has enough storage for logs and APM data. Consider using faster storage classes for better performance.
- Performance: For Logstash, adjust pipeline workers and batch sizes. For APM, tune the sampling rate based on your needs.
- Documentation: Kibana Documentation Logstash Documentation APM Documentation
- Configuration: Use Grafana dashboards to visualize metrics from Prometheus and other data sources.
- Security: Implement OAuth or LDAP for authentication. Use HTTPS for secure connections.
- Documentation: Grafana Documentation Environment Monitoring
- Configuration: Adjust scrape intervals and retention policies to balance between performance and data granularity.
- Storage: Use persistent volumes for long-term metric storage. Consider high-performance storage for large datasets.
- Documentation: Prometheus Documentation
- Configuration: Set up Vault in HA mode with a proper storage backend like Consul for resilience.
- Security: Use auto-unseal feature to automate the unsealing process securely.
- Documentation: Vault Documentation
- Configuration: Set up realms, clients, and roles according to your authentication and authorization needs.
- Performance: Adjust caching settings and session limits to optimize for your workload.
- Documentation: Keycloak Documentation
Each of these components plays a critical role in the Tazama system. By carefully configuring and optimizing them according to the guidelines provided, you can ensure that your system is secure, scalable, and performs optimally. Always refer to the official documentation for the most up-to-date information and advanced configuration options.
In order to get the processor pods to write logs to the lumberjack deployment which then writes the log information to elasticsearch.
- There is a secret that is created for elasticsearch after the HELM install. Duplicate the one created by elasticsearch.
- Change the namespace for
developmenttoprocessor
Example
apiVersion: v1
kind: Secret
metadata:
name: elasticsearch-master-certs
namespace: processor
type: kubernetes.io/tls
data:
ca.crt: >-
tls.crt: >-
tls.key: >-
Secure your ingress with TLS by creating a tlscomsecret in each required namespace:
- Create a secret with your TLS certificate and key:
apiVersion: v1
kind: Secret
metadata:
name: tlscomsecret
namespace: development
type: kubernetes.io/tls
data:
tls.crt: <base64-encoded-cert>
tls.key: <base64-encoded-key>
- Apply this configuration for each relevant namespace (
development,processor,cicd,default).
Customize your ingress resources to match your domain names and assign them to the nginx-ingress-controller's IP address:
Using example.test.com as an example
Please see the example below:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: cicd-ingress
namespace: cicd
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/backend-protocol: HTTP
nginx.ingress.kubernetes.io/cors-allow-headers: X-Forwarded-For
nginx.ingress.kubernetes.io/proxy-body-size: 50m
nginx.ingress.kubernetes.io/use-regex: 'true'
...
spec:
tls:
- hosts:
- example.test.com
secretName: tlscomsecret
rules:
- host: example.test.com
...
Please see the TMS example below:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: test-ingress
namespace: processor
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/backend-protocol: HTTP
nginx.ingress.kubernetes.io/cors-allow-headers: X-Forwarded-For
nginx.ingress.kubernetes.io/proxy-body-size: 50m
nginx.ingress.kubernetes.io/use-regex: 'true'
spec:
tls:
- hosts:
- example.test.com
secretName: tlscomsecret
rules:
- host: example.test.com
http:
paths:
- path: /execute
pathType: ImplementationSpecific
backend:
service:
name: transaction-monitoring-service-rel-1-0-0
port:
number: 3000
- path: /
pathType: ImplementationSpecific
backend:
service:
name: transaction-monitoring-service-rel-1-0-0
port:
number: 3000
- path: /natsPublish
pathType: ImplementationSpecific
backend:
service:
name: nats-utilities-rel-1-0-0
port:
number: 3000
After installing the Vault chart, you'll need to initialize and unseal Vault manually. The process involves generating unseal keys and a root token which you'll use to access and configure Vault further.
For now vault has been integrated into the system but the Jenkins variables haven't been add to vault to be pull through . This will be done sometime.
https://developer.hashicorp.com/vault/docs/concepts/seal
If is set LogLevelto info, error etc.. in your Jenkins environment variables then you will need configure this.
For comprehensive instructions on how to configure logging to Elasticsearch, please refer to the accompanying document. It provides a step-by-step guide that covers all the necessary procedures to ensure your logging system is properly set up, capturing and forwarding logs to Elasticsearch. This includes configuring log shippers, setting up Elasticsearch indices, and establishing the necessary security and access controls. By following this documentation, you can enable efficient log management and monitoring for your services.
If is set APMActive to true (default:true) in your Jenkins environment variables then you will need configure this
Once configured, the APM tool will begin collecting data on application performance metrics, such as response times, error rates, and throughput, which are critical for identifying and resolving performance issues. The collected data is sent to the APM server, where it can be visualized and analyzed. For detailed steps on integrating and configuring APM with your Jenkins environment, please refer to the specific APM setup documentation provided in your APM tool's resources.
Credentials are critical for Jenkins to interact with other services like source control management systems (like GitHub), container registries, and Kubernetes clusters securely. Jenkins provides a centralized credentials store where you can manage all these credentials. Here's a step-by-step guide based on the images you've provided:
- Navigate to Manage Jenkins → Credentials → System → Global credentials (unrestricted).
- Click on Add Credentials.
- Select Username with password from the drop-down menu.
- Enter your GitHub username.
- Enter your GitHub password. If two-factor authentication is enabled, you'll need to use a personal access token in place of your password.
- Set the ID to something memorable, like Github.
- Optionally, provide a description like GitHub Credentials.
- Click Save.
To retrieve your ECR creditials for the aws user you can follow this guide:
Or run this command to retrieve your token:
aws ecr get-login-password --region < --region of the ECR>
The above comand will print out a token copy that token which is used as your password.
Example
Username: AWS Password:
- Follow the first two steps as above to navigate to the Add Credentials page.
- Select Username with password.
- Input the username for your container registry.
- Enter the corresponding password or access token for the registry.
- Assign a unique ID, such as ContainerRegistry.
- Include a description that helps identify the registry, like Login info for the container registry.
- Click Save.
- Again, follow the initial steps to reach the Add Credentials page.
- Select Secret text.
- Enter the personal access token you've created on GitHub with the necessary scopes to read packages.
- Set the ID, for example, githubReadPackage.
- In the description, note the purpose, such as GitHub package read access.
- Click Save.
To configure Jenkins to use Kubernetes secrets for authenticating with Kubernetes services or private registries, you can follow these steps, similar to setting up GitHub package read access:
- Retrieve the Kubernetes Token:
- Access your Kubernetes environment and locate the secret intended for Jenkins authentication, in this case,
scjenkins-secret.- Extract the token value from the secret, which is usually base64-encoded. You have need to decode.
- Add Secret in Jenkins:
- Navigate to the Jenkins dashboard and go to the credentials management section.
- Choose to add new credentials, selecting the "Secret text" type.
- Paste the token you retrieved from the
scjenkins-secretin namespace=processor into the Secret field.
- Configure the Credential ID:
- Set the ID of the new secret to
kubernetespro. This ID will be used to reference these credentials within your Jenkins pipelines or job configurations.
- Add a Description:
- Provide a description for the secret to document its use, such as "Token for authenticating Jenkins with Kubernetes services."
- Save the Configuration:
- Click Save to store the new credentials in Jenkins.
Following this process will allow Jenkins jobs to authenticate with Kubernetes using the token stored in the secret, enabling operations that require Kubernetes access or pulling images from private registries linked to your Kubernetes environment.
Navigate to Manage Jenkins → Managed files
The image shows a Jenkins configuration screen for adding a managed file, specifically an NPM config file (npmrc). Here's a breakdown of the steps and fields:
- Managed Files: This section is for adding configuration files that Jenkins will manage and use across different jobs.
- ID: A unique identifier for the managed file.
- Name: In this case, it's npmrcRulesCredentials. This name helps users identify the file's purpose when selecting it for use in a job.
- Comment: An optional field where you can provide additional information about the managed file, such as its intended use. Here, it is described as 'user config'.
- Enforce NPM version 9 registry format: A checkbox that, when checked, enforces the file to be compatible with the NPM version 9 registry format.
- Add NPM Registry:
- URL: The registry URL field should be filled with the NPM registry's URL. The provided URL, https://npm.pkg.github.com, this config is for accessing packages stored in GitHub Package Registry.
- Credentials: The dropdown is set to github public read package, indicating that the credentials stored in Jenkins should be used to authenticate with this registry.
- Use this registry for specific scoped packages: This option indicates that the registry URL and credentials should only be used for packages with a specific scope. In this case, the scope is frmscoe.
- Registry scopes: Here, you specify the scope for which this registry should be used. Scoped packages are prefixed with the scope in their package name, ie: frmscoe.
-
Content: The text area labeled 'Content' is where you can input the actual content of the .npmrc file. This content typically includes configuration settings like the registry URL, authentication tokens, and various other npm options. always-auth = false will not be always required (usually for public registries).
-
Add: After configuring all the fields, you would click "Add" to save this managed file configuration.
Once you've added this managed file, Jenkins can use it in various jobs that require npm to access private packages or specific registries. The managed file will be placed in the working directory of the job when it runs, ensuring that npm commands use the provided configuration.
Navigate to Manage Jenkins → Tools
- Name: Provide a descriptive name for the Node.js installation (e.g., "NodeJS 20.11.0").
- Install automatically: Check this option to have Jenkins handle the installation of Node.js automatically on the build agent if it is not already installed.
- Install from http://nodejs.org : Select this to install Node.js directly from the official Node.js website.
- Version: Specify the version of Node.js you require for your projects (e.g., "NodeJS 20.11.0"). Ensure to input a valid version number.
- Force 32bit architecture: Check this only if there is a need to install a 32-bit version of Node.js on a 64-bit machine, which is uncommon for most modern applications.
- Global npm packages to install: List any npm packages that should be installed globally, Add newman
- Global npm packages refresh hours: Define how often Jenkins should update the npm package cache. If set to "0", the cache is updated with every build, otherwise, it's updated at the specified interval in hours (e.g., "72" for every three days).
Navigate to Manage Jenkins → Tools
- Name: Assign a name to the Docker installation to be used as a reference within Jenkins (e.g., "Docker Stable Release").
- Install automatically: Enable this setting to have Jenkins take charge of Docker installation on the build agent if Docker isn't already installed.
- Download from http://docker.com : Opt to have Jenkins download Docker directly from the official Docker website.
- Docker version: Specify which version of Docker Jenkins should install. If you want the most recent version, use "latest". Otherwise, provide a specific version number (e.g., "19.03.12").
This needs to be completed before adding the Jenkins Cloud agent.
Please follow the following document to help you build and push the image to the container registry.
Building the Jenkins Agent Image
- Navigate to Manage Jenkins → Clouds → Kubernetes
- Add the Path to Your Kubernetes Instance: Enter the URL of your Kubernetes API server in the Kubernetes URL field. This allows Jenkins to communicate with your Kubernetes cluster.
- Disable HTTPS Certificate Check: If your Kubernetes cluster uses a self-signed certificate or you are in a development environment where certificate validation is not critical, you can disable the HTTPS certificate check. However, for production environments, it is recommended to use a valid SSL certificate and leave this option unchecked for security reasons.
- Add Kubernetes Namespace: Enter
cicdin the Kubernetes Namespace field. This is where your Jenkins agents will run within the Kubernetes cluster. - Add Your Kubernetes Credentials: Select the credentials you have created for Kubernetes access. These credentials will be used by Jenkins to authenticate with the Kubernetes cluster.
- Select WebSocket: Enabling WebSocket is useful for maintaining a stable connection between Jenkins and the Kubernetes cluster, especially when Jenkins is behind a reverse proxy or firewall.
- Add Jenkins URL: This should be the internal service URL for Jenkins within your Kubernetes cluster, like
http://jenkins.cicd.svc.cluster.local - Add Pod Label: Labels are key-value pairs used for identifying resources within Kubernetes. Here, you should add a label with the key
jenkinsand the valueagent. This label will be used to associate the built pods with the Jenkins service.
Add a Pod Template: This step involves defining a new pod template, which Jenkins will use to spin up agents on your Kubernetes cluster.
- Name: Name the pod template
jenkins-builder. This name is used to reference the pod template within Jenkins pipelines or job configurations. - Namespace: Specify
cicdas the namespace where the Jenkins agents will be deployed within the Kubernetes cluster. - Labels: Set
jenkins-agentas the label. This is a key identifier that Jenkins jobs will use to select this pod template when running builds.
Add a Container: In this part of the configuration, you define the container that will run inside the pod created from the pod template.
NOTE This needs to point to the docker image built in this step : Building the Jenkins Agent Image
- Name: The container name is set to
jnlp. This is a conventional name for a Jenkins agent container that uses the JNLP (Java Network Launch Protocol) for the master-agent communication. - Docker Image: The Docker image to use is example.io/jenkins-inbound-agent:1.0.0 . This image is pre-configured with all the necessary tools to run as a Jenkins agent.
- Always Pull Image: This option ensures that Jenkins always pulls the latest version of the specified Docker image before starting a build. This is important to keep your build environment up-to-date with the latest changes to the image.
- Working Directory: The working directory is set to
/home/jenkins/agent. This is the directory inside the container where Jenkins will execute the build steps. - Command to Run: This field is left blank, which means the default command from the Docker image will be used to start the agent.
Run in Privileged Mode: This is an advanced container setting that allows processes within the container to execute with elevated privileges, similar to the root user on a Linux system.
To select "Run in Privileged Mode" in Jenkins Kubernetes plugin:
- Within the container configuration, look for the "Advanced..." button or link (not visible in the screenshot) and click it to expand the advanced options.
- In the advanced settings, find the checkbox labeled "Run in privileged mode" and select it.
Image Pull Secret
Needs to be set to - frmpullsecret - see screenshot below
- Private Registry Authentication: If the container images used by your Jenkins jobs are hosted in a private registry, Kubernetes needs to authenticate with that registry. The image pull secret stores the required credentials (like a username and password or token).
- Adding Image Pull Secret to Pod Template:
- Navigate to the Kubernetes cloud configuration within the Jenkins system settings.
- Under the specific pod template that you are configuring, find the
ImagePullSecretssection. - Enter the name of the Kubernetes secret that contains your private registry credentials in the
Namefield. This secret should already exist within the same namespace as where your Jenkins builder pods are running. - If you have multiple registries or need to pull from multiple private sources, you can add additional image pull secrets by clicking on the “Add Image Pull Secret” dropdown and entering the names of these secrets.
- YAML Merge Strategy: The YAML merge strategy determines how Jenkins should handle the YAML definitions from inherited pod templates. If set to 'Override', it means that the current YAML will completely replace any inherited YAML, which could be important if you need to ensure that the image pull secrets are applied without being altered by any inherited configurations.
By properly configuring image pull secrets in your Jenkins Kubernetes pod templates, you enable Jenkins to pull the necessary private images to run your builds within the Kubernetes cluster. Without these secrets, the image pull would fail, and your builds would not be able to run.
- Accessing Global Configuration:
- Go to the Jenkins dashboard and navigate to Manage Jenkins > Configure System.
- Scroll down to the Global properties section.
- Check the box next to Environment variables to enable the definition of global environment variables.
- Updating Environment Variables:
- You will find a list of predefined variables, which you may need to update with new values relevant to the current deployment. These variables include configuration URLs, passwords, and tokens required by Jenkins jobs during the deployment process.
Passwords: These passwords can be found in your Kubernetes Cluster Secrets, which are autogenerated when the HELM installations are carried out.
Multiple ArangoDB passwords and endpoints: The reason we have different names and passwords for ArangoDB is to keep things organized and safe. Each name points to a different part of the database where different information is kept. Just like having different keys for different rooms. This is useful when you have more than one ArangoDB running at the same time and you want to keep them separate. This way, you can connect to just the part you need.
If you have a single database instance you may be wondering why multiple password variants are needed. For example, if my Configuration, Pseudonyms and TransactionHistory databases are served from the same Arango instance, why must I include single quotes in their password input whereas that requirement was not needed in the ArangoPassword variable.
The ArangoPassword variable is utilised as a CLI argument by newman, for setting up the environment. Where it is called, there is some shell substitution of the ArangoPassword variable but because the substitution involves a special character, $, that has to be surrounded by quotes.
newman {omitted} "arangoPassword=${ArangoPassword}" --disable-unicode
The same reasoning applies to passwords are that explicitly stated to need a single quote around them as they are substituted as is in processors' environments. This means that if your password contains special characters, then you must use single quotes to let the decoder know to interpret them as raw strings, or it will be taken as an indication of substitution
- Variables and Descriptions:
APMActive: A flag to enable or disable Application Performance Monitoring.- Default: True
- value: true / false
ArangoPassword: A secret password required for accessing the Database which is used for populating the Arango configuration.- eg: rm]ukXyA@M
ArangoConfigurationURL: Endpoint for the ArangoDB configuration Database.ArangoConfigurationPassword: A secret password required for accessing the Database. NB: The single quotes need to be added with your password.- eg: 'rm]ukXyA@M'
ArangoDbURL: Endpoint for the ArangoDB configuration Database.ArangoDbPassword: A secret password required for accessing the Database. NB: The single quotes need to be added with your password.- eg: 'rm]ukXyA@M'
ArangoPseudonymsURL: Endpoint for the ArangoDB pseudonym Database.ArangoPseudonymsPassword: A secret password required for accessing the Database. NB: The single quotes need to be added with your password.- eg: 'rm]ukXyA@M'
ArangoTransactionHistoryURL: Endpoint for the ArangoDB transaction history Database.ArangoTransactionHistoryPassword: A secret password required for accessing the Database. NB: The single quotes need to be added with your password.- eg: 'rm]ukXyA@M'
ArangoEvaluationURL: Endpoint for the ArangoDB Evalation Database.ArangoEvaluationPassword: A secret password required for accessing the Database. NB: The single quotes need to be added with your password.- eg: 'rm]ukXyA@M'
Branch: The specific branch in source control that the deployment should target.- value: main
CacheEnabled: A flag to enable or disable caching.- value: true / false
CacheTTL: Time-to-live for the cache in seconds.- eg: 86400
ELASTIC_HOST: The hostname for the Elasticsearch service.ELASTIC_USERNAME: The username for accessing Elasticsearch.- value: elastic
ELASTIC_PASSWORD: The password for accessing Elasticsearch.- eg: ty6&dabnbh
ELASTIC_SEARCH_VERSION: The version of Elasticsearch in use.- value: 8.5.1
EnableQuoting: A flag to enable or disable quoting functionality adding Pain messages in the chain.- value: false
envName: The environment name for the deployment.- eg: dev, prod , etc…
FLUSHBYTES: The byte threshold for flushing data.- value: 10
ImageRepository: The repository for Docker images.- eg: example.io
LogLevel: The verbosity level of logging. NB: The single quotes need to be added in.- eg: 'info', 'error', 'silent', etc…
MaxCPU: The maximum CPU resource limit.- value: leave blank
NATS_SERVER_TYPE: The type of NATS server in use.- value: nats
NATS_SERVER_URL: The URL for the NATS server.- value: nats.development.svc.cluster.local:4222
RedisCluster: A flag to indicate if Redis is running in cluster mode.- value: true
RedisPassword: The password for accessing Redis.- eg: ty6r5*&p0
RedisServers: The hostname for the Redis Cluster service. NB: The single quotes need to be added in to the host string.- value: '[{"host": "redis-cluster.development.svc.cluster.local", "port":6379}]'
Repository: This parameter specifies the name of a repository- value: frmscoe
- Provided is a
jobs.zipfile, which contains job configuration files that you need to add to your Jenkins instance. - Extract the zip file.
- Open a terminal on your local machine and change the directory to where you have stored the
jobs.zipfile and where you unpack it.
cd <path to configuration><path to configuration>is a placeholder for the actual directory path where yourjobs.zipunzipped files are located.
eg: cd "C:\Documents\tasks\Jenkins\jobs"
- Use the
kubectl cpcommand to copy the job configurations from your local machine to the Jenkins pod running in your Kubernetes cluster.
kubectl cp . <name of pod>:/var/jenkins_home/jobs/ -n cicd<name of pod>is a placeholder for the actual name of your Jenkins pod. You need to replace it with the correct pod name which you can find by runningkubectl get pods -n cicd.- The
-n cicdspecifies the namespace where your Jenkins is deployed, which in this case iscicd.
- After copying the job configurations, your Jenkins instance should recognize the new jobs. Jenkins will automatically load job configurations found in the
/var/jenkins_home/jobs/directory.
- You might need to manually reload the Jenkins configuration or restart the Jenkins service for the new job configurations to take effect. This can be done from the Jenkins interface or by restarting the Jenkins pod:
kubectl rollout restart deployment <jenkins-deployment-name> -n cicd- Make sure to replace
<jenkins-deployment-name>with the actual deployment name of your Jenkins instance. - You can also safeRestart through the URL.
eg: http://localhost:52933/safeRestart
The process involves configuring Jenkins to deploy various processors into the Tazama cluster. These processors are essential components of the system and require specific configurations, such as database connections and service endpoints, to function correctly.
Dashboard → Deployments→ ArangoDB
Run the Create Arango Setup and then Populate Arango Configuration jobs to populate the ArangoDB with the correct configuration required by the system. This job would utilize the variables set in the global configuration to connect to ArangoDB and perform the necessary setup.
After importing the Jenkins jobs, you need to configure each job with the appropriate credentials and Kubernetes server endpoint details. This setup is crucial to ensure that each job has the necessary permissions and access to interact with other services and the Kubernetes cluster.
- Access Each Rule Processor Job:
- Navigate to the job configuration for each rule processor, such as TMS, Typology, etc.
- Within each job, look for the section where you can define or edit the repository from which the job will fetch the code or artifacts.
- Repository Configuration:
- Set the Repository URL to the Git repository where the code for the processor is located. This is typically a URL like https://github.com//channel-router-setup-processor/.
- Under Credentials, select the appropriate credentials from the drop-down list, such as Github Creds, which should correspond to the credentials that have access to the repository.
- Kubernetes Configuration:
NOTE- The Kubernetes server endpoint can be copied from your .kubeconfig file under cluster -> server
- Check the option for Setup Kubernetes CLI (kubectl) if not already done.
- Input the Kubernetes server endpoint; this is the API server URL of your Kubernetes cluster.
- Select the Credentials for Kubernetes from the drop-down list, which will typically be a service account token or a kubeconfig file.
- Binding Credentials:
- Under the Bindings section, define the environment variables that the job will use internally.
- For username and password types, such as container registry credentials, set the appropriate Username Variable and Password Variable. Use REG_USER and REG_PASS for registry credentials.
- Choose the specific credentials from the drop-down list, like Login info for the Sybrin Azure container registry.
- For secret texts, such as a GitHub access token, set the Variable to an environment variable name, such as READ_GH_TOKEN, and select the appropriate credentials, like github public read package.
By completing these steps, you ensure that each Jenkins job can access the necessary repositories and services with the correct permissions and interact with your Kubernetes cluster using the right endpoints and credentials. It's essential to review and verify these settings regularly, especially after any changes to the credentials or infrastructure.
Dashboard → Deployments→ Pipelines→ Deploying All Rules and Rule Processors
Run the Jenkins jobs that deploy the processors to the Tazama cluster. These jobs will reference the global environment variables you've configured, ensuring that each processor has the required connections and configurations.
Run the Deploying All Rules and Rule Processors Pipeline Job
Dashboard → Testing→ E2E Test
The "E2E Test" job in Jenkins is an essential component for ensuring the integrity and robustness of the platform. It is specifically designed to perform comprehensive end-to-end testing, replicating user behaviors and interactions to verify that every facet of the platform functions together seamlessly.
- Comprehensive Assessment: The E2E Test tests all aspects of the platform, from individual services to data processing, ensuring that all integrated components are operating together.
- Confidence in Releases: Successful E2E tests will display the stability and readiness of the platform for production releases.
- Test Execution: Trigger the E2E Test job.
- Monitoring Results: Jenkins provides an overview of the job's results, including the last successful run, failures, and test durations.
- Post-Test Actions: Upon completion of the E2E Test, it's crucial to examine the evaluationResults database within ArangoDB.
- Navigate to the transactions collection within the database.
- Confirm the presence of a transaction record, which signifies a successful end-to-end test execution.
To resolve this issue, you would need to:
- Ensure that the
tlscomsecretsecret contains the necessary TLS certificates and keys. - Add the
tlscomsecretto thedevelopmentnamespace, if it's not already present. - After the secret is correctly placed in the namespace, restart the affected pod by deleting the existing pod. Kubernetes will automatically spin up a new pod which should now successfully mount the required volumes, including the TLS secrets, and run as expected.
To address the network access error encountered when deploying containers that require communication with arango.development.svc, follow these steps:
-
Verify that the network policies and service discovery configurations are correctly set up within your cluster to allow connectivity to
arango.development.svc. -
If your deployment is within a Kubernetes environment and you're using network namespaces, consider enabling the Host Network option. This grants the pod access to the host machine's network stack, which can be necessary if the service is only resolvable or accessible in the host's network:
- Navigate to the configuration settings of your pod or service deployment.
- Locate the "Host Network" option, which allows the pod to use the network namespace of the host machine.
- Enable the "Host Network" checkbox to allow direct access to the network services on the host, which can resolve DNS issues if
arango.development.svcis only available on the host's network.
Implementing these steps should help in resolving connectivity issues related to the arango.development.svc hostname not being found, facilitating successful POST requests to the specified endpoints.
If you are experiencing problems with your Kubernetes pods that may be related to environmental variables or configuration issues, such as frequent restarts or failed connections to services like ArangoDB, follow these steps to troubleshoot and resolve the issue:
- Check the Environment Variables in Jenkins:
- Ensure that all required environment variables are properly set in Jenkins. These variables might include database connection strings, service endpoints, credentials, or other configuration parameters necessary for your application to run correctly.
- Review the build and deployment scripts in Jenkins to confirm that the environment variables are being injected into the deployment manifests or pod configurations.
- Verify ArangoDB Configuration:
- Double-check the ArangoDB configuration to ensure that it is correct and aligns with the requirements of your application. This may include database URLs, user credentials, database names, and any other related configuration details.
- If you are using Kubernetes ConfigMaps or Secrets to manage the ArangoDB configuration, make sure they are correctly defined and mounted into your pods.
- Monitor Pod Status and Logs:
- Observe the status of the pods through the Kubernetes dashboard or using
kubectl get podscommand. Take note of any pods that are in a CrashLoopBackOff state or that are frequently restarting. - Use
kubectl describe pod <pod-name>to get more details about the pod's state and events that might indicate what is causing the restarts. - Examine the logs of the restarting pods using
kubectl logs <pod-name>to look for any error messages or stack traces that could point to a configuration problem or a missing environment variable.
- Address Possible Configuration Drifts:
- In a dynamic environment like Kubernetes, configuration drifts can occur where the running state of the system deviates from the defined state. Ensure that all deployments, StatefulSets, or other controller resources match the intended configuration.
- Update and Restart Pods if Necessary:
- Once any necessary changes have been made to the environment variables or ArangoDB configuration, update the relevant Kubernetes resources.
- You can restart the affected pods to apply the changes by deleting them and letting the ReplicaSet create new ones with the correct configuration.
By carefully checking your Jenkins environment variables and ensuring the ArangoDB configuration is correct, you can resolve issues leading to pod instability and ensure that your services run smoothly in the Kubernetes environment.
When encountering authentication errors during a Jenkins build process that involve Kubernetes plugin issues or Docker image push failures, follow these troubleshooting steps:
- Kubernetes Plugin Error:
- The error message suggests a
NullPointerException, which is often due to missing or improperly configured credentials within Jenkins. This could be an issue with the Kubernetes plugin configuration where a required value is not being set, resulting in a null being passed where an object is expected. - Review your Jenkins job configurations and ensure that all the Kubernetes-related credentials are correctly set and that the plugin is properly configured.
- If you are using credential substitution (injected variables), ensure that the substitutions are correctly configured. If necessary, as per the provided instruction, deselect all credential substitutions to see if this resolves the error. This can help isolate the issue by reverting to default or hardcoded credentials, which can then be individually reinstated to identify the problematic substitution.
- Docker Image Push Error:
- The failure to push a Docker image to a registry, with an error indicating "unable to retrieve auth token," typically points to incorrect credentials being used for Docker registry authentication.
- Confirm that the Docker registry credentials set in Jenkins are accurate. You may need to update the username and password or use an access token if the registry requires it.
- Ensure that the credentials are correctly mapped in the Jenkins job and that any credential substitution is correctly applied.
- Rerun the Jenkins Jobs:
- After making the necessary corrections, rerun the Jenkins jobs to confirm that the issue is resolved.
- Monitor the build output for any further authentication-related errors and address them as needed.
- Additional Steps:
- If the issue persists, consider regenerating or re-obtaining the necessary credentials and updating them in Jenkins.
- Check the Jenkins system logs and the specific job's console output for more detailed error messages that can provide additional context for the failure.
By following these steps, you can address the authentication issues that are causing the Jenkins build process to fail, ensuring a successful connection to Kubernetes and Docker registry services.
If for some reason the jenkins agent starts up on your kubernetes instance and then termnates and restarts. You might need to change to frmpullsecret with namespacecicdto .dockerconfigjson data instead of the AWS data.
Or run this command to retrieve your token:
aws ecr get-login-password --region <--Region of the ECR>
The above comand will print out a token copy that token which is used as your password.
Docker Config JSON: Understanding the auth Field
The auth field in the .dockerconfigjson file is a base64-encoded string that combines your Docker registry username and password in the format username:password. Here's how you can construct it:
Steps to Construct the auth Field
-
Combine the Username and Password
Format the string as
username:password. For example, your username isAWSand your password isyourpassword. -
Base64 Encode the String
You can use a command-line tool like base64 or an online base64 encoder to encode the string.
Using a command-line tool:
echo -n 'AWS:yourpassword' | base64This will produce a base64-encoded string, which you then place in the auth field.
Here is an example of what the .dockerconfigjson data in the secret file might look like after encoding:
{"auths":{"registory":{"username":"AWS","password":"token","email":"no@email.local","auth":"QVdTOnlvdXJwYXNzd29yZA=="}}}
Please see the example below:
apiVersion: v1
kind: Secret
metadata:
name: frmpullsecret
namespace: cicd
type: kubernetes.io/dockerconfigjson
data:
.dockerconfigjson: >-With the Helm charts and Jenkins jobs successfully executed, your Tazama (Real-time Monitoring System) should now be operational within your Kubernetes cluster. This comprehensive setup leverages the robust capabilities of Kubernetes orchestrated by Jenkins automation to ensure a seamless deployment process.
As you navigate through the use and potential customization of the Tazama system, keep in mind the importance of maintaining the configurations as documented in this guide. Regularly update your environment variables, manage your credentials securely, and ensure that the pipeline scripts are kept up-to-date with any changes in your infrastructure or workflows.
Should you encounter any issues or have questions regarding the installation and configuration of the Tazama system, support is readily available. You can reach out via email or join the dedicated Slack workspace for collaborative troubleshooting and community support.
For direct assistance:
- Slack: Tazama.slack.com
Joining the FRMS CoE workspace on Slack will connect you with a community of experts and peers who can offer insights and help you leverage the full potential of your FRMS system. Always ensure that you are working within secure communication channels and handling sensitive information with care.