- c-logging
- Authentication integrated in OpenShift and Kubernetes for all exposed dashboards.
- Client authentication supported.
- Save logs and make them accessible for the client via Kibana labeling the namespaces with :
c-logging/enabled: "true" - We do NOT share env logs with the client (delegated in
d-logging) - Elastic logs retention is set to 7 days by default.
- Use standard policy to backup PVs (90 days of retention)
- Capability to add kibana dashboards.
- Multizone support with 3+ Elasticsearch instances (each one in different zones).
- Capability to send logs to an external SIEM following this guide
The diagram show the relationships between each component of the logging stack. It applies both Kubernetes and OpenShift.
The differences between Kubernetes and Openshift are:
- Ingress access to dashboards:
- Kubernetes: the solution use
d-ingresscomponent. - OpensShift:
openshift-default-routerprovides the access to external users.
- Kubernetes: the solution use
- Authentication to dashboards:
- Kubernetes:
- Vanilla/IKS:
d-authis used to authenticate ISCP users and also client users (needs client IdP configured ind-auth)
- Vanilla/IKS:
- OpensShift:
- ROKS: the authentication will be provided by IBM Cloud IAM for all kind of users (ISCP and client)
- Managed OpenShift (onpremise):
d-authfor OCP will be used to authenticate ISCP users (ISCP-SSO) and the clients will the client IdP configured in oauth (e.g. LDAP)
- Kubernetes:
Show full directory structure
├── k8s
│ ├── default --> Default folder for OpenShift deployments without oauth
│ │ ├── crds --> Mandatory CRDs to deploy c-logging stack
│ │ ├── c-logging
| │ │ ├── backup --> K8s sample files with 2 ways to execute the backup (PV or S3).
│ │ │ ├── ilm-slm-policies --> This Jobs for ilm and slm replace curator
│ │ │ ├── eck-operator --> ECK Operator
│ │ │ ├── elasticsearch --> Elastisearch component with k8s sample files to deploy in single o multizone clusters.
│ │ │ ├── elasticsearch-exporter --> Elasticsearch-exporter component
│ │ │ ├── filebeat --> Filebeat component
│ │ │ ├── kibana --> K8s files to deploy kibana in `vanilla` or `iks` clusters
| │ | ├── kibana-index-pattern-creation --> Job for index pattern creation
│ │ │ ├── logstash --> Logstash component with k8s sample files to deploy in single o multizone clusters.
│ │ │ ├── monitoring --> All monitoring objects of all component in the stack.
│ │ ├── ns --> Namespace
│ │ ├── ns-config --> Docker pull secrets
│ │ └── storage --> K8s sample files with PV and PVCs of all component in the stack.
│ ├── oauth --> Folder for OpenShift deployments with oauth
│ │ ├── crds --> Mandatory CRDs to deploy c-logging stack
│ │ ├── c-logging
| │ │ ├── backup --> K8s sample files with 2 ways to execute the backup (PV or S3).
│ │ │ ├── ilm-slm-policies --> This Jobs for ilm and slm replace curator
│ │ │ ├── eck-operator --> ECK Operator
│ │ │ ├── elasticsearch --> Elastisearch component with k8s sample files to deploy in single o multizone clusters.
│ │ │ ├── elasticsearch-exporter --> Elasticsearch-exporter component
│ │ │ ├── filebeat --> Filebeat component
│ │ │ ├── kibana --> K8s files to deploy kibana in `vanilla` or `iks` clusters
| │ | ├── kibana-index-pattern-creation --> Job for index pattern creation
│ │ │ ├── logstash --> Logstash component with k8s sample files to deploy in single o multizone clusters.
│ │ │ └── monitoring --> All monitoring objects of all component in the stack.
│ │ ├── ns --> Namespace
│ │ ├── ns-config --> Docker pull secrets
│ │ └── storage --> K8s sample files with PV and PVCs of all component in the stack.
│
│
|
├── ocp
│ ├── default --> Default folder for OpenShift deployments without oauth
│ │ ├── crds --> Mandatory CRDs to deploy c-logging stack
│ │ ├── c-logging
| │ │ ├── backup --> K8s sample files with 2 ways to execute the backup (PV or S3).
│ │ │ ├── ilm-slm-policies --> This Jobs for ilm and slm replace curator
│ │ │ ├── eck-operator --> ECK Operator
│ │ │ ├── elasticsearch --> Elastisearch component with k8s sample files to deploy in single o multizone clusters.
│ │ │ ├── elasticsearch-exporter --> Elasticsearch-exporter component
│ │ │ ├── filebeat --> Filebeat component
│ │ │ ├── kibana --> K8s files to deploy kibana in `ocp` or `roks` clusters
| │ | ├── kibana-index-pattern-creation --> Job for index pattern creation
│ │ │ ├── logstash --> Logstash component with k8s sample files to deploy in single o multizone clusters.
│ │ │ ├── monitoring --> All monitoring objects of all component in the stack.
│ │ ├── ns --> Namespace
│ │ ├── ns-config --> Docker pull secrets
│ │ └── storage --> K8s sample files with PV and PVCs of all component in the stack.
│ ├── oauth --> Folder for OpenShift deployments with oauth
│ │ ├── crds --> Mandatory CRDs to deploy c-logging stack
│ │ ├── c-logging
| │ │ ├── backup --> K8s sample files with 2 ways to execute the backup (PV or S3).
│ │ │ ├── ilm-slm-policies --> This Jobs for ilm and slm replace curator
│ │ │ ├── eck-operator --> ECK Operator
│ │ │ ├── elasticsearch --> Elastisearch component with k8s sample files to deploy in single o multizone clusters.
│ │ │ ├── elasticsearch-exporter --> Elasticsearch-exporter component
│ │ │ ├── filebeat --> Filebeat component
│ │ │ ├── kibana --> K8s files to deploy kibana in `ocp` or `roks` clusters
| │ | ├── kibana-index-pattern-creation --> Job for index pattern creation
│ │ │ ├── logstash --> Logstash component with k8s sample files to deploy in single o multizone clusters.
│ │ │ ├── monitoring --> All monitoring objects of all component in the stack.
│ │ ├── ns --> Namespace
│ │ ├── ns-config --> Docker pull secrets
│ │ └── storage --> K8s sample files with PV and PVCs of all component in the stack.
|
├── doc
│ ├── files --> Source documentation files
│ ├── img --> Images to be used in README
└── misc
├── img --> ArgoCD files, not used on this release, but needed in the future
└── scripts --> Scripts to generate k8s yaml files from Helm charts used in this repo and Chart values- eck-operator - (Elastic Cloud on Kubernetes simplifies setup, upgrades, snapshots, scaling, high availability, security, and more for running Elasticsearch and Kibana in Kubernetes.)
- elasticsearch - (Elasticsearch is a distributed, RESTful search and analytics engine capable of addressing a growing number of use cases.)
- elasticsearch-exporter - (Prometheus exporter for various metrics about ElasticSearch, written in Go.)
- kibana - (Kibana is a free and open user interface that lets you visualize your Elasticsearch data and navigate the Elastic Stack)
- logstash - (Logstash is a free and open server-side data processing pipeline that ingests data from a multitude of sources, transforms it, and then sends it )
- logstash-exporter - (Prometheus exporter for the metrics available in Logstash).
- filebeat - (Filebeat helps you keep the simple things simple by offering a lightweight way to forward and centralize logs and files)
- busybox - (BusyBox combines tiny versions of many common UNIX utilities into a single small executable)
| Component | CPU req | CPU Limit | Mem req. | Mem limit |
|---|---|---|---|---|
| c-logging-elasticsearch-es-node | 500m | 1 | 4Gi | 6Gi |
| c-logging-elasticsearch-exporter | 50m | 100m | 64Mi | 128Mi |
| c-logging-filebeat-beat-filebeat | 100m | 200m | 100Mi | 250Mi |
| c-logging-filebeat-beat-exporter | 50m | 100m | 32Mi | 32Mi |
| c-logging-kibana-kb | 500m | 1 | 1Gi | 2Gi |
| c-logging-logstash | 500m | 1 | 1Gi | 2Gi |
| c-logging-operator | 100m | 1 | 150Mi | 512Mi |
- elastic-cloud-2.12.0 - Overview of elastic cloud on kubernetes
- docker-images-8.12.2 - Docker images we used to the upgrade for the new images.
This unit has been tested in:
| Component | Platform | Tested versions |
|:------------:|:--------:|:------------------------------------------:|
| c-logging | IKS | <=1.30.x :heavy_check_mark: |
| c-logging | IKS MZ | <=1.30.x :heavy_check_mark: |
| c-logging | Vanilla | <=1.30.x :heavy_check_mark: |
| c-logging | ROKS | <=4.17 :heavy_check_mark: |
| c-logging | ROKS | <=4.17 :heavy_check_mark: |
- For Openshift onpremise, all dashboards has enabled by default
d-authauthentication through ISCP SSO and other IDPs configured ind-auth - For IKS / Kubernetes vanilla, all dashboards has enabled by default
d-authauthentication through ISCP SSO and other IDPs configured ind-auth. We need an Oauthserver (Dex) that it was developed to interact with differents idp "connectors" and serve to apps to connect via proxy to those idps. - For ROKS, the authentication will be manage by IBM Cloud IAM
About the tool enabled by default d-auth is explained here in the new documentation for add the oauth proxy for Kibana.
xpack.security.enabled must be disable in Elasticsearch, in Kibana is complemented in >8.0
| Provider | Platform | Authentication method |
|---|---|---|
| IBM Cloud | Kubernetes self-managed (IKS) | - d-auth* (ISCP-SSO, client LDAP...) |
| IBM Cloud | OpenShift self-managed (ROKS) | - IAM IBM Cloud* |
| vSphere | Kubernetes managed | - d-auth* (ISCP-SSO, client LDAP...) |
| vSphere | OpenShift managed | - oauth-proxy (ISCP-SSO, client LDAP...) |
* Default authentication used by c-logging
- Kibana UI through
d-ingress/Default OpenShift Router
-
K8s:
- Kibana through
d-ingresswith a NGINX Ingress Controller.
c-logging-kibana.<targetCluster>.{{ global.mt.instanceSubDomain }}.{{ global.mt.instanceDomain }}where
{{ global.mt.instanceSubDomain }}{{ global.mt.instanceDomain }}ispro.eu.d-iscp.netExample:c-logging-kibana.oauth.ibc-pro-tst-0001-k01.pro.eu.d-iscp.net - Kibana through
-
Openshift:
- Kibana through routes, components of
default OpenShift Route.
c-logging-kibana.oauth.apps.< subDomain of default OpenShift Router >To find the correct subdomain of default Openshift Route, we can check in this way:
$ oc get ingresscontroller -n openshift-ingress-operator default -o yaml | grep -i "domain" domain: apps.cus01-pro-001-k01.example.ioExample:
c-logging-kibana.oauth.apps.cus01-pro-001-k01.example.io - Kibana through routes, components of
- Standard solution based on namespaces
- Custom dashboards:
| Dashboard name | Description |
|---|---|
| c-logging-elasticsearch-dashboard | Dashboard to view elasticsearch health |
| c-logging-logstash-dashboard | Logstash dashboard |
| c-logging-filebeat-dashboard | Filebeat dashboard |
-
Backups are made by elastic internally. Logging squad configure elastic with two job that configure the delete of index and the backup of those
-
c-logging-ilm runs to configure the delete of index that have more than a week (7d)
-
c-logging-slm runs to configure a nigthly snapshot that save index with a desire retention from the cicd variable 7d
-
It's important once you install or upgrade the logging stack to ensure that both job are completed succssefully
Also with S3 variables you can configure to save the backups on S3 bucket
You can check if the nightly-snapshots job are correctly configured on kibana by checking:
Follow this guide to verify all the prerequisites before the installation of c-logging in any supported container platforms.
For Kubernetes, follow this this guide to check the procedures to install/update c-logging.
For Openshift, follow this this guide to check the procedures to install/update c-logging.
Follow this guide to check the common operational procedures.
Send logs to an external SIEM following this guide.