Skip to content

Latest commit

 

History

History
executable file
·
279 lines (191 loc) · 15.8 KB

README.md

File metadata and controls

executable file
·
279 lines (191 loc) · 15.8 KB
Raw

Monitor the OracleSOASuite instance using Prometheus and Grafana

Using the WebLogic Monitoring Exporter you can scrape runtime information from a running OracleSOASuite instance and monitor them using Prometheus and Grafana.

Prerequisites

  • Have Docker and a Kubernetes cluster running and have ${KUBERNETES_CLI:-kubectl} installed and configured.
  • Have Helm installed.
  • Before installing kube-prometheus-stack (Prometheus, Grafana and Alertmanager), refer link and cleanup if any older CRDs for monitoring services exists in your Kubernetes cluster. Note: Make sure no existing monitoring services is running in the Kubernetes cluster before cleanup. If you do not want to cleanup monitoring services CRDs, refer link for upgrading the CRDs.
  • An OracleSOASuite domain deployed by weblogic-operator is running in the Kubernetes cluster.

Set up monitoring for OracleSOASuite domain

Set up the WebLogic Monitoring Exporter that will collect WebLogic Server metrics and monitor OracleSOASuite domain.

Note: Either of the following methods can be used to set up monitoring for OracleSOASuite domain. Using setup-monitoring.sh does the set up in an automated way.

  1. Set up manually
  2. Set up using setup-monitoring.sh

Set up manually

Deploy Prometheus and Grafana

Refer to the compatibility matrix of Kube Prometheus and clone the release version of the kube-prometheus repository according to the Kubernetes version of your cluster.

  1. Clone the kube-prometheus repository:

    $ git clone https://github.com/coreos/kube-prometheus.git

  2. Change to folder kube-prometheus and enter the following commands to create the namespace and CRDs, and then wait for their availability before creating the remaining resources:

    $ cd kube-prometheus
$ ${KUBERNETES_CLI:-kubectl} create -f manifests/setup
$ until ${KUBERNETES_CLI:-kubectl} get servicemonitors --all-namespaces ; do date; sleep 1; echo ""; done
$ ${KUBERNETES_CLI:-kubectl} create -f manifests/

  3. kube-prometheus requires all nodes in the Kubernetes cluster to be labeled with kubernetes.io/os=linux. If any node is not labeled with this, then you need to label it using the following command:

    $ ${KUBERNETES_CLI:-kubectl} label nodes --all kubernetes.io/os=linux

  4. Enter the following commands to provide external access for Grafana, Prometheus, and Alertmanager:

    $ ${KUBERNETES_CLI:-kubectl} patch svc grafana -n monitoring --type=json -p '[{"op": "replace", "path": "/spec/type", "value": "NodePort" },{"op": "replace", "path": "/spec/ports/0/nodePort", "value": 32100 }]'

$ ${KUBERNETES_CLI:-kubectl} patch svc prometheus-k8s -n monitoring --type=json -p '[{"op": "replace", "path": "/spec/type", "value": "NodePort" },{"op": "replace", "path": "/spec/ports/0/nodePort", "value": 32101 }]'

$ ${KUBERNETES_CLI:-kubectl} patch svc alertmanager-main -n monitoring --type=json -p '[{"op": "replace", "path": "/spec/type", "value": "NodePort" },{"op": "replace", "path": "/spec/ports/0/nodePort", "value": 32102 }]'

    Note:

    • 32100 is the external port for Grafana
    • 32101 is the external port for Prometheus
    • 32102 is the external port for Alertmanager

Generate the WebLogic Monitoring Exporter Deployment Package

The wls-exporter.war package need to be updated and created for each listening ports (Administration Server and Managed Servers) in the domain. Set the below environment values based on your environment and run the script get-wls-exporter.sh to generate the required WAR files at ${WORKDIR}/monitoring-service/scripts/wls-exporter-deploy:

  • adminServerPort
  • wlsMonitoringExporterTosoaCluster
  • soaManagedServerPort
  • wlsMonitoringExporterToosbCluster
  • osbManagedServerPort

For example:

$ cd ${WORKDIR}/monitoring-service/scripts
$ export adminServerPort=7001 
$ export wlsMonitoringExporterTosoaCluster=true
$ export soaManagedServerPort=8001
$ export wlsMonitoringExporterToosbCluster=true
$ export osbManagedServerPort=9001
$ sh get-wls-exporter.sh

Verify whether the required WAR files are generated at ${WORKDIR}/monitoring-service/scripts/wls-exporter-deploy.

$ ls ${WORKDIR}/monitoring-service/scripts/wls-exporter-deploy

Deploy the WebLogic Monitoring Exporter into the OracleSOASuite domain

Follow these steps to copy and deploy the WebLogic Monitoring Exporter WAR files into the OracleSOASuite Domain.

Note: Replace the <xxxx> with appropriate values based on your environment:

$ cd ${WORKDIR}/monitoring-service/scripts
$ ${KUBERNETES_CLI:-kubectl} cp wls-exporter-deploy <namespace>/<admin_pod_name>:/u01/oracle
$ ${KUBERNETES_CLI:-kubectl} cp deploy-weblogic-monitoring-exporter.py <namespace>/<admin_pod_name>:/u01/oracle/wls-exporter-deploy
$ ${KUBERNETES_CLI:-kubectl} exec -it -n <namespace> <admin_pod_name> -- /u01/oracle/oracle_common/common/bin/wlst.sh /u01/oracle/wls-exporter-deploy/deploy-weblogic-monitoring-exporter.py \
-domainName <domainUID> -adminServerName <adminServerName> -adminURL <adminURL> \
-soaClusterName <soaClusterName> -wlsMonitoringExporterTosoaCluster <wlsMonitoringExporterTosoaCluster> \
-osbClusterName <osbClusterName> -wlsMonitoringExporterToosbCluster <wlsMonitoringExporterToosbCluster> \
-username <username> -password <password>

For example:

$ cd ${WORKDIR}/monitoring-service/scripts
$ ${KUBERNETES_CLI:-kubectl} cp wls-exporter-deploy soans/soainfra-adminserver:/u01/oracle
$ ${KUBERNETES_CLI:-kubectl} cp deploy-weblogic-monitoring-exporter.py soans/soainfra-adminserver:/u01/oracle/wls-exporter-deploy
$ ${KUBERNETES_CLI:-kubectl} exec -it -n soans soainfra-adminserver -- /u01/oracle/oracle_common/common/bin/wlst.sh /u01/oracle/wls-exporter-deploy/deploy-weblogic-monitoring-exporter.py \
-domainName soainfra -adminServerName AdminServer -adminURL soainfra-adminserver:7001 \
-soaClusterName soa_cluster -wlsMonitoringExporterTosoaCluster true \
-osbClusterName osb_cluster -wlsMonitoringExporterToosbCluster true \
-username weblogic -password Welcome1

Configure Prometheus Operator

Prometheus enables you to collect metrics from the WebLogic Monitoring Exporter. The Prometheus Operator identifies the targets using service discovery. To get the WebLogic Monitoring Exporter end point discovered as a target, you must create a service monitor pointing to the service.

The service monitor deployment YAML configuration file is available at ${WORKDIR}/monitoring-service/manifests/wls-exporter-ServiceMonitor.yaml.template. Copy the file as wls-exporter-ServiceMonitor.yaml to update with appropriate values as detailed below.

The exporting of metrics from wls-exporter requires basicAuth, so a Kubernetes Secret is created with the user name and password that are base64 encoded. This Secret is used in the ServiceMonitor deployment. The wls-exporter-ServiceMonitor.yaml has namespace as soans and has basicAuth with credentials as username: %USERNAME% and password: %PASSWORD%. Update %USERNAME% and %PASSWORD% in base64 encoded and all occurences of soans based on your environment.

Use the following example for base64 encoded:

$ echo -n "Welcome1" | base64
V2VsY29tZTE=

You need to add RoleBinding and Role for the namespace (soans) under which the WebLogic Servers pods are running in the Kubernetes cluster. These are required for Prometheus to access the endpoints provided by the WebLogic Monitoring Exporters. The YAML configuration files for soans namespace are provided in "${WORKDIR}/monitoring-service/manifests/".

If you are using namespace other than soans, update the namespace details in prometheus-roleBinding-domain-namespace.yaml and prometheus-roleSpecific-domain-namespace.yaml.

Perform the below steps for enabling Prometheus to collect the metrics from the WebLogic Monitoring Exporter:

$ cd ${WORKDIR}/monitoring-service/manifests
$ ${KUBERNETES_CLI:-kubectl} apply -f .

Verify the service discovery of WebLogic Monitoring Exporter

After the deployment of the service monitor, Prometheus should be able to discover wls-exporter and collect the metrics.

  1. Access the Prometheus dashboard at http://mycompany.com:32101/

  2. Navigate to Status to see the Service Discovery details.

  3. Verify that wls-exporter is listed in the discovered Services.

Deploy Grafana Dashboard

You can access the Grafana dashboard at http://mycompany.com:32100/.

  1. Log in to Grafana dashboard with username: admin and password: admin`.

  2. Navigate to + (Create) -> Import -> Upload the weblogic-server-dashboard-import.json file (provided at ${WORKDIR}/monitoring-service/config/weblogic-server-dashboard-import.json).

Set up using setup-monitoring.sh

Alternatively, you can run the helper script setup-monitoring.sh available at ${WORKDIR}/monitoring-service to setup the monitoring for OracleSOASuite domain.

This script creates kube-prometheus-stack(Prometheus, Grafana and Alertmanager), WebLogic Monitoring Exporter and imports weblogic-server-dashboard.json into Grafana for WebLogic Server Dashboard.

Prepare to use the setup monitoring script

The sample scripts for setup monitoring for OracleSOASuite domain are available at ${WORKDIR}/monitoring-service.

You must edit monitoring-inputs.yaml(or a copy of it) to provide the details of your domain. Refer to the configuration parameters below to understand the information that you must provide in this file.

Configuration parameters

The following parameters can be provided in the inputs file.

Parameter Description Default
domainUID domainUID of the OracleSOASuite domain. soainfra
domainNamespace Kubernetes namespace of the OracleSOASuite domain. soans
setupKubePrometheusStack Boolean value indicating whether kube-prometheus-stack (Prometheus, Grafana and Alertmanager) to be installed true
additionalParamForKubePrometheusStack The script install's kube-prometheus-stack with service.type as NodePort and values for service.nodePort as per the parameters defined in monitoring-inputs.yaml. Use additionalParamForKubePrometheusStack parameter to further configure with additional parameters as per values.yaml. Sample value to disable NodeExporter, Prometheus-Operator TLS support, Admission webhook support for PrometheusRules resources and custom Grafana image repository is --set nodeExporter.enabled=false --set prometheusOperator.tls.enabled=false --set prometheusOperator.admissionWebhooks.enabled=false --set grafana.image.repository=xxxxxxxxx/grafana/grafana
monitoringNamespace Kubernetes namespace for monitoring setup. monitoring
adminServerName Name of the Administration Server. AdminServer
adminServerPort Port number for the Administration Server inside the Kubernetes cluster. 7001
soaClusterName Name of the soaCluster. soa_cluster
soaManagedServerPort Port number of the managed servers in the soaCluster. 8001
wlsMonitoringExporterTosoaCluster Boolean value indicating whether to deploy WebLogic Monitoring Exporter to soaCluster. false
osbClusterName Name of the osbCluster. osb_cluster
osbManagedServerPort Port number of the managed servers in the osbCluster. 9001
wlsMonitoringExporterToosbCluster Boolean value indicating whether to deploy WebLogic Monitoring Exporter to osbCluster. false
exposeMonitoringNodePort Boolean value indicating if the Monitoring Services (Prometheus, Grafana and Alertmanager) is exposed outside of the Kubernetes cluster. false
prometheusNodePort Port number of the Prometheus outside the Kubernetes cluster. 32101
grafanaNodePort Port number of the Grafana outside the Kubernetes cluster. 32100
alertmanagerNodePort Port number of the Alertmanager outside the Kubernetes cluster. 32102
weblogicCredentialsSecretName Name of the Kubernetes secret which has Administration Server's user name and password. soainfra-domain-credentials

Note that the values specified in the monitoring-inputs.yaml file will be used to install kube-prometheus-stack (Prometheus, Grafana and Alertmanager) and deploying WebLogic Monitoring Exporter into the OracleSOASuite domain. Hence make the domain specific values to be same as that used during domain creation.

Run the setup monitoring script

Update the values in monitoring-inputs.yaml as per your requirement and run the setup-monitoring.sh script, specifying your inputs file:

$ cd ${WORKDIR}/monitoring-service
$ ./setup-monitoring.sh \
  -i monitoring-inputs.yaml

The script will perform the following steps:

  • Helm install prometheus-community/kube-prometheus-stack if setupKubePrometheusStack is set to true.
  • Deploys WebLogic Monitoring Exporter to Administration Server.
  • Deploys WebLogic Monitoring Exporter to soaCluster if wlsMonitoringExporterTosoaCluster is set to true.
  • Deploys WebLogic Monitoring Exporter to osbCluster if wlsMonitoringExporterToosbCluster is set to true.
  • Exposes the Monitoring Services (Prometheus at 32101, Grafana at 32100 and Alertmanager at 32102) outside of the Kubernetes cluster if exposeMonitoringNodePort is set to true.
  • Imports the WebLogic Server Grafana Dashboard if setupKubePrometheusStack is set to true.

Verify the results

The setup monitoring script will report failure if there was any error. However, verify that required resources were created by the script.

Verify the kube-prometheus-stack

To confirm that prometheus-community/kube-prometheus-stack was installed when setupKubePrometheusStack is set to true, run the following command:

$ helm ls -n <monitoringNamespace>

Replace with value for Kubernetes namespace used for monitoring.

Sample output:

$ helm ls -n monitoring
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                           APP VERSION
monitoring      monitoring      1               2023-03-15 10:31:42.44437202 +0000 UTC  deployed        kube-prometheus-stack-45.7.1    v0.63.0
$

Verify the Prometheus, Grafana and Alertmanager setup

When exposeMonitoringNodePort was set to true, verify that monitoring services are accessible outside of the Kubernetes cluster:

  • 32100 is the external port for Grafana and with credentials admin:admin
  • 32101 is the external port for Prometheus
  • 32102 is the external port for Alertmanager

Verify the service discovery of WebLogic Monitoring Exporter

Verify whether prometheus is able to discover wls-exporter and collect the metrics:

  1. Access the Prometheus dashboard at http://mycompany.com:32101/

  2. Navigate to Status to see the Service Discovery details.

  3. Verify that wls-exporter is listed in the discovered services.

Verify the WebLogic Server dashoard

You can access the Grafana dashboard at http://mycompany.com:32100/.

  1. Log in to Grafana dashboard with username: admin and password: admin.

  2. Navigate to "WebLogic Server Dashboard" under General and verify.

Delete the monitoring setup

To delete the monitoring setup created by Run the setup monitoring script, run the below command:

$ cd ${WORKDIR}/monitoring-service
$ ./delete-monitoring.sh \
  -i monitoring-inputs.yaml