himss_2022_scm_integration

Table of Contents

• 1. Overview
• 2. Technical Discussion Vectors
• 3. Deploy to OpenShift using Ansible
  • 3.1. Pre-reqs:
  • 3.2. Procedure:
• 4. Smoke Test
• 5. Load Test
• 6. Demo Script
  • 6.1. Grafana Dashboard
• 7. Development
  • 7.1. App Metrics
• 8. Reference

1. Overview

TO-DO

2. Technical Discussion Vectors

1. App Throughput / Scalability

   1. scm-frontend

      (Quarkus / RESTEasy / Camel syncronous blocking service)

      1. Throughput of the app JVM can be adjusted via quarkus.vertx.worker-pool-size setting

      2. Auto-scalability beyond app JVM can be accomplished via Knative Serverless

   2. scm-backend

      (Quarkus / Kafka / Camel async non-blocking event consumer)

      1. Throughput is already very high within the app JVM due to the use of the Quarkus/Vert.x IO thread

      2. Auto-scalability beyond app JVM can be accomplished via Keda

2. Ansible

   1. Ansible is used for automated and repeatable deployment of Application Services to OpenShift

3. Monitoring

   1. App Dashboards

      1. Business Dashboards using Grafana’s Postgresql driver

      2. Operational Dashboards using Grafana’s Prometheus driver

      3. Quarkus micro-profile metrics scraped by Prometheus

   2. Operational Dashboards

      OpenShift provides out-of-the-box dashboards that provide awareness as to the health and state of the cluster.

4. Events in Motion

   1. Use of Red Hat AMQ Streams

   2. Visibility of events via KafDrop

5. Polyglot App Frameworks

   1. Quarkus

   2. Camel

3. Deploy to OpenShift using Ansible

Ansible is included to deploy this application to OpenShift in a repeatable manner.

3.1. Pre-reqs:

1. OpenShift Container Platform version 4.11.x

   Resource requirements needed by the app (doesn’t include resource requirements of Openshift to support itself) is as follows:

   1. RAM: 6 GB

   2. CPU: 6

   3. Storage: 5 PVCs of type RWO (no RWX requirement) and each of size 5 GiB

2. cluster-admin credentials to this OpenShift cluster

3. oc utility (corresponding to version of OCP cluster) installed locally

   All versions of this utility are available at either of the following:

   1. https://access.redhat.com/downloads/content/290

      RHN subscription required

   2. https://mirror.openshift.com/pub/openshift-v4/clients/ocp/?C=M;O=D

      Accessible without a RHN subscription

4. ansible installed locally

   ie: dnf install ansible

5. git installed locally

6. Latest oc (4.11), ansible (2.13), ansible modules (community.okd:2.2.0), and potentially jmespath installed via pip

3.2. Procedure:

1. Using the oc utility that corresponds to the version of OpenShift that you will deploy to, log into the cluster:

   $ oc login <OCP API Server url> -u <cluster-admin userId> -p <passwd>

2. Clone the source code of this project:

   $ git clone https://github.com/redhat-naps-da/himss_2022_scm_integration

3. Change to the ansible directory of this project:

   $ cd ansible

4. Update all.yml

   $ vi ./ansible/playbooks/group_vars/all.yml
   # double check `project_admin` and `fips_enabled_cluster`

5. Deploy to OpenShift:

   $ ansible-playbook playbooks/install.yml

If this is an OpenShift cluster with FIPS enabled,then run the following instead:

$ ansible-playbook --extra-vars '{"fips_enabled_cluster": true}' playbooks/install.yml

1. Deployment should complete in about 10 minutes.

2. Notice the creation of a new OCP namespace where the application resides: user1-himss2022-scm

3. At the completion of the installation, expect to see messages similar to the following:

   PLAY RECAP *******************************************************************************************************************************************************************************************************
   
       localhost                  : ok=93   changed=27   unreachable=0    failed=0    skipped=16   rescued=0    ignored=0

   1. Optional: Uninstall from OpenShift:

      $ ansible-playbook playbooks/uninstall.yml

4. Smoke Test

1. Set environment variable to scm-frontend route exposed by Knative:

   $ SCM_FRONTEND_HOST=$(oc get kservice scm-frontend -n user1-himss2022-scm --template='{{ .status.url }}' | sed 's/"//g')

2. Execute smoke test:

   $ cd scm-frontend
   
   $ curl -v -X POST \
       -F "data=@src/test/himss/good/AM3X-034540-6636-2-19710723.tgz" \
       -F "data=@src/test/himss/good/DETM-NKI7I92LX7P-5221-6-20000907.tgz" \
       $SCM_FRONTEND_HOST/gzippedFiles

3. Expected response should be similar to the following:

   * Connection #0 to host scm-frontend-user1-himss2022-scm.apps.cluster-7c8qw.7c8qw.sandbox1691.opentlc.com left intact
   ALL FILES PERSISTED

4. The log of the scm-backend pod should include statements such as the following:

   19:15:25 INFO  [co.re.hi.Routes] (Camel (camel-1) thread #4 - KafkaConsumer[topic-scm-file]) CSVPayloadProcessor.process() DETM-NKI7I92LX7P-5221-6-20000907.txt :   # of rows = 200
   19:15:25 WARN  [co.re.hi.Routes] (Camel (camel-1) thread #4 - KafkaConsumer[topic-scm-file]) will intentionally delay persist by the following millis: 1,000
   19:15:25 INFO  [co.re.hi.Routes] (Camel (camel-1) thread #3 - KafkaConsumer[topic-scm-file]) CSVPayloadProcessor.process() AM3X-034540-6636-2-19710723.txt :   # of rows = 200
   19:15:25 WARN  [co.re.hi.Routes] (Camel (camel-1) thread #3 - KafkaConsumer[topic-scm-file]) will intentionally delay persist by the following millis: 1,000

5. Load Test

This demo includes a custom load testing tool that generates test data and posts to the scm-frontend service.

TO-DO

6. Demo Script

The purpose of this section is to highlight the major components of the demo and propose an outline toward delivering the demo.

TO-DO

6.1. Grafana Dashboard

1. Grafana Admin credentials:

   1. Local environment: admin / admin

   2. OCP environment:

      $ POD=$(oc get pod -n user1-himss2022-scm | grep "^grafana-deployment" | awk '{print $1}')
      $ oc rsh -c grafana $POD env | grep GF_SECURITY_ADMIN_

2. First draft

   

7. Development

7.1. App Metrics

1. View microprofile generated metrics in JSON format

   $ curl -H"Accept: application/json" localhost:8180/q/metrics/application

   Example result:

       {
       "com.redhat.himss.CSVPayloadProcessor.csvProcessed": 5,
       "com.redhat.himss.CSVPayloadProcessor.csvProcessingTimer": {
           "p99": 25.541904,
           "min": 5.51574,
           "max": 25.541904,
           "mean": 14.2945098,
           "p50": 13.453519,
           "p999": 25.541904,
           "stddev": 6.474391748917466,
           "p95": 25.541904,
           "p98": 25.541904,
           "p75": 14.892466,
           "fiveMinRate": 0.013991112279011392,
           "fifteenMinRate": 0.005240759238950647,
           "meanRate": 0.0746185181247482,
           "count": 5,
           "oneMinRate": 0.034748554162230876,
           "elapsedTime": 71.472549
       },
       "com.redhat.himss.ValidationException.dirtyCSVDataCount": 0

2. View microprofile generated metrics in OpenMetrics format:

   $ curl  localhost:8180/q/metrics/application

   Example result:

   # HELP application_com_redhat_himss_CSVPayloadProcessor_csvProcessed_total How many csv payloads have been processed.
   # TYPE application_com_redhat_himss_CSVPayloadProcessor_csvProcessed_total counter
   application_com_redhat_himss_CSVPayloadProcessor_csvProcessed_total 5.0
   # TYPE application_com_redhat_himss_CSVPayloadProcessor_csvProcessingTimer_rate_per_second gauge
   application_com_redhat_himss_CSVPayloadProcessor_csvProcessingTimer_rate_per_second 0.00520411558035244
   # TYPE application_com_redhat_himss_CSVPayloadProcessor_csvProcessingTimer_one_min_rate_per_second gauge
   application_com_redhat_himss_CSVPayloadProcessor_csvProcessingTimer_one_min_rate_per_second 1.1553424142673022E-8
   # TYPE application_com_redhat_himss_CSVPayloadProcessor_csvProcessingTimer_five_min_rate_per_second gauge
   application_com_redhat_himss_CSVPayloadProcessor_csvProcessingTimer_five_min_rate_per_second 7.082833577637795E-4
   # TYPE application_com_redhat_himss_CSVPayloadProcessor_csvProcessingTimer_fifteen_min_rate_per_second gauge
   application_com_redhat_himss_CSVPayloadProcessor_csvProcessingTimer_fifteen_min_rate_per_second 0.001938708318900107
   # TYPE application_com_redhat_himss_CSVPayloadProcessor_csvProcessingTimer_min_seconds gauge
   application_com_redhat_himss_CSVPayloadProcessor_csvProcessingTimer_min_seconds 0.00551574
   # TYPE application_com_redhat_himss_CSVPayloadProcessor_csvProcessingTimer_max_seconds gauge
   application_com_redhat_himss_CSVPayloadProcessor_csvProcessingTimer_max_seconds 0.025541904
   # TYPE application_com_redhat_himss_CSVPayloadProcessor_csvProcessingTimer_mean_seconds gauge
   application_com_redhat_himss_CSVPayloadProcessor_csvProcessingTimer_mean_seconds 0.0142945098
   # TYPE application_com_redhat_himss_CSVPayloadProcessor_csvProcessingTimer_stddev_seconds gauge
   application_com_redhat_himss_CSVPayloadProcessor_csvProcessingTimer_stddev_seconds 0.006474391748917466
   # HELP application_com_redhat_himss_CSVPayloadProcessor_csvProcessingTimer_seconds A measure of how long it takes to process a CSV file.
   # TYPE application_com_redhat_himss_CSVPayloadProcessor_csvProcessingTimer_seconds summary
   application_com_redhat_himss_CSVPayloadProcessor_csvProcessingTimer_seconds_count 5.0
   application_com_redhat_himss_CSVPayloadProcessor_csvProcessingTimer_seconds_sum 0.071472549
   application_com_redhat_himss_CSVPayloadProcessor_csvProcessingTimer_seconds{quantile="0.5"} 0.013453519
   application_com_redhat_himss_CSVPayloadProcessor_csvProcessingTimer_seconds{quantile="0.75"} 0.014892466
   application_com_redhat_himss_CSVPayloadProcessor_csvProcessingTimer_seconds{quantile="0.95"} 0.025541904
   application_com_redhat_himss_CSVPayloadProcessor_csvProcessingTimer_seconds{quantile="0.98"} 0.025541904
   application_com_redhat_himss_CSVPayloadProcessor_csvProcessingTimer_seconds{quantile="0.99"} 0.025541904
   application_com_redhat_himss_CSVPayloadProcessor_csvProcessingTimer_seconds{quantile="0.999"} 0.025541904
   # HELP application_com_redhat_himss_ValidationException_dirtyCSVDataCount_total How many incidents of dirty CSV data.
   # TYPE application_com_redhat_himss_ValidationException_dirtyCSVDataCount_total counter
   application_com_redhat_himss_ValidationException_dirtyCSVDataCount_total 0.0

8. Reference

1. graphics