Troubleshooting-Performance

This project provides a complete ecosystem for throubleshootings with spring-boot, rest/grpc, hibernate and ehcache application.

You can change de hyperparameters like jvm or minmalloc/jemalloc configs and measure the metrics.

To generate requests, change the K6/Jmeter scripts with yours configs like RPS, and thresholds.

Starting Analysis

1. Enable actuator metrics on application.properties (Already configured):

   management.endpoints.web.exposure.include=*

2. Into Dockerfile comment or discomment memory allocations configs and jvm parameters

3. Start Docker environment:

   make.sh

   This step will build projects and start environment with docker-compose.

4. Start K8S environment (IN PROGRESS):

   configure-k8s.sh

5. Run the K6 scripts

   Run performance test using gRPC protocol

   k6 run k6-post-grpc-script.js

   Run performance test using rest

   k6 run k6-post-rest-script.js

   Run consumer rest calling gRPC server. This tests the keep alive configs from server and client

   k6 run k6-post-proxy-grpc-script.js

6. You can take snapshots of threaddump with this command:

   sh generate_threaddump.sh

7. Alternate JVM parameters to see differences of resources:

   #Java Heap
   -Xms500m # Initial java heap
   -Xmx500m # Max java heap
   -Xss1m # Thread memory allocation
   -XX:+UseContainerSupport # Enable container suports for jvm <= 8
   -XX:InitiatingHeapOccupancyPercent=70 # Alternative for fixed -Xms
   -XX:MaxRAMPercentage=50 # Alternative for fixed -Xmx
   #Non Heap
   -Djdk.nio.maxCachedBufferSize=1000000 # Limit buffer cache used into NIO
   -XX:MaxDirectMemorySize=128m
   -XX:CompressedClassSpaceSize=64m
   -XX:MaxMetaspaceSize=128m
   -XX:+UseStringDeduplication
   # GC
   -XX:+UseG1GC
   -XX:MaxGCPauseMillis=200
   -XX:ConcGCThreads=5
   -XX:ParallelGCThreads=20
   -Xloggc:./gc.log
   -XX:+PrintGCDetails
   -XX:+PrintGCDateStamps
   -XX:+PrintGCTimeStamps
   # Monitoring
   -XX:NativeMemoryTracking=summary
   -XX:+UnlockCommercialFeatures
   -XX:+FlightRecorder
   -XX:StartFlightRecording=duration=60s,filename=profiling.jfr

Services and port numbers

Service	Port Number	Type/Tech
Mysql	3306	DB
Application	8080/9090	Spring boot Rest / gRPC
Proxy-gRPC	8081	Spring boot Rest
Prometheus	9091	DB
Grafana	3030	Metrics Monitor

Urls

Endpoints	URL
Application	http://localhost:8080/stock-quotes
Application - Metrics	http://localhost:8080/actuator/prometheus
Application - ThreadDump	http://localhost:8080/actuator/threaddump
Application - HeapDump	http://localhost:8080/actuator/heapdump
Proxy-gRPC	http://localhost:8081/proxy-stock-quotes
Prometheus	http://localhost:9091
Grafana	http://localhost:3000

Grafana

First login:

user: admin

password: admin

Configure Datasource

Configure the datasource using docker-compose service dns and port(prometheus:9090)

Useful Dashboards:

1. Single Pane of Glass (grafana/custom-dash-spg.json)
2. Micrometer - https://grafana.com/grafana/dashboards/4701
3. Spring boot - https://grafana.com/grafana/dashboards/10280

Datascience and Analytics

Into analytics directory follow instructions of Readme.md.

Tests

Decreasing Tomcat and Hikari thread pool size:

Useful links

• JMC tool

• GC explained

  Important:
  
  Actually, any object larger than 50% of the region size is considered humongous. Those objects are not allocated in the
  young generation space, but instead, they are put directly in the Tenured generation. Such objects can increase the
  pause time of the garbage collector and can increase the risk of triggering the Full GC because of running out of
  continued free space.
  

• JVM Footprint 1

• JVM Footprint 2

• Improving Performance and Footprint

• Troubleshooting Problems With Native (Off-Heap) Memory in Java Application

Useful Commands

• Show jvm params:

  java -XX:+PrintFlagsFinal -version | grep CodeCacheFlush

• Native memory:

  See RES (Resident Set Size) of process:

  top

• JCMD:

  jcmd <PID> VM.native_memory

• Memory Map:

  pmap -xx <PID>