DEVNOTES.adoc

Developer Notes

Table of Contents

• 1. [DONE] External Config via JSON File
• 2. [DONE] Runtime Configuration
• 3. [DONE] HttpClient vs OkHttp
• 4. Tulip Documentation
• 5. [DROPPED] Pkl Config Support
• 6. GraalVM native application
• 7. Docker Support
• 8. Tulip Runtime Library
• 9. [DONE] Java Benchmark Support
• 10. Reimplement JSON input (config.json)
• 11. Reimplement JSON output
• 12. [DONE] Micrometer Support
• 13. [DONE] Amper Support
• 14. [DONE] Remove Glowroot support
• 15. [DONE] Add user_actions and user_class to config.json
• 16. [DONE] Add user_params to config.json
• 17. [DONE] Re-write tulip_user.py in Kotlin Script
• 18. Re-write json_print_asciidoc.py in Kotlin Script
• 19. Benchmark Summary
• 20. Performance Requirements
• 21. HdrHistogram
• 22. Summary Statistics
• 23. JCTools Concurrent Queues
• 24. kscript

1. [DONE] External Config via JSON File

config.json

{
    "json_filename": "json_results.txt",
    "contexts": [
        {
            "name": "Scenario-1",
            "num_users": 16,
            "num_threads": 2
        },
        {
            "name": "Scenario-2",
            "num_users": 32,
            "num_threads": 4
        }
    ],
    "benchmarks": [
        {
            "name": "Test0 (Initialize)",
            "enabled": true,
            "time": {
                "startup_duration": 0,
                "warmup_duration": 0,
                "main_duration": 0,
                "main_duration_repeat_count": 1
            },
            "throughput_rate": 0.0,
            "work_in_progress": 1,
            "actions": [
                {
                    "id": 0
                },
                {
                    "id": 7
                }
            ]
        },
        {
            "name": "Test1 (Throughput Test - Max)",
            "enabled": false,
            "time": {
                "startup_duration": 60,
                "warmup_duration": 60,
                "main_duration": 60,
                "main_duration_repeat_count": 1
            },
            "throughput_rate": 0.0,
            "work_in_progress": -1,
            "actions": [
                {
                    "id": 8
                }
            ]
        },
        {
            "name": "Test2 (Throughput Test - Fixed)",
            "enabled": true,
            "time": {
                "startup_duration": 15,
                "warmup_duration": 15,
                "main_duration": 60,
                "main_duration_repeat_count": 4
            },
            "throughput_rate": 100.0,
            "work_in_progress": 0,
            "actions": [
                {
                    "id": 1,
                    "weight": 25
                },
                {
                    "id": 2,
                    "weight": 75
                }
            ]
        }
    ]
}

2. [DONE] Runtime Configuration

Add --config parameter to specify which config.json file to use.

• https://github.com/Kotlin/kotlinx-cli

3. [DONE] HttpClient vs OkHttp

1. Use HttpClient from Java 21, and remove support for OkHttp

   • https://www.baeldung.com/java-httpclient-connection-management

2. Remove unused and optional JAR dependencies

   • http4k

   • …​.

4. Tulip Documentation

Create a user guide for Tulip with Antora

• https://twitter.com/antoraproject

• https://www.baeldung.com/java-httpclient-connection-management

  • -Djdk.httpclient.connectionPoolSize=1

  • -Djdk.httpclient.keepalive.timeout=2

• https://openjdk.org/groups/net/httpclient/recipes.html

5. [DROPPED] Pkl Config Support

Write a config.pkl file to generate config.json

• https://pkl-lang.org/blog/introducing-pkl.html

6. GraalVM native application

Build a native (exe) using GraalVM of a Tulip benchmark application

• https://www.graalvm.org/

$ ./gradlew nativeCompile

$ ./build/native/nativeCompile/tulip -c ./config.json

• https://graalvm.github.io/native-build-tools/latest/gradle-plugin.html

7. Docker Support

Create a Docker container of a Tulip benchmark application using Docker Compose

• https://docs.docker.com/compose/

8. Tulip Runtime Library

Create a Maven Central hosted tulip-core.jar runtime library that can be imported by benchmark applications

• https://jenkov.com/tutorials/maven/publish-to-central-maven-repository.html

<dependency>
    <groupId>io.github.wfouche</groupId>
    <artifactId>tulip-core</artifactId>
    <version>0.8.1</version>
</dependency>

9. [DONE] Java Benchmark Support

Allow benchmark user class to be written in Java or other JVM compatible languages. Add support for:

• Kotlin

• Java

10. Reimplement JSON input (config.json)

Use Kotlin Serialization instead of GSON:

• Support JSON5 format

• Support GraalVM

11. Reimplement JSON output

Re-implement how the json_results.txt file is created. Only use a hierarchy of data classes and GSON to create the JSON output, or kotlinx

• https://www.baeldung.com/kotlin/data-class-json-serialize-default-values

import kotlinx.serialization.Serializable
@Serializable
data class Car(val type: String, @EncodeDefault val color: String = "Blue")

val car = Car("Ford")
val jsonString = Json.encodeToString(car)
assertEquals("{\"type\":\"Ford\",\"color\":\"Blue\"}", jsonString)

• https://discuss.kotlinlang.org/t/request-for-built-in-tostring-json-true-for-data-classes-for-json-output/6720/2

12. [DONE] Micrometer Support

Instrument the benchmark application using Micrometer (http://micrometer.io) and support performance data extraction via Prometheus and Grafana.

• https://grafana.com/docs/grafana/latest/getting-started/get-started-grafana-prometheus/

• $ docker run --name prometheus -d -p 127.0.0.1:9090:9090 prom/prometheus

• https://wfouche.grafana.net/a/cloud-home-app/onboarding-flow/start

• https://grafana.com/docs/grafana-cloud/send-data/metrics/metrics-prometheus/prometheus-config-examples/pivotal-software-inc-micrometer/

Also see docker compose scripts at:

• https://github.com/docker/awesome-compose/blob/master/prometheus-grafana/README.md

13. [DONE] Amper Support

Change the Tulip project to build using Amper/Gradle.

• https://blog.jetbrains.com/blog/2023/11/09/amper-improving-the-build-tooling-user-experience/

14. [DONE] Remove Glowroot support

Remove folder tulip/runtime/glowroot.

See https://glowroot.org/ and https://github.com/glowroot/glowroot

15. [DONE] Add user_actions and user_class to config.json

{
    "user_class": "user.UserHttp",
    "user_actions": {
        "0": "start",
        "1": "DELAY-6ms",
        "2": "DELAY-14ms",
        "3": "REST-posts",
        "4": "REST-comments",
        "5": "REST-albums",
        "6": "REST-photos",
        "7": "REST-todos",
        "8": "login",
        "99": "stop"
    }
}

16. [DONE] Add user_params to config.json

{
    ....
    "user_params": {
        "url": "https://jsonplaceholder.typicode.com",
        ....
    },
    ....
}

17. [DONE] Re-write tulip_user.py in Kotlin Script

• tulip_user.py

• tulip_user.kts

18. Re-write json_print_asciidoc.py in Kotlin Script

• json_print_asciidoc.py

• json_print_asciidoc.kts

19. Benchmark Summary

Display a summary of benchmark results at the end of the benchmark:

Benchmark1

• Name

• Average TPS

• Average response time

• 90th percentile

• Max response time

• Num-failed nnn (%xyz)

Benchmark2

• Name

• Average TPS

• Average response time

• 90th percentile

• Max response time

• Num-failed nnn (%xyz)

Benchmark…​

• Name

• Average TPS

• Average response time

• 90th percentile

• Max response time

• Num-failed nnn (%xyz)

20. Performance Requirements

{
    "performance_requirements": {
        "avg-tps": "12 tps",
        "avg-tps-variance": "10 percent",
        ...
    }
}

21. HdrHistogram

Use HdrHistogram to replace Tulip’s own log-linear quantization logic.

HdrHistogram is a standard used by several load testing tools.

• implementation("org.hdrhistogram:HdrHistogram:2.2.2")

• https://github.com/Hyperfoil/Hyperfoil uses HdrHistogram

///usr/bin/env jbang "$0" "$@" ; exit $?
//DEPS org.hdrhistogram:HdrHistogram:2.2.2

import org.HdrHistogram.Histogram;

import java.util.concurrent.ThreadLocalRandom;

public class test_hdrhistogram {
    public static void main(String[] args) {
        //Histogram histogram = new Histogram(3600*1000*1000L, 3);
        Histogram histogram = new Histogram(3);

        // 6 ms delay (average) with 25% of values
        for (int i=0; i != 250000; i++) {
            histogram.recordValue(ThreadLocalRandom.current().nextLong(12 + 1));
        }

        // 14 ms delay (average) with 75% of values
        for (int i=0; i != 750000; i++) {
            histogram.recordValue(ThreadLocalRandom.current().nextLong(28 + 1));
        }
        // histogram.getMean() = 12.0

        System.out.println(histogram.getTotalCount());
        histogram.outputPercentileDistribution(System.out,1.0);
        System.out.println(histogram.getMean());
        System.out.println(histogram.getStdDeviation());
        System.out.println(histogram.getMaxValue());
        System.out.println(histogram.getValueAtPercentile(50.0));
        System.out.println(histogram.getValueAtPercentile(90.0));
        System.out.println(histogram.getValueAtPercentile(95.0));
        System.out.println(histogram.getValueAtPercentile(99.0));
        System.out.println(histogram.getValueAtPercentile(99.9));
    }
}

22. Summary Statistics

PHASE    METRIC  THROUGHPUT    ACTIONS  MEAN      STD_DEV  p50       p90       p99       p99.9     MAX        SUCCESS   FAILED
example  test    29,41 req/s         1  17,37 ms     0 ms  17,43 ms  17,43 ms  17,43 ms  17,43 ms  17,43 ms         1        0

23. JCTools Concurrent Queues

Replace queues in JC queues.

• https://github.com/JCTools/JCTools

24. kscript

Remove kscript, kotlin 1.9.24 and use jbang