High error rate in load tests for update 9

[xlight05]

Description:
$Subect. Seems like this happened right after we bumped the versions for update9

Ref:
https://github.com/ballerina-platform/module-ballerina-http/blob/master/load-tests/h1_h1_passthrough/results/summary.csv#L465

Steps to reproduce:

Affected Versions:

OS, DB, other environment details and versions:

Related Issues (optional):

Suggested Labels (optional):

Suggested Assignees (optional):

[TharmiganK]

This happened after merging this PR: ballerina-platform/module-ballerina-http#1915

In this PR,

• we have added a new feature to support status code response binding
• we bumped the lang version to update 9

[TharmiganK]

I did the following tests locally and observed high memory usage with update 9:

1. Memory usage with update 9 and the new HTTP feature

2. Memory usage with update 8

3. Memory usage with update 8 and the new HTTP feature

4. Memory usage with update 9 without the new HTTP feature

We can conclude that the high memory usage is due to the lang changes with update 9.

Heap dumps can be found here: https://drive.google.com/drive/folders/1MpFqTrUuE2l3R1XwYsMIXMqByyMOcOB-?usp=sharing

Please find the test source details:

1. Run the netty echo backend using the following command:

$ docker run -d -p 8688:8688 ldclakmal/netty-echo-backend:latest

2. Run the following Ballerina HTTP passthrough service(used the JAR file to run the service with -Xmx1024m option):

import ballerina/http;

final http:Client nettyEP = check new("http://localhost:8688", httpVersion = "1.1");

service /passthrough on new http:Listener(9999, httpVersion = "1.1") {
    resource function post .(http:Request clientRequest) returns http:Response|error {
        http:Response response = check nettyEP->forward("/service/EchoService", clientRequest);
        return response;
    }
}

3. Open VisualVM to monitor the memory usage.
4. Use the jmeter to run the load test with the following jmx file: https://gist.github.com/TharmiganK/6098afe243c8032c69bcb73d42ba380c

[TharmiganK]

Adding @warunalakshitha @MaryamZi for more information on this

[TharmiganK]

@HindujaB I was able to reduce the reproducer with this sample:

import ballerina/http;

service /api on new http:Listener(9090) {

    resource function get greeting(http:Request req) returns string {
        return "Hello, World!";
    }
}

If I remove the http:Request from the resource signature then there was no memory issue

When we have a signature parameter like http:Request we built it from the netty request using the following runtime method: ValueCreator.createObjectValue. Link: https://github.com/ballerina-platform/module-ballerina-http/blob/ea36127c62d925e4df13527466219972269e3587/native/src/main/java/io/ballerina/stdlib/http/api/HttpDispatcher.java#L426

[HindujaB]

The issue is happening due to the internally created strands not garbage collected. These strands are created to invoke record default values in a non-Ballerina-dependent environment (from netty threads). I am currently working on a fix for this.

[TharmiganK]

Tested this change in the linked PR with HTTP and now load tests are having 0% error rate:
https://github.com/ballerina-platform/module-ballerina-http/pull/1948/files
(The snowpeak load test is failing due to some other reason)

Also tested the changes locally by monitoring the memory. There is no high memory usuage, the memory usage is similar to update 8:

[github-actions]

This issue is NOT closed with a proper Reason/ label. Make sure to add proper reason label before closing. Please add or leave a comment with the proper reason label now.

      - Reason/EngineeringMistake - The issue occurred due to a mistake made in the past.
      - Reason/Regression - The issue has introduced a regression.
      - Reason/MultipleComponentInteraction - Issue occured due to interactions in multiple components.
      - Reason/Complex - Issue occurred due to complex scenario.
      - Reason/Invalid - Issue is invalid.
      - Reason/Other - None of the above cases.