Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up BUFFER increases (minimize reallocs) #12792

Merged
merged 3 commits into from
May 2, 2022

Conversation

ktsaou
Copy link
Member

@ktsaou ktsaou commented May 1, 2022

Optimize BUFFER increases for large web requests.

This PR makes buffer increases about 25% faster for large buffers.

In the following test case, using the BUFFER of the master, it finishes in 65 ms. With this PR it finishes in 50 ms.

Test:

int main(int argc, char **argv) {
    BUFFER *wb = buffer_create(1024);

    const char *string = "012345678901234567890123456789012345678901234567890123456789";
    size_t size_of_string = strlen(string);
    size_t total = 0;

    usec_t start = now_realtime_usec();
    while(total < 40*1024*1024) {
        buffer_sprintf(wb, "%s", string);
        total += size_of_string;
    }
    usec_t end = now_realtime_usec();
    printf("Buffer of %zu bytes finished in %llu usec\n", buffer_strlen(wb), end-start);

    log_allocations();

    buffer_free(wb);
    exit(0);
}

Test results from master:

Buffer of 41943060 bytes finished in 65844 usec
MAIN MEMORY ALLOCATIONS: (0697@daemon/main.c:main): Allocated 40961 KiB (+41944078 B), mmapped 0 KiB (+0 B): : malloc 1 (+1), calloc 1 (+1), realloc 40920 (+40920), strdup 0 (+0), free 0 (+0)

Test results from this PR:

Buffer of 41943060 bytes finished in 50036 usec
MAIN MEMORY ALLOCATIONS: (0697@daemon/main.c:main): Allocated 49152 KiB (+50331702 B), mmapped 0 KiB (+0 B): : malloc 1 (+1), calloc 1 (+1), realloc 15 (+15), strdup 0 (+0), free 0 (+0)

real-life effect

Running a parallel test with on a context with 1000 charts, 100 dimensions each, like this:

siege --verbose --concurrent=255 --benchmark --reps=1 "http://localhost:19999/api/v1/data?context=example.random&before=0&after=-60&points=60&options=jsonwrap"

This PR provides a 10% benefit.

The master:

Transactions:                255 hits
Availability:             100.00 %
Elapsed time:             131.46 secs
Data transferred:        2384.86 MB
Response time:            110.83 secs
Transaction rate:           1.94 trans/sec
Throughput:            18.14 MB/sec
Concurrency:              214.98
Successful transactions:         255
Failed transactions:               0
Longest transaction:          131.46
Shortest transaction:          93.43

This PR:

Transactions:                255 hits
Availability:             100.00 %
Elapsed time:             119.09 secs
Data transferred:        2384.76 MB
Response time:             97.45 secs
Transaction rate:           2.14 trans/sec
Throughput:            20.02 MB/sec
Concurrency:              208.66
Successful transactions:         255
Failed transactions:               0
Longest transaction:          119.09
Shortest transaction:          83.44

Copy link
Contributor

@thiagoftsm thiagoftsm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested this PR and I compare with master branch, and I can confirm improvement in our stats.
I also observe that cloud was working fine and the host was visible all the time.
LGTM!

@ktsaou ktsaou merged commit 47fa3d7 into netdata:master May 2, 2022
@ktsaou ktsaou deleted the speedupbufferreallocs branch May 2, 2022 21:30
@ktsaou ktsaou restored the speedupbufferreallocs branch December 28, 2023 21:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants