Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[performance] - UserOperator capacity tests #10018

Merged
merged 10 commits into from
Apr 29, 2024
Merged

[performance] - UserOperator capacity tests #10018

merged 10 commits into from
Apr 29, 2024

Conversation

see-quick
Copy link
Member

@see-quick see-quick commented Apr 23, 2024 •
edited

Type of change

  • Enhancement / new feature
  • Refactoring

Description

This PR introduces parameterized tests aimed at evaluating the capacity and performance of the system under various operational conditions. By adjusting parameters such as controller thread pool size, cache refresh intervals, batch queue size, and user operations thread pool size, the tests simulate different levels of parallel processing, batching strategies, and operational latencies. This helps in understanding how different configurations impact system performance.

We do that incrementally (pseudo-code):

Initialize Test Parameters

while true:
    start = successfulCreations
    end = start + batchSize
    users = getUserBatch(testStorage, username, start, end, authType)

    try:
        createUsersAndWait(testStorage, users, username)
        successfulCreations += batchSize
        log "Batch created and verified from start to end"
    catch exception:
        log "Batch creation failed from start to end with error"
        collectLogs() // this is needed because then we could find a possible problem what is a bottleneck...
        break

Multi-node cluster (ZK-based)

Use Case: capacityDefaultUseCase
+------------+--------------------------+---------------------+---------------------------------+-------------------------------+----------------------+------------------------------+---------------------------------+--------------------------------------+------------------------------------+-------------------------------------+----------------------------------+
| Experiment | IN: OPERATION_TIMEOUT_MS | IN: WORK_QUEUE_SIZE | IN: CONTROLLER_THREAD_POOL_SIZE | IN: CACHE_REFRESH_INTERVAL_MS | IN: BATCH_QUEUE_SIZE | IN: BATCH_MAXIMUM_BLOCK_SIZE | IN: BATCH_MAXIMUM_BLOCK_TIME_MS | IN: USER_OPERATIONS_THREAD_POOL_SIZE | OUT: Successful KafkaUsers Created | jvm_memory_used_megabytes_total.txt | system_load_average_per_core.txt |
| 1          | 300000                   | 10000               | 50                              | 15000                         | 1024                 | 100                          | 100                             | 50                                   | 4800                               | 716.524856                          | 0.23875                          |
| 2          | 300000                   | 10000               | 100                             | 20000                         | 2048                 | 200                          | 50                              | 100                                  | 4800                               | 877.894032                          | 0.2825                           |
| 3          | 300000                   | 10000               | 75                              | 15000                         | 1500                 | 150                          | 75                              | 75                                   | 4800                               | 548.884896                          | 0.2825                           |
| 4          | 300000                   | 10000               | 100                             | 30000                         | 4096                 | 300                          | 100                             | 100                                  | 4800                               | 638.626608                          | 0.243125                         |
| 5          | 300000                   | 10000               | 50                              | 10000                         | 512                  | 50                           | 25                              | 50                                   | 4800                               | 725.024336                          | 0.2375                           |
+------------+--------------------------+---------------------+---------------------------------+-------------------------------+----------------------+------------------------------+---------------------------------+--------------------------------------+------------------------------------+-------------------------------------+----------------------------------+
[2024-04-28T19:39:03.891Z] | Experiment | IN: OPERATION_TIMEOUT_MS | IN: WORK_QUEUE_SIZE | IN: CONTROLLER_THREAD_POOL_SIZE | IN: CACHE_REFRESH_INTERVAL_MS | IN: BATCH_QUEUE_SIZE | IN: BATCH_MAXIMUM_BLOCK_SIZE | IN: BATCH_MAXIMUM_BLOCK_TIME_MS | IN: USER_OPERATIONS_THREAD_POOL_SIZE | OUT: Successful KafkaUsers Created |
[2024-04-28T19:39:03.891Z] | 1          | 300000                   | 10000               | 50                              | 15000                         | 1024                 | 100                          | 100                             | 50                                   | 4800                               |
[2024-04-28T19:39:03.891Z] +------------+--------------------------+---------------------+---------------------------------+-------------------------------+----------------------+------------------------------+---------------------------------+--------------------------------------+------------------------------------+

Multi-node KRaft-based (9624 s -> 2.67333333 hours):

[2024-04-29T00:04:39.287Z] +------------+--------------------------+---------------------+---------------------------------+-------------------------------+----------------------+------------------------------+---------------------------------+--------------------------------------+------------------------------------+
[2024-04-29T00:04:39.287Z] | Experiment | IN: OPERATION_TIMEOUT_MS | IN: WORK_QUEUE_SIZE | IN: CONTROLLER_THREAD_POOL_SIZE | IN: CACHE_REFRESH_INTERVAL_MS | IN: BATCH_QUEUE_SIZE | IN: BATCH_MAXIMUM_BLOCK_SIZE | IN: BATCH_MAXIMUM_BLOCK_TIME_MS | IN: USER_OPERATIONS_THREAD_POOL_SIZE | OUT: Successful KafkaUsers Created |
[2024-04-29T00:04:39.287Z] | 1          | 300000                   | 10000               | 50                              | 15000                         | 1024                 | 100                          | 100                             | 50                                   | 19300                              |
[2024-04-29T00:04:39.288Z] +------------+--------------------------+---------------------+---------------------------------+-------------------------------+----------------------+------------------------------+---------------------------------+--------------------------------------+------------------------------------+

Testing farm (i.e., KRAft-based clusters)

Testing farm:

Use Case: capacityUseCase
+------------+--------------------------+---------------------+---------------------------------+-------------------------------+----------------------+------------------------------+---------------------------------+--------------------------------------+------------------------------------+-------------------------------------+----------------------------------+
| Experiment | IN: OPERATION_TIMEOUT_MS | IN: WORK_QUEUE_SIZE | IN: CONTROLLER_THREAD_POOL_SIZE | IN: CACHE_REFRESH_INTERVAL_MS | IN: BATCH_QUEUE_SIZE | IN: BATCH_MAXIMUM_BLOCK_SIZE | IN: BATCH_MAXIMUM_BLOCK_TIME_MS | IN: USER_OPERATIONS_THREAD_POOL_SIZE | OUT: Successful KafkaUsers Created | jvm_memory_used_megabytes_total.txt | system_load_average_per_core.txt |
| 1          | 300000                   | 10000               | 50                              | 15000                         | 1024                 | 100                          | 100                             | 50                                   | 21100                              | 3326.59004                          | 0.885                            |
+------------+--------------------------+---------------------+---------------------------------+-------------------------------+----------------------+------------------------------+---------------------------------+--------------------------------------+------------------------------------+-------------------------------------+----------------------------------+

Testing farm with default configuration ARM:

Use Case: capacityUseCase
+------------+--------------------------+---------------------+---------------------------------+-------------------------------+----------------------+------------------------------+---------------------------------+--------------------------------------+------------------------------------+-------------------------------------+----------------------------------+
| Experiment | IN: OPERATION_TIMEOUT_MS | IN: WORK_QUEUE_SIZE | IN: CONTROLLER_THREAD_POOL_SIZE | IN: CACHE_REFRESH_INTERVAL_MS | IN: BATCH_QUEUE_SIZE | IN: BATCH_MAXIMUM_BLOCK_SIZE | IN: BATCH_MAXIMUM_BLOCK_TIME_MS | IN: USER_OPERATIONS_THREAD_POOL_SIZE | OUT: Successful KafkaUsers Created | jvm_memory_used_megabytes_total.txt | system_load_average_per_core.txt |
| 1          | 300000                   | 10000               | 50                              | 15000                         | 1024                 | 100                          | 100                             | 50                                   | 26000                              | 5110.567712                         | 0.2990625                        |
+------------+--------------------------+---------------------+---------------------------------+-------------------------------+----------------------+------------------------------+---------------------------------+--------------------------------------+------------------------------------+-------------------------------------+----------------------------------+

Testing farm with default configuration Intel:

Use Case: capacityUseCase
+------------+--------------------------+---------------------+---------------------------------+-------------------------------+----------------------+------------------------------+---------------------------------+--------------------------------------+------------------------------------+-------------------------------------+----------------------------------+
| Experiment | IN: OPERATION_TIMEOUT_MS | IN: WORK_QUEUE_SIZE | IN: CONTROLLER_THREAD_POOL_SIZE | IN: CACHE_REFRESH_INTERVAL_MS | IN: BATCH_QUEUE_SIZE | IN: BATCH_MAXIMUM_BLOCK_SIZE | IN: BATCH_MAXIMUM_BLOCK_TIME_MS | IN: USER_OPERATIONS_THREAD_POOL_SIZE | OUT: Successful KafkaUsers Created | jvm_memory_used_megabytes_total.txt | system_load_average_per_core.txt |
| 1          | 300000                   | 10000               | 50                              | 15000                         | 1024                 | 100                          | 100                             | 50                                   | 48700                              | 10118.300232                        | 0.3009375                        |
+------------+--------------------------+---------------------+---------------------------------+-------------------------------+----------------------+------------------------------+---------------------------------+--------------------------------------+------------------------------------+-------------------------------------+----------------------------------+

Testing farm ARM:

Use Case: capacityUseCase
+------------+--------------------------+---------------------+---------------------------------+-------------------------------+----------------------+------------------------------+---------------------------------+--------------------------------------+------------------------------------+-------------------------------------+----------------------------------+
| Experiment | IN: OPERATION_TIMEOUT_MS | IN: WORK_QUEUE_SIZE | IN: CONTROLLER_THREAD_POOL_SIZE | IN: CACHE_REFRESH_INTERVAL_MS | IN: BATCH_QUEUE_SIZE | IN: BATCH_MAXIMUM_BLOCK_SIZE | IN: BATCH_MAXIMUM_BLOCK_TIME_MS | IN: USER_OPERATIONS_THREAD_POOL_SIZE | OUT: Successful KafkaUsers Created | jvm_memory_used_megabytes_total.txt | system_load_average_per_core.txt |
| 1          | 300000                   | 10000               | 75                              | 15000                         | 1500                 | 150                          | 75                              | 75                                   | 46500                              | 9316.4262                           | 0.4915625                        |
| 2          | 300000                   | 10000               | 100                             | 30000                         | 4096                 | 300                          | 100                             | 100                                  | 45800                              | 9347.137992                         | 0.506875                         |
+------------+--------------------------+---------------------+---------------------------------+-------------------------------+----------------------+------------------------------+---------------------------------+--------------------------------------+------------------------------------+-------------------------------------+----------------------------------+

Testing farm INTEL:

Use Case: capacityUseCase
+------------+--------------------------+---------------------+---------------------------------+-------------------------------+----------------------+------------------------------+---------------------------------+--------------------------------------+------------------------------------+-------------------------------------+----------------------------------+
| Experiment | IN: OPERATION_TIMEOUT_MS | IN: WORK_QUEUE_SIZE | IN: CONTROLLER_THREAD_POOL_SIZE | IN: CACHE_REFRESH_INTERVAL_MS | IN: BATCH_QUEUE_SIZE | IN: BATCH_MAXIMUM_BLOCK_SIZE | IN: BATCH_MAXIMUM_BLOCK_TIME_MS | IN: USER_OPERATIONS_THREAD_POOL_SIZE | OUT: Successful KafkaUsers Created | jvm_memory_used_megabytes_total.txt | system_load_average_per_core.txt |
| 1          | 300000                   | 10000               | 75                              | 15000                         | 1500                 | 150                          | 75                              | 75                                   | 55100                              | 12310.114608                        | 0.3594791666666666               |
| 2          | 300000                   | 10000               | 100                             | 30000                         | 4096                 | 300                          | 100                             | 100                                  | 52500                              | 12179.524968                        | 0.3802083333333333               |
+------------+--------------------------+---------------------+---------------------------------+-------------------------------+----------------------+------------------------------+---------------------------------+--------------------------------------+------------------------------------+-------------------------------------+----------------------------------+

Checklist

  • Write tests
  • Make sure all tests pass

@see-quick
[performance] - UserOperator capacity tests
4947686 
Signed-off-by: see-quick <maros.orsak159@gmail.com>
@see-quick
Copy link
Member Author

/packit test --labels performance

@see-quick see-quick added this to the 0.41.0 milestone Apr 23, 2024
@see-quick
fix alice usecase
ee05330 
Signed-off-by: see-quick <maros.orsak159@gmail.com>
@see-quick
Copy link
Member Author

/packit test --labels performance

@see-quick
update to also collect logs + better deletion of resource via all
0b04715 
Signed-off-by: see-quick <maros.orsak159@gmail.com>
@see-quick
Copy link
Member Author

/packit test --labels performance

@see-quick see-quick marked this pull request as ready for review April 24, 2024 20:51
@see-quick see-quick self-assigned this Apr 24, 2024
@see-quick see-quick requested a review from a team April 24, 2024 20:52
@see-quick
update disk size and ram to see maybe bigger numbers than 21k users?
1d4d659 
Signed-off-by: see-quick <maros.orsak159@gmail.com>
@see-quick
Copy link
Member Author

/packit test --labels performance-capacity

@see-quick
add it to required
66f42b9 
Signed-off-by: see-quick <maros.orsak159@gmail.com>
@see-quick
Copy link
Member Author

/packit test --labels performance-capacity

@see-quick
try with dash dassshhhh
ff48245 
Signed-off-by: see-quick <maros.orsak159@gmail.com>
@see-quick
Copy link
Member Author

/packit test --labels performance-capacity

@see-quick
update a little bit disk and decrease log
b850f44 
Signed-off-by: see-quick <maros.orsak159@gmail.com>
@see-quick
Copy link
Member Author

/packit test --labels performance-capacity

@see-quick
update to fix machine
4731379 
Signed-off-by: see-quick <maros.orsak159@gmail.com>
@see-quick
Copy link
Member Author

/packit test --labels performance-capacity

@see-quick
update
771a319 
Signed-off-by: see-quick <maros.orsak159@gmail.com>
@see-quick
Copy link
Member Author

/packit test --labels performance-capacity

@see-quick
update also perf logging
748de4f 
Signed-off-by: see-quick <maros.orsak159@gmail.com>
@see-quick
Copy link
Member Author

@strimzi-ci run tests --cluster-type=ocp --cluster-version=4.15 --install-type=bundle --profile=performance-capacity

@strimzi-ci
Copy link

▶️ Build started - check Jenkins for more info. ▶️

@strimzi-ci
Copy link

❗ Systemtests Failed (no tests results are present) ❗

@see-quick
Copy link
Member Author

❗ Systemtests Failed (no tests results are present) ❗

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2384 s ✅

@see-quick
Copy link
Member Author

@strimzi-ci run tests --cluster-type=ocp --cluster-version=4.15 --install-type=bundle --profile=performance-capacity --env=STRIMZI_USE_KRAFT_IN_TESTS=true

@strimzi-ci
Copy link

▶️ Build started - check Jenkins for more info. ▶️

@strimzi-ci
Copy link

❗ Systemtests Failed (no tests results are present) ❗

im-konge
im-konge approved these changes Apr 29, 2024
View reviewed changes
Copy link
Member

@im-konge im-konge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, assuming the tests pass :) thanks!

@see-quick see-quick added ready for merge Label for PRs which are ready for merge and removed needs review labels Apr 29, 2024
@see-quick see-quick merged commit c15b3b6 into strimzi:main Apr 29, 2024
14 checks passed
@see-quick see-quick mentioned this pull request May 2, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@im-konge im-konge im-konge approved these changes

Assignees

@see-quick see-quick

Labels
ready for merge Label for PRs which are ready for merge
Projects
None yet
Milestone
0.41.0
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@see-quick @strimzi-ci @im-konge