[Perf] Enhance test.sh and perf_report.py to support multiple iterations.#85
Merged
[Perf] Enhance test.sh and perf_report.py to support multiple iterations.#85
Conversation
gapisback
commented
Jun 16, 2024
| FLD_AVG_ELAPSED_TIME_IDX = 5 | ||
| FLD_SERVER_THROUGHPUT_IDX = 6 | ||
| FLD_CLIENT_THROUGHPUT_IDX = 7 | ||
| FLD_AVG_OPS_PER_THREAD_IDX = 8 |
Collaborator
Author
There was a problem hiding this comment.
New parsing interfaces to pluck metrics fields from comma-separated Summary: line in stdout.
gapisback
commented
Jun 16, 2024
| # Run through diff server-thread parameters, and generate report | ||
| # We make multiple passes extracing metrics for each combination | ||
| # of #-of-server-threads and run-types. | ||
| for threads in list_of_nthreads: |
Collaborator
Author
There was a problem hiding this comment.
This new approach simplifies the post-processing.
We make multiple passes on the output lines, looking for lines of interest to consolidate and to generate aggregate metrics from.
gapisback
commented
Jun 16, 2024
| # using env-var as: | ||
| # | ||
| # SERVER_NUM_THREADS="1 2 4 8" ./test.sh run-all-client-server-perf-tests | ||
| # L3_PERF_SERVER_NUM_THREADS="1 2 4 8" ./test.sh run-all-client-server-perf-tests |
Collaborator
Author
There was a problem hiding this comment.
Slight change in env-var: Use L3_PERF prefix, to namespace the env-vars used by this script.
gapisback
added
enhancement
gregthelaw
approved these changes
Jun 17, 2024
Contributor
gregthelaw
left a comment
gregthelaw
left a comment
There was a problem hiding this comment.
Disclaimer: I've only skimmed over this, but it all looks pretty useful / non-controversial.
|
|
||
| # Key: Run-type 'Baseline - No logging'; Value: [ <metrics> ] | ||
| metrics_by_run = OrderedDict() | ||
| # metrics_by_run = OrderedDict() |
Contributor
There was a problem hiding this comment.
Is this deliberately committed?
Collaborator
Author
There was a problem hiding this comment.
Don't recall. Cruft. Cleaned it up.
gapisback
force-pushed
the
gapisback/run-multi-iters-perf
branch
2 times, most recently
from
24847ed to
7e70955
Compare
…ons.
In previous performance test-runs, we were able to only run one iteration
of the client-server program for each run-type, test-parameter. We were
seeing some fluctuations and unexpected results in the comparison report.
This commit enhances the testing framework to:
a) Run 1 or more iterations of the client/server program
b) Extend the Python perf-comparison report generator to work
with another source of input metrics-line(s).
c) Comparison report generated using the median value of the
relevant metric across the n-iterations.
Note: (b) was needed as runs on AWS were aborted midway and had to
be partially re-run. We add Python parsing logic to extract out
and massage metrics from "Summary: " output lines, that the server
program emits to `stdout`. This expedited the report generation task
w/o having to re-run all perf-tests on AWS instances. This approach
re-casts the new summary data into an input format needed by the
existing report generation logic.
- To run multiple iterations, use env-var L3_PERF_TEST_NUM_ITERS=<n>
Workflow to run the enhanced perf_report.py:
- Run test.sh with diff server-threads and #iterations.
Example:
L3_PERF_TEST_NUM_ITERS=5 L3_PERF_SERVER_NUM_THREADS="1 2 4 8" ./test.sh --clock-default $((1000 * 1000))
- Redirect output to a /tmp/file: > /tmp/perf.out 2>&1
- grep 'Summary:' /tmp/perf.out > /tmp/perf.summary.out
- Feed the summary file to the Python tool using:
./scripts/perf_report.py --summary-file /tmp/perf.summary.out
- tests/pytests/perf_report_test.py:
- Add unit-test cases for new parsing methods added
gapisback
force-pushed
the
gapisback/run-multi-iters-perf
branch
from
7e70955 to
3460fd5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In previous performance test-runs, we were able to only run one iteration of the client-server program for each run-type, test-parameter. We were seeing some fluctuations and unexpected results in the comparison report.
This commit enhances the testing framework to:
a) Run 1 or more iterations of the client/server program
b) Extend the Python perf-comparison report generator to work
with another source of input metrics-line(s).
c) Comparison report generated using median value of the
relevant metric across the n-iterations.
Note: (b) was needed as runs on AWS were aborted midway and had to be partially re-run. We add Python parsing logic to extract out and massage metrics from "Summary: " output lines, that the server program emits to
stdout.Workflow to run the enhanced perf_report.py:
Example:
L3_PERF_TEST_NUM_ITERS=5 L3_PERF_SERVER_NUM_THREADS="1 2 4 8" ./test.sh --clock-default $((1000 * 1000))
Redirect output to a /tmp/file: > /tmp/perf.out 2>&1
grep 'Summary:' /tmp/perf.out > /tmp/perf.summary.out
Feed the summary file to the Python tool using:
./scripts/perf_report.py --summary-file /tmp/perf.summary.out
tests/pytests/perf_report_test.py: