ci: benchmarking updates #1926

armcknight · 2022-06-30T21:25:22Z

We want to change a few things about benchmarking, necessitating refactoring so that the test and configs for actions/saucelabs are all separated from the UI test stuff into which they were previously integrated:

only run on a cron schedule once per day vs. on each PR
~~run on a device from each class we define as low, mid and high~~ -> can't get it working on lower devices, see intermediate commits for what was tried
disable ANR tracking (Disable ANR detection in benchmarks #1961)
capture the raw outputs from benchmarks and print to logs for postprocessing by a new script

script output from the most recent run:

❯ ./scripts/process-benchmark-raw-results.py /Users/andrewmcknight/Downloads/xcuitest-9.log
Benchmark report
----------------
All observations (overhead, %):
['3.776', '3.779', '3.780', '3.785', '3.787', '3.790', '3.790', '3.792', '3.792', '3.793', '3.794', '3.796', '3.796', '3.796', '3.798', '3.798', '3.800', '3.800', '3.802', '3.811']

Median: 3.793
Mean: 3.793
P90: 3.800

#skip-changelog

…90 upon completion of separate test cases

… breaks benchmarks

…nfiguration of 5 test cases running 4 iterations each

philipphofmann · 2022-07-08T12:50:24Z

@philipphofmann this is not ready for review yet, we're still trying to work a couple things out. i'll mark it as ready for review when the time comes

I converted this to a draft. Please change it to ready for review once it is.

armcknight · 2022-07-09T06:36:21Z

@philipphofmann It was actually a draft before, but I finished working on it yesterday so marked it as ready for review 😄

philipphofmann

First pass, will continue tomorrow.

.sauce/benchmarking-config.yml

.github/workflows/benchmarking.yml

Samples/iOS-Swift/PerformanceBenchmarks/SentryProcessInfo.m

Samples/iOS-Swift/PerformanceBenchmarks/SentryProcessInfo.h

Samples/iOS-Swift/iOS-Swift.xcodeproj/xcshareddata/xcschemes/iOS-SwiftUITests copy.xcscheme

Samples/iOS-Swift/iOS-Swift/AppDelegate.swift

Co-authored-by: Philipp Hofmann <philipp.hofmann@sentry.io>

…crash impl

philipphofmann

Just wonder why we dynamically create tests? Apart from that LGTM.

develop-docs/README.md

Samples/iOS-Swift/PerformanceBenchmarks/SentrySDKPerformanceBenchmarkTests.m

philipphofmann

Thanks 🙏

* master: ci: Don't run benchmarks on release (#1971) Don't track OOMs for simulators (#1970) feat: Automatic nest new spans with the ui life cycle function (#1959) docs: update some docs/comments to read a little better (#1966) ci: benchmarking updates (#1926) feat: upload list of slow/frozen rendered frame timestamps during a profile (#1910) feat: Enhance the UIViewController breadcrumbs with more data (#1945) # Conflicts: # Sources/SentryCrash/Recording/Monitors/SentryCrashMonitor_System.h

* master: ref: Fix linter error (#1981) fix: read free_memory when the event is captured, not only at SDK startup (#1962) fix: Remove Sentry keys from cached HTTP request headers (#1975) release: 7.21.0 ci: Don't run benchmarks on release (#1971) Don't track OOMs for simulators (#1970) feat: Automatic nest new spans with the ui life cycle function (#1959) docs: update some docs/comments to read a little better (#1966) ci: benchmarking updates (#1926) feat: upload list of slow/frozen rendered frame timestamps during a profile (#1910)

ref: split benchmarking into separate target/action/sauce-config

8af6508

armcknight force-pushed the armcknight/benchmark-raw-export branch from b17ab9c to 8af6508 Compare June 30, 2022 23:41

armcknight added 7 commits June 30, 2022 15:56

fix: artifact key

480bb49

fix: quote suite names with spaces

bb32159

fix: use custom config

6cb60fa

fix: name of test runner app

4d403af

WIP: run 20 trials

f0cf67d

TEST: figure out where to write the artifact in sauce labs

03dd64d

fixup! TEST: figure out where to write the artifact in sauce labs

c0cdc13

armcknight force-pushed the armcknight/benchmark-raw-export branch from 7842bf6 to c0cdc13 Compare July 1, 2022 08:14

fix: report 20 trials in logs

52c46f4

armcknight force-pushed the armcknight/benchmark-raw-export branch from 829b665 to 52c46f4 Compare July 6, 2022 00:45

fix: fixit for format

5ead763

armcknight force-pushed the armcknight/benchmark-raw-export branch from 043fe79 to 5ead763 Compare July 6, 2022 00:57

fix: increase whole-job timeout

d83ab35

armcknight force-pushed the armcknight/benchmark-raw-export branch from e604e58 to d83ab35 Compare July 6, 2022 01:28

fix: increase timeout more, print raw values from benchmark, assert p…

0d1218f

…90 upon completion of separate test cases

armcknight force-pushed the armcknight/benchmark-raw-export branch from 60c3074 to 0d1218f Compare July 6, 2022 22:37

fix: shuffle raw results to test runner app for logging/analysis

8cc7549

armcknight force-pushed the armcknight/benchmark-raw-export branch from 8c70b94 to 8cc7549 Compare July 6, 2022 23:58

ci: put back to 20 trials

3db365f

armcknight force-pushed the armcknight/benchmark-raw-export branch from 92efeb3 to 3db365f Compare July 7, 2022 00:17

armcknight added 2 commits July 6, 2022 19:52

fix: remove unconditional enablement of user interaction tracing that…

e8b2eb0

… breaks benchmarks

ref: simplify data marshalling, add script to compile report

6794cf9

armcknight force-pushed the armcknight/benchmark-raw-export branch from 15ef7c9 to 6794cf9 Compare July 7, 2022 04:36

ci: make each test case shorter; update docs

7408e19

armcknight force-pushed the armcknight/benchmark-raw-export branch from 65b4385 to 7408e19 Compare July 7, 2022 05:00

getsentry-bot and others added 3 commits July 7, 2022 05:01

Format code

184bfff

ci: add tags to the saucectl invocations fro benchmarking

a02a2cd

fix: increase whole-job timeout to 200m

518343f

armcknight added 3 commits July 7, 2022 18:29

ref: remove unused xcode key from matrix entries

bd273f2

fix: try 20 cases of 1 iteration each

be3a3eb

fix: remove device classes that never finish, put back to original co…

59b3ae9

…nfiguration of 5 test cases running 4 iterations each

armcknight marked this pull request as ready for review July 8, 2022 04:47

armcknight requested review from philipphofmann and brustolin as code owners July 8, 2022 04:47

armcknight requested a review from indragiek July 8, 2022 04:48

philipphofmann marked this pull request as draft July 8, 2022 12:50

armcknight marked this pull request as ready for review July 9, 2022 06:36

philipphofmann reviewed Jul 11, 2022

View reviewed changes

armcknight and others added 6 commits July 11, 2022 09:23

Update Samples/iOS-Swift/PerformanceBenchmarks/SentryProcessInfo.m

ce67d78

Co-authored-by: Philipp Hofmann <philipp.hofmann@sentry.io>

Update .github/workflows/benchmarking.yml

68df84e

Co-authored-by: Philipp Hofmann <philipp.hofmann@sentry.io>

ref: remove unneeded duplicate scheme

4439031

ci: put back to always run on PRs (and release branches)

127b688

ci: bring down timeout to 30 minutes (observing 24-28 minutes)

9a2554b

fix: link to qa article on proc flags; add proc flags init to sentry …

2285b8d

…crash impl

armcknight requested a review from philipphofmann July 11, 2022 23:40

Format code

62a245c

philipphofmann approved these changes Jul 12, 2022

View reviewed changes

armcknight and others added 3 commits July 12, 2022 12:27

fix: disable ANR tracking during benchmarks

d5f3c62

docs: more explainers and cross references for test setup

ee199e7

Format code

146d9c3

philipphofmann approved these changes Jul 13, 2022

View reviewed changes

philipphofmann mentioned this pull request Jul 13, 2022

Disable ANR detection in benchmarks #1961

Closed

armcknight merged commit e7beac2 into master Jul 13, 2022

armcknight deleted the armcknight/benchmark-raw-export branch July 13, 2022 19:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: benchmarking updates #1926

ci: benchmarking updates #1926

armcknight commented Jun 30, 2022 •

edited

philipphofmann commented Jul 8, 2022 •

edited

armcknight commented Jul 9, 2022

philipphofmann left a comment

philipphofmann left a comment

philipphofmann left a comment

ci: benchmarking updates #1926

ci: benchmarking updates #1926

Conversation

armcknight commented Jun 30, 2022 • edited

philipphofmann commented Jul 8, 2022 • edited

armcknight commented Jul 9, 2022

philipphofmann left a comment

Choose a reason for hiding this comment

philipphofmann left a comment

Choose a reason for hiding this comment

philipphofmann left a comment

Choose a reason for hiding this comment

armcknight commented Jun 30, 2022 •

edited

philipphofmann commented Jul 8, 2022 •

edited