-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
contact_list apdex: Establish baseline #90
Comments
Background
ConfigsThe baseline "BEFORE" apdex score will be obtained from running the test suite in these three configs
DevicesTo be confirmed User type
DataFor CHWs
IterationsWe will execute the automation tests for 1 user and simulate 5 days, with the suite running 10 times each day. Clickpath for measuring
|
Hi @michaelkohn
|
Hi @michaelkohn, due to the issues I am facing with the automated testing I decided to get a baseline using manual testing. I ran the manual testing 30 times how I outlined it:
Kenya - Phone Tecno Pop 7 - contact_list:load = 0.77, contact_list:query = 0.88"contact_list:load": {
"sum": 267504,
"min": 2092,
"max": 162416,
"count": 30,
"sumsqr": 26868328860
},
"contact_list:load:apdex:frustrated": {
"sum": 162416,
"min": 162416,
"max": 162416,
"count": 1,
"sumsqr": 26378957056
},
"contact_list:load:apdex:satisfied": {
"sum": 42145,
"min": 2092,
"max": 2939,
"count": 17,
"sumsqr": 106401163
},
"contact_list:load:apdex:tolerable": {
"sum": 62943,
"min": 3008,
"max": 8022,
"count": 12,
"sumsqr": 382970641
},
"contact_list:query": {
"sum": 528342,
"min": 1158,
"max": 162013,
"count": 113,
"sumsqr": 41920686882
},
"contact_list:query:apdex:frustrated": {
"sum": 298102,
"min": 14690,
"max": 162013,
"count": 3,
"sumsqr": 41201725470
},
"contact_list:query:apdex:satisfied": {
"sum": 133062,
"min": 1158,
"max": 2897,
"count": 90,
"sumsqr": 208120866
},
"contact_list:query:apdex:tolerable": {
"sum": 97178,
"min": 3471,
"max": 7512,
"count": 20,
"sumsqr": 510840546
},
Mali - Phone Tecno Pop 7 - contact_list:load = 1.00, contact_list:query = 0.99 (Mali CHWs don't have reports) "contact_list:load": {
"sum": 43652,
"min": 295,
"max": 2745,
"count": 31,
"sumsqr": 76434764
},
"contact_list:load:apdex:satisfied": {
"sum": 43652,
"min": 295,
"max": 2745,
"count": 31,
"sumsqr": 76434764
},
"contact_list:query": {
"sum": 111828,
"min": 112,
"max": 39022,
"count": 90,
"sumsqr": 1613727640
},
"contact_list:query:apdex:frustrated": {
"sum": 39022,
"min": 39022,
"max": 39022,
"count": 1,
"sumsqr": 1522716484
},
"contact_list:query:apdex:satisfied": {
"sum": 72806,
"min": 112,
"max": 2335,
"count": 89,
"sumsqr": 91011156
}, Nepal - Phone Tecno Pop 7 - contact_list:load = 0.97, contact_list:query = 0.98 "contact_list:load": {
"sum": 85369,
"min": 680,
"max": 38758,
"count": 30,
"sumsqr": 1584216519
},
"contact_list:load:apdex:frustrated": {
"sum": 38758,
"min": 38758,
"max": 38758,
"count": 1,
"sumsqr": 1502182564
},
"contact_list:load:apdex:satisfied": {
"sum": 46611,
"min": 680,
"max": 2609,
"count": 29,
"sumsqr": 82033955
},
"contact_list:query": {
"sum": 229962,
"min": 159,
"max": 84752,
"count": 148,
"sumsqr": 8853099192
},
"contact_list:query:apdex:frustrated": {
"sum": 123075,
"min": 38323,
"max": 84752,
"count": 2,
"sumsqr": 8651553833
},
"contact_list:query:apdex:satisfied": {
"sum": 94346,
"min": 159,
"max": 2393,
"count": 144,
"sumsqr": 119663414
},
"contact_list:query:apdex:tolerable": {
"sum": 12541,
"min": 4997,
"max": 7544,
"count": 2,
"sumsqr": 81881945
} Togo - Phone Tecno Pop 7 - contact_list:load = 0.59, contact_list:query = 0.90 "contact_list:load": {
"sum": 122034,
"min": 964,
"max": 19578,
"count": 29,
"sumsqr": 808812564
},
"contact_list:load:apdex:frustrated": {
"sum": 19578,
"min": 19578,
"max": 19578,
"count": 1,
"sumsqr": 383298084
},
"contact_list:load:apdex:satisfied": {
"sum": 9144,
"min": 964,
"max": 2071,
"count": 6,
"sumsqr": 15516228
},
"contact_list:load:apdex:tolerable": {
"sum": 93312,
"min": 3212,
"max": 6279,
"count": 22,
"sumsqr": 409998252
},
"contact_list:query": {
"sum": 321022,
"min": 178,
"max": 78021,
"count": 145,
"sumsqr": 7601213600
},
"contact_list:query:apdex:frustrated": {
"sum": 121479,
"min": 19243,
"max": 78021,
"count": 3,
"sumsqr": 7043935715
},
"contact_list:query:apdex:satisfied": {
"sum": 110372,
"min": 178,
"max": 2753,
"count": 120,
"sumsqr": 176227142
},
"contact_list:query:apdex:tolerable": {
"sum": 89171,
"min": 3046,
"max": 6663,
"count": 22,
"sumsqr": 381050743
}, |
Nice @Benmuiruri. Thanks for the clear data... it's really interesting to see how much worse Togo is, I wonder why?? I assume the Mali and Nepal users you set up are only seeing 2-3 rows in the contact_list (due to where they are in the hierarchy), so it's not surprising they are scoring very high. Since we have a good dataset of, well, bad data with Kenya and Togo, I'm fine if you just proceed comparing the results of before / after code changes for Kenya and Togo only since there really isn't any room to improve Mali/Nepal anyway. The one thing that could be useful, but not totally necessary, would be to show that Mali/Nepal didn't get slower after the code changes, but it's not necessary since I imagine it's a very low chance it gets slower. |
contact_list
The text was updated successfully, but these errors were encountered: