[ci] Check APK sizes on Azure Pipelines #4133
Conversation
This target was overlooked in the initial split up and porting efforts for our APK tests. This target calculates the sizes of or test APKs and compares them against a baseline. This change ensures that we will report a failure in our Azure Pipelines build if any project's APK size drifts too far from the baseline.
When comparing APK sizes, we need to make sure the correct `TestsFlavor`
metadata is set. In order for `-Aot` to be included in the test name,
the `AotAssemblies` property must be explicitly set.
Fixes:
error MSB4018: The "ProcessApkSizes" task failed unexpectedly. [/Users/runner/runners/2.164.3/work/1/s/tests/Xamarin.Forms-Performance-Integration/Droid/Xamarin.Forms.Performance.Integration.Droid.csproj]
error MSB4018: System.Collections.Generic.KeyNotFoundException: The given key 'apk-Release-Profiled' was not present in the dictionary. [/Users/runner/runners/2.164.3/work/1/s/tests/Xamarin.Forms-Performance-Integration/Droid/Xamarin.Forms.Performance.Integration.Droid.csproj]
There was a problem hiding this comment.
Looks like the test triggered and the size difference is more than the threshold of 100kb.
error : the reference 16540799 and actual size 16995788 for apk-Release differs too much (>100kb). [/Users/runner/runners/2.164.3/work/1/s/src/Mono.Android/Test/Mono.Android-Tests.csproj]
error : please check if it is correct and update the reference apk sizes files if it is. [/Users/runner/runners/2.164.3/work/1/s/src/Mono.Android/Test/Mono.Android-Tests.csproj]
error : it can be done by running `make CONFIGURATION=<Debug or Release> update-apk-sizes-reference` in XA root directory [/Users/runner/runners/2.164.3/work/1/s/src/Mono.Android/Test/Mono.Android-Tests.csproj]
That would need more investigation as the size measurements were not recorded for some time, until fixed recently. https://jenkins.mono-project.com/view/Xamarin.Android/job/xamarin-android/plot/Tests%20size/ - there is about 450k increase during the gap for that test :(
It happened between Jenkins builds 776 and 1029.
So we need to look into that. Hopefully the azure artifacts should contain the measurements.
|
Build 900: Release apk size 16548991 So it looks like the increase is result of 130905e (build 1009 https://jenkins.mono-project.com/view/Xamarin.Android/job/xamarin-android/1009/) |
|
I'm trying to figure out why the APK size comparison is failing on Azure Pipelines but not Jenkins. Is it at all possible that the .apk files we produce when building against a system installation would be slightly larger than the ones created against a build tree? That seems unlikely... |
|
The numbers on Jenkins from the last builds are equal to those reported in the error. So I am trying to trace it now in my above comment. |
|
Ok great, thanks for the investigation. I'll prepare a diff to bump the baseline sizes as part of this PR if we determine that these increases are acceptable. |
|
I will try to look at it closer next week. Opened the above issue for it so that I don't forget. |
|
Release apk size diff: diff --git a/tests/apk-sizes-reference/TestResult-Mono.Android_Tests-values-Release.csv b/tests/apk-sizes-reference/TestResult-Mono.Android_Tests-values-Release.csv
index 73a25954..9fce41ef 100644
--- a/tests/apk-sizes-reference/TestResult-Mono.Android_Tests-values-Release.csv
+++ b/tests/apk-sizes-reference/TestResult-Mono.Android_Tests-values-Release.csv
@@ -1,2 +1,2 @@
-apk-Release,Java.Interop.dll-Release,Mono.Android.dll-Release,mscorlib.dll-Release,monosgen-armeabi-v7a-Release,apk-Release-Aot,Java.Interop.dll-Release-Aot,Mono.Android.dll-Release-Aot,mscorlib.dll-Release-Aot,monosgen-armeabi-v7a-Release-Aot,Java.Interop.dll.so-Release-Aot,Mono.Android.dll.so-Release-Aot,mscorlib.dll.so-Release-Aot,apk-Release-Profiled-Aot,Java.Interop.dll-Release-Profiled-Aot,Mono.Android.dll-Release-Profiled-Aot,mscorlib.dll-Release-Profiled-Aot,monosgen-armeabi-v7a-Release-Profiled-Aot,Mono.Android.dll.so-Release-Profiled-Aot,mscorlib.dll.so-Release-Profiled-Aot,Java.Interop.dll.so-Release-Profiled-Aot,apk-Release-Bundle,monosgen-armeabi-v7a-Release-Bundle
-16540799,161792,1400832,2112000,3843248,32985861,161792,1400832,2112000,3843248,842432,5908196,8641772,18506501,161792,1400832,2112000,3843248,421036,1685856,220904,14229016,3843248
+apk-Release,Java.Interop.dll-Release,Mono.Android.dll-Release,mscorlib.dll-Release,monosgen-armeabi-v7a-Release,apk-Release-Aot,Java.Interop.dll-Release-Aot,Mono.Android.dll-Release-Aot,mscorlib.dll-Release-Aot,monosgen-armeabi-v7a-Release-Aot,Java.Interop.dll.so-Release-Aot,Mono.Android.dll.so-Release-Aot,mscorlib.dll.so-Release-Aot,apk-Release-Profiled-Aot,Java.Interop.dll-Release-Profiled-Aot,Mono.Android.dll-Release-Profiled-Aot,mscorlib.dll-Release-Profiled-Aot,monosgen-armeabi-v7a-Release-Profiled-Aot,Mono.Android.dll.so-Release-Profiled-Aot,Java.Interop.dll.so-Release-Profiled-Aot,mscorlib.dll.so-Release-Profiled-Aot,apk-Release-Bundle,monosgen-armeabi-v7a-Release-Bundle
+16995788,173568,1410560,2103808,3844692,35870532,173568,1410560,2103808,3844692,1194160,6762616,10086236,20154180,173568,1410560,2103808,3844692,419856,235520,1695084,14392856,3844692
diff --git a/tests/apk-sizes-reference/TestResult-Mono.Android_TestsMultiDex-values-Release.csv b/tests/apk-sizes-reference/TestResult-Mono.Android_TestsMultiDex-values-Release.csv
index 256e89fc..da361b49 100644
--- a/tests/apk-sizes-reference/TestResult-Mono.Android_TestsMultiDex-values-Release.csv
+++ b/tests/apk-sizes-reference/TestResult-Mono.Android_TestsMultiDex-values-Release.csv
@@ -1,2 +1,2 @@
apk-Release,Mono.Android.dll-Release,mscorlib.dll-Release,monosgen-armeabi-v7a-Release,apk-Release-Aot,Mono.Android.dll-Release-Aot,mscorlib.dll-Release-Aot,monosgen-armeabi-v7a-Release-Aot,apk-Release-Profiled-Aot,Mono.Android.dll-Release-Profiled-Aot,mscorlib.dll-Release-Profiled-Aot,monosgen-armeabi-v7a-Release-Profiled-Aot,apk-Release-Bundle,Mono.Android.dll-Release-Bundle,mscorlib.dll-Release-Bundle,monosgen-armeabi-v7a-Release-Bundle
-16192740,987136,2112000,3843248,16192740,987136,2112000,3843248,16192740,987136,2112000,3843248,16192740,987136,2112000,3843248
+16659977,1008128,2103808,3844692,16659977,1008128,2103808,3844692,16659977,1008128,2103808,3844692,16659977,1008128,2103808,3844692
diff --git a/tests/apk-sizes-reference/TestResult-Xamarin.Forms_Tests-values-Release.csv b/tests/apk-sizes-reference/TestResult-Xamarin.Forms_Tests-values-Release.csv
index 09d448da..22ab0b28 100644
--- a/tests/apk-sizes-reference/TestResult-Xamarin.Forms_Tests-values-Release.csv
+++ b/tests/apk-sizes-reference/TestResult-Xamarin.Forms_Tests-values-Release.csv
@@ -1,2 +1,2 @@
apk-Release,Java.Interop.dll-Release,Mono.Android.dll-Release,mscorlib.dll-Release,monosgen-armeabi-v7a-Release,apk-Release-Aot,Java.Interop.dll-Release-Aot,Mono.Android.dll-Release-Aot,mscorlib.dll-Release-Aot,monosgen-armeabi-v7a-Release-Aot,Java.Interop.dll.so-Release-Aot,mscorlib.dll.so-Release-Aot,Mono.Android.dll.so-Release-Aot,apk-Release-Profiled-Aot,Java.Interop.dll-Release-Profiled-Aot,Mono.Android.dll-Release-Profiled-Aot,mscorlib.dll-Release-Profiled-Aot,monosgen-armeabi-v7a-Release-Profiled-Aot,Mono.Android.dll.so-Release-Profiled-Aot,Java.Interop.dll.so-Release-Profiled-Aot,mscorlib.dll.so-Release-Profiled-Aot,apk-Release-Bundle,monosgen-armeabi-v7a-Release-Bundle
-21069561,162816,2226688,2113024,3843248,46788671,162816,2226688,2113024,3843248,831180,8460084,9131876,25268287,162816,2226688,2113024,3843248,759148,285668,2053044,15935912,3843248
+21069561,162816,2226688,2113024,3843248,47058467,164352,2228224,2103808,3844692,872920,8410108,9426332,25268287,162816,2226688,2113024,3843248,759148,285668,2053044,15935912,3843248 |
|
I think once #4153 is in, hopefully the apk size test should not fail anymore. |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
pjcollins
force-pushed
the
azp-check-apk-sizes
branch
from
ee29ec8
to
20bef54
Compare
|
We should analyze when the XF test apk sizes changed. |
|
@radekdoulik Could you link me to the Jenkins page which contains the APK size information? I'm not exactly sure how to track this one down and how to determine if we've seen the same increase in the XF test app on Jenkins. |
The page is here https://jenkins.mono-project.com/view/Xamarin.Android/job/xamarin-android/plot/Tests%20size/ Unfortunately we are missing sizes between builds 776 and 1029. So I had to download all the xa-test-results*zip files and extract the csv files from there :-) |
|
For the record, here is a chart of master builds which changed the size
|
|
Interestingly the nice drop in build 960 is a mono bump to |
|
I have created a new issue for the AOT apk size increase, so that we don't forget about it. #4169 Otherwise the PR looks good to me. |
This target was overlooked in the initial split up and porting efforts
for our APK tests. This target calculates the sizes of or test APKs and
compares them against a baseline. This change ensures that we will
report a failure in our Azure Pipelines build if any project's APK size
drifts too far from the baseline.