Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[mono][ci] Improve CI perf by excluding intermediate dir from iOS app bundles #94304

Merged
merged 1 commit into from
Nov 3, 2023
Merged

[mono][ci] Improve CI perf by excluding intermediate dir from iOS app bundles #94304

merged 1 commit into from
Nov 3, 2023

Conversation

ivanpovazan
Copy link
Member

@ivanpovazan ivanpovazan commented Nov 2, 2023 •
edited
Loading

Description

This PR improves CI performance for ios/tvos lanes across different pipelines.

When we introduced AOT compilation on Helix, by mistake, we started including intermediate build directories in the app bundles.

This negatively affected:

  • size of the generated bundle
  • time to deploy the bundle to the device for testing

With this PR we improve this in the following way:

  • the app bundle size shrinks about 5,7 x
  • time to deploy the app is improved about 2,8 x

Outcome

It is not uncommon that we sometimes experience timeouts on Apple Helix queues, this fix should hopefully positively impact those issues (eg: #93784 ) and reduce the cost of running tests on ios/tvos devices.

The impact is not that significant on runtime pipeline as we are not running the full test-suite on this pipeline.
However on runtime-extra-platforms the impact is much more noticeable as it can be seen in the breakdown below (expand the dedicated analyses sections).

Detailed analysis

Here is the breakdown of comparisons between CI runs on the main branch and on this PR, for both runtime pipeline and runtime-extra-platforms

Deployment time

Deployment time

As I was not able to measure this on the Helix machine, I ran benchmarks locally (as-if I was building/deploying on a Helix machine) by comparing the time it takes to deploy/run bundles on main branch and with this PR:

System.Runtime.Tests main PR improvement (x)
Bundle size (Mb) 1848,1 312,1
Run 1 (mm:ss) 03:06 01:03
Run 2 (mm:ss) 02:51 01:04
Run 3 (mm:ss) 03:05 01:04
Average (mm:ss) 03:01 01:04 2,84

Bundle size

runtime pipeline

ios-arm64 Release AllSubsets_Mono

Test suite main - AppBundle (Mb) PR - AppBundle size (Mb) improvement (x)
System.Runtime.Tests 1713,8 344,2 5,0
iOS.Device.Aot.Test 1412,1 277,2 5,1
iOS.Device.ExportManagedSymbols.Test 1411,9 277,2 5,1

tvos-arm64 Release AllSubsets_Mono

Test suite main - AppBundle (Mb) PR - AppBundle size (Mb) improvement (x)
System.Runtime.Tests 1713,2 344,1 5,0
runtime-extra-platforms pipeline

ios-arm64 Release AllSubsets_Mono

Test suite main - AppBundle (Mb) PR - AppBundle size (Mb) improvement (x)
System.IO.Hashing.Tests 1600,4 279,5 5,7
System.IO.MemoryMappedFiles.Tests 1668,9 292,9 5,7
System.Runtime.Numerics.Tests 1664,1 290,8 5,7
System.Runtime.Tests 1893,8 344,2 5,5
iOS.Device.Aot.Test 1589,2 277,3 5,7
iOS.Device.ExportManagedSymbols.Test 1589,6 277,2 5,7

tvos-arm64 Release AllSubsets_Mono

Test suite main - AppBundle (Mb) PR - AppBundle size (Mb) improvement (x)
IcuAppLocal.Tests 1634,2 285,9 5,7
Invariant.Tests 1590,8 277,3 5,7
Microsoft.Bcl.AsyncInterfaces.Tests 1595,5 278,5 5,7
Microsoft.Bcl.Cryptography.Tests 1590,7 277,3 5,7
Microsoft.Bcl.Numerics.Tests 1588,5 277,1 5,7
Microsoft.Bcl.TimeProvider.Tests 1591,9 277,8 5,7
Microsoft.Extensions.Configuration.Binder.Tests 1618,3 283,4 5,7
Microsoft.Extensions.Configuration.CommandLine.Tests 1594,8 278,5 5,7
Microsoft.Extensions.Configuration.EnvironmentVariables.Tests 1594,9 278,5 5,7
Microsoft.Extensions.Configuration.FileExtensions.Tests 1643,4 288,6 5,7
Microsoft.Extensions.Configuration.Functional.Tests 1630,2 286,6 5,7
Microsoft.Extensions.Configuration.Ini.Tests 1601,2 280,1 5,7
Microsoft.Extensions.Configuration.Json.Tests 1602,1 280,3 5,7
Microsoft.Extensions.Configuration.Tests 1642,2 288,1 5,7
Microsoft.Extensions.Configuration.UserSecrets.Tests 1599,8 279,9 5,7
Microsoft.Extensions.Configuration.Xml.Tests 1626,1 285,4 5,7
Microsoft.Extensions.DependencyModel.Tests 1751,9 310 5,7
Microsoft.Extensions.Diagnostics.Abstractions.Tests 1617,8 284,1 5,7
Microsoft.Extensions.Diagnostics.Tests 1672,1 294,2 5,7
Microsoft.Extensions.FileProviders.Composite.Tests 1634,9 286,5 5,7
Microsoft.Extensions.FileProviders.Physical.Tests 1652 290,4 5,7
Microsoft.Extensions.FileSystemGlobbing.Tests 1594,4 278,3 5,7
Microsoft.Extensions.HostFactoryResolver.Tests 1653 292,2 5,7
Microsoft.Extensions.Hosting.Systemd.Tests 1652,7 291,9 5,7
Microsoft.Extensions.Http.Tests 1690,4 299 5,7
Microsoft.Extensions.Logging.Console.Tests 1683,1 296,6 5,7
Microsoft.Extensions.Logging.Testing.Tests 1624,7 285,3 5,7
Microsoft.Extensions.Logging.Tests 1691,4 299,5 5,6
Microsoft.Extensions.Options.Tests 1657 292,9 5,7
Microsoft.Extensions.Primitives.Tests 1641,1 287,8 5,7
Microsoft.VisualBasic.Core.Tests 1671,2 292,7 5,7
Microsoft.Win32.Primitives.Tests 1589,1 277 5,7
Microsoft.XmlSerializer.Generator.Tests 1589,3 277,8 5,7
System.AppContext.Tests 1588,4 277 5,7
System.Buffers.Tests 1640 287,3 5,7
System.CodeDom.Tests 1679,8 294,3 5,7
System.Collections.Concurrent.Tests 1686 296,3 5,7
System.Collections.Immutable.Tests 1887,5 340,4 5,5
System.Collections.NonGeneric.Tests 1647,8 288,6 5,7
System.Collections.Specialized.Tests 1598,3 279,3 5,7
System.Collections.Tests 1708,5 302,4 5,6
System.ComponentModel.Annotations.Tests 1645,9 288 5,7
System.ComponentModel.Composition.Registration.Tests 1614,1 283,1 5,7
System.ComponentModel.EventBasedAsync.Tests 1638 286,4 5,7
System.ComponentModel.Primitives.Tests 1591,2 277,6 5,7
System.ComponentModel.Tests 1588,3 277,2 5,7
System.ComponentModel.TypeConverter.Tests 1713,7 302 5,7
System.Composition.AttributeModel.Tests 1604,8 280,9 5,7
System.Composition.Convention.Tests 1599,2 279,5 5,7
System.Composition.Hosting.Tests 1593,4 278,1 5,7
System.Composition.Runtime.Tests 1589,6 277,4 5,7
System.Composition.TypedParts.Tests 1597,7 279,2 5,7
System.Configuration.ConfigurationManager.Tests 1661,1 292 5,7
System.Console.Manual.Tests 1590,6 277 5,7
System.Console.Tests 1648,5 288,5 5,7
System.Data.Common.Tests 1715,5 301,1 5,7
System.Data.DataSetExtensions.Tests 1591,5 277,6 5,7
System.Data.Odbc.Tests 1598,9 279,3 5,7
System.Diagnostics.Contracts.Tests 1588,4 277 5,7
System.Diagnostics.Debug.Tests 1590,4 277,5 5,7
System.Diagnostics.DiagnosticSource.Switches.Tests 1589,1 277,1 5,7
System.Diagnostics.FileVersionInfo.Tests 1588,7 277,1 5,7
System.Diagnostics.Process.Tests 1660,9 292 5,7
System.Diagnostics.StackTrace.Tests 1637,4 286,2 5,7
System.Diagnostics.TextWriterTraceListener.Tests 1636,7 286,1 5,7
System.Diagnostics.Tools.Tests 1587,6 277 5,7
System.Diagnostics.TraceSource.Config.Tests 1653,9 290,2 5,7
System.Diagnostics.TraceSource.Tests 1658,6 291,2 5,7
System.Drawing.Primitives.Tests 1595,3 278,3 5,7
System.Formats.Asn1.Tests 1599 279,3 5,7
System.Formats.Tar.Manual.Tests 1638,9 286,6 5,7
System.Formats.Tar.Tests 1711,3 303,4 5,6
System.Globalization.Calendars.IOS.Tests 1593,7 276,9 5,8
System.Globalization.Calendars.Tests 1595,1 278,4 5,7
System.Globalization.CalendarsWithConfigSwitch.Tests 1587,1 277 5,7
System.Globalization.Extensions.Tests 1593,6 280,8 5,7
System.Globalization.Extensions.iOS.Tests 1588,8 277,1 5,7
System.Globalization.IOS.Tests 1604,7 296,3 5,4
System.Globalization.Tests 1670,8 297,9 5,6
System.IO.Compression.Brotli.Tests 1701 362,8 4,7
System.IO.Compression.Tests 1760,1 374,4 4,7
System.IO.Compression.ZipFile.Tests 1720,3 365,7 4,7
System.IO.FileSystem.DisabledFileLocking.Tests 1677,7 294,7 5,7
System.IO.FileSystem.DriveInfo.Tests 1588,6 277,1 5,7
System.IO.FileSystem.Manual.Tests 1587,8 277 5,7
System.IO.FileSystem.Primitives.Tests 1588,2 277 5,7
System.IO.FileSystem.Tests 1716,3 302,7 5,7
System.IO.IsolatedStorage.Tests 1597,3 279,6 5,7
System.IO.Packaging.Tests 1596,3 278,8 5,7
System.IO.Pipelines.Tests 1683,9 296,5 5,7
System.IO.Pipes.Tests 1704,5 302 5,6
System.IO.Tests 1697,8 298,6 5,7
System.IO.UnmanagedMemoryStream.Tests 1617,8 283 5,7
System.Linq.Parallel.Tests 1698,9 298,9 5,7
System.Linq.Queryable.Tests 1607,9 281 5,7
System.Linq.Tests 1657,1 290,8 5,7
System.Memory.Data.Tests 1597,6 278,9 5,7
System.Memory.Tests 1731,3 303,1 5,7
System.Net.Http.Enterprise.Tests 1637,1 286,2 5,7
System.Net.Http.Json.Functional.Tests 1620,7 284,1 5,7
System.Net.Http.Json.Unit.Tests 1591,9 277,9 5,7
System.Net.Http.Unit.Tests 1679,6 295,2 5,7
System.Net.HttpListener.Tests 1635,3 286,8 5,7
System.Net.Mail.Functional.Tests 1646 288,1 5,7
System.Net.Mail.Unit.Tests 1596,8 279 5,7
System.Net.NameResolution.Functional.Tests 1653,5 289,7 5,7
System.Net.NameResolution.Pal.Tests 1591,3 277,7 5,7
System.Net.Ping.Functional.Tests 1643 287,6 5,7
System.Net.Primitives.Functional.Tests 1644 287,6 5,7
System.Net.Primitives.Pal.Tests 1591,6 277,5 5,7
System.Net.Primitives.UnitTests.Tests 1595,2 278,2 5,7
System.Net.Security.Enterprise.Tests 1593,5 278,1 5,7
System.Net.Security.Unit.Tests 1596,1 278,7 5,7
System.Net.ServicePoint.Tests 1636,3 285,9 5,7
System.Net.Sockets.Tests 1736,1 307,1 5,7
System.Net.WebClient.Tests 1606,2 280,8 5,7
System.Net.WebHeaderCollection.Tests 1588,6 277,1 5,7
System.Net.WebProxy.Tests 1588,2 277,1 5,7
System.Net.WebSockets.Client.Tests 1643,8 288,8 5,7
System.Net.WebSockets.Tests 1597,5 278,9 5,7
System.Numerics.Tensors.Tests 1613,5 280,3 5,8
System.Numerics.Vectors.Tests 1641,9 284,9 5,8
System.ObjectModel.Tests 1613 281,9 5,7
System.Private.Uri.ExtendedFunctional.Tests 1589,1 277,2 5,7
System.Private.Uri.Functional.Tests 1594,1 278,3 5,7
System.Private.Uri.Unit.Tests 1589,2 277,1 5,7
System.Private.Xml.Tests 1742,5 301,9 5,8
System.Reflection.Context.Tests 1593,2 278,3 5,7
System.Reflection.CoreCLR.Tests 1588,1 277 5,7
System.Reflection.Extensions.Tests 1590,5 277,4 5,7
System.Reflection.Metadata.Tests 1622,3 283,8 5,7
System.Reflection.MetadataLoadContext.Tests 1620,2 283,9 5,7
System.Reflection.TypeExtensions.Tests 1594,1 277,8 5,7
System.Resources.Extensions.Tests 1647,2 290,1 5,7
System.Resources.Reader.Tests 1588,8 277,1 5,7
System.Resources.ResourceManager.Tests 1643,5 287,5 5,7
System.Resources.Writer.Tests 1588,7 277 5,7
System.Runtime.Caching.Tests 1617,7 283,7 5,7
System.Runtime.CompilerServices.Unsafe.Tests 1591,7 277,6 5,7
System.Runtime.CompilerServices.VisualC.Tests 1588,3 277 5,7
System.Runtime.Extensions.Tests 1682,7 294,2 5,7
System.Runtime.Handles.Tests 1587,9 277 5,7
System.Runtime.IOS.Tests 1638 285 5,7
System.Runtime.InteropServices.RuntimeInformation.Tests 1637,3 286,2 5,7
System.Runtime.InteropServices.Tests 1664,1 291,6 5,7
System.Runtime.Intrinsics.Tests 1661,6 292,9 5,7
System.Runtime.Loader.DefaultContext.Tests 1588,6 277 5,7
System.Runtime.Loader.RefEmitLoadContext.Tests 1588,5 277 5,7
System.Runtime.Loader.Tests 1640,3 287 5,7
System.Runtime.ReflectionInvokeEmit.Tests 1680,8 295 5,7
System.Runtime.ReflectionInvokeInterpreted.Tests 1681,9 295,1 5,7
System.Runtime.Serialization.Formatters.Tests 1733,2 310,4 5,6
System.Runtime.Serialization.Json.ReflectionOnly.Tests 1611,1 281,5 5,7
System.Runtime.Serialization.Json.Tests 1610,4 281,4 5,7
System.Runtime.Serialization.Primitives.Tests 1588,3 277 5,7
System.Runtime.Serialization.Schema.Tests 1648,6 288,8 5,7
System.Runtime.Serialization.Xml.ReflectionOnly.Tests 1639,7 288,6 5,7
System.Runtime.Serialization.Xml.Tests 1642,5 289,1 5,7
System.Runtime.Tests 1893,7 344,1 5,5
System.Security.Claims.Tests 1636,6 286 5,7
System.Security.Cryptography.Cose.Tests 1659 290,4 5,7
System.Security.Cryptography.Csp.Tests 1598,8 279,5 5,7
System.Security.Cryptography.Pkcs.Tests 1620,3 284,2 5,7
System.Security.Cryptography.ProtectedData.Tests 1588,6 277,2 5,7
System.Security.Cryptography.Tests 1777,2 317 5,6
System.Security.Cryptography.Xml.Tests 1621,4 284,3 5,7
System.Security.SecureString.Tests 1588,2 277,1 5,7
System.ServiceModel.Syndication.Tests 1615 283,2 5,7
System.Text.Encoding.CodePages.Tests 1637,6 286,3 5,7
System.Text.Encoding.Extensions.Tests 1587,7 277,1 5,7
System.Text.Encoding.Tests 1732,9 305,6 5,7
System.Text.Encodings.Web.Tests 1603,3 280,7 5,7
System.Text.Json.Tests 2630,4 496,2 5,3
System.Text.RegularExpressions.Unit.Tests 1603,5 280,1 5,7
System.Threading.Overlapped.Tests 1637,7 286,3 5,7
System.Threading.RateLimiting.Tests 1637,1 287,1 5,7
System.Threading.Tasks.Dataflow.Tests 1735,2 306,9 5,7
System.Threading.Tasks.Extensions.Tests 1668,6 292,6 5,7
System.Threading.Tasks.Tests 1679,2 294,5 5,7
System.Threading.Tests 1650,8 288,8 5,7
System.Threading.Thread.Tests 1639,1 286,7 5,7
System.Threading.ThreadPool.Tests 1641 287 5,7
System.Threading.Timer.Tests 1641,5 287,1 5,7
System.Transactions.Local.Tests 1644,5 287,7 5,7
System.ValueTuple.Tests 1605,2 280,1 5,7
System.Web.HttpUtility.Tests 1589,2 277,1 5,7
System.Xml.Linq.Axes.Tests 1588,3 277 5,7
System.Xml.Linq.Events.Tests 1608,3 282,6 5,7
System.Xml.Linq.Misc.Tests 1611,9 283,2 5,7
System.Xml.Linq.Properties.Tests 1609,5 282,7 5,7
System.Xml.Linq.SDMSample.Tests 1595,3 278,3 5,7
System.Xml.Linq.Streaming.Tests 1607,4 282,2 5,7
System.Xml.Linq.TreeManipulation.Tests 1614,2 283,6 5,7
System.Xml.Linq.xNodeBuilder.Tests 1613,7 283,6 5,7
System.Xml.Linq.xNodeReader.Tests 1613,3 283,6 5,7
System.Xml.Schema.Extensions.Tests 1588,6 277 5,7
System.Xml.XmlSerializer.ReflectionOnly.Tests 1609,9 281,4 5,7
tvOS.Device.Aot.Test 1588,7 277 5,7

@ivanpovazan ivanpovazan added NO-MERGE The PR is not ready for merge yet (see discussion for detailed reasons) NO-REVIEW Experimental/testing PR, do NOT review it labels Nov 2, 2023
@ivanpovazan ivanpovazan self-assigned this Nov 2, 2023
@ghost
Copy link

ghost commented Nov 2, 2023

Tagging subscribers to this area: @directhex
See info in area-owners.md if you want to be subscribed.

Issue Details

Testing the PR on Helix for now.
TODO: Add description.

Author: ivanpovazan
Assignees: ivanpovazan
Labels:

NO-MERGE, NO-REVIEW, area-Build-mono, area-Infrastructure-mono
Milestone: -

@ivanpovazan
Copy link
Member Author

/azp run runtime-ioslike

@azure-pipelines Azure Pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@build-analysis build-analysis bot mentioned this pull request Nov 2, 2023
Closed
@ivanpovazan
Copy link
Member Author

/azp run runtime-ioslike

@azure-pipelines Azure Pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@ivanpovazan ivanpovazan changed the title WIP: [mono][ci] Explicitly set intermediate directory path to avoid including it in the app bundle [mono][ci] Explicitly set intermediate directory path to avoid including it in the app bundle Nov 2, 2023
@ivanpovazan ivanpovazan changed the title [mono][ci] Explicitly set intermediate directory path to avoid including it in the app bundle [mono][ci] Improve CI perf by excluding intermediate dir from iOS app bundles Nov 2, 2023
@ivanpovazan
Copy link
Member Author

/azp run runtime-ioslike

@azure-pipelines Azure Pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@ivanpovazan ivanpovazan marked this pull request as ready for review November 3, 2023 00:15
@ivanpovazan ivanpovazan removed NO-MERGE The PR is not ready for merge yet (see discussion for detailed reasons) NO-REVIEW Experimental/testing PR, do NOT review it labels Nov 3, 2023
@ivanpovazan
Copy link
Member Author

The failures are unrelated.

@ivanpovazan ivanpovazan merged commit aac9923 into dotnet:main Nov 3, 2023
104 of 107 checks passed
@ivanpovazan ivanpovazan deleted the helix-ios-improvements branch November 3, 2023 15:27
@ghost ghost locked as resolved and limited conversation to collaborators Dec 3, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@steveisok steveisok steveisok approved these changes

@kotlarmilos kotlarmilos kotlarmilos approved these changes

@akoeplinger akoeplinger Awaiting requested review from akoeplinger

@SamMonoRT SamMonoRT Awaiting requested review from SamMonoRT

@marek-safar marek-safar Awaiting requested review from marek-safar marek-safar is a code owner

Assignees

@ivanpovazan ivanpovazan

Labels
area-Infrastructure-mono
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@ivanpovazan @steveisok @kotlarmilos