[msbuild] Don't process assemblies over and over again in the UnpackLibraryResources target. Fixes #16377. #16416

dellis1972 · 2022-10-24T13:27:25Z

MSbuild has the ability to run Target partially. However in order to make sure we can use this, both the Inputs and Outputs need to have a 1-1 mapping. MSBuild will then use this mapping to figure out which files have actually changed.
Then it will call the Target with only those files.

In Xamarin.Android we make use of stamp files to provide this 1-1 mapping. We have a pre Target which will calculate an ItemGroup which will include a StampFile piece of metadata. We then use this new ItemGroup for both the Inputs and Outputs, thus meeting the 1-1 requirement.

This PR updates the _UnpackLibraryResources to run partially. This will hopefully reduce the build time on incremental builds.

dellis1972 · 2022-10-24T13:28:03Z

This is still a work in progress,

msbuild/Xamarin.Shared/Xamarin.Shared.targets

rolfbjarne · 2022-11-08T15:15:36Z

It fails like this:

/Users/builder/azdo/_work/1/s/xamarin-macios/builds/downloads/dotnet-sdk-7.0.100-rtm.22531.3/packs/Microsoft.iOS.Sdk/16.1.530-ci.pr.gh16416/tools/msbuild/iOS/Xamarin.Shared.targets(1808,3): error MSB3371: The file "obj/Debug/net7.0-ios/ios-arm64/resourcestamps/Users/builder/azdo/_work/1/s/xamarin-macios/builds/downloads/dotnet-sdk-7.0.100-rtm.22531.3/packs/Microsoft.NETCore.App.Ref/7.0.0-rtm.22511.4/ref/net7.0/Microsoft.CSharp.dll.stamp" cannot be created. Could not find a part of the path '/Users/builder/azdo/_work/1/s/xamarin-macios/msbuild/Xamarin.HotRestart.PreBuilt/Xamarin.PreBuilt.iOS/obj/Debug/net7.0-ios/ios-arm64/resourcestamps/Users/builder/azdo/_work/1/s/xamarin-macios/builds/downloads/dotnet-sdk-7.0.100-rtm.22531.3/packs/Microsoft.NETCore.App.Ref/7.0.0-rtm.22511.4/ref/net7.0/Microsoft.CSharp.dll.stamp'. [/Users/builder/azdo/_work/1/s/xamarin-macios/msbuild/Xamarin.HotRestart.PreBuilt/Xamarin.PreBuilt.iOS/Xamarin.PreBuilt.iOS.csproj]
/Users/builder/azdo/_work/1/s/xamarin-macios/builds/downloads/dotnet-sdk-7.0.100-rtm.22531.3/packs/Microsoft.iOS.Sdk/16.1.530-ci.pr.gh16416/tools/msbuild/iOS/Xamarin.Shared.targets(1808,3): error MSB3371: The file "obj/Debug/net7.0-ios/ios-arm64/resourcestamps/Users/builder/azdo/_work/1/s/xamarin-macios/builds/downloads/dotnet-sdk-7.0.100-rtm.22531.3/packs/Microsoft.iOS.Ref/16.1.530-ci.pr.gh16416/ref/net7.0/Microsoft.iOS.dll.stamp" cannot be created. Could not find a part of the path '/Users/builder/azdo/_work/1/s/xamarin-macios/msbuild/Xamarin.HotRestart.PreBuilt/Xamarin.PreBuilt.iOS/obj/Debug/net7.0-ios/ios-arm64/resourcestamps/Users/builder/azdo/_work/1/s/xamarin-macios/builds/downloads/dotnet-sdk-7.0.100-rtm.22531.3/packs/Microsoft.iOS.Ref/16.1.530-ci.pr.gh16416/ref/net7.0/Microsoft.iOS.dll.stamp'. [/Users/builder/azdo/_work/1/s/xamarin-macios/msbuild/Xamarin.HotRestart.PreBuilt/Xamarin.PreBuilt.iOS/Xamarin.PreBuilt.iOS.csproj]
/Users/builder/azdo/_work/1/s/xamarin-macios/builds/downloads/dotnet-sdk-7.0.100-rtm.22531.3/packs/Microsoft.iOS.Sdk/16.1.530-ci.pr.gh16416/tools/msbuild/iOS/Xamarin.Shared.targets(1808,3): error MSB3371: The file "obj/Debug/net7.0-ios/ios-arm64/resourcestamps/Users/builder/azdo/_work/1/s/xamarin-macios/builds/downloads/dotnet-sdk-7.0.100-rtm.22531.3/packs/Microsoft.NETCore.App.Ref/7.0.0-rtm.22511.4/ref/net7.0/Microsoft.VisualBasic.Core.dll.stamp" cannot be created. Could not find a part of the path '/Users/builder/azdo/_work/1/s/xamarin-macios/msbuild/Xamarin.HotRestart.PreBuilt/Xamarin.PreBuilt.iOS/obj/Debug/net7.0-ios/ios-arm64/resourcestamps/Users/builder/azdo/_work/1/s/xamarin-macios/builds/downloads/dotnet-sdk-7.0.100-rtm.22531.3/packs/Microsoft.NETCore.App.Ref/7.0.0-rtm.22511.4/ref/net7.0/Microsoft.VisualBasic.Core.dll.stamp'. [/Users/builder/azdo/_work/1/s/xamarin-macios/msbuild/Xamarin.HotRestart.PreBuilt/Xamarin.PreBuilt.iOS/Xamarin.PreBuilt.iOS.csproj]
...

dellis1972 · 2022-11-09T16:04:44Z

It fails like this:

/Users/builder/azdo/_work/1/s/xamarin-macios/builds/downloads/dotnet-sdk-7.0.100-rtm.22531.3/packs/Microsoft.iOS.Sdk/16.1.530-ci.pr.gh16416/tools/msbuild/iOS/Xamarin.Shared.targets(1808,3): error MSB3371: The file "obj/Debug/net7.0-ios/ios-arm64/resourcestamps/Users/builder/azdo/_work/1/s/xamarin-macios/builds/downloads/dotnet-sdk-7.0.100-rtm.22531.3/packs/Microsoft.NETCore.App.Ref/7.0.0-rtm.22511.4/ref/net7.0/Microsoft.CSharp.dll.stamp" cannot be created. Could not find a part of the path '/Users/builder/azdo/_work/1/s/xamarin-macios/msbuild/Xamarin.HotRestart.PreBuilt/Xamarin.PreBuilt.iOS/obj/Debug/net7.0-ios/ios-arm64/resourcestamps/Users/builder/azdo/_work/1/s/xamarin-macios/builds/downloads/dotnet-sdk-7.0.100-rtm.22531.3/packs/Microsoft.NETCore.App.Ref/7.0.0-rtm.22511.4/ref/net7.0/Microsoft.CSharp.dll.stamp'. [/Users/builder/azdo/_work/1/s/xamarin-macios/msbuild/Xamarin.HotRestart.PreBuilt/Xamarin.PreBuilt.iOS/Xamarin.PreBuilt.iOS.csproj]
/Users/builder/azdo/_work/1/s/xamarin-macios/builds/downloads/dotnet-sdk-7.0.100-rtm.22531.3/packs/Microsoft.iOS.Sdk/16.1.530-ci.pr.gh16416/tools/msbuild/iOS/Xamarin.Shared.targets(1808,3): error MSB3371: The file "obj/Debug/net7.0-ios/ios-arm64/resourcestamps/Users/builder/azdo/_work/1/s/xamarin-macios/builds/downloads/dotnet-sdk-7.0.100-rtm.22531.3/packs/Microsoft.iOS.Ref/16.1.530-ci.pr.gh16416/ref/net7.0/Microsoft.iOS.dll.stamp" cannot be created. Could not find a part of the path '/Users/builder/azdo/_work/1/s/xamarin-macios/msbuild/Xamarin.HotRestart.PreBuilt/Xamarin.PreBuilt.iOS/obj/Debug/net7.0-ios/ios-arm64/resourcestamps/Users/builder/azdo/_work/1/s/xamarin-macios/builds/downloads/dotnet-sdk-7.0.100-rtm.22531.3/packs/Microsoft.iOS.Ref/16.1.530-ci.pr.gh16416/ref/net7.0/Microsoft.iOS.dll.stamp'. [/Users/builder/azdo/_work/1/s/xamarin-macios/msbuild/Xamarin.HotRestart.PreBuilt/Xamarin.PreBuilt.iOS/Xamarin.PreBuilt.iOS.csproj]
/Users/builder/azdo/_work/1/s/xamarin-macios/builds/downloads/dotnet-sdk-7.0.100-rtm.22531.3/packs/Microsoft.iOS.Sdk/16.1.530-ci.pr.gh16416/tools/msbuild/iOS/Xamarin.Shared.targets(1808,3): error MSB3371: The file "obj/Debug/net7.0-ios/ios-arm64/resourcestamps/Users/builder/azdo/_work/1/s/xamarin-macios/builds/downloads/dotnet-sdk-7.0.100-rtm.22531.3/packs/Microsoft.NETCore.App.Ref/7.0.0-rtm.22511.4/ref/net7.0/Microsoft.VisualBasic.Core.dll.stamp" cannot be created. Could not find a part of the path '/Users/builder/azdo/_work/1/s/xamarin-macios/msbuild/Xamarin.HotRestart.PreBuilt/Xamarin.PreBuilt.iOS/obj/Debug/net7.0-ios/ios-arm64/resourcestamps/Users/builder/azdo/_work/1/s/xamarin-macios/builds/downloads/dotnet-sdk-7.0.100-rtm.22531.3/packs/Microsoft.NETCore.App.Ref/7.0.0-rtm.22511.4/ref/net7.0/Microsoft.VisualBasic.Core.dll.stamp'. [/Users/builder/azdo/_work/1/s/xamarin-macios/msbuild/Xamarin.HotRestart.PreBuilt/Xamarin.PreBuilt.iOS/Xamarin.PreBuilt.iOS.csproj]
...

Yup I accidently used %(Identity) rather than %(FileName) for the stamp file metadata. The Identity contains the full path 🤦

github-actions · 2022-11-11T10:10:32Z

⚠️ Your code has been reformatted. ⚠️

If this is not desired, add the actions-disable-autoformat label, and revert the reformatting commit.

If files unrelated to your change were modified, try reverting the reformatting commit + merging with the target branch (and push those changes).

msbuild/Xamarin.Shared/Xamarin.Shared.targets

vs-mobiletools-engineering-service2 · 2022-11-16T15:06:54Z

📚 [PR Build] Artifacts 📚

Artifacts were not provided.

Pipeline on Agent XAMBOT-1161.Monterey'
Hash: cee7d283359651e1431d51d451b5c1cc1dafbf7d [PR build]

vs-mobiletools-engineering-service2 · 2022-11-16T15:11:53Z

💻 [PR Build] Tests on macOS M1 - Mac Big Sur (11.5) passed 💻

✅ All tests on macOS M1 - Mac Big Sur (11.5) passed.

Pipeline on Agent
Hash: cee7d283359651e1431d51d451b5c1cc1dafbf7d [PR build]

vs-mobiletools-engineering-service2 · 2022-11-16T15:19:55Z

✅ API diff for current PR / commit

Legacy Xamarin (No breaking changes)

~~iOS~~ (no change detected)
~~tvOS~~ (no change detected)
~~watchOS~~ (no change detected)
~~macOS~~ (no change detected)

NET (empty diffs)

~~iOS~~: (empty diff detected)
~~tvOS~~: (empty diff detected)
~~MacCatalyst~~: (empty diff detected)
~~macOS~~: (empty diff detected)

❗ API diff vs stable (Breaking changes)

Legacy Xamarin (:heavy_exclamation_mark: Breaking changes :heavy_exclamation_mark:)

iOS: vsdrops gist (:heavy_exclamation_mark: Breaking changes :heavy_exclamation_mark:)
tvOS: vsdrops gist (No breaking changes)
watchOS: vsdrops gist (No breaking changes)
macOS: vsdrops gist (No breaking changes)

.NET (:heavy_exclamation_mark: Breaking changes :heavy_exclamation_mark:)

iOS: vsdrops gist (:heavy_exclamation_mark: Breaking changes :heavy_exclamation_mark:)
tvOS: vsdrops gist (No breaking changes)
MacCatalyst: vsdrops gist (No breaking changes)
macOS: vsdrops gist (No breaking changes)
Microsoft.iOS vs Microsoft.MacCatalyst: vsdrops gist

Legacy Xamarin (stable) vs .NET

iOS: vsdrops gist
tvOS: vsdrops gist
macOS: vsdrops gist

✅ Generator diff

Generator diff is empty

Pipeline on Agent
Hash: cee7d283359651e1431d51d451b5c1cc1dafbf7d [PR build]

vs-mobiletools-engineering-service2 · 2022-11-16T19:30:57Z

🚀 [CI Build] Test results 🚀

Test results

✅ All tests passed on VSTS: simulator tests.

🎉 All 223 tests passed 🎉

Tests counts

✅ bcl: All 69 tests passed. Html Report (VSDrops) Download
✅ cecil: All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests: All 1 tests passed. Html Report (VSDrops) Download
✅ fsharp: All 7 tests passed. Html Report (VSDrops) Download
✅ framework: All 8 tests passed. Html Report (VSDrops) Download
✅ generator: All 2 tests passed. Html Report (VSDrops) Download
✅ interdependent_binding_projects: All 7 tests passed. Html Report (VSDrops) Download
✅ install_source: All 1 tests passed. Html Report (VSDrops) Download
✅ introspection: All 8 tests passed. Html Report (VSDrops) Download
✅ linker: All 65 tests passed. Html Report (VSDrops) Download
✅ mac_binding_project: All 1 tests passed. Html Report (VSDrops) Download
✅ mmp: All 2 tests passed. Html Report (VSDrops) Download
✅ mononative: All 12 tests passed. Html Report (VSDrops) Download
✅ monotouch: All 23 tests passed. Html Report (VSDrops) Download
✅ msbuild: All 2 tests passed. Html Report (VSDrops) Download
✅ mtouch: All 1 tests passed. Html Report (VSDrops) Download
✅ xammac: All 3 tests passed. Html Report (VSDrops) Download
✅ xcframework: All 8 tests passed. Html Report (VSDrops) Download
✅ xtro: All 2 tests passed. Html Report (VSDrops) Download

Pipeline on Agent
Hash: cee7d283359651e1431d51d451b5c1cc1dafbf7d [PR build]

rolfbjarne · 2023-01-13T23:07:01Z

/sudo backport release/7.0.2xx

vs-mobiletools-engineering-service2 · 2023-01-13T23:07:23Z

Backport Job to branch release/7.0.2xx Created! The magic is happening here

vs-mobiletools-engineering-service2 · 2023-01-13T23:10:36Z

Hooray! Backport succeeded! Please see https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=7195173 for more details.

…ain in the UnpackLibraryResources target. Fixes #16377. (#17234) Fixes #16377 MSbuild has the ability to run `Target` partially. However in order to make sure we can use this, both the `Inputs` and `Outputs` need to have a 1-1 mapping. MSBuild will then use this mapping to figure out which files have actually changed. Then it will call the Target with only those files. In Xamarin.Android we make use of stamp files to provide this 1-1 mapping. We have a `pre` Target which will calculate an ItemGroup which will include a StampFile piece of metadata. We then use this new ItemGroup for both the `Inputs` and `Outputs`, thus meeting the 1-1 requirement. This PR updates the `_UnpackLibraryResources` to run partially. This will hopefully reduce the build time on incremental builds. Backport of #16416 Co-authored-by: Dean Ellis <dellis1972@googlemail.com> Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>

dellis1972 added the enhancement The issue or pull request is an enhancement label Oct 24, 2022

This comment has been minimized.

Sign in to view

jonathanpeppers reviewed Nov 7, 2022

View reviewed changes

msbuild/Xamarin.Shared/Xamarin.Shared.targets Outdated Show resolved Hide resolved

This comment has been minimized.

Sign in to view

dellis1972 force-pushed the Issue16377 branch from a47e186 to ac17c1b Compare November 8, 2022 10:15

This comment has been minimized.

Sign in to view

dellis1972 force-pushed the Issue16377 branch from 48f9f3c to 086e84a Compare November 11, 2022 10:05

This comment has been minimized.

Sign in to view

jonathanpeppers reviewed Nov 11, 2022

View reviewed changes

msbuild/Xamarin.Shared/Xamarin.Shared.targets Outdated Show resolved Hide resolved

This comment has been minimized.

Sign in to view

dellis1972 force-pushed the Issue16377 branch from 8726a13 to 41d0f6c Compare November 12, 2022 22:34

This comment has been minimized.

Sign in to view

Initial Commit

213b41d

dellis1972 added 2 commits November 14, 2022 10:24

Fix the Metadata Item to use %(FileName)

031fbe0

Move the MakeDir call

a59b692

dellis1972 force-pushed the Issue16377 branch from 41d0f6c to a59b692 Compare November 14, 2022 10:26

dellis1972 requested a review from jonathanpeppers November 14, 2022 10:26

jonathanpeppers approved these changes Nov 14, 2022

View reviewed changes

msbuild/Xamarin.Shared/Xamarin.Shared.targets Outdated Show resolved Hide resolved

This comment has been minimized.

Sign in to view

Fix pathing issue

2559156

This comment has been minimized.

Sign in to view

dellis1972 marked this pull request as ready for review November 16, 2022 09:47

dellis1972 requested review from emaf, mauroa and rolfbjarne as code owners November 16, 2022 09:47

rolfbjarne approved these changes Nov 16, 2022

View reviewed changes

msbuild/Xamarin.Shared/Xamarin.Shared.targets Outdated Show resolved Hide resolved

Update Touch to use the Output parameter

cee7d28

emaf approved these changes Nov 17, 2022

View reviewed changes

rolfbjarne changed the title ~~Don't process assemblies over and over again in the UnpackLibraryResources target~~ [msbuild] Don't process assemblies over and over again in the UnpackLibraryResources target. Fixes #16377. Nov 17, 2022

rolfbjarne merged commit 30405fd into xamarin:main Nov 17, 2022

vs-mobiletools-engineering-service2 mentioned this pull request Jan 13, 2023

[release/7.0.2xx] [msbuild] Don't process assemblies over and over again in the UnpackLibraryResources target. Fixes #16377. #17234

Merged

[msbuild] Don't process assemblies over and over again in the UnpackLibraryResources target. Fixes #16377. #16416

[msbuild] Don't process assemblies over and over again in the UnpackLibraryResources target. Fixes #16377. #16416

Conversation

dellis1972 commented Oct 24, 2022

dellis1972 commented Oct 24, 2022

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

rolfbjarne commented Nov 8, 2022

dellis1972 commented Nov 9, 2022

github-actions bot commented Nov 11, 2022

⚠️ Your code has been reformatted. ⚠️

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

vs-mobiletools-engineering-service2 commented Nov 16, 2022

📚 [PR Build] Artifacts 📚

vs-mobiletools-engineering-service2 commented Nov 16, 2022

💻 [PR Build] Tests on macOS M1 - Mac Big Sur (11.5) passed 💻

vs-mobiletools-engineering-service2 commented Nov 16, 2022

✅ API diff for current PR / commit

❗ API diff vs stable (Breaking changes)

✅ Generator diff

vs-mobiletools-engineering-service2 commented Nov 16, 2022

🚀 [CI Build] Test results 🚀

Test results

Tests counts

rolfbjarne commented Jan 13, 2023

vs-mobiletools-engineering-service2 commented Jan 13, 2023

vs-mobiletools-engineering-service2 commented Jan 13, 2023