Alpine Linux dotnet31 / dotnet5 package

[ayakael]

I've been working on an Alpine Linux package for dotnet31, dotnet5 and dotnet6 package for the past few months. dotnet31 and dotnet5 are finally ready, and is in the process of being merged here:

dotnet31: https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/28197
dotnet5: https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/28195

Build for dotnet31 and dotnet5 required many patches, as the build process wasn't without kinks on Alpine. Below you'll find the patch notes for each build. For upstreaming these fixes, I'm going ahead and sending merge requests to their respective repos.

Patch notes for dotnet31

## application-insights_fix-net40-location.patch (see microsoft/ApplicationInsights-dotnet#2498)
   Because paths on Linux are case sensitive. Applications insights looks
   for <dotnet/applications-insights>/src/Core/Managed/net40 in net40 
   while the repo has a Net40. Patch moves whatever is in Net40 to net40.
## build_coreclr-tools-path.patch (see dotnet/source-build#2697)
   Various steps in the build process looks for ilasm in ildasm in 
   $builddir/src/dotnet-3.1/Tools/source-built/coreclr-tools/x64 while they are
   built to $builddir/src/dotnet-3.1/Tools/source-built/coreclr-tools.
   This patches <dotnet/source-build> to look for ilasm and ildasm
   in correct path.
## build_darc-fix-alpine.patch
   Darc has a segmentation fault on Alpine due to not chosing the correct
   binary architecture. This patch deletes all the wrong ones so that it
   is forced to chose the correct one. See issue dotnet/source-build dotnet/installer#1868
   This issue only occurs when using dotnet-stage0-runtime, thus libunwind is
   suspected to be the issue as stage0 wasn't built with system libunwind.
## coreclr_fix-build-on-Alpine-edge-45352.patch (see dotnet/coreclr#28226)
   Fixes a build error for Alpine Linux. It has since been fixed in dotnet-5.0,
   see issue dotnet/runtime 45352.
## corefx_fix-build-clang10.patch (see dotnet/corefx#43130)
   Fixes a build error for clang 10
## core-setup_rid-plat-generation-on-alpine-fix.patch (see dotnet/core-setup#9195)
   <dotnet/core-setup>/src/corehost/build.sh generates wrong plat_rid on Alpine
   Generated RID is expected to be alpine.x.xx-xx, while the computed RID
   is linux-musl-xxx. This patches it by matching what's expected.
## coreclr_non-portable-distrorid-fix-alpine.patch (see dotnet/coreclr#28227)
   <dotnet/coreclr>/init-distro-id.sh generates incorrect DistroRid on Alpine.
   While the expected DistroRid (alpine-x.xx-xxx) should only include macro
   version, init-distro-id.sh includes the micro version. This patches it
   to cut off the trailing subversion off of VERSION_ID by treating it 
   the same way RHEL's VERSION_ID is treated.
## core-sdk_generate-layout-core-setup-blob-path-fix.patch
   <dotnet/core-setup> blobs are written to $builddir/artifacts/obj/x64/Release/
   blobs/Runtime/3.1.20-alpine.3.15 while installer expects them in 3.1.20.
   This patches <dotnet/installer>/src/redist/targets/GenerateLayout.targets
   to point to actual path.

Patch notes for dotnet5:

## application-insights_fix-net40-location.patch (see microsoft/ApplicationInsights-dotnet#2498)
   Because paths on Linux are case sensitive. Applications insights looks
   for <dotnet/applicationsinsights>/src/Core/Managed/net40 in net40 
   while the repo has a Net40. Patch moves whatever is in Net40 to net40
## build_darc-use-available-runtime.patch (see dotnet/source-build#2698)
   Default version of Darc needs dotnet 3.1 runtimes, and fails to run on
   Alpine. This patch fixes this by pushing Darc's version and disabling runtime
   retrieval. Can't push it too much, though, as latest version breaks with
   application-insights as it doesn't include a eng/Version.Details.xml file
## build_darc-fix-alpine.patch (see dotnet/source-build#1868)
   Darc has a segmentation fault on Alpine due to not chosing the correct
   binary architecture. This patch deletes all the wrong ones so that it
   is forced to chose the correct one. See issue dotnet/source-build dotnet/installer#1868
   This issue only occurs when using dotnet-stage0-runtime, thus libunwind is
   suspected to be the issue as stage0 wasn't built with system libunwind.
## runtime_add-rid-for-alpine-315.patch (see dotnet/runtime#62938)
   As of version 5.0.12, runtime does not have the RIDs for Alpine Linux 3.15.
   This patch adds them. See dotnet/core issue dotnet/installer#6985 
## runtime_link-order.patch
   For some reason, runtime does not link in the right order. This fixes that.
## runtime_non-portable-distrorid-fix-alpine.patch (see dotnet/runtime#62942)
   Runtime adds the extra subversion in its calculation of Alpine's DistroRID
   when a non-portable build, but does so inconsistently. This creates an error
   when it generates its nuget package.
## runtime_use-latest-microsoftnetcoreplatformspackage.patch
   In stock build, an old version of MicrosoftNETCorePlatformsPackage is used.
   Thus, dotnet5-runtime is shipped without RIDs for alpine 3.13 and up, causing
   a segmentation fault for dotnet projects. Related to dotnet/runtime#24189 and
   dotnet/runtime#50739. This patches eng/Versions.props to use latest version.
## sdk_telemetry-optout.patch
   Optouts of telemetry gathering

I'm open to any comments on the Alpine package. I've heard from various voices that this is something many have been wating for, so I'd like this space to also be a place of discussion in case there's a better way to do certain things on my APKBUILD. I have very little experience with the .NET framework, I'm just someone who likes creating packaging scripts.

For dotnet6, please see #2782

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[ayakael]

According to am11 on dotnet/runtime#62942, future best practice is moving away from non-portable builds given the maintenance headaches it represents for very little upside. I am currently playing around with this on the APKBUILDs, which would slim down the patchset considerably.

[safern]

cc: @dagood

[dagood]

You should consider following the standard package naming (used by Microsoft, Red Hat, and Fedora): https://docs.microsoft.com/en-us/dotnet/core/distribution-packaging. As far as I can tell, the names dotnet5 etc. are being invented by your implementation, and it's missing the distinct runtime and sdk (and ...) packages that people used to the .NET ecosystem expect to be able to install individually.

As for non-portable builds, I haven't worked on .NET since Nov 2021, but dotnet/runtime#62942 seems to me to only be highlighting the Microsoft perspective, not the needs of Linux distros. That thread talks about it as a legacy feature, but last I saw, it is in active use by Red Hat and Fedora. A "portable" build is an unusual concept for a tool that ships in a Linux distro. That said, as far as your work goes: if Alpine doesn't need non-portable builds, you don't need to implement them.

I'll let the source-build triage process get around to this issue for an official response, but putting in my two cents. 😄

[ayakael]

Thanks for your two cents, much appreciated.

You should consider following the standard package naming (used by Microsoft, Red Hat, and Fedora): https://docs.microsoft.com/en-us/dotnet/core/distribution-packaging. As far as I can tell, the names dotnet5 etc. are being invented by your implementation, and it's missing the distinct runtime and sdk (and ...) packages that people used to the .NET ecosystem expect to be able to install individually.

Indeed, I had first followed the standard package naming, but after various conversations it was deemed more desirable to follow Alpine's convention for multiversionned packages (i.e python3 and php8). That said, I have not received a response from any Alpine developpers on that matter, so it could go either way.

As for subpackages, the following are implemented: dotnet-host, dotnet-sdk, dotnet-runtime, dotnet-targeting-pack, aspnet-runtime, aspnet-targeting-pack, and netstandard-targeting-pack.

As for non-portable builds, I haven't worked on .NET since Nov 2021, but dotnet/runtime#62942 seems to me to only be highlighting the Microsoft perspective, not the needs of Linux distros. That thread talks about it as a legacy feature, but last I saw, it is in active use by Red Hat and Fedora. A "portable" build is an unusual concept for a tool that ships in a Linux distro. That said, as far as your work goes: if Alpine doesn't need non-portable builds, you don't need to implement them.

Quite so! I found the position odd, given that it seems counter-intuitive for portable builds on non-portable packages. Why have portable bits generated for non-portable applications? On the other hand, given the headache RIDs bring to the table for package maintainance, why bother when package managers insure library installation? Just yesterday, I had a reoccurence of bug dotnet/runtime#50739, causing an odd segmentation fault. It was due to source-build using version 5.0.0 of MicrosoftNETCorePlatformsPackage rather than the up-to-date (and patched by myself for Alpine 3.15) 5.0.4. Version 5.0.0 only had RIDs till 5.12, causing an obscure segmentation fault with error code 0x000000000000e5c6 in ?? () on any runtime builds past that version.

Could you expand on the benefits of non-portable builds? I'm partial to keeping the non-portable builds, I'm just still bewildered on why they exist given that, as far as I can tell, no distro-specific optimizations have been implemented.

With that said, portable builds bring their own set of obscure build failures on Alpine, so my APKBUILD will maintain the non-portable route. I suppose the ideal workflow is for portable bits to be generated for use by the bootstrap subpackage, but keep non-portable bits for use by the distros-specific subpackage.

[dagood]

As for subpackages, the following are implemented: dotnet-host, dotnet-sdk, dotnet-runtime, dotnet-targeting-pack, aspnet-runtime, aspnet-targeting-pack, and netstandard-targeting-pack.

Ah! Cool, I didn't spot those. I'm not sure how they work (not super familiar with this packaging system), but that sounds right. 👍

Could you expand on the benefits of non-portable builds?

My understanding has been that dynamic linking has performance benefits over dlopen, but I can't find any info on this.

A distro tooling benefit IIRC is with dynamic linking, standard packaging tools determine what the output binaries depend on by inspecting them. This allows package dependencies to be set up automatically, rather than manually tweaking it while figuring out how to build the package. (And then maintained over time.)

Really, Red Hat and Fedora would be the right people to ask. I never dug into these requirements very far, but it was always a high priority for the source-build team to keep non-portable builds working.

Source-build team: I think it would be great to maintain this info in some doc. 😄 dotnet/runtime and other folks could use this to justify this kind of source-build-only requirement, and keep track of new developments (if there have been any).

I suppose the ideal workflow is for portable bits to be generated for use by the bootstrap subpackage, but keep non-portable bits for use by the distros-specific subpackage.

I think a portable SDK (prebuilt by Microsoft) would be used at the start, but the first SDK you build (to put in a bootstrap package) would make sense as a non-portable SDK. That actually seems best, so that it matches the final output and resembles a "real" N-1 => N build that will be done in future patch versions. Maybe this is what you're saying--I'm rusty on bootstrap terminology. Here's a doc on how the source-build team tends to think about it: https://github.com/dotnet/source-build/blob/main/Documentation/EngineeringPlanLinuxDistroRepos.md. (Note: hasn't been updated in a while, but the broad requirements haven't changed much AFAIK.)

[crummel]

Hi @ayakael, thanks for working on this, it's great to see! As I understand it Davis is correct on the benefits of non-portable vs portable, but @omajid may have more details. Portable bits are generated even in non-portable builds because ASP.NET relies on having the portable runtime available. This is something which I believe is fixed in the 6.0 SDK. I think your segfault is probably due to source-build having an old version of Microsoft.NETCore.Platforms from https://github.com/dotnet/source-build-reference-packages - we tend to only update these when we run into issues. We're working on making it easier for everyone to contribute to SBRP so you'll be able to submit fixes like that yourself, but if you have it working now that should be fine.

Thanks as well for upstreaming all your work - we really appreciate it! Just ping me or dotnet/source-build-internal on the PRs if any of the other maintainers have source-build-specific questions you don't have an answer for.

[ayakael]

@crummel I've been a lurker for too long, happy to finally share in the open source spirit :)

Thanks for the explanation for the segfault! Indeed, I fixed it by patching eng/Versions.props to use the lastest 2.0.4 patched with added Alpine 3.15. Simple fix for, thankfully, a simple problem.

As for portable, the package currently does not generate portable bits as I encounter strange issues while trying to build them, thus the build.sh currently with /p:SkipPortableRuntimeBuild=true. In what cases does ASP.NET rely on the portable runtime? Is it something that I should hold merge for as I try to figure out the issues with the portable build?

Also, another question: there's a step towards the end when a large Private.Source.Build.Artifacts is downloaded. Sometimes this fails, thus halting the build. I understand that there's a way to locally generate that file, is there not? Any pre-built binaries should be provided by the -stage0 package, while anything built using the main package should be locally built, and provided by the -bootstrap-* subpackages.

It also happens to be my current roadblock with dotnet 6, as the downloaded Private.Source.Build.Artifacts does not have musl binaries, thus failing the build when ilasm fails with error code 127 (edit, actually error code 139) (presumably referring to not finding the required libc libraries).

[ayakael]

Just as I finished writing the last post, @omajid answered the Artifacts question on #2782. That'll at least allow me to go forward on dotnet6, although I'm still lost on how to locally build Private.Source.Build.Artifacts for proper N-1 => N so that all binaries in the post-initial build bootstrap have been built within an Alpine environment.

Indeed, dotnet31 and dotnet5 currently fails to build against the current state of Alpine's edge repository (rolling testing repo), presumably due to a change in libraries that are yet to be identified. In dotnet31's case, ilasm is missing something, but the segmentation fault leaves much to be desired as far as useful information. I have yet to find time to debug the dotnet5 failure. I'm naively hoping that whatever issues will clear up because debugging segmentation faults is my least favorite activity. At least everything builds find against the current (edIt: stable) version of Alpine.

edit
Omajid answered that first answer: Private.Source.Build.Artifacts is built on dotnet31, dotnet5, and dotnet6, thus I just need to package it properly.

edit jan-07-22
Fixed the segmentation fault, caused by lttng-ust version 2.13 see dotnet/runtime#57784

[ayakael]

A contributor on Alpine's aports has brought up the issue of EOL. Given dotnet5's EOL of May 2022, is it worthwhile for me to continue working on this package? Any chances that they'll be an LTS for 5.0?

Also note, since I've removed dotnet5's need for dotnet31-runtime, their two builds are now decoupled, thus the packages are now in two different merge requests on Alpine's gitlab for ease of development.
dotnet31: https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/28197
dotnet5: https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/28195

[omajid]

Given dotnet5's EOL of May 2022, is it worthwhile for me to continue working on this package?

I can't put myself in your shoes, but if someone had asked me to add .NET 5 to a Linux distro, I would simply tell me that I won't; it's too close to EOL.

If a user has been using .NET 5 on alpine until now using some other mechanism, they can continue doing that for the next 4 months or so too. If they are looking to adapt .NET for a fresh project, 5.0 is a bad idea to start with since it's EOL is so close.

Any chances that they'll be an LTS for 5.0?

None that I can see. Microsoft has stated that alternate releases are LTS. 3.1 is LTS (EOL December 3, 2022) and 6.0 is LTS (EOL November 8, 2024). They have also said that next LTS will be 8.0. 5.0 is not and will not be LTS. See https://dotnet.microsoft.com/en-us/platform/support/policy/dotnet-core#cadence for details.

[omajid]

Hey, @ayakael are you also packaging up https://github.com/dotnet/source-build-reference-packages/ ? Both 3.1 and 5.0 need it if you are going to use the N-1 build to build the N version of source-build. 6.0 also uses it, but automagically under the hood. For 3.1 and 5.0, you will need to explicitly build it and consume it in source-build, by placing it at packages/archive/. Also, watch out, there are different branches in that repo for 3.1 and 5.0, and they are not compatible with each other. We had to package each as a separate package in Fedora.

[ayakael]

@omajid Thanks for the heads up! No I don't, I'll get to doing that shortly.

[ayakael]

@omajid Is SBRP designed to be built after the first local build?

That is to say, current practice does the following:

1. dotnet5-stage0/APKBUILD builds stage0 N-1
   1.1) using Microsoft binaries + Private.Artifacts,
   1.1) creating dotnet5-stage0-{sdk,runtime,artifacts},
   1.1.1) which provides dotnet5-bootstrap-{sdk,runtime,artifacts}
2. dotnet5/APKBUILD builds first N-1
   2.1) using dotnet5-stage0-{sdk,runtime,artifacts} (via virtual package dotnet5-bootstrap-{sdk,runtime,artifacts))
   2.2) creating dotnet5-{sdk,runtime,artifacts,targeting-packs}
   2.2.1) which provides dotnet5-{sdk,runtime,artifacts} provides dotnet5-bootstrap-{sdk,runtime,artifacts}, a
   2.3) creating aspnet5-{runtime,targeting-packs}
   2.4) creating netstandard5-targeting-pack.
   2.4.1) which provides netstandard21-targeting-pack to allow shared package between dotnet31 and dotnet6.
3. dotnet5/APKBUILD builds "true" N
   3.1) using dotnet5-{sdk,runtime,artifacts} (via virtual package dotnet5-bootstrap-{sdk,runtime,artifacts})
   3.2) creating above "true" N packages.

In this, where does SBRP fit? Should dotnet5-stage0 build it in step 1, creating a dotnet5-stage0-reference package, providing dotnet5-bootstrap-reference? Or is this something that should be built between step 2 and 3 as its own special package that should only be built with dotnet5-{sdk,runtime,artifacts} and not dotnet5-stage0-{sdk,runtime,artifacts}? Or both?

As it is right now, ilasm pulled by SBRB fails to execute due to pulling libc ilasm rather than musl ilasm. I suspect this would be fixed with the Private.Artifacts file built in step 2, thus suggesting this 3rd package should be built between step 2 and 3.

[omajid]

The prebuilt version of SBRP (Prviate.SourceBuild.ReferencePackages.tar.gz) should be a part of stage 0 N-1 too. Then the N-1 build can use that prebuilt (or bootstrap) binary to build.

When you have the N-1 build of source-build, you can use that to build SBRP.

When you have built SBRP yourself, it should be used to build the "true N" build.

[ayakael]

I'm hitting a wall in step 2: Private.SourceBuild.ReferencePackages.$ver.tar.gz and Private.SourceBuilt.Artifacts.$ver.tar.gz aren't being pulled by dotnet5 build. (i.e., online versions are downloaded regardless of them being there or not)

edit fixed by removing /p:ArchiveDownloadedPackages=true
edit 2 or not... sigh known-good still downloading those two files

[ayakael]

For Artifacts and ReferencePackages, is there a way for source-build to injest *.nuget files rather than *.tar.gz? I'm bothered that I'm packaging (i.e. compressing) a file that's already compressed. Alpine's APK agrees as it doesn't like managing large (2GB+) files. I could always recompress them on build for source-build to decompress them.

And indeed, known-good still downloads those files. I thought I had fixed it, but I was wrong. I'm not quite sure what's up, but I'll investigate tomorrow and see if they're any flags I should set.

Also, what does /p:ArchiveDownloadedPackages=true do?

[omajid]

And indeed, known-good still downloads those files. I thought I had fixed it, but I was wrong. I'm not quite sure what's up, but I'll investigate tomorrow and see if they're any flags I should set.

Is this 5.0? Looking at the sources, repos/known-good.proj shouldn't download anything, it should only copy the stuff that was just built into the right locations. build.proj downloads then, but only if $(OfflineBuild) != 'true'. OfflineBuild is true when you build using build.sh, so that shouldn't be the case either. Maybe try setting SkipDownloadingPreviouslySourceBuiltPackages and SkipDownloadingReferencePackage to true to force this? Do you have logs that we can look at?

Also, what does /p:ArchiveDownloadedPackages=true do?

Not quite 100% sure I understand ArchiveDownloadedPackages myself. From reading the code it sounds like when ArchiveDownloadedPackages is true, the build systems saves all nuget packages that were downloaded - for the main build or for running tests - to a separate tarball.

If that understanding is correct, this would have a couple of uses:

• It's used during the development (that is, pre GA) phase of a .NET release to track and build .NET (even offline) while it needed extra (non-source-built) pieces to build.
• It's used to find regressions in dependencies. If a new dependency suddenly appears in a component, we can see it ending up here at the source-build level and then try and figure out why that happened.
• While a security release is under embargo, we can have a tarball of not-yet-public nuget packages that are needed to run tests and verify .NET works, while being able to share that with others included in the security embargo, even if they can't otherwise access the embargoed nuget servers.

[ayakael]

I'm trying a rebuild with those flags which will hopefully work, but for the problematic logs, voilà: https://gitlab.alpinelinux.org/ayakael/aports/-/jobs/588313/raw look for Private.SourceBuild.ReferencePackages, you'll see known-good redownloading those files.

[ayakael]

No dice for ReferencePackages with SkipDownloadingReferencePackage, but SkipDownloadingPreviouslySourceBuiltPackages works for Artifacts. See, https://gitlab.alpinelinux.org/ayakael/aports/-/jobs/589171/raw

[omajid]

Sorry, I had a typo. It's SkipDownloadingReferencePackages (extra s at the end).

[ayakael]

Perfect, that did it!

[ayakael]

I see that Fedora runs with /p:SkipPortableRuntimeBuild=true, as do I in the Alpine build. Is there any issue to skipping the building of the portable runtimes, other than not being able to go across distro versions? I figured that I should eventually get portable runtime builds going for proper bootstrap packages to be able to run across distro versions. Why does Fedora skip that?

(edit got note from Alpine developpers that portable runtimes won't be necessary for bootstrapping, as they'll automatically build stage0 package at every new-version avoiding manually transferring N-1 bootstraps. As long as lack of portable runtimes won't make anything unhappy, I'll do like Fedora pkg and skip building them)

Also, in investigating getting the arm64 builds going, I see that Microsoft does not publish SDKs of dotnet-3.1 for linux-musl-arm64, but does publish runtimes for linux-musl-arm64. Are there long forgotten SDK binaries somewhere I can use to have stage0 be able to pull something for arm64?

[ayakael]

I also see that dotnet-build-reference-packages pkg on Fedora is at version 11. I have yet to see any mention of a versioning scheme for SBRP much to my dismay. Where did you get that version number? Is it in-house to fedora?

[ayakael]

Of note: Alpine devs indeed prefer maintaining the $pkgname$macroversion-$subpkgname naming scheme rather than Microsoft's $pkgname-$subpkgname-$macroversion. I've mitigated this issue by creating virtual packages that translate Microsoft -> Alpine. Thus: apk add dotnet-runtime-5.0 will pull dotnet5-runtime without a fuss.

I'm wrapping up dotnet5 and dotnet31, now that references and artifacts are now packaged (in dotnetx-artifacts and dotnetx-references, respectively). I've also added a zsh and bash completions provided here in their own subpackages, per Alpine's standards.

[omajid]

I see that Fedora runs with /p:SkipPortableRuntimeBuild=true, as do I in the Alpine build. Is there any issue to skipping the building of the portable runtimes, other than not being able to go across distro versions? I figured that I should eventually get portable runtime builds going for proper bootstrap packages to be able to run across distro versions. Why does Fedora skip that?

AFAIK, it was a bunch of accidents that lead to this state. Fedora was probably the earliest consumer of source-build. In fact, I think it started consuming source-build before the portable runtime build was added (IOW, only non-portable was being built when .NET was bootstrapped and added to Fedora). When the portable build was added to source-build, we could either re-bootstrap or skip the portable runtime build. Skipping the build worked in the sense it produced working artifacts, so that's what we ended up doing, rather than the more expensive (in terms of dev time) re-bootstrap.

It hasn't caused any issues until now, so we have kept skipping the portable build for 3.1 and 5.0.

I also see that dotnet-build-reference-packages pkg on Fedora is at version 11. I have yet to see any mention of a versioning scheme for SBRP much to my dismay. Where did you get that version number? Is it in-house to fedora?

SBRP doesn't have official release numbers, so the versioning in Fedora is made up. The Fedora SBRP package tries to follow Fedora's suggestions for what to use as a number when upstream (ie, SBRP) doesn't do releases.

What you want to do is to make sure the version of SBRP is the same as the commit listed in the eng/Version.Details.xml file of source-build:

 grep -A1 source-build-reference-package eng/Version.Details.xml 
      <Uri>https://github.com/dotnet/source-build-reference-packages</Uri>
      <Sha>5f10a4b951be1d0f2966aca0702b0348bbf5f7c1</Sha>

If/when the commit sha doesn't match, you should update SBRP to match this commit sha.

I've also added a zsh and bash completions provided here in their own subpackages, per Alpine's standards.

Nice! The bash completion script is also in the repo, and you can just cp that: https://github.com/dotnet/cli/blob/master/scripts/register-completions.bash

The zsh completion is actually broken if you install it as a standard completion. There's a (stalled) fix here: dotnet/sdk#14239

[ayakael]

Awesome! Thanks for all the notes. A relief that skipping runtime builds is okay, it would've been a process to fix 'em on musl. Completely implemented a references package. Unfortunately, the merge request is stalling because of a versioning issue. Current building infrastructure does not support assigning versions to each package. I've found a possible solution, but it's going to take a moment to figure out if that's going to break the build servers haha. I'm not liking the idea of omitting either SDK versions or runtime versions, and whatever versioning scheme we decide on, it'll be an awkward process changing it in the future. Might as well get it right the first time.

Time to fully tackle dotnet6!

[ayakael]

I'm encountering an issue with dotnet5's roslyn relating to ingesting Private.SourceBuilt.Artifacts. It is unable to find Microsoft.NETCore.App.Host.alpine.3.15-x64 when the package clearly contains Microsoft.NETCore.App.Host.alpine.3.15-x64.5.0.12.nupkg. The Private.SourceBuilt.Artifacts.5.0.0-dev.tar.gz is available under packages/archive.

Compilers.sln:
/var/build/aports/testing/dotnet5/src/source-build-5.0.209.1-SDK/bootstrap/sdk/5.0.209/MSBuild.dll /nologo -maxcpucount /m -verbosity:m /v:minimal /bl:/var/build/aports/testing/dotnet5/src/source-build-5.0.209.1-SDK/artifacts/src/roslyn.f8111b17631aa5e2703dc11acd671c6ea22c3af1/artifacts/log/Release/Build.binlog /clp:Summary /nr:true /p:TreatWarningsAsErrors=false /p:ContinuousIntegrationBuild=false /p:Configuration=Release /p:Projects=/var/build/aports/testing/dotnet5/src/source-build-5.0.209.1-SDK/artifacts/src/roslyn.f8111b17631aa5e2703dc11acd671c6ea22c3af1/Compilers.sln /p:RepoRoot=/var/build/aports/testing/dotnet5/src/source-build-5.0.209.1-SDK/artifacts/src/roslyn.f8111b17631aa5e2703dc11acd671c6ea22c3af1 /p:Restore=true /p:Build=true /p:Rebuild=false /p:Test=false /p:Pack=true /p:Publish=false /p:RunAnalyzersDuringBuild=false /p:BootstrapBuildPath= /p:ContinuousIntegrationBuild=false /p:TreatWarningsAsErrors=true /p:TestRuntimeAdditionalArguments= /p:DotNetBuildFromSource=false /p:TreatWarningsAsErrors=false /p:OfficialBuild=true /p:ApplyPartialNgenOptimization=false /p:EnablePartialNgenOptimization=false /p:PublishWindowsPdb=false /p:DotNetPackageVersionPropsPath=/var/build/aports/testing/dotnet5/src/source-build-5.0.209.1-SDK/artifacts/obj/x64/Release/PackageVersions.props /p:DotNetBuildFromSource=true /p:UsingToolMicrosoftNetCompilers=false /var/build/aports/testing/dotnet5/src/source-build-5.0.209.1-SDK/Tools/source-built/Microsoft.DotNet.Arcade.Sdk/tools/Build.proj
  Determining projects to restore...
  Nothing to do. None of the projects specified contain packages to restore.
[SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.Sdk. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/5.0.0-beta.20431.1' at '/var/build/aports/testing/dotnet5/src/source-build-5.0.209.1-SDK/Tools/source-built/Microsoft.DotNet.Arcade.Sdk/'
[SourceBuiltSdkResolver] Looking for SDK Microsoft.DotNet.Arcade.Sdk. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/5.0.0-beta.20431.1' at '/var/build/aports/testing/dotnet5/src/source-build-5.0.209.1-SDK/Tools/source-built/Microsoft.DotNet.Arcade.Sdk/'
[SourceBuiltSdkResolver] Overriding Microsoft.DotNet.Arcade.Sdk with 'ARCADE'
[SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.ILLink.Tasks. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/5.0.0-beta.20431.1' at '/var/build/aports/testing/dotnet5/src/source-build-5.0.209.1-SDK/Tools/source-built/Microsoft.DotNet.Arcade.Sdk/'
  Determining projects to restore...
  Restored /var/build/aports/testing/dotnet5/src/source-build-5.0.209.1-SDK/artifacts/src/roslyn.f8111b17631aa5e2703dc11acd671c6ea22c3af1/src/NuGet/Microsoft.Net.Compilers.Toolset/Microsoft.Net.Compilers.Toolset.Package.csproj (in 2.52 sec).
  Restored /var/build/aports/testing/dotnet5/src/source-build-5.0.209.1-SDK/artifacts/src/roslyn.f8111b17631aa5e2703dc11acd671c6ea22c3af1/src/NuGet/Microsoft.NETCore.Compilers/Microsoft.NETCore.Compilers.Package.csproj (in 2.52 sec).
  Restored /var/build/aports/testing/dotnet5/src/source-build-5.0.209.1-SDK/artifacts/src/roslyn.f8111b17631aa5e2703dc11acd671c6ea22c3af1/src/CodeStyle/Tools/CodeStyleConfigFileGenerator.csproj (in 3.6 sec).
  Restored /var/build/aports/testing/dotnet5/src/source-build-5.0.209.1-SDK/artifacts/src/roslyn.f8111b17631aa5e2703dc11acd671c6ea22c3af1/src/Compilers/Core/Portable/Microsoft.CodeAnalysis.csproj (in 4.28 sec).
  Restored /var/build/aports/testing/dotnet5/src/source-build-5.0.209.1-SDK/artifacts/src/roslyn.f8111b17631aa5e2703dc11acd671c6ea22c3af1/src/Compilers/CSharp/Portable/Microsoft.CodeAnalysis.CSharp.csproj (in 4.28 sec).
  Restored /var/build/aports/testing/dotnet5/src/source-build-5.0.209.1-SDK/artifacts/src/roslyn.f8111b17631aa5e2703dc11acd671c6ea22c3af1/src/NuGet/Microsoft.CodeAnalysis.Package.csproj (in 4.38 sec).
  Restored /var/build/aports/testing/dotnet5/src/source-build-5.0.209.1-SDK/artifacts/src/roslyn.f8111b17631aa5e2703dc11acd671c6ea22c3af1/src/Compilers/VisualBasic/vbc/vbc.csproj (in 4.46 sec).
  Restored /var/build/aports/testing/dotnet5/src/source-build-5.0.209.1-SDK/artifacts/src/roslyn.f8111b17631aa5e2703dc11acd671c6ea22c3af1/src/Compilers/CSharp/csc/csc.csproj (in 4.47 sec).
  Restored /var/build/aports/testing/dotnet5/src/source-build-5.0.209.1-SDK/artifacts/src/roslyn.f8111b17631aa5e2703dc11acd671c6ea22c3af1/src/Compilers/Server/VBCSCompiler/VBCSCompiler.csproj (in 4.47 sec).
  Restored /var/build/aports/testing/dotnet5/src/source-build-5.0.209.1-SDK/artifacts/src/roslyn.f8111b17631aa5e2703dc11acd671c6ea22c3af1/src/Workspaces/Core/Portable/Microsoft.CodeAnalysis.Workspaces.csproj (in 4.49 sec).
  Restored /var/build/aports/testing/dotnet5/src/source-build-5.0.209.1-SDK/artifacts/src/roslyn.f8111b17631aa5e2703dc11acd671c6ea22c3af1/src/Workspaces/CSharp/Portable/Microsoft.CodeAnalysis.CSharp.Workspaces.csproj (in 4.5 sec).
  Restored /var/build/aports/testing/dotnet5/src/source-build-5.0.209.1-SDK/artifacts/src/roslyn.f8111b17631aa5e2703dc11acd671c6ea22c3af1/src/Compilers/Core/MSBuildTask/Microsoft.Build.Tasks.CodeAnalysis.csproj (in 4.67 sec).
  Restored /var/build/aports/testing/dotnet5/src/source-build-5.0.209.1-SDK/artifacts/src/roslyn.f8111b17631aa5e2703dc11acd671c6ea22c3af1/src/Compilers/VisualBasic/Portable/Microsoft.CodeAnalysis.VisualBasic.vbproj (in 4.76 sec).
  Restored /var/build/aports/testing/dotnet5/src/source-build-5.0.209.1-SDK/artifacts/src/roslyn.f8111b17631aa5e2703dc11acd671c6ea22c3af1/src/Workspaces/VisualBasic/Portable/Microsoft.CodeAnalysis.VisualBasic.Workspaces.vbproj (in 4.83 sec).
  Restored /var/build/aports/testing/dotnet5/src/source-build-5.0.209.1-SDK/artifacts/src/roslyn.f8111b17631aa5e2703dc11acd671c6ea22c3af1/src/CodeStyle/VisualBasic/Analyzers/Microsoft.CodeAnalysis.VisualBasic.CodeStyle.vbproj (in 4.94 sec).
  Restored /var/build/aports/testing/dotnet5/src/source-build-5.0.209.1-SDK/artifacts/src/roslyn.f8111b17631aa5e2703dc11acd671c6ea22c3af1/src/CodeStyle/Core/Analyzers/Microsoft.CodeAnalysis.CodeStyle.csproj (in 4.95 sec).
  Restored /var/build/aports/testing/dotnet5/src/source-build-5.0.209.1-SDK/artifacts/src/roslyn.f8111b17631aa5e2703dc11acd671c6ea22c3af1/src/CodeStyle/CSharp/Analyzers/Microsoft.CodeAnalysis.CSharp.CodeStyle.csproj (in 4.95 sec).
/var/build/aports/testing/dotnet5/src/source-build-5.0.209.1-SDK/artifacts/src/roslyn.f8111b17631aa5e2703dc11acd671c6ea22c3af1/src/Tools/Source/CompilerGeneratorTools/Source/CSharpSyntaxGenerator/CSharpSyntaxGenerator.csproj : error NU1101: Unable to find package Microsoft.NETCore.App.Host.alpine.3.15-x64. No packages exist with this id in source(s): dotnet-eng, dotnet-public, dotnet-tools, dotnet5, dotnet6, source-built, vs-buildservices, vs-impl, vssdk [/var/build/aports/testing/dotnet5/src/source-build-5.0.209.1-SDK/artifacts/src/roslyn.f8111b17631aa5e2703dc11acd671c6ea22c3af1/Compilers.sln]
  Restored /var/build/aports/testing/dotnet5/src/source-build-5.0.209.1-SDK/artifacts/src/roslyn.f8111b17631aa5e2703dc11acd671c6ea22c3af1/src/CodeStyle/Core/CodeFixes/Microsoft.CodeAnalysis.CodeStyle.Fixes.csproj (in 4.96 sec).
  Restored /var/build/aports/testing/dotnet5/src/source-build-5.0.209.1-SDK/artifacts/src/roslyn.f8111b17631aa5e2703dc11acd671c6ea22c3af1/src/CodeStyle/VisualBasic/CodeFixes/Microsoft.CodeAnalysis.VisualBasic.CodeStyle.Fixes.vbproj (in 4.98 sec).
  Restored /var/build/aports/testing/dotnet5/src/source-build-5.0.209.1-SDK/artifacts/src/roslyn.f8111b17631aa5e2703dc11acd671c6ea22c3af1/src/CodeStyle/CSharp/CodeFixes/Microsoft.CodeAnalysis.CSharp.CodeStyle.Fixes.csproj (in 4.98 sec).
  Failed to restore /var/build/aports/testing/dotnet5/src/source-build-5.0.209.1-SDK/artifacts/src/roslyn.f8111b17631aa5e2703dc11acd671c6ea22c3af1/src/Tools/Source/CompilerGeneratorTools/Source/CSharpSyntaxGenerator/CSharpSyntaxGenerator.csproj (in 5 sec).

Build FAILED.

/var/build/aports/testing/dotnet5/src/source-build-5.0.209.1-SDK/artifacts/src/roslyn.f8111b17631aa5e2703dc11acd671c6ea22c3af1/src/Tools/Source/CompilerGeneratorTools/Source/CSharpSyntaxGenerator/CSharpSyntaxGenerator.csproj : error NU1101: Unable to find package Microsoft.NETCore.App.Host.alpine.3.15-x64. No packages exist with this id in source(s): dotnet-eng, dotnet-public, dotnet-tools, dotnet5, dotnet6, source-built, vs-buildservices, vs-impl, vssdk [/var/build/aports/testing/dotnet5/src/source-build-5.0.209.1-SDK/artifacts/src/roslyn.f8111b17631aa5e2703dc11acd671c6ea22c3af1/Compilers.sln]
    0 Warning(s)
    1 Error(s)

Time Elapsed 00:00:15.34

Using prebuilt artifacts doesn't have this issue, and dotnet31 builds without issue using post-build artifacts. Any idea what this might be?

edit

build() {
        msg "Building $pkgname-$pkgver"
        export SOURCE_BUILD_SKIP_SUBMODULE_CHECK=1

        # Disable tracing, which is incompatible with certain versions of
        # lttng See https://github.com/dotnet/runtime/issues/57784. The
        # suggested compile-time change doesn't work, unfrotunately.
        export COMPlus_LTTng=0

        ./build.sh \
                --with-sdk bootstrap \
                /p:SkipDownloadingPreviouslySourceBuiltPackages=true \
                /p:SkipDownloadingReferencePackages=true \
                /p:ContinueOnPrebuiltBaselineError=true \
                /p:SkipPortableRuntimeBuild=true \
                /p:SkipPrebuiltEnforcement=true \
                /p:UseSystemLibraries=true \
                /p:UseSystemLibunwind=true
}

[ayakael]

Further investigations has pulled up that roslyn's artifacts/src/roslyn.f8111b17631aa5e2703dc11acd671c6ea22c3af1/NuGet.config pulls NuPkg from ./artifacts/obj/x64/Release/blob-feed/packages/

So if I copy Microsoft.NETCore.App.Host.alpine.3.15-x64.5.0.12.nupkg manually to ./artifacts/obj/x64/Release/blob-feed/packages/ it works. Tempted to answer my request regarding pulling all artifacts directly from the loose *.nupkg in /usr/share/dotnet/artifacts/5.0.209/, rather than compressing a tar at build, I copied everything from the latter directory to the former. Of course, it failed as blob-feed/packages seems to also be where source-build sends its built nupkg file after each step. Source-build then fails as it finds it weird that all of its packages are already built.

This leads me to wonder if there's a build flag that can add a nuget feed, allowing pulls from other directories than blob-feed-packages. I need to further investigate Private.Sourcebuilt/x*.tar.gz ingesting logics.

[ayakael]

Fixed the above issue by adding a NuGet repo pointing to /usr/share/dotnet/artifacts, allowing it to pull the missing nupkg. Terrific news: this means that both dotnet31 and dotnet5 are now fully capable of building on themselves!

[ayakael]

Generally, what causes the following errors, other than not having that exact NuPkg?

##vso[task.logissue type=error;sourcepath=/var/build/dotnet31/testing/dotnet31-bootstrap/src/source-build-3.1.416-tarball/src/corefx.93bbacb3437ddb3b5a45ea269ea049d64b5825ec/external/tools/tools.depproj;linenumber=0;columnnumber=0;code=NU1102;]Unable to find package runtime.linux-x64.Microsoft.NETCore.ILAsm with version (>= 3.1.22-servicing)%0A  - Found 1 version(s) in previously-source-built [ Nearest version: 3.1.20-servicing %5D%0A  - Found 0 version(s) in reference-packages%0A  - Found 0 version(s) in source-built%0A  - Found 0 version(s) in prebuilt

Is there a way to regenerate PackageVersions.props, or have build process be more relaxed on versioning?

[omajid]

Hm. Are you trying to build 3.1.416 using an older version? Unfortunately the previous version was a 3.1.1xx release and going from 3.1.1xx to 3.1.4xx requires a re-bootstrap.

Other than that, going from one minor 3.1 release to next (without skipping versions), I wouldn't expect to see this error.

[ayakael]

Package was merged into Alpine's aport succesfully. Packaging is thus done for dotnet31 and dotnet5. Closing.