Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for FreeBSD #14537

Open
ghuntley opened this issue May 4, 2015 · 793 comments · Fixed by #58085
Open

Support for FreeBSD #14537

ghuntley opened this issue May 4, 2015 · 793 comments · Fixed by #58085
Labels
area-Meta enhancement Product code improvement that does NOT require public API changes/additions needs-further-triage Issue has been initially triaged, but needs deeper consideration or reconsideration os-freebsd FreeBSD OS
Milestone

Comments

@ghuntley
Copy link
Member

ghuntley commented May 4, 2015

Updated proposal from 2017/9

Proposal (by @karelz - https://github.com/dotnet/corefx/issues/1626#issuecomment-329840518) will be updated in top-post based on further discussion and proposal changes.

We discussed community-driven port for FreeBSD with @RussellHaley (from FreeBSD community) and @wfurt (from .NET Core team) who both expressed interest in the work.
Here's a plan proposal we put together (feedback / suggestions are welcome):

  1. Produce binaries in CoreCLR & CoreFX repo targeting FreeBSD - using hacks is fine
    • Hard to parallelize, @wfurt will work on that
    • The build can be mix of builds from other platforms (Mac, Linux) targeting FreeBSD
    • We will need documented steps (on FreeBSD wiki) to reproduce the build with FreeBSD-specific bug fixes
  2. Run & stabilize CoreCLR tests (using corerun)
    • Tests may be built on another platform
    • Goal: Provides basic quality of runtime
  3. Run & stabilize CoreFX tests (using corerun)
    • Tests may be built on another platform
    • Note this requires xunit. We believe, based on our past porting experience, once [2] is done, xunit will just work.
    • This can be in theory parallelized with [2] - it may require shortcutting xunit (e.g. generate static execution recipe on another platform)
    • We can expose new OSPlatform API for FreeBSD when the pass rate is reasonable: see dotnet/corefx#23989
  4. Full stack build on FreeBSD (using corerun as bootstrapper from [1]-[3])
    • We will need all tools (nuget, msbuild, roslyn) to work on boostrapping .NET Core
  5. Installers (FreeBSD ports)
    • First-stage: Using product binaries from nuget feeds
    • Second-stage: Build product from source (blocked on build from source effort)
    • Requires FreeBSD community expertise and guidance on design
    • Note: We can link FreeBSD packages also from official .NET Core download pages as community-support packages
  6. Regular build and test runs on FreeBSD
    • Goal: Make sure changes in .NET Core repos breaking FreeBSD are known early
    • Design needed
    • Requires FreeBSD community expertise and guidance on design

Operation principles:

  • Changes in [2]-[4] should be done primarily in CoreCLR/CoreFX repos (due to CLA signing requirements, code reviews from .NET Core team experts/members. etc.)
  • We will track high-level work on this issue. Specific bugs will be filed as separate issues.

If anyone is interested in helping, please let us know here. We can easily distribute work items from [2] & [3] above once we are far enough with [1].


Original proposal from @ghuntley from 2015/5

This issue is to discuss unit(s) of work to actually produce FreeBSD assemblies for corefx.

@stephentoub - There's what's likely a more pressing issue, which is actually building for FreeBSD. Today, when we need to specialize an assembly for a particular platform, we effectively have three builds, producing three different managed assemblies: Windows, Linux, OSX. Sounds like at least for now we'll need a fourth, FreeBSD. I suggest you start by modifying the build to support an IsFreeBSD property (or just IsBSD of you think there's a high chance that the implementations across BSDs will be the same even with varied kernels) along with the appropriate OSGroup targets. That can then be used in the csproj files as needed to specialize an assembly with FreeBSD-specific code.

Related issue(s)

/cc: @janhenke @josteink

@josteink
Copy link
Member

josteink commented May 4, 2015

There seems to be agreement as far as https://github.com/dotnet/corefx/issues/1576 is concerned.

When we also have a decision on https://github.com/dotnet/corefx/issues/1625 we should be able to start shipping some code.

@ghuntley
Copy link
Member Author

Agreement on #14536 has been reached by the portteam, unless MSFT chooses otherwise it will be FreeBSD. Issue dotnet/corefx#1999 will potentially be the issue that introduces the definition into the public API.

@josteink
Copy link
Member

Agreement on #14536 has been reached by the portteam, unless MSFT chooses otherwise it will be FreeBSD

If I read that right, this means that when dotnet/corefx#1999 is merged, we can consider this MSFT approving of the new public API, and can therefore press forward on the remaining issues with regular pull-requests without need for MSFT approval.

If so, that sounds good to me.

@ghuntley
Copy link
Member Author

Next steps as per dotnet/corefx#1999 (comment) are:

  1. The "FreeBSD port team" continues their work to get a FreeBSD version of CoreFX produced (tracked by dotnet/corefx#1626).
  2. The port team brings up enough of the CoreFX and CoreCLR stack on FreeBSD such that we can start running the CoreFX unit tests on FreeBSD.
  3. The tests reach some minimal quality level. I don't know exactly what this looks like yet, but I expect it means something like a majority of the tests pass. Ideally we would not have a bunch of specific tests disabled for only FreeBSD (compared to Linux and OSX, we wouldn't want to hold FreeBSD to a higher standard than the other *NIX platforms we have there).
  4. Working with the FreeBSD port team, the CoreFX team gets the CoreFX tests added to our CI system running on FreeBSD.
  5. Discuss merging a PR based for issue [public api] System.Runtime.Environment - OSName("FreeBSD") or OSName("BSD") #14536, which adds the property.

@josteink
Copy link
Member

That sounds like a fully reasonable plan to me.

@janhenke
Copy link
Member

Okay, then let's start the work on getting corefx to work.

@josteink
Copy link
Member

First obstacle in building corefx on FreeBSD seems to be mono. The build-script insists version 4.1 is required. @ajensenwaud did some work on this on the Frankfurt-host, but I'm not sure how complete it is.

I'll queue a build for now and see what the output looks like.

Edit: The (mono) build crashes with the following kicker at the end:

Making all in mini
make[1]: "/usr/home/josteink/mono/mono/mini/Makefile" line 2906: warning: duplicate script for target "%.exe" ignored
make[1]: "/usr/home/josteink/mono/mono/mini/Makefile" line 2899: warning: using previous script for "%.exe" defined here
  CC       genmdesc-genmdesc.o
In file included from genmdesc.c:9:0:
mini.h:17:34: fatal error: ./mono/metadata/loader.h: Too many levels of symbolic links
 #include <mono/metadata/loader.h>
                                  ^
compilation terminated.
*** Error code 1

Stop.
make[1]: stopped in /usr/home/josteink/mono/mono/mini
*** Error code 1

Stop.

@stephentoub
Copy link
Member

First obstacle in building corefx on FreeBSD seems to be mono

FWIW, I personally don't think this is the first obstacle. There are two build related issues:

  1. Building assemblies that work correctly on FreeBSD
  2. Building those assemblies on FreeBSD

(1) is critical, and is I believe what this issue is meant to be about. (2) is very nice to have, but lack of it doesn't prevent the creation of a great system for running managed code on FreeBSD.

You're of course free to prioritize however you see fit, but my recommendation would be to focus on (1) rather than (2).

Note that we still have issues building corefx on Linux and building it on OSX, such that our CI system builds the assemblies for those platforms on Windows; it then shuttles the resulting assemblies over to the target platform to execute the tests.

@josteink
Copy link
Member

That's fair enough. I just assumed that it would be easier to get general FreeBSD platform support baked into corefx if we could actually build it ourselves on FreeBSD.

I'll make do with Windows-initiated building for now and attempt to ninja together a build-configuration.

@akoeplinger
Copy link
Member

@josteink btw. corefx should now build on Mono 4.0.1.44.

@josteink
Copy link
Member

@akoeplinger Nice. That leaves me some hope we can get it running on FreeBSD too :)

@ajensenwaud
Copy link
Contributor

Good points. However if we really want corefx to be part of the FreeBSD environment, we really need it to be able to compile from source to get it into the Ports system.

I did hear that Mono 4.0.1.44 fixes a lot of these issues but have not had time to play with it yet. I know the ports team are updating the port Makefile as well as we speak with a new patch.

On 12 Jun 2015, at 20:21, Stephen Toub notifications@github.com wrote:

First obstacle in building corefx on FreeBSD seems to be mono

FWIW, I personally don't think this is the first obstacle. There are two build related issues:

Building assemblies that work correctly on FreeBSD
Building those assemblies on FreeBSD
(1) is critical, and is I believe what this issue is meant to be about. (2) is very nice to have, but lack of it doesn't prevent the creation of a great system for running managed code on FreeBSD.

You're of course free to prioritize however you see fit, but my recommendation would be to focus on (1) rather than (2).

Note that we barely have corefx building-on-Linux and building-on-OSX, such that our CI system builds the assemblies for those platforms on Windows; it then shuttles the resulting assemblies over to the target platform to execute the tests.


Reply to this email directly or view it on GitHub.

@stephentoub
Copy link
Member

Yes, I'm in no way disagreeing... being able to build corefx on Linux, OSX, and FreeBSD is important. I'm simply suggesting that from a priority perspective it's more important to be able to actually run corefx on Linux, OSX, and FreeBSD. 😉 If both can be worked on in parallel, all the better.

@ghost
Copy link

ghost commented Jun 13, 2015

@ghuntley,
would be super 🆒 if we have a markdown task checklist outlining what what is remaining:

- [x] task 1
- [ ] task 2
- [ ] task 3

renders as:

  • task 1
  • task 2
  • task 3

This will probably encourage others to score those feats and FreeBSD support will land rather sooner than anticipated! 😎

@janhenke
Copy link
Member

To my knowledge the following pieces of work in CoreFX are required for FreeBSD support:

13 Assemblies do not compile on their own and need FreeBSD specific changes. Mostly the Interop pieces that already exist for Linux/OS X (order by the occurrence in the build output):

I will try to update that list based on PRs opened and merged.

@josteink
Copy link
Member

FYI: PR dotnet/corefx#2039 merged

@josteink
Copy link
Member

Just trying to be ahead of the curve here... How do we plan to implement System.IO.FileSystem.Watcher ?

Iirc FreeBSD has no inotify such as Linux and Windows does (which is also why there is no Dropbox last time I checked). Will this be a potential source of trouble coming our way? Or does anyone have an idea for how to work around this?

@janhenke
Copy link
Member

I suggest we stub that out for the moment and throw a PlatformNotSupportedException as Stephen Toub suggested in the other topic (dotnet/corefx#2021 (comment)). Then we have at least a complete set of assemblies and we can continue to work on that particular issue without blocking further steps.

Would you mind opening a separate issue for that?

@ghuntley
Copy link
Member Author

Let's move System.IO.FileSystem.Watcher discussions to dotnet/corefx#2046

@ghost
Copy link

ghost commented Jun 15, 2015

Guys is there any such blocker for System.Diagnostics.Process?

@ghuntley
Copy link
Member Author

@jasonwilliams200OK added FreeBSD to S.RT.I.RI early this morning which was merged but the FreeBSD tests within CheckPlatformTests had to be backed out until dotnet/buildtools is updated.

@ghuntley
Copy link
Member Author

@jasonwilliams200OK there were some discussions last night about System.Diagnostics.Process in gitter which have been formalized into https://github.com/dotnet/corefx/issues/2070

@ghost
Copy link

ghost commented Jun 16, 2015

@ghuntley, thanks. I actually read those messages. System.Diagnostics.Process is a tricky one. AFAIK, io.js team had similar challenges with FreeBSD process management. Mono team has probably nailed it, so lets hope if @akoeplinger and co. could enlighten us on this matter? :)

@josteink
Copy link
Member

System.IO.FileSystem.DriveInfo

As discussed in the gitter, For this one I tried looking into basic usage of getmntinfo:

#include <sys/param.h>
#include <sys/ucred.h>
#include <sys/mount.h>
#include <stdio.h>

int main() {
  struct statfs *mntbuf;
  int mntsize = getmntinfo(&mntbuf, MNT_NOWAIT);

  for( int i = 0; i < mntsize; i++ ) {
    printf("%s\n", mntbuf[i].f_mntonname);
  }
}

Running that sample yielded this output:

$ ./a.out
/
/dev
/tmp
/usr/home
/usr/ports
/usr/src
/var/crash
/var/log
/var/mail
/var/tmp
/dev/fd
/usr/compat/linux/proc
/proc
$

So it seems it does what we need. The question is, should we do any type of filtering on the results?

Looking at the "intent" of the DriveInfo object, coming from the Windows world of .NET it has often been to enumerate the available locations to store or retrieve files (C:, D:, etc). But when using Unix hierarchical file-systems, returning / would be adequate to cover those needs.

So what should we return? What would be useful? Should even consider it being useful or not?

The Linux-version just dumps everything, except things set to be ignored:

https://github.com/dotnet/corefx/blob/master/src/System.IO.FileSystem.DriveInfo/src/System/IO/DriveInfo.Linux.cs#L98-L99

I tried putting in the following filter, but it didn't really change anything in terms of output:

    if ((mntbuf[i].f_flags != MNT_IGNORE)) {
        printf("%s\n", mntbuf[i].f_mntonname);
    }

Any opinions?

@ghost
Copy link

ghost commented Jun 21, 2015

@josteink, great diggings! Based on https://github.com/dotnet/corefx/issues/815#issuecomment-113825960 and https://github.com/dotnet/corefx/issues/1729, I think we should collaborate with @sokket to come up with a solution with works across different Unices.

@sec
Copy link
Contributor

sec commented Feb 20, 2024

Reading this state that Each major release of .NET requires [bootstrapping](https://github.com/dotnet/source-build/blob/main/Documentation/bootstrapping-guidelines.md). You will not be able to build 9.0 with an 8.0 SDK/artifacts. and I can say it's true, becuase I tried :)

@arrowd
Copy link
Contributor

arrowd commented Feb 20, 2024

This is extremely sad and annoying. This means that I wouldn't be able to maintain the port myself without you guys. Anyone willing to take the maintainership?

@sec
Copy link
Contributor

sec commented Feb 20, 2024

For now 8 build fine, even .1 and .2 releases can be build from the same port without any changes (you can bump the port to 8.0.2 btw). As for 9, I need to figure out myself how to bootstrap/build this properly first...

@joperator
Copy link

I've already tried doing crossbuilds without any problems, but for VMR we need Private.SourceBuilt.Artifacts which I don't know how to build either from crossbuilds or from repo's directly.

This is exactly the problem that @sec has already addressed: Cross-building new major versions has become relatively easy in the meantime and I've already done this for .NET 6, 7 and 8. However, this approach isn't suitable for maintaining a FreeBSD dotnet port. But as the Makefile indicates, @arrowd has already created a .NET 8 version of Private.SourceBuilt.Artifacts for FreeBSD and successfully used it with the VMR. It would be really nice to know how to get there, as I've also no idea how to do that.

@arrowd
Copy link
Contributor

arrowd commented Feb 21, 2024

But as the Makefile indicates, @arrowd has already created a .NET 8 version of Private.SourceBuilt.Artifacts for FreeBSD and successfully used it with the VMR. It would be really nice to know how to get there, as I've also no idea how to do that.

These are created from the ports build results by running make bootstrap-makesum after the build finishes: https://github.com/freebsd/freebsd-ports/blob/d3af6c1c0caa1585dc88e07c6981f5ce1b817ec8/lang/dotnet/Makefile#L105

Artifacts are actually created by the build itself, I only compute checksums and reupload archives.

@sec
Copy link
Contributor

sec commented Feb 21, 2024

But as the Makefile indicates, @arrowd has already created a .NET 8 version of Private.SourceBuilt.Artifacts for FreeBSD and successfully used it with the VMR. It would be really nice to know how to get there, as I've also no idea how to do that.

These are created from the ports build results by running make bootstrap-makesum after the build finishes: https://github.com/freebsd/freebsd-ports/blob/d3af6c1c0caa1585dc88e07c6981f5ce1b817ec8/lang/dotnet/Makefile#L105

Artifacts are actually created by the build itself, I only compute checksums and reupload archives.

Yes, the question was rather to maybe someone from .net team and/or someone who maybe know something about this part - as those artifcats are created as product of VMR build - if those could be created/automated from each repo build somehow etc.

I will try to find some time and go through docs etc. trying to produce those artifacts for v9, as getting crossbuild SDK is relative easy right now - then those can be used to easily use port to build 9.

@sec
Copy link
Contributor

sec commented Feb 21, 2024

I was able to bootstrap 9 for both x64 and arm64, reading helps and understand what you read helps even more :)

The steps that were needed, for future as I will mostly forget:

  1. Cross-compile runtime, aspnetcore and installer under Linux, to FreeBSD. First check what SDK version is needed to compile from VMR. All 3 commits/versions should match, the key is to select the right commit:
  • output from dotnet --info will give you SDK commit and Host (runtime) commit
  • for aspnet core commit, grep strings from any of dll's from Linux SDK (ex. strings shared/Microsoft.AspNetCore.App/9.0.0-alpha.1.24061.8/Microsoft.AspNetCore.dll|grep -i commit)
  • checkout those 3 commits, generate OfficialBuildId (look here)
  • apply needed patches (the ones from the port for correct tree, either 8 or 9)
  • for installer one extra change is needed, otherwise it will fail to generate - dotnet/installer@9ba09a7 - don't know if it make sense to upstream this? @Thefrank I believe this one came from you at some point while ago, what do you think?
  • cross-compile is a walk in the park, you need to create rootfs, pass correct parameters and copy output from previous result into new repo to have success build, etc. (I have helper scripts for that in my repo somewhere)
  • this should produce working SDK under FreeBSD
  1. Download VMR under Linux, correct tag, run ./prep.sh - it will download SDK and Private.SourceBuilt.Artifacts.9.0.100-alpha.1.24067.1.centos.8-x64.tar.gz - just example
  • now you need to insert FreeBSD specific runtimes into this package
  • for example
Microsoft.AspNetCore.App.Runtime.freebsd-arm64.9.0.0-alpha.1.24061.8.nupkg
Microsoft.NETCore.App.Crossgen2.freebsd-arm64.9.0.0-alpha.1.24061.26.nupkg
Microsoft.NETCore.App.Host.freebsd-arm64.9.0.0-alpha.1.24061.26.nupkg
Microsoft.NETCore.App.Runtime.freebsd-arm64.9.0.0-alpha.1.24061.26.nupkg
runtime.freebsd-arm64.Microsoft.DotNet.ILCompiler.9.0.0-alpha.1.24061.26.nupkg
runtime.freebsd-arm64.Microsoft.NETCore.DotNetAppHost.9.0.0-alpha.1.24061.26.nupkg
runtime.freebsd-arm64.Microsoft.NETCore.ILAsm.9.0.0-alpha.1.24061.26.nupkg
runtime.freebsd-arm64.Microsoft.NETCore.ILDAsm.9.0.0-alpha.1.24061.26.nupkg
runtime.freebsd-arm64.Microsoft.NETCore.TestHost.9.0.0-alpha.1.24061.26.nupkg

those should be under artifacts/packages/Release/Shipping and artifacts/packages/Release/NonShipping either from runtime or aspnetcore build.
3. Now you will have working SDK and matching artifacts
4. I took existing port and made slight changes for 9.preview.1 - commits should be here https://github.com/sec/freebsd-ports/tree/dotnet9
5. Copy SDK and artifacts into ports/distfiles/dotnet with names (replace x64 with arm64 if needed) that will match port:

  • dotnet-sdk-9.0.100-freebsd-x64.tar.gz
  • Private.SourceBuilt.Artifacts.9.0.100-alpha.1.24067.1.freebsd.13-x64.tar.gz
  1. Then make NO_CHECKSUM=1 should work
  2. I hit one error, with missing Nuget for System.CommandLine - I just edited work/dotnet-9.0.0-preview.1.24080.9/src/nuget-client/Directory.Packages.props, line 11, changed version to 2.0.0-beta4.24060.1 (the one from artifacts) - hope this is just temp error :)
  3. And there it is - https://github.com/sec/dotnet-core-freebsd-source-build/releases/tag/9.0.0-preview.1-vmr - 9.0.100-preview.1 for x64 and arm64
  4. Next preview builds of 9, should be easy, just bump number of VMR tag and it should work - we'll see
  5. Now I just need to finally get some even more time and read/figure out how to run runtime/aspnet tests - if any1 already done that, with step-by-step for dummies, please share :) (I was following docs some some time ago, but it was failing with lot of weird errors)
  6. Also rerunning the same builds, but using this native SDK as bootstrap - just replace 2 files and re-run - after next next free time - or wait for preview.2 :)

@arrowd - as you are port master - do you think it will make sense to also have dotnet9 flavor of the port - is it possible to have common base with 8 and 9 (9 require almost no patches, for preview.1 there are few, but most of them are already upstreamed and I hope they will be included in later previews). If yes, then some changes are needed to match naming for rtm and preview build, as it fail at finishing port build, just cosmetic changes if you ask me, but I'm not an expert.

Having both SDK and Private.SourceBuilt.Artifacts should be all needed to build new versions of SDK, both for 8 (I've already done builds for 8.0.1 and 8.0.2 just by bumping numbers in Makefile on my machines) and 9 (time will tell).

@joperator That's how I got those :)

@arrowd
Copy link
Contributor

arrowd commented Feb 22, 2024

@arrowd - as you are port master - do you think it will make sense to also have dotnet9 flavor of the port - is it possible to have common base with 8 and 9 (9 require almost no patches, for preview.1 there are few, but most of them are already upstreamed and I hope they will be included in later previews).

Yes, absolutely. Just like we have several LLVM's in Ports, in the same way we can have several .NET's. We indeed should structure these ports to have some common base - this is achieved by so-called master/slave ports.

As I know nothing of .NET - which version should be default? Or even there should be no default at all? In other words, should we have lang/dotnetX, lang/dotnetY, etc rather than lang/dotnetX, lang/dotnetY and lang/dotnet?

If yes, then some changes are needed to match naming for rtm and preview build, as it fail at finishing port build, just cosmetic changes if you ask me, but I'm not an expert.

Yep, I had a feeling that I was doing something wrong in all that versioning.

P.S.

Then make NO_CHECKSUM=1 should work

You can use make makesum to regenerate the distinfo.

@Thefrank
Copy link
Contributor

@sec 's workflow is the better way of approaching too-new dotNET versions.

  • I opted to use the ports framework for the VMR build and Microsoft's docker containers with AZP for the cross-builds. Both methods work. Pick whatever you are more familiar. I am lazy and look for automation where possible.
  • Cross builds are easy. We have tags and the linux sdk usually has runtime, sdk or installer, and aspnetcore commits if you want to fish them for native builds they usually work :D
  • I still manually make a patch like dotnet/installer@9ba09a7 so it will bundle the correct items. Mine are with RID injection for older dotNET versions too but hardly a requirement. I never upstreamed the linux packaging workaround because I thought it too hacky and did not know if it would break packaging elsewhere
  • For ports 8.0.100's output can be used for 8.0.101 and that can be used for 8.0.102...etc. AFAIK VMR will stick with .1xx SDKs. We already have 8.0.20x tags elsewhere for portable builds.
  • The (official) port also handles the hosting of the artifacts needed instead of using some random GH. FreeBSD hosting is much more trusted than "trust me, this repo is fine". Replicating prep.sh via ports Makefile was a fun exercise but added far to much bloat to what could be reduced down to a few manually added items and alternative file hosting. I am still bad at awk


  • OpenSSL3 support for FreeBSD is still pending in PR
  • Build fix for FreeBSD 14 still needs to be made PRable. The official port already uses it as-is.
  • That should(?) cover it for NET9... Outside of test failures.
  • I still need to set my nightly build+test system back up and get back to working on test failures
  • I still need to clean up and publish my "95%-brain-off cross-build YML". Not sure who else would use this but hey.

@joperator
Copy link

@sec I haven't tried it yet, but your instructions sound very promising to me. Thank you very much for investigating this.

@arrowd I used your dotnet package to build and test the Azure Pipelines Agent with my adjustments and it just worked immediately! So, as soon as they are merged and find their way into an official release of the azure-pipelines-agent, we can provide a port for it. One problem could be that building .NET apps is often heavily dependent on NuGet packages. In the FreeBSD ports collection, these must therefore be restored in advance of the actual build, which isn't always trivial.

As I know nothing of .NET - which version should be default? Or even there should be no default at all? In other words, should we have lang/dotnetX, lang/dotnetY, etc rather than lang/dotnetX, lang/dotnetY and lang/dotnet?

We should definitely have a lang/dotnetX port for each major version of .NET. I think a default lang/dotnet meta port pointing to the latest version would also be useful, as this also seems to be the case on Linux. I'm not an expert on this, but perhaps someone from Microsoft will share the design decisions for the package infrastructure on Linux so we can decide if they're suitable for FreeBSD as well?
I'd even go one step further and provide separate ports for Runtime, ASP.NET, SDK and so on, as this is also done on Linux. With lang/dotnetX in BUILD_DEPENDS I guess it should be no problem to create them.

@arrowd
Copy link
Contributor

arrowd commented Feb 22, 2024

One problem could be that building .NET apps is often heavily dependent on NuGet packages. In the FreeBSD ports collection, these must therefore be restored in advance of the actual build, which isn't always trivial.

The important thing here is that all files required for the build should be fetched beforehand. The port should stop accessing network right after make fetch stage. This becomes a problem for some languages which is usually solved by

  1. Adding a way to infer all the dependencies from the initial distfile.
  2. Implementing a way to download all the dependencies separately and to hook them into the build.

For example, rust ports have a special CARGO_CRATES knob listing all Rust packages that are required for building the port. This knob is auto-generated by code from Mk/Uses/cargo.mk. The values from this knob are then used to populate MASTER_SITES and DISTFILES and to fill distinfo. Then during make extract all these pre-fetched packages gets extracted together with the port itself.

I guess, we'll need to come up with same machinery for ports that use .NET? Are NuGet package hosted on some centralized CDN where we can fetch them by direct URL?

@joperator
Copy link

Are NuGet package hosted on some centralized CDN where we can fetch them by direct URL?

For NuGet packages there is nuget.org, which appears to be a CDN, where you can also download them by URL. However, NuGet allows the specification of other so-called packages sources, which is done, for example, in the NuGet.Config files of azure-pipelines-agent.

@rmszc81
Copy link

rmszc81 commented Mar 5, 2024

Hello guys, first, thank you so much for your effort. It's great to see this finally happen.

Today, I decided to move some of my personal .NET 8 projects to FreeBSD 14, inside a jail, of course, and then, I got 2 problems:

1st, it was necessary to set allow.mlock=1 to the jail configuration. otherwise, it wasn't possible even to run a dotnet new command. problem solved.

then, 2nd, now I'm facing the following issue when I'm running dotnet publish -c Release -o /app/bin:

error NU1101: Unable to find package Microsoft.NETCore.App.Host.freebsd-x64. No packages exist with this id in source(s): nuget.org

any idea?

Once again, thanks a lot.
Best!

@rmszc81

@sec
Copy link
Contributor

sec commented Mar 5, 2024

@rmszc81 First problem is just the way it is, you have to do it.

For the second part, you have two choices:

  • either create local directory, download those nuget's (either from port build output or from mine's or @Thefrank releases) and add that directory as nuget source. You can do that per project or .nuget/NuGet/NuGet.Config
  • second option is to use private nuget feed with those, as we don't have nothing "semi-official", for example I'm maintaining my own feed with outputs both for x64 and arm64 - check the info/link in here https://github.com/sec/dotnet-freebsd-nuget-feed
  • you can albo build dotnet from ports, it will generate Private.SourceBuilt.Artifacts which should also contain all the needed nuget's (or you can use the one that's bootstraping dotnet port build, just make fetch should get them, then extract that to local dir and reference in nuget.config).

@nkosi23
Copy link

nkosi23 commented Mar 5, 2024

@sec Interesting! Since the NuGet packages are generated when building the port, would it be a good idea to add this custom NuGet feed automatically as a post-install script of the package?

We could for example copy the relevant packages under Private.SourceBuilt.Artifacts to a location such as: /usr/local/dotnet/nuget and create the .nuget/NuGet/NuGet.Config file pointing to this directory

@sec
Copy link
Contributor

sec commented Mar 5, 2024

That's good idea, we could copy FreeBSD specific nuget's after build and add pkg-message informing user to reference this directory in nuget.config - @arrowd what do you think?

@akoeplinger
Copy link
Member

akoeplinger commented Mar 5, 2024

Normally these packages are in the packs folder in the sdk root when source-building .NET for Linux distros. I think that doesn't happen for non-source-build but you can put them there.

@arrowd
Copy link
Contributor

arrowd commented Mar 5, 2024

Sounds good. @sec can you make a patch for the port yourself? It should be too hard and you seem to already know what and where to put stuff.

@akoeplinger
Copy link
Member

akoeplinger commented Mar 5, 2024

@rmszc81
Copy link

rmszc81 commented Mar 5, 2024

Hello everyone!

My solution was to run make fetch (as suggested by @sec), then, I extracted the contents of the artifacts package into /var/cache/nuget and added this directory to my Nuget.config file configuration.

So far, so good, the project was built, published to /app/bin inside the jail but, now I have a problem with the encrypted SQLite database, aka sqlcipher that says that some .so files are missing. And well, they are, lol.

As this problem is not related to the .NET framework, I'll go after the solution in the appropriated place. Besides, as my app has support to Postgres as well, not having sqlcipher is not an issue at all.

Anyway, I'd like to say thanks again to @sec, @nkosi23, @akoeplinger and @arrowd for the support. I went through all the links and I found a lot of cool stuff.

See you guys around.
Best!

@rmszc81

@sec
Copy link
Contributor

sec commented Mar 5, 2024

As for SQLite, it's old known issue (ericsink/SQLitePCL.raw#176) - simplest you can do is to either create symbolic link e_sqlite3.so in /usr/local/lib to your SQLite version.

@rmszc81
Copy link

rmszc81 commented Mar 6, 2024

hello @sec,
in this case, it's the e_sqlcipher.so.

I'll have a look on this in another time. But thanks anyway, you helped a lot already ^^

@sec
Copy link
Contributor

sec commented Mar 14, 2024

Well... As building 8.0.3 was no problem... Then 9.preview-2 to build from VMR require 9.preview-2 (just few versions behind, using preview-1 doesn't work), I really don't get it...

@davidchisnall
Copy link

1st, it was necessary to set allow.mlock=1 to the jail configuration. otherwise, it wasn't possible even to run a dotnet new command. problem solved.

If I remember correctly, the mlock call is there to provide support for the asymmetric lock, which requires executing a full barrier on other threads. This is implemented on FreeBSD by updating a page mapping that must be guaranteed to trigger IPIs.

FreeBSD 13.3 and 14 both include a Linux-compatible membarrier, and so the fallback code should no longer be required. This should remove the need for mlock.

Unfortunately, even though this work was sponsored by the FreeBSD Foundation, it landed without an accompanying man page so you have to poke at the headers and the git history to know that it's there (@emaste, perhaps the Foundation could also sponsor the docs?).

Note that, unfortunately, you can't use the header to detect the features that are supported because the header provides a full set of the Linux #defines but the kernel implements only a subset. I believe the subset is sufficient for .NET.

@Thefrank
Copy link
Contributor

So as the SDK does not accept "General Support" via issues...

How is the installer built now that the Installer repo is gone for all versions of dotNET? The SDK has always come across as "magic" to me.

From the old installer repo:
The RID was patched in GenerateBundledVersions.targets and if crossbuilding a patch was made to GetRuntimeInformation.targets so that it would include the targeted OS items instead of the host OS items. The former is gone entirely (how are supported RIDs generated?) and the later was trimmed down to a few lines, removing the FreeBSD reference entirely. Right now, I am just looking at dotNET6 but I assume this is similar across all in-support versions.

@jkotas
Copy link
Member

jkotas commented Jun 13, 2024

Installer repo is gone for all versions of dotNET?

The branches for shipped version of .NET are still there and in use.

@Thefrank
Copy link
Contributor

ah! I see my mistake. I was tracking SDK branches until a new one would come up, then follow that: 6.0.1xx -> 6.0.2xx -> etc. I was looking for a 6.0.423 which does not exist. From the core repo I see it only lists a 6.0.1xx branch for the SDK and there is a 6.0.131 in the installer repo even though it is unlisted in core. I will give this a shot!

pointy hat squarely on me here.

@Foxtrek64
Copy link
Contributor

Can we update the OP with what's left of the to-do list on this? I'm definitely interested in contributing if I can, especially considering that I personally want to see BSD support. There was a post here which aggregated a to-do list, but it has everything completed.

@Thefrank
Copy link
Contributor

@Foxtrek64

A quick summary of os-freebsd FreeBSD OS

Documentation:
#31238

Implementation:
#3320

  • Major.Minor is a bit more important as FreeBSD ABI changes can impact dotnet

#26633

  • We use libinotify shim instead of native kqueue.
  • This would remove a dependency but I am not sure how much faster / better it would be as neither libinotify nor kqueue are designed for monitoring large number of files; this hits mediaplayers and backup tools built with dotnet.
  • Something "better" would need to come from the FreeBSD side.

#10519

  • Not sure on this one. I think this is more FreeBSD issue than dotnet issue? We don't have libthr2 on FreeBSD

#8544

  • This covers all non-windows builds

No issue but mentioned:

#14537 (comment)


Pending PRs:

#103187

  • Checks system uses a FreeBSD 13 based image so this was missed.

Others I missed?

@Thefrank
Copy link
Contributor

Thefrank commented Jun 29, 2024

WALL OF STUFF WARNING

dotNET9 preview 5 cross compiles from linux with two small caveats:

  • GetRuntimeInformation.targets patch needed to be reapplied to the new file location. Native builds do not need this. AFAIK native builds require no patching currently!
  • ILCompiler made by runtime still contains Linux ELFs despite being labeled for freebsd-x64

/p:BundleRuntimePacks=true seems to work. The packs path contains the runtime and aspnetcore for FreeBSD as mostly symlinks.
/p:BundleNativeAotCompiler=true is not in preview 5 and hopefully will operate like BundleRuntimePacks

frank@freebsd:~/sdktest $ uname -a
FreeBSD freebsd 14.1-RELEASE FreeBSD 14.1-RELEASE releng/14.1-n267679-10e31f0946d8 GENERIC amd64
---- Built using the CBL Mariner image with FreeBSD 13.2 crossrootfs

frank@freebsd:~/sdktest $ ./dotnet --info
.NET SDK:
 Version:           9.0.100-preview.5.24307.3
 Commit:            35b2c21ea6
 Workload version:  9.0.100-manifests.6407b7e4
 MSBuild version:   17.11.0-preview-24279-02+b963c24ef

Runtime Environment:
 OS Name:     FreeBSD
 OS Version:  14
 OS Platform: FreeBSD
 RID:         freebsd-x64
 Base Path:   /home/frank/sdktest/sdk/9.0.100-preview.5.24307.3/

.NET workloads installed:
Configured to use loose manifests when installing new manifests.
There are no installed workloads to display.

Host:
  Version:      9.0.0-preview.5.24306.7
  Architecture: x64
  Commit:       a5cc707d97

.NET SDKs installed:
  9.0.100-preview.5.24307.3 [/home/frank/sdktest/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 9.0.0-preview.5.24306.11 [/home/frank/sdktest/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 9.0.0-preview.5.24306.7 [/home/frank/sdktest/shared/Microsoft.NETCore.App]

Other architectures found:
  None

Environment variables:
  Not set

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

----

frank@freebsd:~/sdktest $ dotnet publish -r freebsd-x64 console/console.csproj /p:PublishAot=true
-sh: dotnet: not found
frank@freebsd:~/sdktest $ ./dotnet publish -r freebsd-x64 console/console.csproj /p:PublishAot=true
    /home/frank/sdktest/console/console.csproj : error NU1101: Unable to find package runtime.freebsd-x64.Microsoft.DotNet.ILCompiler. No packages exist with this id in source(s): nuget.org
    /home/frank/sdktest/console/console.csproj : error NU1101: Unable to find package runtime.freebsd-x64.Microsoft.DotNet.ILCompiler. No packages exist with this id in source(s): nuget.org

---

Restore failed with 2 error(s) in 3.6s
frank@freebsd:~/sdktest $ ./dotnet publish -r freebsd-x64 console/console.csproj /p:PublishReadyToRun=true
    /home/frank/sdktest/console/console.csproj : error NU1101: Unable to find package Microsoft.NETCore.App.Crossgen2.freebsd-x64. No packages exist with this id in source(s): nuget.org

Restore failed with 1 error(s) in 3.9s

neither of these work without external NuGet packages.
https://reviews.freebsd.org/D44561 might be the better solution if items like crossgen2 and ilc are not able to be added outside of source building

Crossgen2 NuGet correctly contains FreeBSD ELFs:

$file crossgen2 
crossgen2: ELF 64-bit LSB pie executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 13.2, FreeBSD-style, BuildID[sha1]=54a7f1c2a4752435c2cffd15eeb959f609966907, stripped

ILCompiler does not: it contains a mix of FreeBSD libraries and Linux ELFs

$ ./ * | xargs file
./:                                           directory
./ILCompiler.RyuJit.pdb:                      Microsoft Roslyn C# debugging symbols version 1.0
./libclrjit_unix_x64_x64.so:                  ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, for FreeBSD 13.2, BuildID[sha1]=66177aebc4ab51f16fe1e6a5faa90a7ade09b674, stripped
./libclrjit_win_x86_x64.so:                   ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, for FreeBSD 13.2, BuildID[sha1]=79ecdf1053497bde0393928dee1a727bc6b6b6a1, stripped
./libclrjit_universal_arm_x64.so:             ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, for FreeBSD 13.2, BuildID[sha1]=b15d888e793cca18b6dd42b3b672f7144fbe45ec, stripped
./libjitinterface_x64.so:                     ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, for FreeBSD 13.2, BuildID[sha1]=02a2d4a17bcbd0ff35f3caca9252853f95529a3c, stripped
./ILCompiler.TypeSystem.pdb:                  Microsoft Roslyn C# debugging symbols version 1.0
./ILCompiler.DependencyAnalysisFramework.pdb: Microsoft Roslyn C# debugging symbols version 1.0
./ILCompiler.Compiler.pdb:                    Microsoft Roslyn C# debugging symbols version 1.0
./libclrjit_universal_arm64_x64.so:           ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, for FreeBSD 13.2, BuildID[sha1]=0b476dc684291af72ab673b77dccbbf7f386cbf8, stripped
./libclrjit_win_x64_x64.so:                   ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, for FreeBSD 13.2, BuildID[sha1]=a58827fd1b7ac68612408fa5e13c7db9091938a2, stripped
./ilc.pdb:                                    Microsoft Roslyn C# debugging symbols version 1.0
./ilc:                                        ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=41cb9e347020ad19b6402190528b968b6850c46f, stripped

Other than Linux ELFs in FreeBSD packs everything is going smooth from the cross compile side for .net 9

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area-Meta enhancement Product code improvement that does NOT require public API changes/additions needs-further-triage Issue has been initially triaged, but needs deeper consideration or reconsideration os-freebsd FreeBSD OS
Projects
No open projects
Status: Untriaged
Development

Successfully merging a pull request may close this issue.