Support installing workloads in user folder

[tmds]

Fixes #18104

@dsplaisted I've started with the plumbing to get the user location into the commands. Can you take a look and give feedback? Especially about the 4 places I marked with TODO.

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label.

[dsplaisted]

Thanks for getting this started! This looks like it covers the basics of reading workloads from a user local folder.

Were you also planning to update the installation logic so that it would install workloads to the user local folder?

[dsplaisted]

Hey @tmds, any updates on this? Let us know if you need anything.

[omajid]

Hey, @dsplaisted, just an fyi that @tmds is on vacation for about 2 more weeks. It's unlikely that there will be more progress on this until he gets back.

[marcpopMSFT]

@omajid Thanks for the heads up. To give you a sense of schedule, the SDK does not lock down as early as the runtime so we have some more time for .NET 6 but not a ton. We'll be asked to start locking down on changes going into the SDK around mid-September so it'd be good to have this ready by then if you're targeting 6.0.100.

[tmds]

@dsplaisted I've updated the PR based on your feedback. Can you take a look?

There may be some logical mistakes as I don't fully understand the relationship between the metadata, sdk-manifests and packs. Specifically: which of these come with the installer, and which are updated later by the workload command?

I put some things in the "Common" folder but you may prefer them in another location.

I've limited the string userProfileDir argument to the classes that use it, and not propagate it upwards. This parameter can be used in tests, which I haven't added yet.

[tmds]

@dsplaisted I'm looking forward to get some feedback here.

[dsplaisted]

@dsplaisted I'm looking forward to get some feedback here.

Thanks, I'm going to try to review this soon!

[dsplaisted]

This is looking great, thanks a bunch!

As discussed, this should be rebased onto the release/6.0.1xx branch (which currently targets rc2).

@sfoslund Do you think we can / should add automated tests for user local workload installs?

[sfoslund]

Do you think we can / should add automated tests for user local workload installs?

Yes, I think that would be great. You could model the tests off of this one: https://github.com/dotnet/sdk/blob/main/src/Tests/dotnet-workload-install.Tests/GivenDotnetWorkloadInstall.cs#L249 and feel free to ping me if you have any questions.

[tmds]

@dsplaisted thanks for the review.
I have two open questions: #18823 (comment) and #18823 (comment).

[tmds]

You could model the tests off of this one: https://github.com/dotnet/sdk/blob/main/src/Tests/dotnet-workload-install.Tests/GivenDotnetWorkloadInstall.cs#L249 and feel free to ping me if you have any questions.

I have changed GivenDotnetWorkloadInstall so it runs tests against with a userlocal install also. Can you take a look and give feedback?
After that, I'll update some more GivenDotnetWorkload* tests.

[tmds]

@dsplaisted @sfoslund can you do another review?

I did an integration test as follows:

# build
./build.sh

# mark sdk as userlocal
mkdir -p artifacts/bin/redist/Debug/dotnet/metadata/workloads/6.0.100
touch artifacts/bin/redist/Debug/dotnet/metadata/workloads/6.0.100/userlocal

# make dotnet root read only
chmod -R -w artifacts/bin/redist/Debug/dotnet

# create a home folder
mkdir /tmp/home
export HOME=/tmp/home

# test
source eng/dogfood.sh 
dotnet workload list
dotnet workload search
dotnet workload install wasm-tools
dotnet workload list

This succesfully installed wasm-tools into the /tmp/home folder.

[tmds]

@dsplaisted what would be a good place to create the userlocal file in a source-built .NET?

[dleeapho]

what would be a good place to create the userlocal file in a source-built .NET?
@dseefeld can you help?

[dseefeld]

what would be a good place to create the userlocal file in a source-built .NET?
@dseefeld can you help?

Under what conditions should the userlocal file be created?

[dsplaisted]

I believe it should be included in all source-built versions of the .NET SDK. That means that when they install workloads the workloads will always be written to a user-local folder.

For .NET 7 we may need to adjust how this works, as there will probably be some workloads that themselves should be source-built, and not necessarily installed in the user folder.

[tmds]

I believe it should be included in all source-built versions of the .NET SDK. That means that when they install workloads the workloads will always be written to a user-local folder.

Yes.

The file should be created under the dotnet root at /metadata/workloads/<sdkfeatureband>/userlocal. The <sdkfeatureband> version does not include the -suffix. So at 6.0.100 (not 6.0.100-rc1).

For .NET 7 we may need to adjust how this works, as there will probably be some workloads that themselves should be source-built, and not necessarily installed in the user folder.

It would be good to write/update the design doc to be sure different scenarios are covered for source-build.

[dsplaisted]

Thanks a bunch @tmds. 🎉🎉🎉

Could you write a short description of the paths that this uses? IE what it will use as the user local folder, including the various fallbacks on different OS's, as well as where it looks for the marker file.

[tmds]

@dsplaisted no problem. I'm happy we can have a nice UX also for source-build .NET when using workloads. Thanks you and @sfoslund for the quick reviews.

Could you write a short description of the paths that this uses? IE what it will use as the user local folder, including the various fallbacks on different OS's, as well as where it looks for the marker file.

When the dotnet installation has a file at <DOTNET_ROOT>/metadata/workloads/<sdkfeatureband>/userlocal it will use a user folder as the preferred location for installing workloads, and resolving them (in addition to <DOTNET_ROOT>).

The folders that used for workloads under <DOTNET_ROOT> (like sdk-manifests, packs, ...) will be created under <HOME>/.dotnet.

<HOME> is determined from the HOME envvar on Linux and USERPROFILE on Windows. The location can be set also using the DOTNET_CLI_HOME envvar.

[dseefeld]

I believe it should be included in all source-built versions of the .NET SDK. That means that when they install workloads the workloads will always be written to a user-local folder.

Yes.

The file should be created under the dotnet root at /metadata/workloads/<sdkfeatureband>/userlocal. The <sdkfeatureband> version does not include the -suffix. So at 6.0.100 (not 6.0.100-rc1).

Can the file be created when the sdk layout is created? The way to get it only for source-built versions is to condition it on '$(DotNetBuildFromSource)' == 'true'.

[tmds]

@omajid the /metadata/workloads/<sdkfeatureband>/userlocal file needs to end up in one of our packages. The sdk package is probably the right one.

Can the file be created when the sdk layout is created?

If it can place the metadata directory under <DOTNET_ROOT>, it can.

What causes the blazor manifest file to be added in source-build?

[lmcwhirt-insight]

@tmds I am working in a very restricted environment and need to have workloads stored in my Linux userpath. How can I use this to make that happen?

[tmds]

@lmcwhirt-insight consider opening an issue for this use-case in the repo.

A source-built SDK has a file at $DOTNET_ROOT/metadata/workloads/8.0.100/userlocal that causes the SDK to install workloads under the user's home. You can try adding that file in your .NET installation.
Note that the last two digits are always 00. So when your SDK is at version 8.0.101 the file must still be under the 8.0.100 directory.