Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Validate container improvements with .NET 6 #53149

Closed
richlander opened this issue May 23, 2021 · 4 comments
Closed

Validate container improvements with .NET 6 #53149

richlander opened this issue May 23, 2021 · 4 comments

Comments

@richlander
Copy link
Member

richlander commented May 23, 2021

Validate container improvements with .NET 6

We've made various changes in .NET 6 to improve aspects of container resource limits governance, particularly for Windows containers. This issue demonstrates the new capabilities and their current behavior using the dotnetapp sample app.

Issues:

Changes:

The last change is to a sample, which will be used to demonstrate the others changes. It is a .NET 5 app. As a result, various roll-forward settings are used to coerce the app to run on .NET 6.

At the time of writing, the relevant changes have not been released in a public .NET 6 preview. As a result, I updated and locally rebuilt various Dockerfiles with daily builds from dotnet/installer.

Relevant docs:

Linux -- CPU and memory limits

The following test demonstrates the .NET runtime honoring CPU and resource limits.

C:\git\dotnet-docker\samples\dotnetapp>docker run --rm --cpus 3 -m 100mb -v C:\git\dotnet-docker\samples\dotnetapp\bin\Debug\net5.0:/app -w /app -e DOTNET_ROLL_FORWARD=LatestMajor -e DOTNET_ROLL_FORWARD_TO_PRERELEASE=1 mcr.microsoft.com/dotnet/runtime:6.0 dotnet dotnetapp.dll
         42
         42              ,d                             ,d
         42              42                             42
 ,adPPYb,42  ,adPPYba, MM42MMM 8b,dPPYba,   ,adPPYba, MM42MMM
a8"    `Y42 a8"     "8a  42    42P'   `"8a a8P_____42   42
8b       42 8b       d8  42    42       42 8PP"""""""   42
"8a,   ,d42 "8a,   ,a8"  42,   42       42 "8b,   ,aa   42,
 `"8bbdP"Y8  `"YbbdP"'   "Y428 42       42  `"Ybbd8"'   "Y428

.NET 6.0.0-preview.2.21154.6
Debian GNU/Linux bullseye/sid

OSArchitecture: X64
ProcessorCount: 3
TotalAvailableMemoryBytes: 75.00 MiB
cfs_quota_us: 300000
usage_in_bytes: 25821184 24.63 MiB
limit_in_bytes: 104857600 100.00 MiB

Similar, but with CPU sets (AKA CPU affinity) is used instead.

C:\git\dotnet-docker\samples\dotnetapp>docker run --rm --cpuset-cpus "1-3,5" -m 100mb -v C:\git\dotnet-docker\samples\dotnetapp\bin\Debug\net5.0:/app -w /app -e DOTNET_ROLL_FORWARD=LatestMajor -e DOTNET_ROLL_FORWARD_TO_PRERELEASE=1 mcr.microsoft.com/dotnet/runtime:6.0 dotnet dotnetapp.dll
         42
         42              ,d                             ,d
         42              42                             42
 ,adPPYb,42  ,adPPYba, MM42MMM 8b,dPPYba,   ,adPPYba, MM42MMM
a8"    `Y42 a8"     "8a  42    42P'   `"8a a8P_____42   42
8b       42 8b       d8  42    42       42 8PP"""""""   42
"8a,   ,d42 "8a,   ,a8"  42,   42       42 "8b,   ,aa   42,
 `"8bbdP"Y8  `"YbbdP"'   "Y428 42       42  `"Ybbd8"'   "Y428

.NET 6.0.0-preview.2.21154.6
Debian GNU/Linux bullseye/sid

OSArchitecture: X64
ProcessorCount: 4
TotalAvailableMemoryBytes: 75.00 MiB
usage_in_bytes: 6459392 6.16 MiB
limit_in_bytes: 104857600 100.00 MiB

The latest .NET 6 Preview is used for these examples, since these capabilities are all pre .NET 6.

Linux -- custom processor count

The following test demonstrates the .NET runtime honoring the new DOTNET_PROCESSOR_COUNT ENV, which provides a different value to Environment.ProcessorCount and the equivalent setting in the native runtime. It doesn't directly related to and does not affect the docker run --cpus mechanism, but is intended as a higher-level signal for scaling algorithms.

C:\git\dotnet-docker\samples\dotnetapp>docker run --rm --cpus 3 -m 100mb -e DOTNET_PROCESSOR_COUNT=10 -v C:\git\dotnet-docker\samples\dotnetapp\bin\Debug\net5.0:/app -w /app -e DOTNET_ROLL_FORWARD=LatestMajor -e DOTNET_ROLL_FORWARD_TO_PRERELEASE=1 dotnet6runtime dotnet dotnetapp.dll
         42
         42              ,d                             ,d
         42              42                             42
 ,adPPYb,42  ,adPPYba, MM42MMM 8b,dPPYba,   ,adPPYba, MM42MMM
a8"    `Y42 a8"     "8a  42    42P'   `"8a a8P_____42   42
8b       42 8b       d8  42    42       42 8PP"""""""   42
"8a,   ,d42 "8a,   ,a8"  42,   42       42 "8b,   ,aa   42,
 `"8bbdP"Y8  `"YbbdP"'   "Y428 42       42  `"Ybbd8"'   "Y428

.NET 6.0.0-preview.6.21276.13
Debian GNU/Linux bullseye/sid

OSArchitecture: X64
ProcessorCount: 10
TotalAvailableMemoryBytes: 75.00 MiB
cfs_quota_us: 300000
usage_in_bytes: 6885376 6.57 MiB
limit_in_bytes: 104857600 100.00 MiB

I used a .NET 6 Preview 6 build to demonstrate this change. It works as expected.

Let's double check that we get the right behavior if no CPU limits are set.

C:\git\dotnet-docker\samples\dotnetapp>docker run --rm -m 100mb -e DOTNET_PROCESSOR_COUNT=10 -v C:\git\dotnet-docker\samples\dotnetapp\bin\Debug\net5.0:/app -w /app -e DOTNET_ROLL_FORWARD=LatestMajor -e DOTNET_ROLL_FORWARD_TO_PRERELEASE=1 dotnet6runtime dotnet dotnetapp.dll
         42
         42              ,d                             ,d
         42              42                             42
 ,adPPYb,42  ,adPPYba, MM42MMM 8b,dPPYba,   ,adPPYba, MM42MMM
a8"    `Y42 a8"     "8a  42    42P'   `"8a a8P_____42   42
8b       42 8b       d8  42    42       42 8PP"""""""   42
"8a,   ,d42 "8a,   ,a8"  42,   42       42 "8b,   ,aa   42,
 `"8bbdP"Y8  `"YbbdP"'   "Y428 42       42  `"Ybbd8"'   "Y428

.NET 6.0.0-preview.6.21276.13
Debian GNU/Linux bullseye/sid

OSArchitecture: X64
ProcessorCount: 10
TotalAvailableMemoryBytes: 75.00 MiB
usage_in_bytes: 6713344 6.40 MiB
limit_in_bytes: 104857600 100.00 MiB

We do.

Let's check if we get the correct value with CPU affinity.

C:\git\dotnet-docker\samples\dotnetapp>docker run --rm --cpuset-cpus "1-3,5" -m 100mb -e DOTNET_PROCESSOR_COUNT=10 -v C:\git\dotnet-docker\samples\dotnetapp\bin\Debug\net5.0:/app -w /app -e DOTNET_ROLL_FORWARD=LatestMajor -e DOTNET_ROLL_FORWARD_TO_PRERELEASE=1 dotnet6runtime dotnet dotnetapp.dll
         42
         42              ,d                             ,d
         42              42                             42
 ,adPPYb,42  ,adPPYba, MM42MMM 8b,dPPYba,   ,adPPYba, MM42MMM
a8"    `Y42 a8"     "8a  42    42P'   `"8a a8P_____42   42
8b       42 8b       d8  42    42       42 8PP"""""""   42
"8a,   ,d42 "8a,   ,a8"  42,   42       42 "8b,   ,aa   42,
 `"8bbdP"Y8  `"YbbdP"'   "Y428 42       42  `"Ybbd8"'   "Y428

.NET 6.0.0-preview.6.21276.13
Debian GNU/Linux bullseye/sid

OSArchitecture: X64
ProcessorCount: 10
TotalAvailableMemoryBytes: 75.00 MiB
usage_in_bytes: 5898240 5.63 MiB
limit_in_bytes: 104857600 100.00 MiB

We do.

Windows -- CPU and memory limits

The major container improvement changes were made on Windows in .NET 6. Let's try with the .NET 6 preview 3. We're expecting that CPU limits are not honored with process isolated containers.

C:\git\dotnet-docker\samples\dotnetapp>docker run --rm --cpus 3 -m 100mb --isolation=process -v C:\git\dotnet-docker\samples\dotnetapp\bin\Debug\net5.0:C:\app -w C:\app -e DOTNET_ROLL_FORWARD=LatestMajor mcr.microsoft.com/dotnet/runtime:6.0 dotnet dotnetapp.dll
         42
         42              ,d                             ,d
         42              42                             42
 ,adPPYb,42  ,adPPYba, MM42MMM 8b,dPPYba,   ,adPPYba, MM42MMM
a8"    `Y42 a8"     "8a  42    42P'   `"8a a8P_____42   42
8b       42 8b       d8  42    42       42 8PP"""""""   42
"8a,   ,d42 "8a,   ,a8"  42,   42       42 "8b,   ,aa   42,
 `"8bbdP"Y8  `"YbbdP"'   "Y428 42       42  `"Ybbd8"'   "Y428

.NET 6.0.0-preview.3.21201.4
Microsoft Windows 10.0.19042

OSArchitecture: X64
ProcessorCount: 16
TotalAvailableMemoryBytes: 75.00 MiB

As expected, processor count is incorrect.

Note: You will see the correct behavior with Hyper-V isolated containers. The functionality gap is with process-isolated containers.

Let's try .NET 6 Preview 6.

C:\git\dotnet-docker\samples\dotnetapp>docker run --rm --cpus 3 -m 100mb --isolation=process -v C:\git\dotnet-docker\samples\dotnetapp\bin\Debug\net5.0:C:\app -w C:\app -e DOTNET_ROLL_FORWARD=LatestMajor dotnet6runtime dotnet dotnetapp.dll
         42
         42              ,d                             ,d
         42              42                             42
 ,adPPYb,42  ,adPPYba, MM42MMM 8b,dPPYba,   ,adPPYba, MM42MMM
a8"    `Y42 a8"     "8a  42    42P'   `"8a a8P_____42   42
8b       42 8b       d8  42    42       42 8PP"""""""   42
"8a,   ,d42 "8a,   ,a8"  42,   42       42 "8b,   ,aa   42,
 `"8bbdP"Y8  `"YbbdP"'   "Y428 42       42  `"Ybbd8"'   "Y428

.NET 6.0.0-preview.6.21272.4
Microsoft Windows 10.0.19042

OSArchitecture: X64
ProcessorCount: 3
TotalAvailableMemoryBytes: 75.00 MiB

Perfect.

Now let's try CPU affinity, like we did with Linux.

C:\git\dotnet-docker\samples\dotnetapp>docker run --rm --cpuset-cpus="1-3" -m 100mb --isolation=process -v C:\git\dotnet-docker\samples\dotnetapp\bin\Debug\net5.0:C:\app -w C:\app -e DOTNET_ROLL_FORWARD=LatestMajor -e DOTNET_PROCESSOR_COUNT=10 dotnet6runtime dotnet dotnetapp.dll
docker: Error response from daemon: invalid option: Windows does not support CpusetCpus.
See 'docker run --help'.

It isn't supported. I didn't know that.

Windows -- custom processor count

Just like with Linux, let's try setting a custom processor count.

C:\git\dotnet-docker\samples\dotnetapp>docker run --rm --cpus 3 -m 100mb -e DOTNET_PROCESSOR_COUNT=10 --isolation=process -v C:\git\dotnet-docker\samples\dotnetapp\bin\Debug\net5.0:C:\app -w C:\app -e DOTNET_ROLL_FORWARD=LatestMajor dotnet6runtime dotnet dotnetapp.dll
         42
         42              ,d                             ,d
         42              42                             42
 ,adPPYb,42  ,adPPYba, MM42MMM 8b,dPPYba,   ,adPPYba, MM42MMM
a8"    `Y42 a8"     "8a  42    42P'   `"8a a8P_____42   42
8b       42 8b       d8  42    42       42 8PP"""""""   42
"8a,   ,d42 "8a,   ,a8"  42,   42       42 "8b,   ,aa   42,
 `"8bbdP"Y8  `"YbbdP"'   "Y428 42       42  `"Ybbd8"'   "Y428

.NET 6.0.0-preview.6.21276.13
Microsoft Windows 10.0.19042

OSArchitecture: X64
ProcessorCount: 10
TotalAvailableMemoryBytes: 75.00 MiB

As expected, it behaves just like demonstrated with Linux.

Validating isolation mode

Here's what I did to validate the isolation mode.

C:\git\dotnet-docker\samples\dotnetapp>docker run --rm -it --isolation=process --name dotnet6runtime dotnet6runtime

And then in another command prompt.

C:\>docker ps
CONTAINER ID   IMAGE            COMMAND                    CREATED          STATUS          PORTS     NAMES
ee11156d0488   dotnet6runtime   "c:\\windows\\system32…"   15 seconds ago   Up 14 seconds             dotnet6runtime

C:\>docker inspect dotnet6runtime | findstr Isolation
            "Isolation": "process",
@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

@jeffschwMSFT
Copy link
Member

cc @mangod9

@mangod9 mangod9 added area-VM-coreclr and removed untriaged New issue has not been triaged by the area owner labels May 24, 2021
@mangod9 mangod9 added this to the 6.0.0 milestone May 24, 2021
@richlander
Copy link
Member Author

richlander commented May 28, 2021

I updated the testing data above with the latest build. Everything looks great. Thanks!

@AntonLapounov @AaronRobinsonMSFT @jkotas

This issue can be closed at any time.

@mangod9
Copy link
Member

mangod9 commented May 28, 2021

Nice.

@msftbot msftbot bot locked as resolved and limited conversation to collaborators Jun 27, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
Development

No branches or pull requests

4 participants