[Broken Build]: Linux build with ipv6 connectivity

[ya-mouse]

Issue Description

On a system having ipv6 connectivity, the build is failing with a lot of Connection reset by peer messages:

$ ./build.sh
...
  Restored /home/mouse/.nuget/packages/microsoft.dotnet.arcade.sdk/8.0.0-beta.23463.1/tools/Tools.proj (in 1.16 min).
  Determining projects to restore...
  Retrying 'FindPackagesByIdAsync' for source 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/d1622942-d16f-48e5-bc83-96f4539e7601/nuget/v3/flat2/system.threading.timer/index.json'.
  An error occurred while sending the request.
    Unable to read data from the transport connection: Connection reset by peer.
    Connection reset by peer
  Retrying 'FindPackagesByIdAsync' for source 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/825db618-e3eb-4426-ba54-b1d6e6c944d8/nuget/v3/flat2/system.security.cryptography.primitives/index.json'.
...

However, I can download individual packages via ipv6:

wget https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/45bacae2-5efb-47c8-91e5-8ec20c22b4f8/nuget/v3/flat2/system.text.json/7.0.3/system.text.json.7.0.3.nupkg
--2023-11-25 16:22:51--  https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/45bacae2-5efb-47c8-91e5-8ec20c22b4f8/nuget/v3/flat2/system.text.json/7.0.3/system.text.json.7.0.3.nupkg
Resolving pkgs.dev.azure.com (pkgs.dev.azure.com)... 2620:1ec:21::20, 13.107.42.20
Connecting to pkgs.dev.azure.com (pkgs.dev.azure.com)|2620:1ec:21::20|:443... connected.
HTTP request sent, awaiting response... 303 See Other
Location: https://p2ovsblobprodcus312.vsblob.vsassets.io/b-b55de4ed4b5a4215a8e40a0a5f71e7d8/078804D398BAF5B006EBD157302B95899EB13E0C504861FCF943C85E887A291A00.blob?sv=2019-07-07&sr=b&si=1&sig=t0zFKQDxcCJ55KHfvhMWPlzx2d4lnqvC8uMPPTa4uWg%3D&spr=https&se=2023-11-26T10%3A03%3A39Z&rscl=x-e2eid-b06be6ab-dc3f4ff6-ae7ba0ed-ded7bbbf-session-b06be6ab-dc3f4ff6-ae7ba0ed-ded7bbbf&rscd=attachment%3B%20filename%3D%22system.text.json.7.0.3.nupkg%22&P1=1700936571&P2=2&P3=2&P4=zspUl9cI%2bQ3dWjHCSi6uoYgi26mnshprCDdcfjiD3GE%3d [following]
--2023-11-25 16:22:51--  https://p2ovsblobprodcus312.vsblob.vsassets.io/b-b55de4ed4b5a4215a8e40a0a5f71e7d8/078804D398BAF5B006EBD157302B95899EB13E0C504861FCF943C85E887A291A00.blob?sv=2019-07-07&sr=b&si=1&sig=t0zFKQDxcCJ55KHfvhMWPlzx2d4lnqvC8uMPPTa4uWg%3D&spr=https&se=2023-11-26T10%3A03%3A39Z&rscl=x-e2eid-b06be6ab-dc3f4ff6-ae7ba0ed-ded7bbbf-session-b06be6ab-dc3f4ff6-ae7ba0ed-ded7bbbf&rscd=attachment%3B%20filename%3D%22system.text.json.7.0.3.nupkg%22&P1=1700936571&P2=2&P3=2&P4=zspUl9cI%2bQ3dWjHCSi6uoYgi26mnshprCDdcfjiD3GE%3d
Resolving p2ovsblobprodcus312.vsblob.vsassets.io (p2ovsblobprodcus312.vsblob.vsassets.io)... 2620:1ec:a92::175, 13.107.6.175
Connecting to p2ovsblobprodcus312.vsblob.vsassets.io (p2ovsblobprodcus312.vsblob.vsassets.io)|2620:1ec:a92::175|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1734391 (1.7M) [application/octet-stream]
Saving to: ‘system.text.json.7.0.3.nupkg’

system.text.json.7.0.3.nupkg          100%[=========================================================================>]   1.65M   591KB/s    in 2.9s    

2023-11-25 16:22:54 (591 KB/s) - ‘system.text.json.7.0.3.nupkg’ saved [1734391/1734391]

If I change in /etc/gai.conf the preference to ipv4, then build.sh works without issues.

#
#    For sites which prefer IPv4 connections change the last line to
#
precedence ::ffff:0:0/96  100

$ ./build.sh
...
  Restored /home/mouse/RPM/msbuild/src/Package/MSBuild.VSSetup.Arm64/MSBuild.VSSetup.Arm64.csproj (in 4.02 sec).
  Restored /home/mouse/RPM/msbuild/src/Deprecated/Engine/Microsoft.Build.Engine.csproj (in 5.8 sec).
  Restored /home/mouse/RPM/msbuild/src/Package/Localization/Localization.csproj (in 4.92 sec).
...
Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:01:34.26

Steps to Reproduce

1. Have a system with IPv6 internet connectivity
2. Try to run ./build.sh

Expected Behavior

Successful build

Actual Behavior

Build failed on downloading dependencies

Ask us questions

No response

[baronfel]

cc @zivkan - are there any ipv6 known issues for the Restore task that you know of? Or should we open a new issue at nuget/home (since we can't transfer).

[zivkan]

NuGet basically just uses HttpClient.SendAsync(string url), so IMO it sounds more like a dotnet/runtime issue 🤷

But no, no known ipv6 tracking issue in NuGet/Home.

[ya-mouse]

An strace that doesn't help too much:

1081005 socket(AF_INET6, SOCK_STREAM|SOCK_CLOEXEC, IPPROTO_TCP <unfinished ...>
1081005 <... socket resumed>)           = 1470
1081005 setsockopt(1470, SOL_IPV6, IPV6_V6ONLY, [1], 4 <unfinished ...>
1081005 setsockopt(1470, SOL_IPV6, IPV6_V6ONLY, [0], 4) = 0
1081005 setsockopt(1470, SOL_TCP, TCP_NODELAY, [1], 4) = 0
1081002 connect(1470, {sa_family=AF_INET6, sin6_port=htons(443), sin6_flowinfo=htonl(0), inet_pton(AF_INET6, "2620:1ec:21::20", &sin6_addr), sin6_scope_id=0}, 28 <unfinished ...>
1081006 getsockopt(1470, SOL_SOCKET, SO_ERROR,  <unfinished ...>
1081006 getpeername(1470, {sa_family=AF_INET6, sin6_port=htons(443), sin6_flowinfo=htonl(0), inet_pton(AF_INET6, "2620:1ec:21::20", &sin6_addr), sin6_scope_id=0}, [28]) = 0
1081006 sendto(1470, "\26\3\....", 567, 0, NULL, 0 <unfinished ...>
1081006 <... sendto resumed>)           = 567
...
1081007 recvfrom(1470, "\4;...Microsoft Corporation1.0,\6\3U\4\3\23%Microsoft Azure RSA TLS Issuing CA 040\36\27\r231121155025Z\27\r231221155025Z0\0371\0350\33\6\3U\4\3\23\24AzureRSA04 OCSP Cert0"..., 3872, 0, NULL, NULL) = 1693
1081007 recvfrom(1470, 0x7ef45b7fcbf0, 1, MSG_PEEK, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
1081003 recvfrom(1470,  <unfinished ...>
1081003 <... recvfrom resumed>"\26\...", 8192, 0, NULL, NULL) = 342
1081009 sendto(1470, "\...f", 505, 0, NULL, 0 <unfinished ...>
1081009 <... sendto resumed>)           = 505
1081009 recvfrom(1470, 0x7ef45a7fa0d0, 1, MSG_PEEK, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
1080982 recvfrom(1470, 0x7ef67b01a2c8, 32768, 0, NULL, NULL) = -1 ECONNRESET (Connection reset by peer)
1080982 shutdown(1470, SHUT_RDWR)       = -1 ENOTCONN (Transport endpoint is not connected)

Seems like different threads handle the same socket descriptor for send/recv and nothing suspicious.

[baronfel]

Thanks for the check @zivkan!

In that case I'll forward this to dotnet/runtime and let the experts there tell us if there's something nefarious afoot.

[ya-mouse]

Some logs with -v diag:
build.v4.txt
build.v6.txt

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Issue Description

On a system having ipv6 connectivity, the build is failing with a lot of Connection reset by peer messages:

$ ./build.sh
...
  Restored /home/mouse/.nuget/packages/microsoft.dotnet.arcade.sdk/8.0.0-beta.23463.1/tools/Tools.proj (in 1.16 min).
  Determining projects to restore...
  Retrying 'FindPackagesByIdAsync' for source 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/d1622942-d16f-48e5-bc83-96f4539e7601/nuget/v3/flat2/system.threading.timer/index.json'.
  An error occurred while sending the request.
    Unable to read data from the transport connection: Connection reset by peer.
    Connection reset by peer
  Retrying 'FindPackagesByIdAsync' for source 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/825db618-e3eb-4426-ba54-b1d6e6c944d8/nuget/v3/flat2/system.security.cryptography.primitives/index.json'.
...

However, I can download individual packages via ipv6:

wget https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/45bacae2-5efb-47c8-91e5-8ec20c22b4f8/nuget/v3/flat2/system.text.json/7.0.3/system.text.json.7.0.3.nupkg
--2023-11-25 16:22:51--  https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/45bacae2-5efb-47c8-91e5-8ec20c22b4f8/nuget/v3/flat2/system.text.json/7.0.3/system.text.json.7.0.3.nupkg
Resolving pkgs.dev.azure.com (pkgs.dev.azure.com)... 2620:1ec:21::20, 13.107.42.20
Connecting to pkgs.dev.azure.com (pkgs.dev.azure.com)|2620:1ec:21::20|:443... connected.
HTTP request sent, awaiting response... 303 See Other
Location: https://p2ovsblobprodcus312.vsblob.vsassets.io/b-b55de4ed4b5a4215a8e40a0a5f71e7d8/078804D398BAF5B006EBD157302B95899EB13E0C504861FCF943C85E887A291A00.blob?sv=2019-07-07&sr=b&si=1&sig=t0zFKQDxcCJ55KHfvhMWPlzx2d4lnqvC8uMPPTa4uWg%3D&spr=https&se=2023-11-26T10%3A03%3A39Z&rscl=x-e2eid-b06be6ab-dc3f4ff6-ae7ba0ed-ded7bbbf-session-b06be6ab-dc3f4ff6-ae7ba0ed-ded7bbbf&rscd=attachment%3B%20filename%3D%22system.text.json.7.0.3.nupkg%22&P1=1700936571&P2=2&P3=2&P4=zspUl9cI%2bQ3dWjHCSi6uoYgi26mnshprCDdcfjiD3GE%3d [following]
--2023-11-25 16:22:51--  https://p2ovsblobprodcus312.vsblob.vsassets.io/b-b55de4ed4b5a4215a8e40a0a5f71e7d8/078804D398BAF5B006EBD157302B95899EB13E0C504861FCF943C85E887A291A00.blob?sv=2019-07-07&sr=b&si=1&sig=t0zFKQDxcCJ55KHfvhMWPlzx2d4lnqvC8uMPPTa4uWg%3D&spr=https&se=2023-11-26T10%3A03%3A39Z&rscl=x-e2eid-b06be6ab-dc3f4ff6-ae7ba0ed-ded7bbbf-session-b06be6ab-dc3f4ff6-ae7ba0ed-ded7bbbf&rscd=attachment%3B%20filename%3D%22system.text.json.7.0.3.nupkg%22&P1=1700936571&P2=2&P3=2&P4=zspUl9cI%2bQ3dWjHCSi6uoYgi26mnshprCDdcfjiD3GE%3d
Resolving p2ovsblobprodcus312.vsblob.vsassets.io (p2ovsblobprodcus312.vsblob.vsassets.io)... 2620:1ec:a92::175, 13.107.6.175
Connecting to p2ovsblobprodcus312.vsblob.vsassets.io (p2ovsblobprodcus312.vsblob.vsassets.io)|2620:1ec:a92::175|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1734391 (1.7M) [application/octet-stream]
Saving to: ‘system.text.json.7.0.3.nupkg’

system.text.json.7.0.3.nupkg          100%[=========================================================================>]   1.65M   591KB/s    in 2.9s    

2023-11-25 16:22:54 (591 KB/s) - ‘system.text.json.7.0.3.nupkg’ saved [1734391/1734391]

If I change in /etc/gai.conf the preference to ipv4, then build.sh works without issues.

#
#    For sites which prefer IPv4 connections change the last line to
#
precedence ::ffff:0:0/96  100

$ ./build.sh
...
  Restored /home/mouse/RPM/msbuild/src/Package/MSBuild.VSSetup.Arm64/MSBuild.VSSetup.Arm64.csproj (in 4.02 sec).
  Restored /home/mouse/RPM/msbuild/src/Deprecated/Engine/Microsoft.Build.Engine.csproj (in 5.8 sec).
  Restored /home/mouse/RPM/msbuild/src/Package/Localization/Localization.csproj (in 4.92 sec).
...
Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:01:34.26

Steps to Reproduce

1. Have a system with IPv6 internet connectivity
2. Try to run ./build.sh

Expected Behavior

Successful build

Actual Behavior

Build failed on downloading dependencies

Ask us questions

No response

Author:	ya-mouse
Assignees:	-
Labels:	area-System.Net, untriaged
Milestone:	-

[antonfirsov]

It's very hard to tell anything based on the provided information. @ya-mouse what is your environment? Does it reproduce consistently? Do you see this happening in other environments? If it's a container can you repro on another machine with the same container? Can you get packet captures?

[wfurt]

I think packet captures will be crucial. This looks like some environmental instability as .NET relies purely on OS to do TCP/IP.

This issue has been marked needs-author-action and may be missing some important information.