Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Microsoft.NET.Build.Containers.IntegrationTests are pulling from Docker Hub #42026

Open
akoeplinger opened this issue Jul 8, 2024 · 9 comments
Assignees
Labels

Comments

@akoeplinger
Copy link
Member

akoeplinger commented Jul 8, 2024

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=733130
Build error leg or test failing: Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.WriteToPrivateBasicRegistry
Pull request: https://github.com/dotnet/sdk.git/pull/42019

We need to stop pulling from upstream Docker Hub since we're hitting rate limits.

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "You have reached your pull rate limit",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=733130
Error message validated: [You have reached your pull rate limit]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 7/8/2024 7:20:41 PM UTC

Report

Build Definition Test Pull Request
855463 dotnet/sdk Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation #44494
855453 dotnet/sdk Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation #44477
854912 dotnet/sdk Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation #44203
852292 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry #44405
851428 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.EndToEndTests.ApiEndToEndWithLocalLoad #44406
851247 dotnet/sdk Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation #44276
851248 dotnet/sdk Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation #44405
851260 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.EndToEndTests.ApiEndToEndWithLocalLoad #44407
851225 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry #44403
851073 dotnet/sdk Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation #44320
850518 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry #44376
849555 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.EndToEndTests.ApiEndToEndWithLocalLoad #44345
847314 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry #44296
847105 dotnet/sdk Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation #44267
845628 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry #44207
845611 dotnet/sdk Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation #44254
845094 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry #44239
845069 dotnet/sdk Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation #44198
845022 dotnet/sdk Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation #44236
845006 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry #43627
844980 dotnet/sdk Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation #44153
844947 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry #44233
839289 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry #44050
844722 dotnet/sdk Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation #44218
844721 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry #44230
844726 dotnet/sdk Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation #44219
844665 dotnet/sdk Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation #44014
844642 dotnet/sdk Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation #44225
844639 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry #44224
844636 dotnet/sdk Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation #44011
844621 dotnet/sdk Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation #44223
844093 dotnet/sdk Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation #44181
844021 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry #43988
842283 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry #44084
842245 dotnet/sdk Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.CreateNewImage_RootlessBaseImage #43015
842238 dotnet/sdk Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation #44153
842026 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry #44148
842024 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry #44146
841145 dotnet/sdk Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation #44014
841139 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.EndToEndTests.ApiEndToEndWithLocalLoad #44139
841119 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry #44134
841113 dotnet/sdk Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation #44130
840845 dotnet/sdk Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation #44135
840849 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry #44136
840833 dotnet/sdk Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation #44133
840839 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry #44134
840827 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.EndToEndTests.EndToEnd_MultiProjectSolution #44132
840248 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.EndToEndTests.ApiEndToEndWithLocalLoad
840184 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.EndToEndTests.EndToEnd_MultiProjectSolution #43920
840094 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry #44069
839852 dotnet/sdk Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation #44104
839920 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry #44111
839882 dotnet/sdk Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation #44108
839834 dotnet/sdk Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation #44101
839870 dotnet/sdk Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation #44107
839819 dotnet/sdk Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation #44099
839862 dotnet/sdk Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation #44106
839847 dotnet/sdk Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation #44103
837986 dotnet/sdk Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation #44064
838031 dotnet/sdk Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation #44067
837988 dotnet/sdk Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation #44065
838020 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry #44059
837971 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry #44062
837948 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry #44061
837723 dotnet/sdk Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation #44057

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
0 0 65
@akoeplinger
Copy link
Member Author

@marcpopMSFT who owns Microsoft.NET.Build.Containers?

@baronfel
Copy link
Member

baronfel commented Jul 8, 2024

@dotnet/sdk-container-builds-maintainers

@baronfel
Copy link
Member

baronfel commented Jul 8, 2024

@akoeplinger the image in question is registry:2, which we need/use to validate our logic for communicating with container registries via the container registry APIs. Do you know if this image is mirrored anywhere that we can pull from? It's not currently on mcr.microsoft.com.

@nagilson
Copy link
Member

Bumping this thread since it's affecting more PRs

@nagilson
Copy link
Member

We don't have a path forward to fix this yet - Aspire moved to using their own ACR for storing the images they use, but we need an engineering-systems-managed solution for our entire stack. There's an issue tracking this request but I can't find it at the moment.

@baronfel
Copy link
Member

Could be fixed at a root cause level by dotnet/dnceng#3389

@MichaelSimons
Copy link
Member

@akoeplinger the image in question is registry:2, which we need/use to validate our logic for communicating with container registries via the container registry APIs. Do you know if this image is mirrored anywhere that we can pull from? It's not currently on mcr.microsoft.com.

@baronfel - It sounds like the test has this dependency and is what is pulling the image. One possible option to workaround scenarios where tests are pulling images directly from DockerHub would be to create a wrapper buildtools image. The Dockerfile would just be a FROM statement. The buildtools images are stored on MCR and therefore you would be able to get around this rate limiting issue.

@nagilson
Copy link
Member

@dotnet/sdk-container-builds-maintainers Do you agree with @MichaelSimons suggestion? This has a pretty large impact on PR failure rates. If this is still an issue going forward a month or two down the line, I will sadly suggest that we just disable these tests

@marcpopMSFT
Copy link
Member

@donJoseLuis @MichalPavlik I'm told by baronfel that there may be a solution to this of setting up our own ACR mirror of these so we don't get blocked by docker from pulling too often. He said to talk to @eerhardt about that. Is that something you can potentially drive getting set up?

As you can see above, this fails a couple of dozen PRs every week that have to be rerun. It's not a crazy blocker but that's high enough to be something we want eyes on to get fixed eventually.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

6 participants