Pushing with kaniko is slow

[suleimi]

Actual behavior
Pushing large images (e.g 5Gb) with kaniko to some https registries (e.g ecr, quay or gitlab contianer registry) has been identified to be quite slow when using kaniko . After some preliminary investigation (in https://gitlab.com/gitlab-org/gitlab/-/issues/241996 )it appears the slowness comes from the issue mentioned in golang/go#47840 and is related to the http/2 implementation of net/http

Because of the slowness caused by golang/go#47840 docker and buildx registry clients have opted to only push on http/1.1 connection. see: moby/buildkit#1420

A few issues have been raised on (go-containerregistry) which kaniko uses that alludes to a similar http2 bottleneck:

• Slow uploads to quay.io google/go-containerregistry#1193
• Push to registry.New() hangs with HTTP/2. google/go-containerregistry#1210

Expected behavior

Pushing a 5Gb layer using kaniko to gitlab contianer registry, ecr or quay should not take > 30mins on arguably fast internet connections

To Reproduce
Steps to reproduce the behavior:

1. Push a random 5Gb image to gitlab container registry
2. Push a random 5Gb image to gitlab container registry with http 2 disabled (by setting GODEBUG="http2client=0")
3. Observe it takes almost > 40mins with http 2 enabled

Additional Information

• Setup: https://gitlab.com/suleimiahmed/registry-speed-test/-/tree/test-ff?ref_type=heads
  • includes docker file
  • Any kaniko that defaults to http 2 has this issue

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

[slamer59]

From 40 minutes to 2 minutes with workaround... Thanks !

[stocktondr]

From 40 minutes to 2 minutes with workaround... Thanks !

We are having the same problem.
The workaround also worked for us.

[zijiwork]

How to solve the problem of slow push in tekton kaniko

[keepdying]

We also saw improvements on builds while pushing caches and image to gcr and gar. From 13 mins to 7 mins.
Thanks for workaround!

[psarka]

In my case it went from 38min to 2min. @zijiwork this is how you do it in gitlab:

build-ci-image:
    stage: build-ci-image
    variables:
         GODEBUG: "http2client=0"
    image:
        name: gcr.io/kaniko-project/executor:v1.9.0-debug
        entrypoint: [ "" ]
    script:
        - /kaniko/executor
            --context ...
            --dockerfile ...
            --destination ...