Unable to push images to artifactory

[Srujankumar99]

Hi,

When I'm trying to push the image to artifactory with tag name as 123, 1.0.19, 1.0.12345 or any numerals. I don't see any issues pushing the image to artifactory.
When I try to push the image with the tag as stable, 1.0-stable, 3.5.57-jdkjava. Then it always fails, I'm unable to push the images if it has alphabets in the tag

jfrog rt dp $BUILDNAME:$BUILDNUM gitlab-runner-docker --build-name=$BUILDNAME --build-number=$BUILDNUM
WARNING! Your password will be stored unencrypted in /home/gitlab-runner/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
The push refers to repository [gitlab-runner-docker.artifactory.srujankumar.com/srujankumar-build-utilities]
13956cd511a8: Preparing
d1d71233ce95: Preparing
c5b7b136938a: Preparing
b6101be29fba: Preparing
32f947af7493: Preparing
143ea9cb87f0: Preparing
eeb4ac0cd3dc: Preparing
cc1b4cbc7710: Preparing
f972d139738d: Preparing
143ea9cb87f0: Waiting
eeb4ac0cd3dc: Waiting
cc1b4cbc7710: Waiting
f972d139738d: Waiting
c5b7b136938a: Layer already exists
32f947af7493: Layer already exists
d1d71233ce95: Layer already exists
13956cd511a8: Layer already exists
b6101be29fba: Layer already exists
f972d139738d: Layer already exists
143ea9cb87f0: Layer already exists
eeb4ac0cd3dc: Layer already exists
cc1b4cbc7710: Layer already exists
1.0.116197: digest: sha256:e1ae51dd971b3dea69ea74cae3af03cc1b813c65d1f077f50442028cf39e2904 size: 2205
[Info] Setting properties...
[Info] [Thread 2] Setting properties on: gitlab-runner-docker-dev-local/srujankumar-build-utilities/1.0.116197/manifest.json
[Info] [Thread 0] Setting properties on: gitlab-runner-docker-dev-local/srujankumar-build-utilities/1.0.116197/sha256__7c44e63a1fcb41ace2e8aecc05f50344391b4a4f092125b86ea29020d625a1bf
[Info] [Thread 1] Setting properties on: gitlab-runner-docker-dev-local/srujankumar-build-utilities/1.0.116197/sha256__aeb7866da422acc7e93dcf7323f38d7646f6269af33bcdb6647f2094fc4b3bf7
[Info] [Thread 0] Setting properties on: gitlab-runner-docker-dev-local/srujankumar-build-utilities/1.0.116197/sha256__78ac0a1119f1388ecf33e4b5b40492c0be0b2eea413adb12819318feaeb970b9
[Info] [Thread 1] Setting properties on: gitlab-runner-docker-dev-local/srujankumar-build-utilities/1.0.116197/sha256__1b5bfdcacbad5f3b3a86b757a9f8f961b54ef76f0a98a84790c2460f27d7d9d6
[Info] [Thread 2] Setting properties on: gitlab-runner-docker-dev-local/srujankumar-build-utilities/1.0.116197/sha256__be01a918d00625ecef01a5dc8781dc6626ba546602bcc7e7ee5d8e4cc6e7a57a
[Info] [Thread 0] Setting properties on: gitlab-runner-docker-dev-local/srujankumar-build-utilities/1.0.116197/sha256__ed24bca6e3c75e86a0050477e184fd99bc18c71b2d9bfdcbb7614883a516d8ea
[Info] [Thread 1] Setting properties on: gitlab-runner-docker-dev-local/srujankumar-build-utilities/1.0.116197/sha256__01fe65cf4ba069fd35198337685680cd0a53068de7a11798d64e53a913eca6d7
[Info] [Thread 2] Setting properties on: gitlab-runner-docker-dev-local/srujankumar-build-utilities/1.0.116197/sha256__5dfc1afd9de4058a087de14b27f124cc017e490aed61801f0e81297e54f8148e
[Info] [Thread 1] Setting properties on: gitlab-runner-docker-dev-local/srujankumar-build-utilities/1.0.116197/sha256__f16b2dd69036631c0a9f59652689ad66c2e807986e43773a79863bf6be9786db
[Info] [Thread 2] Setting properties on: gitlab-runner-docker-dev-local/srujankumar-build-utilities/1.0.116197/sha256__f40e1f68eda7c46eb1d2416e8ba8519d6180e154a603520e9bbdd2561b3731aa
[Info] Done setting properties.
$ jfrog rt bp $BUILDNAME $BUILDNUM
[Info] Deploying build info...
[Info] Build info successfully deployed. Browse it in Artifactory under https://artifactory.artifactory.srujankumar.com/artifactory/webapp/builds/gitlab-runner-docker.artifactory.srujankumar.com/srujankumar-build-utilities/1.0.116197

====================================================

$ if [[ $CI_COMMIT_REF_NAME == master ]] # collapsed multi-line command
WARNING! Your password will be stored unencrypted in /home/gitlab-runner/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
The push refers to repository [gitlab-runner-docker.artifactory.srujankumar.com/srujankumar-build-utilities]
13956cd511a8: Preparing
d1d71233ce95: Preparing
c5b7b136938a: Preparing
b6101be29fba: Preparing
32f947af7493: Preparing
143ea9cb87f0: Preparing
eeb4ac0cd3dc: Preparing
cc1b4cbc7710: Preparing
f972d139738d: Preparing
eeb4ac0cd3dc: Waiting
f972d139738d: Waiting
cc1b4cbc7710: Waiting
143ea9cb87f0: Waiting
13956cd511a8: Layer already exists
d1d71233ce95: Layer already exists
b6101be29fba: Layer already exists
32f947af7493: Layer already exists
c5b7b136938a: Layer already exists
eeb4ac0cd3dc: Layer already exists
f972d139738d: Layer already exists
cc1b4cbc7710: Layer already exists
143ea9cb87f0: Layer already exists
1.0-stable: digest: sha256:e1ae51dd971b3dea69ea74cae3af03cc1b813c65d1f077f50442028cf39e2904 size: 2205
panic: Found incorrect manifest.json file, expecting image ID: sha256:7c44e63a1fcb41ace2e8aecc05f50344391b4a4f092125b86ea29020d625a1bf

goroutine 1 [running]:
panic(0xabd680, 0xc4203cf8a0)
/var/lib/jenkins/tools/org.jenkinsci.plugins.golang.GolangInstallation/go-1.7.1/src/runtime/panic.go:500 +0x1a1
github.com/jfrog/jfrog-cli-go/jfrog-cli/utils/cliutils.PanicOnError(0xf295a0, 0xc4203cf8a0, 0x38, 0xc4203f05a0)
/var/lib/jenkins/workspace/EcoSystem/jfrog-cli-release/temp/src/github.com/jfrog/jfrog-cli-go/jfrog-cli/utils/cliutils/utils.go:36 +0x6b
github.com/jfrog/jfrog-cli-go/jfrog-cli/artifactory/utils/docker.getManifest(0xc4203f05a0, 0x47, 0xc4203993b0, 0xc4201bce80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/var/lib/jenkins/workspace/EcoSystem/jfrog-cli-release/temp/src/github.com/jfrog/jfrog-cli-go/jfrog-cli/artifactory/utils/docker/buildinfo.go:187 +0x3ee
github.com/jfrog/jfrog-cli-go/jfrog-cli/artifactory/utils/docker.(*buildInfoBuilder).updateArtifactsAndDependencies(0xc4203f2000, 0xc4203f05a0, 0x47)
/var/lib/jenkins/workspace/EcoSystem/jfrog-cli-release/temp/src/github.com/jfrog/jfrog-cli-go/jfrog-cli/artifactory/utils/docker/buildinfo.go:103 +0xad
github.com/jfrog/jfrog-cli-go/jfrog-cli/artifactory/utils/docker.(*buildInfoBuilder).Build(0xc4203f2000, 0xc4203f2000, 0x0, 0x0)
/var/lib/jenkins/workspace/EcoSystem/jfrog-cli-release/temp/src/github.com/jfrog/jfrog-cli-go/jfrog-cli/artifactory/utils/docker/buildinfo.go:55 +0x86
github.com/jfrog/jfrog-cli-go/jfrog-cli/artifactory/commands/docker.PushDockerImage(0x7fff068aa965, 0x41, 0x7fff068aa9a7, 0x14, 0xc4201b2140, 0x36, 0xc420159ab0, 0xa, 0xc420208c48, 0xc4200a7e00, ...)
/var/lib/jenkins/workspace/EcoSystem/jfrog-cli-release/temp/src/github.com/jfrog/jfrog-cli-go/jfrog-cli/artifactory/commands/docker/push.go:58 +0x2cc
github.com/jfrog/jfrog-cli-go/jfrog-cli/artifactory.dockerPushCmd(0xc4200a7e00)
/var/lib/jenkins/workspace/EcoSystem/jfrog-cli-release/temp/src/github.com/jfrog/jfrog-cli-go/jfrog-cli/artifactory/cli.go:1231 +0x1f2
github.com/jfrog/jfrog-cli-go/jfrog-cli/artifactory.GetCommands.func25(0xc4200a7e00)
/var/lib/jenkins/workspace/EcoSystem/jfrog-cli-release/temp/src/github.com/jfrog/jfrog-cli-go/jfrog-cli/artifactory/cli.go:358 +0x2b
github.com/jfrog/jfrog-cli-go/vendor/github.com/codegangsta/cli.Command.Run(0xb9055b, 0xb, 0x0, 0x0, 0xc420159030, 0x1, 0x1, 0xb90cb2, 0xc, 0xbb2185, ...)
/var/lib/jenkins/workspace/EcoSystem/jfrog-cli-release/temp/src/github.com/jfrog/jfrog-cli-go/vendor/github.com/codegangsta/cli/command.go:174 +0xbbe
github.com/jfrog/jfrog-cli-go/vendor/github.com/codegangsta/cli.(*App).RunAsSubcommand(0xc4200a7b80, 0xc4200a7a40, 0x0, 0x0)
/var/lib/jenkins/workspace/EcoSystem/jfrog-cli-release/temp/src/github.com/jfrog/jfrog-cli-go/vendor/github.com/codegangsta/cli/app.go:298 +0x9f8
github.com/jfrog/jfrog-cli-go/vendor/github.com/codegangsta/cli.Command.startApp(0xb88ece, 0x2, 0x0, 0x0, 0x0, 0x0, 0x0, 0xb96c1d, 0x14, 0x0, ...)
/var/lib/jenkins/workspace/EcoSystem/jfrog-cli-release/temp/src/github.com/jfrog/jfrog-cli-go/vendor/github.com/codegangsta/cli/command.go:249 +0x564
github.com/jfrog/jfrog-cli-go/vendor/github.com/codegangsta/cli.Command.Run(0xb88ece, 0x2, 0x0, 0x0, 0x0, 0x0, 0x0, 0xb96c1d, 0x14, 0x0, ...)
/var/lib/jenkins/workspace/EcoSystem/jfrog-cli-release/temp/src/github.com/jfrog/jfrog-cli-go/vendor/github.com/codegangsta/cli/command.go:65 +0x16a3
github.com/jfrog/jfrog-cli-go/vendor/github.com/codegangsta/cli.(*App).Run(0xc4200a7540, 0xc42000c150, 0x7, 0x7, 0x0, 0x0)
/var/lib/jenkins/workspace/EcoSystem/jfrog-cli-release/temp/src/github.com/jfrog/jfrog-cli-go/vendor/github.com/codegangsta/cli/app.go:187 +0xa0c
main.main()
/var/lib/jenkins/workspace/EcoSystem/jfrog-cli-release/temp/src/github.com/jfrog/jfrog-cli-go/jfrog-cli/jfrog/main.go:81 +0x27c

[eyalbe4]

@Srujankumar99,
It looks like JFrog CLI fails when it tries to create build-info for the image it pushes to Artifactory.

Here's what JFrog CLI does, to create build-info for a docker image:

1. Push the image to Artofactory.
2. Read the manifet.json of the pushed image (by downloading it from Artifactory).
3. Set properties on the pushed layers in Artifactory.
4. Create the build-info and push it to Artifactory.

It appears that the build fails on step #2 with the following error message (from the logs you shared):
1.0-stable: digest: sha256:e1ae51dd971b3dea69ea74cae3af03cc1b813c65d1f077f50442028cf39e2904 size: 2205
panic: Found incorrect manifest.json file, expecting image ID: sha256:7c44e63a1fcb41ace2e8aecc05f50344391b4a4f092125b86ea29020d625a1bf

The above failure happens, right after JFrog CLI reads the manifest.json. It compares to the manifet.json it expects to find and fails, because it got a different manifet.

Why can this happen?
In order to answer the above question, we first need to understand how JFrog CLI fetches the manifet.
Well, the second argument of the jfrog rt dp command is the name of the docker repository in Artifactory to which the image is pushed. JFrog CLI uses this argument for steps #2 and #3 above. It does not use it for pushing the image.
That can explain why only step #1 succeeds or the rest fail.

So what can be wrong with the repository argument?
If the repository sent is different than the one to which the image was pushed, the above validation can fail.

And what if this is the correct repository?
If two processes push the same image to this repository at the exact same time, then one of the processes can fail with the above error, when it tries validate it got the right manifest.

Pleas let us know of the above helps you figure out the cause of failure.

[Srujankumar99]

Thanks for your reply @eyalbe4

Here is what I'm doing. Please let me know if I'm doing anything wrong.

jfrog rt config artifactory --url=$URL--apikey=$ARTIFACTORY_API_KEY
echo -e " Current working branch == $CI_COMMIT_REF_NAME"
jfrog rt dp gitlab-runner-docker.srujankumar.com/srujankumar-build-utilities:1.0.12345 gitlab-runner-docker --build-name=gitlab-runner-docker.srujankumar.com/srujankumar-build-utilities --build-number=1.0.12345
jfrog rt bp gitlab-runner-docker 1.0.12345
####### This Step is success, And it never fails ##########

    - |
        if [[ $CI_COMMIT_REF_NAME ==  master ]]
        then
##### This keeps failing ######
            jfrog rt dp gitlab-runner-docker.srujankumar.com/srujankumar-build-utilities:1.0-stable gitlab-runner-docker --build-name=gitlab-runner-docker.srujankumar.com/srujankumar-build-utilities --build-number=1.0-stable
            jfrog rt bp gitlab-runner-docker 1.0-stable
        fi

Also, we are not at all pushing the same image at the exact time. This was working fine in the past and keeps failing now, no change in the jfrog version, no changes in the code. After seeing the build failures I changed the tag to 1.0 instead of 1.0-stable and I didn't see any failures again

[alexmnyc]

this happens with DOCKER_BUILDKIT=1 flag set for me

[barbelity]

@Srujankumar99 @alexmnyc I couldn't reproduce this issue, also with the 'DOCKER_BUILDKIT' variable enabled.
I am using JFrog CLI version 1.39.4, docker version 19.03.8 on Mac OS.

My steps were:

1. Enable DOCKER_BUILDKIT.
2. Run 'docker build' on the jfrog-cli project Dockerfile.
3. Tag the image twice -> '1.0' and '1.0-stable'.
4. Push the images to Artifactory using JFrog CLI, collecting and publishing build-info for both:
   jfrog rt dp <image-name>:<tag> --build-name=<build> --build-number=<number>
jfrog rt bp <build> <number>

As it worked for me, I suggest the following steps:

1. Upgrade your JFrog CLI version.
2. Try running the steps I did -> or alternatively, suggest an example project of your own.
3. If the issue persists, share with us more details about your environment and logs, and your example project.

[shuvadipc]

No response and outdated.