Earthly image --push to Artifactory fails with 401 #1017
Assignees
Labels
type:bug
Something isn't working
Comments
|
Interesting. Logs from the repository end could be telling, for sure. Do you have any other "stuff" in your Another thing to try: Build locally with Earthly, and then push with your logged-in Docker. Does that work, or have the same issue? |
|
Hello, building locally and then pushing with Docker works perfectly and is our current workaround so this isn't a massive issue, but would be nice to fix as it would simplify our CI a bit. When the build starts, the only additional stuff in our The auth token is added to The username:password does contain some non word characters (e.g. }, = and @), in case that could be the cause of the issue (I have tried configuring JFrog container registry with the same password and it is fine though)? Not sure if this is relevant but we're running I'm still trying to get the Artifatory logs. |
|
In order to connect to the Artifactory, is the proxy required? It could be the fact that we don't yet support proxy for the interaction with the registry. |
|
Artifactory has a free trial; I can grab their self-hosted one and try to reproduce this. |
|
I may have just got this to work in one environment but I am not entirely sure how or why, so give me a bit more time to investigate before you spend time on it. |
|
I have not made much progress on this but I have seen some really weird behaviour which it would be great to get some thoughts on... When I run Now, on my local machine I used Earthly to build them image and then So I'm not sure if this is possible but it seems that there is potentially some caching at play? Could Earthly/Artifactory be seeing that the image is the same and not pushing it (which would explain the successful pushes after the This the buildkitd log from the successful pushAnd the failed pushMany thanks! |
I believe that the buildkit daemon caches the authentication in-memory for a while (15m or so?).. or the authenticated connection? I forgot what exactly. But I think that's what may have happened there if the daemon was reused. I'll let @dchw investigate further. But in the meantime, do you know if you have a credential helper in your docker config by any chance? It should be in your |
|
There is no credential helper so the auth token is just stored encoded in |
|
Hi, I have encounter a very similar issue, although in my case the error is a 404:
We are trying to push the new image to an internal docker registry, from the host we can do a docker push without any issues. Can we pass to Earthly the DOCKER_AUTH_CONFIG env variable from the Host or the config.json? |
|
@jrodrigv I think I need more information before I can properly poke around. Do you have any additional logs? If you could also obtain the buildkit logs, that would be great too. Also, what registry are you pushing to? @jamesalucas, I am digging around in your logs. Sorry for the delay. A couple interesting things I noticed from your description and your logs is that the successful run doesn't appear to have pushed anything. At least, all failures with suggested scopes only suggest needing |
|
@dchw thanks for taking a look.
Yes, I came to the same conclusion - the only time Earthly finished the push with success was when the same image had been previously built (with earthly) and pushed with If I remove the tag from artifactory and re-run the earthly build (with push), the push then fails. Is that what you mean? Thank you! |
Almost. I'm trying to eliminate the successful(?) run by your admin from when you said;
This would help me know if I should start searching permissions, or if the bug is in a different place. Currently, I do believe it is a permissions thing? What might help me make sure I can reproduce and debug if needed is more knowledge of the permissions your user has against that repository (and what is "public" on it too). I am on Earthly's slack channel if you would like to pass those details in more privacy. |
|
@dchw I think I have worked this out, so apologies if I have wasted your time. We are currently porting from Nexus to Artifactory and our Earthfiles look a bit like this: We can then produce images for either registry by passing in build-args when we invoke earthly, e.g.
Our Nexus registry is not exposed via https, hence the So we can't switch using build-args but that's no big deal. Thanks for the help. |
Learning about how people are using Earthly is never a waste of time. :D I do wonder if theres more we can do to help? How do you decide which image goes to which registry when? |
|
We have a medium sized monorepo that is built via Jenkins jobs with Docker images being deployed to Nexus. We're in the process of porting all of this over to hosted Gitlab infrastructure that pushes images to Artifactory. The Jenkins builds have some Groovy scripting and Git calls to work out what has changed so that only updated code is built but as part of the move to Gitlab we're throwing all that away and using Earthly to build everything and getting performance from caching rather than from our Groovy Git glue. |
|
You could work around it with our You would just need to swap out the "insecure" for your registry address (assuming the registry is part of the tag). Otherwise, sounds like you have it figured out. Let us know if you need anything else! |
|
Fantastic, thanks very much! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I'm using Earthly with the
--pushflag to push images to Artifactory and upon doing so Earthly returns the error:and the earthly-buildkitd logs show:
I issue a
docker loginbefore attempting the Earthly build.Using docker to push the image works as does
docker buildx build --pushso Artifactory is accepting image uploads from other sources, I just can't get it to work directly from Earthly.I have tried this with some other registries and they all succeed:
Would you have any suggestions for what could be wrong?
This is a corporate Artifactory server so I don't currently have logs from it but I am trying to get them to see if they reveal anything.
Thanks!
The text was updated successfully, but these errors were encountered: