KO not authenticating against the target docker registry/repo

[pksurferdad]

After successfully using KO to build the NATS channel controller and dispatcher, KO failed to deploy the image to the target registry, in this case, docker.io

below are the execution steps to replicate this issue

1. set KO_DOCKER_REPO = pkaisharis [account name]
2. run ko apply -f natss/config
3. execution results

2019/09/24 15:00:54 Publishing index.docker.io/pkaisharis/channel_controller-c1ecaddae73c4ef9fc778b143614b0c2:latest
2019/09/24 15:00:54 Publishing index.docker.io/pkaisharis/channel_dispatcher-c87d09956961b64d037b8b7969b36e36:latest
2019/09/24 15:00:56 error processing import paths in "natss/config/500-dispatcher.yaml": unsupported status code 401

Below is the docker.io auth config from .docker/config.json

{
	"auths": {
		"https://index.docker.io/v1/": {}
	},
	"HttpHeaders": {
		"User-Agent": "Docker-Client/19.03.2 (linux)"
	},
	"credsStore": "secretservice"
}

[jonjohnsonjr]

And you're able to "docker push" just fine with this config?

[pksurferdad]

yes, but i specify a specific repo when i push, like pkaisharis/nats-controller

[jonjohnsonjr]

Weird!

Don't paste the output without censoring it (contains credentials), but if you run this:

echo 'docker.io' | docker-credential-secretservice get

Does it succeed? Is the Username <token> or your own?

[pksurferdad]

no, it does not succeed...

credentials not found in native keychain

[jonjohnsonjr]

Hmm... what about:

echo "https://index.docker.io/v1/" | docker-credential-secretservice get

Possibly hitting google/go-containerregistry#456 (comment)

[pksurferdad]

yeah, that does

{"ServerURL":"https://index.docker.io/v1/","Username":"pkaisharis","Secret":"<removed>"}

[jonjohnsonjr]

Awesome, thanks for helping me debug this! I'm going to try to land a proper fix for this upstream soon but if that takes too long I'll add a simple workaround for dockerhub.

[jonjohnsonjr]

Didn't realize I could close issues cross-repo!

[pksurferdad]

do i get latest just be running GO111MODULE=on go get github.com/google/ko/cmd/ko

[jonjohnsonjr]

I'm just going to merge it and we'll roll back if I break the world, then it will be a lot easier to try.

[pksurferdad]

sorry, but i'm still seeing the 401 response, but i did see any error (below in steps) after running GO111MODULE=on go get github.com/google/ko/cmd/ko. I'm probably missing some go module?

steps

• ran GO111MODULE=on go get github.com/google/ko/cmd/ko
  go: finding github.com/knative/eventing v0.7.1
  go: finding github.com/google/uuid v1.1.1
  go: finding github.com/onsi/gomega v1.5.0
  go: finding github.com/knative/pkg v0.0.0-20190624141606-d82505e6c5b4
  go: finding github.com/hashicorp/go-hclog v0.9.1
  go: github.com/google/go-github@v0.0.0-20180926004559-f55b50f38167: go.mod has post-v0 module path "github.com/google/go-github/v18" at revision f55b50f38167
  go: finding github.com/sirupsen/logrus v1.4.2
  go: finding github.com/operator-framework/operator-sdk v0.9.0
  go: finding go.opencensus.io v0.21.0
  go: finding k8s.io/klog v0.4.0
  go: finding go.uber.org/multierr v0.0.0-20180122172545-ddea229ff1df
  go: finding github.com/go-openapi/spec v0.19.2
  go: finding k8s.io/utils v0.0.0-20190801114015-581e00157fb1
  go: finding github.com/pborman/uuid v1.2.0
  go: finding github.com/Azure/go-ansiterm v0.0.0-20170929234023-d6e3b3328b78
  go: finding github.com/pascaldekloe/goe v0.1.0
  go: finding sigs.k8s.io/controller-runtime v0.1.10
  go: finding github.com/knative/serving v0.7.1
  go: finding github.com/fatih/structs v1.1.0
  go: finding github.com/xiang90/probing v0.0.0-20160813154853-07dd2e8dfe18
  go: finding github.com/tmc/grpc-websocket-proxy v0.0.0-20170815181823-89b8d40f7ca8
  go: finding github.com/DataDog/datadog-go v2.2.0+incompatible
  go: finding github.com/circonus-labs/circonusllhist v0.1.3
  go: finding k8s.io/apimachinery v0.0.0-20190612125636-6a5db36e93ad
  go: finding golang.org/x/sys v0.0.0-20190712062909-fae7ac547cb7
  go: finding github.com/radovskyb/watcher v1.0.6
  go: finding k8s.io/api v0.0.0-20190612125737-db0771252981
  go: finding github.com/gogo/protobuf v1.2.2-0.20190723190241-65acae22fc9d
  go: finding github.com/scylladb/go-set v1.0.2
  go: finding github.com/onsi/ginkgo v1.8.0
  go: finding golang.org/x/crypto v0.0.0-20190701094942-4def268fd1a4
  go: finding github.com/hashicorp/raft-boltdb v0.0.0-20171010151810-6e5ba93211ea
  go: finding github.com/stoewer/go-strcase v1.0.2
  go: finding github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0
  go: finding gotest.tools v2.2.0+incompatible
  go: finding k8s.io/client-go v11.0.0+incompatible
  go: finding k8s.io/kube-openapi v0.0.0-20190510232812-a01b7d5d6c22
  go: finding github.com/coreos/prometheus-operator v0.29.0
  go: finding github.com/coreos/etcd v3.3.15+incompatible
  go: finding github.com/pkg/profile v1.2.1
  go: finding github.com/stretchr/testify v1.3.0
  go: finding k8s.io/kube-openapi v0.0.0-20190816220812-743ec37842bf
  go: finding github.com/grpc-ecosystem/go-grpc-middleware v0.0.0-20190222133341-cfaf5686ec79
  go: finding sigs.k8s.io/structured-merge-diff v0.0.0-20190817042607-6149e4549fca
  go: finding github.com/alecthomas/jsonschema v0.0.0-20190122210438-a6952de1bbe6
  go: finding github.com/grpc-ecosystem/grpc-gateway v1.3.0
  go: finding github.com/hashicorp/go-retryablehttp v0.5.3
  go: finding github.com/jonboulle/clockwork v0.1.0
  go: finding github.com/google/go-containerregistry v0.0.0-20190206233756-dbc4da98389f
  go: finding gonum.org/v1/netlib v0.0.0-20181029234149-ec6d1f5cefe6
  go: finding github.com/coreos/pkg v0.0.0-20180108230652-97fdf19511ea
  go: finding gopkg.in/square/go-jose.v2 v2.2.2
  go: finding gonum.org/v1/gonum v0.0.0-20181121035319-3f7ecaa7e8ca
  go: finding github.com/coreos/bbolt v1.3.1-coreos.6
  go: finding github.com/hashicorp/golang-lru v0.5.1
  go: finding github.com/docker/docker v0.7.3-0.20190327010347-be7ac8be2ae0
  go: finding golang.org/x/crypto v0.0.0-20190611184440-5c40567a22f8
  go: finding golang.org/x/net v0.0.0-20190812203447-cdfb69ac37fc
  go: finding github.com/soheilhy/cmux v0.1.3
  go: finding github.com/coreos/go-oidc v2.1.0+incompatible
  go: finding go.uber.org/atomic v0.0.0-20181018215023-8dc6146f7569
  go: finding cloud.google.com/go v0.43.0
  go: finding github.com/pquerna/cachecontrol v0.0.0-20171018203845-0dec1b30a021
  go: finding github.com/munnerz/goautoneg v0.0.0-20120707110453-a547fc61f48d
  go: finding github.com/jpillora/backoff v0.0.0-20170918002102-8eab2debe79d
  go: finding google.golang.org/grpc v1.23.0
  go: finding github.com/boltdb/bolt v1.3.1
  go: finding golang.org/x/exp v0.0.0-20180321215751-8460e604b9de
  go: finding github.com/coreos/go-semver v0.3.0
  go: finding github.com/google/gofuzz v1.0.0
  go: finding github.com/emicklei/go-restful v2.9.5+incompatible
  go: finding gopkg.in/natefinch/lumberjack.v2 v2.0.0
  go: finding github.com/openshift/api v3.9.0+incompatible
  go: finding github.com/tv42/httpunix v0.0.0-20150427012821-b75d8614f926
  go: finding github.com/sirupsen/logrus v1.4.1
  go: finding k8s.io/apimachinery v0.0.0-20190913075813-344bcc0201c9
  go: finding github.com/Masterminds/semver v1.4.2
  go: finding golang.org/x/net v0.0.0-20190628185345-da137c7871d7
  go: finding k8s.io/api v0.0.0-20190913080256-21721929cffa
  go: finding github.com/coreos/go-systemd v0.0.0-20180511133405-39ca1b05acc7
  go: finding k8s.io/client-go v0.0.0-20190913080825-6f3bc4ba9215
  go: finding golang.org/x/sys v0.0.0-20190523142557-0e01d883c5c5
  go: finding google.golang.org/genproto v0.0.0-20190502173448-54afdca5d873
  go: finding github.com/hashicorp/go-immutable-radix v1.0.0
  go: finding github.com/circonus-labs/circonus-gometrics v2.3.1+incompatible
  go: finding k8s.io/component-base v0.0.0-20190913081755-3f8a0e53a4bc
  go: finding github.com/rs/xid v1.2.1
  go: finding github.com/openzipkin/zipkin-go v0.1.6
  go: error loading module requirements

follow steps executed from ~/go/src/knative.dev/eventing-contrib

• ran docker login and authenticated successfully
• set KO_DOCKER_REPO=pkaisharis [my docker.io account]
• ran ko publish ./natss/cmd/channel_controller

2019/09/26 15:04:16 Using base gcr.io/distroless/static:latest for knative.dev/eventing-contrib/natss/cmd/channel_controller
2019/09/26 15:04:17 Building knative.dev/eventing-contrib/natss/cmd/channel_controller
2019/09/26 15:04:20 Publishing index.docker.io/ko-local/channel_controller-c1ecaddae73c4ef9fc778b143614b0c2:latest
2019/09/26 15:04:21 failed to publish images: error publishing knative.dev/eventing-contrib/natss/cmd/channel_controller: unsupported status code 401

[jonjohnsonjr]

It seems like installing it didn't actually work?

go: error loading module requirements

ls -l $(which ko) should tell you when it was built, at least.

[pksurferdad]

looks like it was built successfully -rwxr-xr-x 1 paulkaisharis paulkaisharis 38009184 Sep 26 14:40 /home/paulkaisharis/go/bin/ko

[jonjohnsonjr]

Can you try cloning repo and running go install ./cmd/ko? I want to make sure that go get error isn't causing problems.

[pksurferdad]

sorry, 401 still. before i cloned the repo, i removed ko locally and cloned to ~/go/src/github.com/google and it looks like it got built -rwxr-xr-x 1 paulkaisharis paulkaisharis 40122438 Sep 26 16:40. i also tried setting KO_DOCKER_REPO to pkaisharis/channel_controller which is a repo i created on docker.io, but still got the 401

[cezkuj]

@jonjohnsonjr I believe it's not fully working yet.
KO_DOCKER_REPO=index.docker.io/myuser seems ok.
KO_DOCKER_REPO=myuser seems to have trouble finding credentials as it is treated like "plain registry". https://github.com/google/ko/pull/94/files - NewRegistry doesn't return error and NewRepository isn't called anymore. If NewRepository call is forced (or credentials are renamed to myuser) then everything looks fine.

[jonjohnsonjr]

@cezkuj ah that's definitely annoying. I'm going to reopen #93 to track that -- I think this issue was a different root cause.