Third Party S3 configuration results in runtime panic #752
Labels
lifecycle/rotten
Denotes an issue or PR that has aged beyond stale and will be auto-closed.
Comments
|
@nate-duke can you try |
|
Thanks for taking a look @dmage. Looks the same to me. Let me know if you need more info. The bucket does already exist, and the credentials configured in Come to think of it ... it's odd that the operator controller is trying to reach out to the bucket at all. It doesn't have the credentials available to it unless it's just reading the secret directly through the api. storage:
s3:
bucket: okd4-dev-registry
region: us-east-1
regionEndpoint: powerscale-system-hostname:9021E0204 12:23:11.412691 1 runtime.go:78] Observed a panic: runtime.boundsError{x:3, y:2, signed:true, code:0x3} (runtime error: slice bounds out of range [3:2])
goroutine 455 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x2af0d20, 0xc000f716f8)
/go/src/github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0x95
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
/go/src/github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x86
panic(0x2af0d20, 0xc000f716f8)
/usr/lib/golang/src/runtime/panic.go:965 +0x1b9
github.com/aws/aws-sdk-go/aws/signer/v4.getURIPath(0xc001159b00, 0x0, 0x2cb7c29)
/go/src/github.com/openshift/cluster-image-registry-operator/vendor/github.com/aws/aws-sdk-go/aws/signer/v4/uri_path.go:14 +0x15e
github.com/aws/aws-sdk-go/aws/signer/v4.(*signingCtx).buildCanonicalString(0xc0020b4ba8)
/go/src/github.com/openshift/cluster-image-registry-operator/vendor/github.com/aws/aws-sdk-go/aws/signer/v4/v4.go:656 +0xd0
github.com/aws/aws-sdk-go/aws/signer/v4.(*signingCtx).build(0xc0020b4ba8, 0x3203f00, 0xc000054098, 0xc0014c1e40)
/go/src/github.com/openshift/cluster-image-registry-operator/vendor/github.com/aws/aws-sdk-go/aws/signer/v4/v4.go:533 +0x30d
github.com/aws/aws-sdk-go/aws/signer/v4.Signer.signWithBody(0xc000d6be80, 0x0, 0x31a2c80, 0xc000e21b10, 0x10100, 0x2e2b398, 0x0, 0xc001123900, 0x31bbb80, 0xc000f19ee0, ...)
/go/src/github.com/openshift/cluster-image-registry-operator/vendor/github.com/aws/aws-sdk-go/aws/signer/v4/v4.go:350 +0x3b3
github.com/aws/aws-sdk-go/aws/signer/v4.SignSDKRequestWithCurrentTime(0xc0009ca000, 0x2e2b398, 0xc000e21b38, 0x1, 0x1)
/go/src/github.com/openshift/cluster-image-registry-operator/vendor/github.com/aws/aws-sdk-go/aws/signer/v4/v4.go:481 +0x2d5
github.com/aws/aws-sdk-go/aws/signer/v4.BuildNamedHandler.func1(0xc0009ca000)
/go/src/github.com/openshift/cluster-image-registry-operator/vendor/github.com/aws/aws-sdk-go/aws/signer/v4/v4.go:436 +0x52
github.com/aws/aws-sdk-go/aws/request.(*HandlerList).Run(0xc0009ca1e8, 0xc0009ca000)
/go/src/github.com/openshift/cluster-image-registry-operator/vendor/github.com/aws/aws-sdk-go/aws/request/handlers.go:267 +0x99
github.com/aws/aws-sdk-go/aws/request.(*Request).Sign(0xc0009ca000, 0x10cb297640, 0x45cb5c0)
/go/src/github.com/openshift/cluster-image-registry-operator/vendor/github.com/aws/aws-sdk-go/aws/request/request.go:429 +0xd6
github.com/aws/aws-sdk-go/aws/request.(*Request).Send(0xc0009ca000, 0x0, 0x0)
/go/src/github.com/openshift/cluster-image-registry-operator/vendor/github.com/aws/aws-sdk-go/aws/request/request.go:526 +0xe6
github.com/aws/aws-sdk-go/service/s3.(*S3).HeadBucketWithContext(0xc000e21b30, 0x3203f48, 0xc000054098, 0xc000dff9e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/openshift/cluster-image-registry-operator/vendor/github.com/aws/aws-sdk-go/service/s3/api.go:5355 +0xb3
github.com/openshift/cluster-image-registry-operator/pkg/storage/s3.(*driver).bucketExists(0xc001329140, 0xc0012dc270, 0x21, 0x0, 0x203000)
/go/src/github.com/openshift/cluster-image-registry-operator/pkg/storage/s3/s3.go:439 +0x106
github.com/openshift/cluster-image-registry-operator/pkg/storage/s3.(*driver).CreateStorage(0xc001329140, 0xc0020aaa80, 0xc000620900, 0xc0011260f0)
/go/src/github.com/openshift/cluster-image-registry-operator/pkg/storage/s3/s3.go:503 +0x2e30
github.com/openshift/cluster-image-registry-operator/pkg/resource.(*Generator).syncStorage(0xc000be69f0, 0xc0020aaa80, 0x0, 0x0)
/go/src/github.com/openshift/cluster-image-registry-operator/pkg/resource/generator.go:149 +0x15b
github.com/openshift/cluster-image-registry-operator/pkg/resource.(*Generator).Apply(0xc000be69f0, 0xc0020aaa80, 0x0, 0x378)
/go/src/github.com/openshift/cluster-image-registry-operator/pkg/resource/generator.go:219 +0x4d
github.com/openshift/cluster-image-registry-operator/pkg/operator.(*Controller).createOrUpdateResources(0xc000bedea0, 0xc0020aaa80, 0x7, 0xc0012b8e01)
/go/src/github.com/openshift/cluster-image-registry-operator/pkg/operator/controller.go:210 +0x18c
github.com/openshift/cluster-image-registry-operator/pkg/operator.(*Controller).sync(0xc000bedea0, 0x0, 0xc001295d90)
/go/src/github.com/openshift/cluster-image-registry-operator/pkg/operator/controller.go:264 +0x1328
github.com/openshift/cluster-image-registry-operator/pkg/operator.(*Controller).eventProcessor.func1(0xc000bedea0, 0x2791760, 0x3169160)
/go/src/github.com/openshift/cluster-image-registry-operator/pkg/operator/controller.go:367 +0xb2
github.com/openshift/cluster-image-registry-operator/pkg/operator.(*Controller).eventProcessor(0xc000bedea0)
/go/src/github.com/openshift/cluster-image-registry-operator/pkg/operator/controller.go:374 +0x56
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000dff720)
/go/src/github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000dff720, 0x31a5440, 0xc0015d7170, 0xc000b24f01, 0xc000f29c80)
/go/src/github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0x9b
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000dff720, 0x3b9aca00, 0x0, 0x1, 0xc000f29c80)
/go/src/github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(0xc000dff720, 0x3b9aca00, 0xc000f29c80)
/go/src/github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x4d
created by github.com/openshift/cluster-image-registry-operator/pkg/operator.(*Controller).Run
/go/src/github.com/openshift/cluster-image-registry-operator/pkg/operator/controller.go:445 +0x1a5
panic: runtime error: slice bounds out of range [3:2] [recovered]
panic: runtime error: slice bounds out of range [3:2]
goroutine 455 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
/go/src/github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0x109
panic(0x2af0d20, 0xc000f716f8)
/usr/lib/golang/src/runtime/panic.go:965 +0x1b9
github.com/aws/aws-sdk-go/aws/signer/v4.getURIPath(0xc001159b00, 0x0, 0x2cb7c29)
/go/src/github.com/openshift/cluster-image-registry-operator/vendor/github.com/aws/aws-sdk-go/aws/signer/v4/uri_path.go:14 +0x15e
github.com/aws/aws-sdk-go/aws/signer/v4.(*signingCtx).buildCanonicalString(0xc0020b4ba8)
/go/src/github.com/openshift/cluster-image-registry-operator/vendor/github.com/aws/aws-sdk-go/aws/signer/v4/v4.go:656 +0xd0
github.com/aws/aws-sdk-go/aws/signer/v4.(*signingCtx).build(0xc0020b4ba8, 0x3203f00, 0xc000054098, 0xc0014c1e40)
/go/src/github.com/openshift/cluster-image-registry-operator/vendor/github.com/aws/aws-sdk-go/aws/signer/v4/v4.go:533 +0x30d
github.com/aws/aws-sdk-go/aws/signer/v4.Signer.signWithBody(0xc000d6be80, 0x0, 0x31a2c80, 0xc000e21b10, 0x10100, 0x2e2b398, 0x0, 0xc001123900, 0x31bbb80, 0xc000f19ee0, ...)
/go/src/github.com/openshift/cluster-image-registry-operator/vendor/github.com/aws/aws-sdk-go/aws/signer/v4/v4.go:350 +0x3b3
github.com/aws/aws-sdk-go/aws/signer/v4.SignSDKRequestWithCurrentTime(0xc0009ca000, 0x2e2b398, 0xc000e21b38, 0x1, 0x1)
/go/src/github.com/openshift/cluster-image-registry-operator/vendor/github.com/aws/aws-sdk-go/aws/signer/v4/v4.go:481 +0x2d5
github.com/aws/aws-sdk-go/aws/signer/v4.BuildNamedHandler.func1(0xc0009ca000)
/go/src/github.com/openshift/cluster-image-registry-operator/vendor/github.com/aws/aws-sdk-go/aws/signer/v4/v4.go:436 +0x52
github.com/aws/aws-sdk-go/aws/request.(*HandlerList).Run(0xc0009ca1e8, 0xc0009ca000)
/go/src/github.com/openshift/cluster-image-registry-operator/vendor/github.com/aws/aws-sdk-go/aws/request/handlers.go:267 +0x99
github.com/aws/aws-sdk-go/aws/request.(*Request).Sign(0xc0009ca000, 0x10cb297640, 0x45cb5c0)
/go/src/github.com/openshift/cluster-image-registry-operator/vendor/github.com/aws/aws-sdk-go/aws/request/request.go:429 +0xd6
github.com/aws/aws-sdk-go/aws/request.(*Request).Send(0xc0009ca000, 0x0, 0x0)
/go/src/github.com/openshift/cluster-image-registry-operator/vendor/github.com/aws/aws-sdk-go/aws/request/request.go:526 +0xe6
github.com/aws/aws-sdk-go/service/s3.(*S3).HeadBucketWithContext(0xc000e21b30, 0x3203f48, 0xc000054098, 0xc000dff9e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/openshift/cluster-image-registry-operator/vendor/github.com/aws/aws-sdk-go/service/s3/api.go:5355 +0xb3
github.com/openshift/cluster-image-registry-operator/pkg/storage/s3.(*driver).bucketExists(0xc001329140, 0xc0012dc270, 0x21, 0x0, 0x203000)
/go/src/github.com/openshift/cluster-image-registry-operator/pkg/storage/s3/s3.go:439 +0x106
github.com/openshift/cluster-image-registry-operator/pkg/storage/s3.(*driver).CreateStorage(0xc001329140, 0xc0020aaa80, 0xc000620900, 0xc0011260f0)
/go/src/github.com/openshift/cluster-image-registry-operator/pkg/storage/s3/s3.go:503 +0x2e30
github.com/openshift/cluster-image-registry-operator/pkg/resource.(*Generator).syncStorage(0xc000be69f0, 0xc0020aaa80, 0x0, 0x0)
/go/src/github.com/openshift/cluster-image-registry-operator/pkg/resource/generator.go:149 +0x15b
github.com/openshift/cluster-image-registry-operator/pkg/resource.(*Generator).Apply(0xc000be69f0, 0xc0020aaa80, 0x0, 0x378)
/go/src/github.com/openshift/cluster-image-registry-operator/pkg/resource/generator.go:219 +0x4d
github.com/openshift/cluster-image-registry-operator/pkg/operator.(*Controller).createOrUpdateResources(0xc000bedea0, 0xc0020aaa80, 0x7, 0xc0012b8e01)
/go/src/github.com/openshift/cluster-image-registry-operator/pkg/operator/controller.go:210 +0x18c
github.com/openshift/cluster-image-registry-operator/pkg/operator.(*Controller).sync(0xc000bedea0, 0x0, 0xc001295d90)
/go/src/github.com/openshift/cluster-image-registry-operator/pkg/operator/controller.go:264 +0x1328
github.com/openshift/cluster-image-registry-operator/pkg/operator.(*Controller).eventProcessor.func1(0xc000bedea0, 0x2791760, 0x3169160)
/go/src/github.com/openshift/cluster-image-registry-operator/pkg/operator/controller.go:367 +0xb2
github.com/openshift/cluster-image-registry-operator/pkg/operator.(*Controller).eventProcessor(0xc000bedea0)
/go/src/github.com/openshift/cluster-image-registry-operator/pkg/operator/controller.go:374 +0x56
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000dff720)
/go/src/github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000dff720, 0x31a5440, 0xc0015d7170, 0xc000b24f01, 0xc000f29c80)
/go/src/github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0x9b
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000dff720, 0x3b9aca00, 0x0, 0x1, 0xc000f29c80)
/go/src/github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(0xc000dff720, 0x3b9aca00, 0xc000f29c80)
/go/src/github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x4d
created by github.com/openshift/cluster-image-registry-operator/pkg/operator.(*Controller).Run
/go/src/github.com/openshift/cluster-image-registry-operator/pkg/operator/controller.go:445 +0x1a5
|
|
Experiencing the same issue with Sorry for not providing any new info here (besides other storage endpoints are affected as well), but I can add some once we get a response. |
|
Please update your regionEndpoint so that it starts with http:// or https://. |
Thank you, that fixed the error for us. |
|
Issues go stale after 90d of inactivity. Mark the issue as fresh by commenting If this issue is safe to close now please do so with /lifecycle stale |
openshift-ci
bot
added
the
lifecycle/stale
|
Stale issues rot after 30d of inactivity. Mark the issue as fresh by commenting If this issue is safe to close now please do so with /lifecycle rotten |
openshift-ci
bot
added
lifecycle/rotten
|
Rotten issues close after 30d of inactivity. Reopen the issue by commenting /close |
|
@openshift-bot: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
in #485 it was mentioned that any compatible S3 storage back should more-or-less work. I'm trying to get the image-registry operator deployed and using an S3-compatible appliance (EMC PowerScale in my case) and am running into the following:
We're not in AWS if that helps. All IPI on vSphere.
Operator configured with:
which results in:
The source and documentation indicate that region is "optional" but I realize that there's a lot of 'convention' encoded into much of the tooling around s3 libraries so I humored it and fill in a region and am met with the following fun crash no matter what i put into the region attribute. S3 clients (tested with mc and s3cmd) work just fine against this thing no mater what region i set.
The text was updated successfully, but these errors were encountered: