Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

processHTTPRetryResponse Panic when deploying ILB with missing permissions #287

Closed
julienstroheker opened this issue Jan 28, 2020 · 2 comments
Closed

processHTTPRetryResponse Panic when deploying ILB with missing permissions #287

julienstroheker opened this issue Jan 28, 2020 · 2 comments

Comments

@julienstroheker
Copy link

What happened:

When deploying ILB with wrong cloud provider permissions looks like cloud provider is panicing :

I0128 16:22:26.295175       1 azure_vmss.go:620] EnsureHostInPool begins to update vmssVM(infra-1580226559-000001) with new backendPoolID /subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/0041ce54-3b66-4982-9b15-40470720cec9/providers/Microsoft.Network/loadBalancers/kubernetes-internal/backendAddressPools/kubernetes
I0128 16:22:26.296235       1 azure_vmss.go:620] EnsureHostInPool begins to update vmssVM(infra-1580226559-000000) with new backendPoolID /subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/0041ce54-3b66-4982-9b15-40470720cec9/providers/Microsoft.Network/loadBalancers/kubernetes-internal/backendAddressPools/kubernetes
I0128 16:22:26.295175       1 azure_vmss.go:620] EnsureHostInPool begins to update vmssVM(infra-1580226559-000002) with new backendPoolID /subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/0041ce54-3b66-4982-9b15-40470720cec9/providers/Microsoft.Network/loadBalancers/kubernetes-internal/backendAddressPools/kubernetes
I0128 16:22:26.303357       1 azure_vmss.go:620] EnsureHostInPool begins to update vmssVM(compute-1580226559-000000) with new backendPoolID /subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/0041ce54-3b66-4982-9b15-40470720cec9/providers/Microsoft.Network/loadBalancers/kubernetes-internal/backendAddressPools/kubernetes
I0128 16:22:26.355941       1 azure_vmss.go:623] EnsureHostInPool update backing off vmssVM(infra-1580226559-000000) with new backendPoolID /subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/0041ce54-3b66-4982-9b15-40470720cec9/providers/Microsoft.Network/loadBalancers/kubernetes-internal/backendAddressPools/kubernetes, err: compute.VirtualMachineScaleSetVMsClient#Update: Failure sending request: StatusCode=403 -- Original Error: Code="LinkedAuthorizationFailed" Message="The client 'efe8dd0c-0f98-4cda-9800-5a9a316f2909' with object id 'efe8dd0c-0f98-4cda-9800-5a9a316f2909' has permission to perform action 'Microsoft.Compute/virtualMachineScaleSets/virtualmachines/write' on scope '/subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/0041ce54-3b66-4982-9b15-40470720cec9/providers/Microsoft.Compute/virtualMachineScaleSets/ss-infra-1580226559/virtualmachines/0'; however, it does not have permission to perform action 'Microsoft.Network/networkSecurityGroups/join/action' on the linked scope(s) '/subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/0041ce54-3b66-4982-9b15-40470720cec9/providers/Microsoft.Network/networkSecurityGroups/nsg-worker' or the linked scope(s) are invalid."
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x210d229]
goroutine 5668 [running]:
github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/cloudprovider/providers/azure.processHTTPRetryResponse(0x0, 0x77f8300, 0xc4282d8680, 0xc4243718e0, 0x2, 0x2)
	/builddir/build/BUILD/atomic-openshift-git-0.dfe38da/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/cloudprovider/providers/azure/azure_backoff.go:364 +0x69
github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/cloudprovider/providers/azure.(*Cloud).UpdateVmssVMWithRetry.func1(0xc424371940, 0x413c28, 0xb0)
	/builddir/build/BUILD/atomic-openshift-git-0.dfe38da/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/cloudprovider/providers/azure/azure_backoff.go:300 +0x24a
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff(0x12a05f200, 0x3ff8000000000000, 0x3ff0000000000000, 0x6, 0xc4258c9130, 0xa692400, 0xc400000000)
	/builddir/build/BUILD/atomic-openshift-git-0.dfe38da/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:203 +0x9c
github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/cloudprovider/providers/azure.(*Cloud).UpdateVmssVMWithRetry(0xc421367200, 0x7857200, 0xc423e94540, 0xc425e3f0e0, 0x24, 0xc42924f800, 0x13, 0xc422e6cf19, 0x1, 0x0, ...)
	/builddir/build/BUILD/atomic-openshift-git-0.dfe38da/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/cloudprovider/providers/azure/azure_backoff.go:297 +0x19f
github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/cloudprovider/providers/azure.(*scaleSet).EnsureHostInPool(0xc425f48380, 0xc42a182554, 0xc, 0xc427c67f40, 0x17, 0xc4294db450, 0xc4, 0x0, 0x0, 0xc425ec6701, ...)
	/builddir/build/BUILD/atomic-openshift-git-0.dfe38da/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/cloudprovider/providers/azure/azure_vmss.go:624 +0xcfb
github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/cloudprovider/providers/azure.(*scaleSet).EnsureHostsInPool.func1(0x87da5a, 0x1)
	/builddir/build/BUILD/atomic-openshift-git-0.dfe38da/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/cloudprovider/providers/azure/azure_vmss.go:660 +0x130
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/errors.AggregateGoroutines.func1(0xc4272c7560, 0xc4272c6960)
	/builddir/build/BUILD/atomic-openshift-git-0.dfe38da/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/errors/errors.go:189 +0x27
created by github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/errors.AggregateGoroutines
	/builddir/build/BUILD/atomic-openshift-git-0.dfe38da/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/errors/errors.go:189 +0x8d

What you expected to happen:

Cloud provider should not panic

How to reproduce it:

Creating ILB as following with missing permissions such as Microsoft.Network/networkSecurityGroups/join/action

ILB Test template used

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"
  creationTimestamp: null
  name: ilb
  selfLink: /api/v1/namespaces/e2e-test-t31lq/services/ilb
spec:
  externalTrafficPolicy: Cluster
  ports:
  - name: port
    nodePort: 31621
    port: 8080
    protocol: TCP
    targetPort: 8080
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer: {}

Error :

I0128 16:22:26.355941       1 azure_vmss.go:623] EnsureHostInPool update backing off vmssVM(infra-1580226559-000000) with new backendPoolID /subscriptions/XXXXXXX-XXXX-XXXX-XXXXXXXXX/resourceGroups/XXXXXXX-XXXX-XXXX-XXXXXXXXX/providers/Microsoft.Network/loadBalancers/kubernetes-internal/backendAddressPools/kubernetes, err: compute.VirtualMachineScaleSetVMsClient#Update: Failure sending request: StatusCode=403 -- Original Error: Code="LinkedAuthorizationFailed" Message="The client 'XXXXXXX-XXXX-XXXX-XXXXXXXXX' with object id 'XXXXXXX-XXXX-XXXX-XXXXXXXXX' has permission to perform action 'Microsoft.Compute/virtualMachineScaleSets/virtualmachines/write' on scope '/subscriptions/XXXXXXX-XXXX-XXXX-XXXXXXXXX/resourceGroups/XXXXXXX-XXXX-XXXX-XXXXXXXXX/providers/Microsoft.Compute/virtualMachineScaleSets/ss-infra-XXXXXXXX/virtualmachines/0'; however, it does not have permission to perform action 'Microsoft.Network/networkSecurityGroups/join/action' on the linked scope(s) '/subscriptions/XXXXXXX-XXXX-XXXX-XXXXXXXXX/resourceGroups/XXXXXXX-XXXX-XXXX-XXXXXXXXX/providers/Microsoft.Network/networkSecurityGroups/nsg-worker' or the linked scope(s) are invalid."

Anything else we need to know?:

This is happening in an OpenShift v3.x cluster deployed via Azure Red Hat OpenShift

Environment:

  • Kubernetes version (use kubectl version):
Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.0+d4cacc0", GitCommit:"d4cacc0", GitTreeState:"clean", BuildDate:"2019-12-02T08:30:15Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}
@feiskyer
Copy link
Member

@julienstroheker Thanks for reporting the issue. It has been fixed by kubernetes/kubernetes#68210 and have been included in v1.12.0 and above versions. Have you tried the same things on v1.12+ clusters?

@julienstroheker
Copy link
Author

@feiskyer Thanks for your answer !

Good to know, I'll check, but ARO is running openshift 3.11 with mean k8s v1.11 + some cherry-pick.

Closing the issue then.

Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@feiskyer @julienstroheker