How to debug a non-working authorization webhook mode config?

[letthefireflieslive]

General information

• Minishift version: v1.24.0+8a904d0
• OS: macOS
• Hypervisor: VirtualBox

Steps to reproduce

1. oc login -u system:admin
2. oc new-project demo
3. oc label ns demo opa-controlled=true
4. oc adm policy add-scc-to-user privileged -z opa-sa
5. oc new-project opa
6. Clone https://github.com/raffaelespazzoli/openshift-opa
7. cd inside
8. helm template ./charts/open-policy-agent --namespace opa --set kubernetes_policy_controller.image_tag=2.0 --set kubernetes_policy_controller.image=quay.io/raffaelespazzoli/kubernetes-policy-controller --set caBundle=$CA_BUNDLE --set log_level=debug | oc apply -f - -n opa
9. oc create cm test1 --from-file=./examples/authorization-webhooks/unreadable_secrets.rego -n opa
10. minishift ssh
11. cd into /var/lib/minisifhit/openshift.local.config/master
12. Create opa-policy-controller.kubeconfig to current path with

# Kubernetes API version
apiVersion: v1
# kind of the API object
kind: Config
# clusters refers to the remote service.
clusters:
  - name: opa-server
    cluster:
      # CA for verifying the remote service.
      certificate-authority: /var/lib/minishift/openshift.local.config/master/ca-bundle.crt
      # URL of remote service to query. Must use 'https'. May not include parameters.
      server: https://opa.opa.svc

# users refers to the API Server's webhook configuration.
users:
  - name: opa-user
    user:
      client-certificate: /var/lib/minishift/openshift.local.config/master/master.kubelet-client.crt # cert for the webhook plugin to use
      client-key: /var/lib/minishift/openshift.local.config/master/master.kubelet-client.key          # key matching the cert

# kubeconfig files require a context. Provide one for the API Server.
current-context: opa-webhook
contexts:
- context:
    cluster: opa-server
    user: opa-user
  name: opa-webhook

13. chown docker:docker opa-policy-controller.kubeconfig
14. chmod 7777 opa-policy-controller.kubeconfig
15. Create audit-policy.yaml to current path

apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
  - level: RequestResponse
    userGroups: ["system:authenticated"]
    resources:
      - group: ""
        resources: ["serviceaccounts", "secrets"]
    omitStages:
      - "RequestReceived"

16. chown docker:docker audit-policy.yaml
17. chmod 7777 audit-policy.yaml
18. Update master-config.yaml to

admissionConfig:
  pluginConfig:
    MutatingAdmissionWebhook:
      configuration:
        apiVersion: apiserver.config.k8s.io/v1alpha1
        kind: WebhookAdmission
        kubeConfigFile: /dev/null
    ValidatingAdmissionWebhook:
      configuration:
        apiVersion: apiserver.config.k8s.io/v1alpha1
        kind: WebhookAdmission
        kubeConfigFile: /dev/null
    GenericAdmissionWebhook:
      configuration:
        apiVersion: v1
        disable: false
        kind: DefaultAdmissionConfig
      location: ""
    openshift.io/ImagePolicy:
      configuration:
        apiVersion: v1
        executionRules:
          - matchImageAnnotations:
              - key: images.openshift.io/deny-execution
                value: "true"
            name: execution-denied
            onResources:
              - resource: pods
              - resource: builds
            reject: true
            skipOnResolutionFailure: true
        kind: ImagePolicyConfig
      location: ""
aggregatorConfig:
  proxyClientInfo:
    certFile: aggregator-front-proxy.crt
    keyFile: aggregator-front-proxy.key
apiLevels:
  - v1
apiVersion: v1
auditConfig:
  enabled: true
  policyFile: audit-policy.yaml
  auditFilePath: audit.log
  logFormat: json
authConfig:
  requestHeader:
    clientCA: front-proxy-ca.crt
    clientCommonNames:
      - aggregator-front-proxy
    extraHeaderPrefixes:
      - X-Remote-Extra-
    groupHeaders:
      - X-Remote-Group
    usernameHeaders:
      - X-Remote-User
controllerConfig:
  controllers:
    - '*'
  election: null
  serviceServingCert:
    signer:
      certFile: service-signer.crt
      keyFile: service-signer.key
controllerLeaseTTL: 0
controllers: '*'
corsAllowedOrigins:
  - //127\.0\.0\.1(:|$)
  - //192\.168\.99\.108:8443$
  - //localhost(:|$)
disabledFeatures: null
dnsConfig:
  allowRecursiveQueries: true
  bindAddress: 0.0.0.0:8053
  bindNetwork: tcp4
etcdClientInfo:
  ca: ca.crt
  certFile: master.etcd-client.crt
  keyFile: master.etcd-client.key
  urls:
    - https://127.0.0.1:4001
etcdConfig:
  address: 127.0.0.1:4001
  peerAddress: 127.0.0.1:7001
  peerServingInfo:
    bindAddress: 0.0.0.0:7001
    bindNetwork: tcp4
    certFile: etcd.server.crt
    clientCA: ca.crt
    keyFile: etcd.server.key
    namedCertificates: null
  servingInfo:
    bindAddress: 0.0.0.0:4001
    bindNetwork: tcp4
    certFile: etcd.server.crt
    clientCA: ca.crt
    keyFile: etcd.server.key
    namedCertificates: null
  storageDirectory: /var/lib/origin/openshift.local.etcd
etcdStorageConfig:
  kubernetesStoragePrefix: kubernetes.io
  kubernetesStorageVersion: v1
  openShiftStoragePrefix: openshift.io
  openShiftStorageVersion: v1
imageConfig:
  format: openshift/origin-${component}:v3.9.0
  latest: false
imagePolicyConfig:
  allowedRegistriesForImport:
    - domainName: docker.io
    - domainName: '*.docker.io'
    - domainName: '*.redhat.com'
    - domainName: gcr.io
    - domainName: quay.io
    - domainName: registry.centos.org
    - domainName: registry.redhat.io
    - domainName: '*.amazonaws.com'
  disableScheduledImport: false
  maxImagesBulkImportedPerRepository: 5
  maxScheduledImageImportsPerMinute: 60
  scheduledImageImportMinimumIntervalSeconds: 900
jenkinsPipelineConfig:
  autoProvisionEnabled: true
  parameters: null
  serviceName: jenkins
  templateName: jenkins-persistent
  templateNamespace: openshift
kind: MasterConfig
kubeletClientInfo:
  ca: ca.crt
  certFile: master.kubelet-client.crt
  keyFile: master.kubelet-client.key
  port: 10250
kubernetesMasterConfig:
  admissionConfig:
    pluginConfig: null
  apiLevels: null
  apiServerArguments:
    authorization-mode:
      - Node
      - Webhook
      - RBAC
    authorization-webhook-config-file:
      - opa-policy-controller.kubeconfig
    runtime-config:
      - apis/admissionregistration.k8s.io/v1alpha1=true
    storage-backend:
      - etcd3
    storage-media-type:
      - application/vnd.kubernetes.protobuf
  controllerArguments: null
  disabledAPIGroupVersions: {}
  masterCount: 1
  masterEndpointReconcileTTL: 15
  masterIP: 127.0.0.1
  podEvictionTimeout: 5m
  proxyClientInfo:
    certFile: master.proxy-client.crt
    keyFile: master.proxy-client.key
  schedulerArguments: null
  schedulerConfigFile: ""
  servicesNodePortRange: 30000-32767
  servicesSubnet: 172.30.0.0/16
  staticNodeNames: null
masterClients:
  externalKubernetesClientConnectionOverrides:
    acceptContentTypes: application/vnd.kubernetes.protobuf,application/json
    burst: 400
    contentType: application/vnd.kubernetes.protobuf
    qps: 200
  externalKubernetesKubeConfig: ""
  openshiftLoopbackClientConnectionOverrides:
    acceptContentTypes: application/vnd.kubernetes.protobuf,application/json
    burst: 600
    contentType: application/vnd.kubernetes.protobuf
    qps: 300
  openshiftLoopbackKubeConfig: openshift-master.kubeconfig
masterPublicURL: https://192.168.99.108:8443
networkConfig:
  clusterNetworkCIDR: 10.128.0.0/14
  clusterNetworks:
    - cidr: 10.128.0.0/14
      hostSubnetLength: 9
  externalIPNetworkCIDRs: null
  hostSubnetLength: 9
  ingressIPNetworkCIDR: 172.29.0.0/16
  networkPluginName: ""
  serviceNetworkCIDR: 172.30.0.0/16
oauthConfig:
  alwaysShowProviderSelection: false
  assetPublicURL: https://192.168.99.108:8443/console/
  grantConfig:
    method: auto
    serviceAccountMethod: prompt
  identityProviders:
    - challenge: true
      login: true
      mappingMethod: claim
      name: anypassword
      provider:
        apiVersion: v1
        kind: AllowAllPasswordIdentityProvider
  masterCA: ca-bundle.crt
  masterPublicURL: https://192.168.99.108:8443
  masterURL: https://127.0.0.1:8443
  sessionConfig:
    sessionMaxAgeSeconds: 300
    sessionName: ssn
    sessionSecretsFile: ""
  templates: null
  tokenConfig:
    accessTokenMaxAgeSeconds: 86400
    authorizeTokenMaxAgeSeconds: 300
pauseControllers: false
policyConfig:
  bootstrapPolicyFile: policy.json
  openshiftInfrastructureNamespace: openshift-infra
  openshiftSharedResourcesNamespace: openshift
  userAgentMatchingConfig:
    defaultRejectionMessage: ""
    deniedClients: null
    requiredClients: null
projectConfig:
  defaultNodeSelector: ""
  projectRequestMessage: ""
  projectRequestTemplate: ""
  securityAllocator:
    mcsAllocatorRange: s0:/2
    mcsLabelsPerProject: 5
    uidAllocatorRange: 1000000000-1999999999/10000
routingConfig:
  subdomain: 192.168.99.108.nip.io
serviceAccountConfig:
  limitSecretReferences: false
  managedNames:
    - default
    - builder
    - deployer
  masterCA: ca-bundle.crt
  privateKeyFile: serviceaccounts.private.key
  publicKeyFiles:
    - serviceaccounts.public.key
servingInfo:
  bindAddress: 0.0.0.0:8443
  bindNetwork: tcp4
  certFile: master.server.crt
  clientCA: ca.crt
  keyFile: master.server.key
  maxRequestsInFlight: 1200
  namedCertificates: null
  requestTimeoutSeconds: 3600
volumeConfig:
  dynamicProvisioningEnabled: true

19. chown docker:docker master-config.yaml
20. chmod 7777 master-config.yaml
21. minishift stop
22. minishift start
23. minishift ssh
24. cd into /var/lib/minisifhit/openshift.local.config/master
25. oc login -u system:admin -n demo
    26.a tail -n 100 -F audit.log
    26.b oc get secrets / oc get secret [samplesecret] -o yaml

Expected

1. I should not be able to view secrets
2. Or at least (for debugging) see some logs if authorization webhook mode is properly configured or not (or what's the problem) was ittrying to send it to the webhook or not

Actual

1. Secrets/tokens are still viewable
2. Can't see 'webhook' or 'SubjectAccessReview' string in the audit.log whenever I trying to oc get/describe secrets in the demo namespace

Logs

https://gist.github.com/lambotchi/0e9016cc33165dc212caea3f386da84f

You can start Minishift with minishift start --show-libmachine-logs -v5 to collect logs.
Please consider posting this on http://gist.github.com/ and post the link in the issue.

[jeicoo]

Trying to integrate OPA with OpenShift 3.9, unfortunately, Kubernetes API is not calling the OPA controller as well. Same master-config.yaml. I really need to integrate OPA authorization in our cluster.

Openshift: 3.9
Kubernetes: 1.9

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.