Skip to content
This repository has been archived by the owner on Oct 24, 2023. It is now read-only.

Upgrade to 1.15.11 ends with podsecuritypolicies.extensions not found #3726

Closed
jprecuch opened this issue Aug 18, 2020 · 10 comments
Closed

Upgrade to 1.15.11 ends with podsecuritypolicies.extensions not found #3726

jprecuch opened this issue Aug 18, 2020 · 10 comments
Labels
bug Something isn't working stale
Projects
backlog

Comments

@jprecuch
Copy link

jprecuch commented Aug 18, 2020 •
edited

Describe the bug
I've tried to upgrade to 1.15.11 today using aks-engine 0.54.1 but master never finished full bringup. In this upgrade I want to change image to be 18.04. It ends same if I upgrade from 1.14.7 or force from 1.15.11.
It seems it ended on

+ retrycmd 120 5 30 /usr/local/bin/kubectl --kubeconfig=/home/ubuntu/.kube/config get podsecuritypolicy privileged restricted
+ retries=120
+ wait_sleep=5
+ timeout=30
+ shift
+ shift
+ shift
++ seq 1 120
+ for i in $(seq 1 $retries)
+ timeout 30 /usr/local/bin/kubectl --kubeconfig=/home/ubuntu/.kube/config get podsecuritypolicy privileged restricted
+ '[' 1 -eq 120 ']'
+ sleep 5
+ for i in $(seq 1 $retries)
+ timeout 30 /usr/local/bin/kubectl --kubeconfig=/home/ubuntu/.kube/config get podsecuritypolicy privileged restricted
Error from server (NotFound): podsecuritypolicies.extensions "privileged" not found
Error from server (NotFound): podsecuritypolicies.extensions "restricted" not found
+ '[' 2 -eq 120 ']'
+ sleep 5
+ for i in $(seq 1 $retries)
+ timeout 30 /usr/local/bin/kubectl --kubeconfig=/home/ubuntu/.kube/config get podsecuritypolicy privileged restricted
Error from server (NotFound): podsecuritypolicies.extensions "privileged" not found
Error from server (NotFound): podsecuritypolicies.extensions "restricted" not found
+ '[' 3 -eq 120 ']'
+ sleep 5

Steps To Reproduce
Try to upgrade from 1.14.7 or 1.15.11 aks-engine (v0.48.0) to 1.15.11 using aks-engine 0.54.1.
Change image from 16.04 to 18.04 and enable pod-security-policy extension.

  "apiVersion": "vlabs",
  "location": "centralus",
  "properties": {
    "orchestratorProfile": {
      "orchestratorType": "Kubernetes",
      "orchestratorRelease": "1.15",
      "orchestratorVersion": "1.15.11",
      "kubernetesConfig": {
        "kubernetesImageBase": "k8s.gcr.io/",
        "kubernetesImageBaseType": "gcr",
        "mcrKubernetesImageBase": "mcr.microsoft.com/",
        "clusterSubnet": "10.244.0.0/16",
        "dnsServiceIP": "10.0.0.10",
        "serviceCidr": "10.0.0.0/16",
        "networkPlugin": "kubenet",
        "containerRuntime": "docker",
        "dockerBridgeSubnet": "172.17.0.1/16",
        "mobyVersion": "3.0.11",
        "useInstanceMetadata": true,
        "enableRbac": true,
        "enableSecureKubelet": true,
        "enableAggregatedAPIs": true,
        "privateCluster": {
          "enabled": true
        },
        "gchighthreshold": 85,
        "gclowthreshold": 80,
        "etcdVersion": "3.3.18",
        "etcdDiskSizeGB": "1024",
        "addons": [
          {
            "name": "heapster",
            "enabled": false
          },
          {
            "name": "tiller",
            "enabled": false
          },
          {
            "name": "aci-connector",
            "enabled": false
          },
          {
            "name": "cluster-autoscaler",
            "enabled": false
          },
          {
            "name": "blobfuse-flexvolume",
            "enabled": true,
            "containers": [
              {
                "name": "blobfuse-flexvolume",
                "image": "mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8",
                "cpuRequests": "50m",
                "memoryRequests": "100Mi",
                "cpuLimits": "50m",
                "memoryLimits": "100Mi"
              }
            ]
          },
          {
            "name": "smb-flexvolume",
            "enabled": false
          },
          {
            "name": "keyvault-flexvolume",
            "enabled": true,
            "containers": [
              {
                "name": "keyvault-flexvolume",
                "image": "mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.16",
                "cpuRequests": "50m",
                "memoryRequests": "100Mi",
                "cpuLimits": "50m",
                "memoryLimits": "100Mi"
              }
            ]
          },
          {
            "name": "kubernetes-dashboard",
            "enabled": true,
            "containers": [
              {
                "name": "kubernetes-dashboard",
                "image": "k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1",
                "cpuRequests": "300m",
                "memoryRequests": "150Mi",
                "cpuLimits": "300m",
                "memoryLimits": "150Mi"
              }
            ]
          },
          {
            "name": "rescheduler",
            "enabled": false
          },
          {
            "name": "metrics-server",
            "enabled": true,
            "containers": [
              {
                "name": "metrics-server",
                "image": "k8s.gcr.io/metrics-server-amd64:v0.2.1"
              }
            ]
          },
          {
            "name": "nvidia-device-plugin",
            "enabled": false
          },
          {
            "name": "container-monitoring",
            "enabled": false
          },
          {
            "name": "azure-cni-networkmonitor",
            "enabled": false
          },
          {
            "name": "azure-npm-daemonset",
            "enabled": false
          },
          {
            "name": "cloud-node-manager",
            "enabled": false
          },
          {
            "name": "ip-masq-agent",
            "enabled": true,
            "containers": [
              {
                "name": "ip-masq-agent",
                "image": "k8s.gcr.io/ip-masq-agent-amd64:v2.5.0",
                "cpuRequests": "50m",
                "memoryRequests": "50Mi",
                "cpuLimits": "50m",
                "memoryLimits": "250Mi"
              }
            ],
            "config": {
              "enable-ipv6": "false",
              "non-masq-cni-cidr": "",
              "non-masquerade-cidr": "10.244.0.0/16",
              "secondary-non-masquerade-cidr": ""
            }
          },
          {
            "name": "dns-autoscaler",
            "enabled": false
          },
          {
            "name": "calico-daemonset",
            "enabled": false
          },
          {
            "name": "cilium",
            "enabled": false
          },
          {
            "name": "aad-pod-identity",
            "enabled": false
          },
          {
            "name": "appgw-ingress",
            "enabled": false
          },
          {
            "name": "azuredisk-csi-driver",
            "enabled": false
          },
          {
            "name": "azurefile-csi-driver",
            "enabled": false
          },
          {
            "name": "azure-policy",
            "enabled": false
          },
          {
            "name": "node-problem-detector",
            "enabled": false
          },
          {
            "name": "kube-dns",
            "enabled": false
          },
          {
            "name": "coredns",
            "enabled": true,
            "containers": [
              {
                "name": "coredns",
                "image": "k8s.gcr.io/coredns:1.6.7"
              }
            ],
            "config": {
              "clusterIP": "10.0.0.10",
              "domain": "cluster.local"
            }
          },
          {
            "name": "kube-proxy",
            "enabled": true,
            "containers": [
              {
                "name": "kube-proxy",
                "image": "k8s.gcr.io/hyperkube-amd64:v1.15.11"
              }
            ],
            "config": {
              "cluster-cidr": "10.244.0.0/16",
              "featureGates": "{}",
              "proxy-mode": "iptables"
            }
          },
          {
            "name": "pod-security-policy",
            "enabled": true
          },
          {
            "name": "audit-policy",
            "enabled": true
          },
          {
            "name": "azure-cloud-provider",
            "enabled": true
          },
          {
            "name": "aad",
            "enabled": false
          },
          {
            "name": "antrea",
            "enabled": false
          },
          {
            "name": "flannel",
            "enabled": false
          },
          {
            "name": "scheduled-maintenance",
            "enabled": false
          }
        ],
        "components": [
          {
            "name": "kube-scheduler",
            "enabled": true,
            "containers": [
              {
                "name": "kube-scheduler",
                "image": "k8s.gcr.io/hyperkube-amd64:v1.15.11"
              }
            ],
            "config": {
              "command": "\"/hyperkube\", \"kube-scheduler\""
            }
          },
          {
            "name": "kube-controller-manager",
            "enabled": true,
            "containers": [
              {
                "name": "kube-controller-manager",
                "image": "k8s.gcr.io/hyperkube-amd64:v1.15.11"
              }
            ],
            "config": {
              "command": "\"/hyperkube\", \"kube-controller-manager\""
            }
          },
          {
            "name": "cloud-controller-manager",
            "enabled": false
          },
          {
            "name": "kube-apiserver",
            "enabled": true,
            "containers": [
              {
                "name": "kube-apiserver",
                "image": "k8s.gcr.io/hyperkube-amd64:v1.15.11"
              }
            ],
            "config": {
              "command": "\"/hyperkube\", \"kube-apiserver\""
            }
          },
          {
            "name": "kube-addon-manager",
            "enabled": true,
            "containers": [
              {
                "name": "kube-addon-manager",
                "image": "k8s.gcr.io/kube-addon-manager-amd64:v9.0.2"
              }
            ]
          }
        ],
        "kubeletConfig": {
          "--address": "0.0.0.0",
          "--anonymous-auth": "false",
          "--authorization-mode": "Webhook",
          "--azure-container-registry-config": "/etc/kubernetes/azure.json",
          "--cgroups-per-qos": "true",
          "--client-ca-file": "/etc/kubernetes/certs/ca.crt",
          "--cloud-config": "/etc/kubernetes/azure.json",
          "--cloud-provider": "azure",
          "--cluster-dns": "10.0.0.10",
          "--cluster-domain": "cluster.local",
          "--enforce-node-allocatable": "pods",
          "--event-qps": "0",
          "--eviction-hard": "memory.available<750Mi,nodefs.available<10%,nodefs.inodesFree<5%",
          "--feature-gates": "RotateKubeletServerCertificate=true",
          "--image-gc-high-threshold": "85",
          "--image-gc-low-threshold": "80",
          "--image-pull-progress-deadline": "30m",
          "--keep-terminated-pod-volumes": "false",
          "--kubeconfig": "/var/lib/kubelet/kubeconfig",
          "--max-pods": "110",
          "--network-plugin": "kubenet",
          "--node-status-update-frequency": "10s",
          "--non-masquerade-cidr": "0.0.0.0/0",
          "--pod-infra-container-image": "mcr.microsoft.com/k8s/core/pause:1.2.0",
          "--pod-manifest-path": "/etc/kubernetes/manifests",
          "--pod-max-pids": "-1",
          "--rotate-certificates": "true",
          "--streaming-connection-idle-timeout": "4h",
          "--tls-cert-file": "/etc/kubernetes/certs/kubeletserver.crt",
          "--tls-cipher-suites": "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256",
          "--tls-private-key-file": "/etc/kubernetes/certs/kubeletserver.key"
        },
        "controllerManagerConfig": {
          "--allocate-node-cidrs": "true",
          "--cloud-config": "/etc/kubernetes/azure.json",
          "--cloud-provider": "azure",
          "--cluster-cidr": "10.244.0.0/16",
          "--cluster-name": "k8scentralus-stagedc",
          "--cluster-signing-cert-file": "/etc/kubernetes/certs/ca.crt",
          "--cluster-signing-key-file": "/etc/kubernetes/certs/ca.key",
          "--configure-cloud-routes": "true",
          "--controllers": "*,bootstrapsigner,tokencleaner",
          "--feature-gates": "LocalStorageCapacityIsolation=true,ServiceNodeExclusion=true",
          "--kubeconfig": "/var/lib/kubelet/kubeconfig",
          "--leader-elect": "true",
          "--node-monitor-grace-period": "40s",
          "--pod-eviction-timeout": "5m0s",
          "--profiling": "false",
          "--root-ca-file": "/etc/kubernetes/certs/ca.crt",
          "--route-reconciliation-period": "10s",
          "--service-account-private-key-file": "/etc/kubernetes/certs/apiserver.key",
          "--terminated-pod-gc-threshold": "5000",
          "--use-service-account-credentials": "true",
          "--v": "2"
        },
        "cloudControllerManagerConfig": {
          "--allocate-node-cidrs": "true",
          "--cloud-config": "/etc/kubernetes/azure.json",
          "--cloud-provider": "azure",
          "--cluster-cidr": "10.244.0.0/16",
          "--cluster-name": "k8scentralus-stagedc",
          "--configure-cloud-routes": "true",
          "--kubeconfig": "/var/lib/kubelet/kubeconfig",
          "--leader-elect": "true",
          "--route-reconciliation-period": "10s",
          "--v": "2"
        },
        "apiServerConfig": {
          "--advertise-address": "<advertiseAddr>",
          "--allow-privileged": "true",
          "--anonymous-auth": "false",
          "--audit-log-maxage": "30",
          "--audit-log-maxbackup": "10",
          "--audit-log-maxsize": "100",
          "--audit-log-path": "/var/log/kubeaudit/audit.log",
          "--audit-policy-file": "/etc/kubernetes/addons/audit-policy.yaml",
          "--authorization-mode": "Node,RBAC",
          "--bind-address": "0.0.0.0",
          "--client-ca-file": "/etc/kubernetes/certs/ca.crt",
          "--cloud-config": "/etc/kubernetes/azure.json",
          "--cloud-provider": "azure",
          "--enable-admission-plugins": "NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,ValidatingAdmissionWebhook,ResourceQuota,ExtendedResourceToleration",
          "--enable-bootstrap-token-auth": "true",
          "--etcd-cafile": "/etc/kubernetes/certs/ca.crt",
          "--etcd-certfile": "/etc/kubernetes/certs/etcdclient.crt",
          "--etcd-keyfile": "/etc/kubernetes/certs/etcdclient.key",
          "--etcd-servers": "https://127.0.0.1:2379",
          "--feature-gates": "VolumeSnapshotDataSource=true",
          "--insecure-port": "8080",
          "--kubelet-client-certificate": "/etc/kubernetes/certs/client.crt",
          "--kubelet-client-key": "/etc/kubernetes/certs/client.key",
          "--profiling": "false",
          "--proxy-client-cert-file": "/etc/kubernetes/certs/proxy.crt",
          "--proxy-client-key-file": "/etc/kubernetes/certs/proxy.key",
          "--requestheader-allowed-names": "",
          "--requestheader-client-ca-file": "/etc/kubernetes/certs/proxy-ca.crt",
          "--requestheader-extra-headers-prefix": "X-Remote-Extra-",
          "--requestheader-group-headers": "X-Remote-Group",
          "--requestheader-username-headers": "X-Remote-User",
          "--secure-port": "443",
          "--service-account-key-file": "/etc/kubernetes/certs/apiserver.key",
          "--service-account-lookup": "true",
          "--service-cluster-ip-range": "10.0.0.0/16",
          "--storage-backend": "etcd3",
          "--tls-cert-file": "/etc/kubernetes/certs/apiserver.crt",
          "--tls-cipher-suites": "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA",
          "--tls-private-key-file": "/etc/kubernetes/certs/apiserver.key",
          "--v": "4"
        },
        "schedulerConfig": {
          "--kubeconfig": "/var/lib/kubelet/kubeconfig",
          "--leader-elect": "true",
          "--profiling": "false",
          "--v": "2"
        },
        "cloudProviderBackoffMode": "v2",
        "cloudProviderBackoff": false,
        "cloudProviderBackoffRetries": 6,
        "cloudProviderBackoffJitter": 1,
        "cloudProviderBackoffDuration": 5,
        "cloudProviderBackoffExponent": 1.5,
        "cloudProviderRateLimit": true,
        "cloudProviderRateLimitQPS": 10,
        "cloudProviderRateLimitQPSWrite": 10,
        "cloudProviderRateLimitBucket": 100,
        "cloudProviderRateLimitBucketWrite": 100,
        "cloudProviderDisableOutboundSNAT": false,
        "loadBalancerSku": "Standard",
        "excludeMasterFromStandardLB": true,
        "maximumLoadBalancerRuleCount": 250,
        "kubeProxyMode": "iptables",
        "outboundRuleIdleTimeoutInMinutes": 30
      }
    },
    "masterProfile": {
      "count": 3,
      "dnsPrefix": "k8scentralus-stagedc",
      "subjectAltNames": null,
      "vmSize": "Standard_D2s_v3",
      "osDiskSizeGB": 100,
      "vnetSubnetID": "/subscriptions/xxx/resourceGroups/anf.dc.mgmt.centralus-stage.rg/providers/Microsoft.Network/virtualNetworks/anf.dc.mgmt.centralus-stage.vnet/subnets/anf.dc.mgmt.centralus-stage.subnet",
      "firstConsecutiveStaticIP": "10.31.0.50",
      "storageProfile": "ManagedDisks",
      "oauthEnabled": false,
      "preProvisionExtension": null,
      "extensions": [],
      "distro": "aks-ubuntu-18.04",
      "kubernetesConfig": {
        "kubeletConfig": {
          "--address": "0.0.0.0",
          "--anonymous-auth": "false",
          "--authorization-mode": "Webhook",
          "--azure-container-registry-config": "/etc/kubernetes/azure.json",
          "--cgroups-per-qos": "true",
          "--client-ca-file": "/etc/kubernetes/certs/ca.crt",
          "--cloud-config": "/etc/kubernetes/azure.json",
          "--cloud-provider": "azure",
          "--cluster-dns": "10.0.0.10",
          "--cluster-domain": "cluster.local",
          "--enforce-node-allocatable": "pods",
          "--event-qps": "0",
          "--eviction-hard": "memory.available<750Mi,nodefs.available<10%,nodefs.inodesFree<5%",
          "--feature-gates": "RotateKubeletServerCertificate=true",
          "--image-gc-high-threshold": "85",
          "--image-gc-low-threshold": "80",
          "--image-pull-progress-deadline": "30m",
          "--keep-terminated-pod-volumes": "false",
          "--kubeconfig": "/var/lib/kubelet/kubeconfig",
          "--max-pods": "110",
          "--network-plugin": "kubenet",
          "--node-status-update-frequency": "10s",
          "--non-masquerade-cidr": "0.0.0.0/0",
          "--pod-infra-container-image": "mcr.microsoft.com/k8s/core/pause:1.2.0",
          "--pod-manifest-path": "/etc/kubernetes/manifests",
          "--pod-max-pids": "-1",
          "--protect-kernel-defaults": "true",
          "--rotate-certificates": "true",
          "--streaming-connection-idle-timeout": "4h",
          "--tls-cert-file": "/etc/kubernetes/certs/kubeletserver.crt",
          "--tls-cipher-suites": "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256",
          "--tls-private-key-file": "/etc/kubernetes/certs/kubeletserver.key"
        },
        "cloudProviderBackoffMode": ""
      },
      "availabilityProfile": "AvailabilitySet",
      "availabilityZones": [
        "1",
        "2",
        "3"
      ],
      "platformFaultDomainCount": null,
      "platformUpdateDomainCount": 3,
      "sysctldConfig": {
        "net.core.message_burst": "80",
        "net.core.message_cost": "40",
        "net.core.somaxconn": "16384",
        "net.ipv4.neigh.default.gc_thresh1": "4096",
        "net.ipv4.neigh.default.gc_thresh2": "8192",
        "net.ipv4.neigh.default.gc_thresh3": "16384",
        "net.ipv4.tcp_max_syn_backlog": "16384",
        "net.ipv4.tcp_retries2": "8"
      },
      "cosmosEtcd": false
    },
    "agentPoolProfiles": [
      {
        "name": "node",
        "count": 9,
        "vmSize": "Standard_D8s_v3",
        "osDiskSizeGB": 100,
        "osType": "Linux",
        "availabilityProfile": "VirtualMachineScaleSets",
        "storageProfile": "ManagedDisks",
        "vnetSubnetID": "/subscriptions/xxxx/resourceGroups/anf.dc.mgmt.centralus-stage.rg/providers/Microsoft.Network/virtualNetworks/anf.dc.mgmt.centralus-stage.vnet/subnets/anf.dc.mgmt.centralus-stage.subnet",
        "distro": "aks-ubuntu-18.04",
        "kubernetesConfig": {
          "kubeletConfig": {
            "--address": "0.0.0.0",
            "--anonymous-auth": "false",
            "--authorization-mode": "Webhook",
            "--azure-container-registry-config": "/etc/kubernetes/azure.json",
            "--cgroups-per-qos": "true",
            "--client-ca-file": "/etc/kubernetes/certs/ca.crt",
            "--cloud-config": "/etc/kubernetes/azure.json",
            "--cloud-provider": "azure",
            "--cluster-dns": "10.0.0.10",
            "--cluster-domain": "cluster.local",
            "--enforce-node-allocatable": "pods",
            "--event-qps": "0",
            "--eviction-hard": "memory.available<750Mi,nodefs.available<10%,nodefs.inodesFree<5%",
            "--feature-gates": "RotateKubeletServerCertificate=true",
            "--image-gc-high-threshold": "85",
            "--image-gc-low-threshold": "80",
            "--image-pull-progress-deadline": "30m",
            "--keep-terminated-pod-volumes": "false",
            "--kubeconfig": "/var/lib/kubelet/kubeconfig",
            "--max-pods": "110",
            "--network-plugin": "kubenet",
            "--node-status-update-frequency": "10s",
            "--non-masquerade-cidr": "0.0.0.0/0",
            "--pod-infra-container-image": "mcr.microsoft.com/k8s/core/pause:1.2.0",
            "--pod-manifest-path": "/etc/kubernetes/manifests",
            "--pod-max-pids": "-1",
            "--protect-kernel-defaults": "true",
            "--rotate-certificates": "true",
            "--streaming-connection-idle-timeout": "4h",
            "--tls-cert-file": "/etc/kubernetes/certs/kubeletserver.crt",
            "--tls-cipher-suites": "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256",
            "--tls-private-key-file": "/etc/kubernetes/certs/kubeletserver.key"
          },
          "cloudProviderBackoffMode": ""
        },
        "acceleratedNetworkingEnabled": true,
        "acceleratedNetworkingEnabledWindows": false,
        "vmssOverProvisioningEnabled": false,
        "auditDEnabled": false,
        "fqdn": "",
        "preProvisionExtension": null,
        "extensions": [],
        "singlePlacementGroup": true,
        "platformFaultDomainCount": null,
        "platformUpdateDomainCount": 3,
        "availabilityZones": [
          "1",
          "2",
          "3"
        ],
        "enableVMSSNodePublicIP": false,
        "sysctldConfig": {
          "net.core.message_burst": "80",
          "net.core.message_cost": "40",
          "net.core.somaxconn": "16384",
          "net.ipv4.neigh.default.gc_thresh1": "4096",
          "net.ipv4.neigh.default.gc_thresh2": "8192",
          "net.ipv4.neigh.default.gc_thresh3": "16384",
          "net.ipv4.tcp_max_syn_backlog": "16384",
          "net.ipv4.tcp_retries2": "8"
        }
      }
    ],
    "linuxProfile": {
      "adminUsername": "ubuntu",
      "ssh": {
        "publicKeys": [
          {
            "keyData": ""
          }
        ]
      }
    },
    "servicePrincipalProfile": {
      "clientId": "",
      "secret": ""
    },
    "certificateProfile": {
 
    "telemetryProfile": {
      "applicationInsightsKey": "xxx"
    }
  }
}

Expected behavior
Upgrade works
AKS Engine version
0.54.1
Kubernetes version
1.14.7
Additional context
@jprecuch jprecuch added the bug Something isn't working label Aug 18, 2020
@CecileRobertMichon
Copy link
Contributor

@jprecuch I believe you're running into #3656, which should have been fixed by #3673

@jackfrancis @mboersma when is the above fix expected to be in a release?

@jprecuch
Copy link
Author

jprecuch commented Aug 19, 2020 •
edited

@CecileRobertMichon not really. #3656 says that extension should be enabled and then it should work. Which we did enable but issue is different now.

          {
            "name": "pod-security-policy",
            "enabled": true
          },

Are there some additional option needful to be added/enabed in apimodel?

Error from server (NotFound): podsecuritypolicies.extensions "privileged" not found
Error from server (NotFound): podsecuritypolicies.extensions "restricted" not found

@CecileRobertMichon
Copy link
Contributor

@jackfrancis any ideas about the error above?

@jackfrancis
Copy link
Member

@jprecuch, are you able to log onto one of the master VMs after a failure and report if you have the pod-security-policy spec in the /etc/kubernetes/addons/ directory?

@jprecuch
Copy link
Author

@jackfrancis @CecileRobertMichon This is what is inside of that directory

root@k8s-master-13989919-0:~# ls -la /etc/kubernetes/addons/
total 64
drwxr-xr-x 3 root root 4096 Aug 18 11:10 .
drwxr-xr-x 6 root root 4096 Aug 18 11:10 ..
-rw-r--r-- 1 root root  841 Aug 18 11:08 audit-policy.yaml
-rw-r--r-- 1 root root 3027 Aug 18 11:08 azure-cloud-provider.yaml
-rw-r--r-- 1 root root 1386 Aug 18 11:08 blobfuse-flexvolume.yaml
-rw-r--r-- 1 root root 8250 Aug 18 11:08 coredns.yaml
drwxr-xr-x 2 root root 4096 Aug 18 11:10 init
-rw-r--r-- 1 root root 1757 Aug 18 11:08 ip-masq-agent.yaml
-rw-r--r-- 1 root root 1466 Aug 18 11:08 keyvault-flexvolume.yaml
-rw-r--r-- 1 root root 2437 Aug 18 11:08 kube-proxy.yaml
-rw-r--r-- 1 root root 7210 Aug 18 11:08 kubernetes-dashboard.yaml
-rw-r--r-- 1 root root 3377 Aug 18 11:08 metrics-server.yaml
-rw-r--r-- 1 root root 2880 Aug 18 11:10 pod-security-policy.yaml

root@k8s-master-13989919-0:~# cat /etc/kubernetes/addons/pod-security-policy.yaml
apiVersion: extensions/v1beta1
kind: PodSecurityPolicy
metadata:
  name: privileged
  annotations:
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: "*"
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  privileged: true
  allowPrivilegeEscalation: true
  allowedCapabilities:
  - "*"
  volumes:
  - "*"
  hostNetwork: true
  hostPorts:
  - min: 0
    max: 65535
  hostIPC: true
  hostPID: true
  runAsUser:
    rule: RunAsAny
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny
---
apiVersion: extensions/v1beta1
kind: PodSecurityPolicy
metadata:
  name: restricted
  annotations:
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
    apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
    seccomp.security.alpha.kubernetes.io/defaultProfileName:  docker/default
    apparmor.security.beta.kubernetes.io/defaultProfileName:  runtime/default
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  privileged: false
  allowPrivilegeEscalation: false
  requiredDropCapabilities:
    - ALL
  volumes:
    - configMap
    - emptyDir
    - projected
    - secret
    - downwardAPI
    - persistentVolumeClaim
  hostNetwork: false
  hostIPC: false
  hostPID: false
  runAsUser:
    rule: MustRunAsNonRoot
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: MustRunAs
    ranges:
      - min: 1
        max: 65535
  fsGroup:
    rule: MustRunAs
    ranges:
      - min: 1
        max: 65535
  readOnlyRootFilesystem: false
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: psp:privileged
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
rules:
- apiGroups: ['extensions']
  resources: ['podsecuritypolicies']
  verbs:     ['use']
  resourceNames:
  - privileged
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: psp:restricted
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
rules:
- apiGroups: ['extensions']
  resources: ['podsecuritypolicies']
  verbs:     ['use']
  resourceNames:
  - restricted
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: default:restricted
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: psp:restricted
subjects:
- kind: Group
  name: system:authenticated
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: default:privileged
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: psp:privileged
subjects:
- kind: Group
  name: system:authenticated
  apiGroup: rbac.authorization.k8s.io
- kind: Group
  name: system:nodes
  apiGroup: rbac.authorization.k8s.io

@jackfrancis
Copy link
Member

@jprecuch thank you, I'll attempt a repro today

@jackfrancis
Copy link
Member

O.K., I was unable to repro using the following flow:

  1. create a simple 1 master, 1 node v1.14.7 cluster using aks-engine v0.48.0 w/ pod-security-policy addon disabled
  2. upgrade to v1.15.11 using aks-engine v0.54.1, and updating the apimodel.json before running upgrade to enable the pod-security-policy addon

@jackfrancis
Copy link
Member

Here's the relevant CSE (CustomScriptExtension, i.e., the bootstrap script that runs on the VM) output (in /var/log/azure/cluster-provision.log) that ran during my upgrade operation:

+ retrycmd 120 5 30 /usr/local/bin/kubectl --kubeconfig=/home/azureuser/.kube/config get podsecuritypolicy privileged restricted
+ retries=120
+ wait_sleep=5
+ timeout=30
+ shift
+ shift
+ shift
++ seq 1 120
+ for i in '$(seq 1 $retries)'
+ timeout 30 /usr/local/bin/kubectl --kubeconfig=/home/azureuser/.kube/config get podsecuritypolicy privileged restricted
NAME         PRIV    CAPS   SELINUX    RUNASUSER          FSGROUP     SUPGROUP    READONLYROOTFS   VOLUMES
privileged   true    *      RunAsAny   RunAsAny           RunAsAny    RunAsAny    false            *
restricted   false          RunAsAny   MustRunAsNonRoot   MustRunAs   MustRunAs   false            configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
+ break
+ echo Executed '"/usr/local/bin/kubectl' --kubeconfig=/home/azureuser/.kube/config get podsecuritypolicy privileged 'restricted"' 1 times
Executed "/usr/local/bin/kubectl --kubeconfig=/home/azureuser/.kube/config get podsecuritypolicy privileged restricted" 1 times

So, in other words, simply enabling the pod-security-policy addon prior to upgrading did the things we'd expect:

  1. It actuallly installed the PodSecurityPolicy resources we expect
  2. It engaged the CSE checks we'd expect, and those checks passed

There must be some other edge case happening. @jprecuch can you confirm that the privileged and restricted PodSecurityPolicy resources aren't present on your cluster after upgrading?

@jprecuch
Copy link
Author

jprecuch commented Aug 26, 2020 •
edited

@jackfrancis I believe it has something with first master not being primary.
There are no podsecurity policies created. I can see it touch the file on master but that's all

2020-08-26 05:38:23,491 - util.py[DEBUG]: Writing to /etc/kubernetes/addons/pod-security-policy.yaml - wb: [644] 2885 bytes
2020-08-26 05:38:23,491 - util.py[DEBUG]: Changing the ownership of /etc/kubernetes/addons/pod-security-policy.yaml to 0:-1

I've copied the file it creates from /etc/kubernetes/addons/pod-security-policy.yaml and applied within my cluster manually and it created those policies fine. After that I was able to upgrade to 1.15.12.

  1. Did you change ubuntu from 16.04 to 18.04 as part of upgrade?
  2. Can you test this with 3 masters? Might be that if master is not primary it won't allow him to create these policies maybe?

@craiglpeters craiglpeters added this to To do in backlog Sep 3, 2020
@stale
Copy link

stale bot commented Nov 16, 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale label Nov 16, 2020
@stale stale bot closed this as completed Dec 19, 2020
backlog automation moved this from To do to Done Dec 19, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
bug Something isn't working stale
Projects
No open projects
backlog
  
Done
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jackfrancis @CecileRobertMichon @jprecuch