OpenShift hitting external kubernetes for internal OpenShift resources

[paralin]

Running against external kube version:

kubectl version
Client Version: version.Info{Major:"1", Minor:"3+", GitVersion:"v1.3.0-alpha.5-dirty", GitCommit:"914163247d9a16b46921e83f7dbedb572859b3e4", GitTreeState:"dirty", BuildDate:"2016-06-14T18:43:00Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.5", GitCommit:"b0deb2eb8f4037421077f77cb163dbb4c0a2a9f5", GitTreeState:"clean", BuildDate:"2016-08-11T20:21:58Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}

SecurityContextConstraint does not exist in Kubernetes API, I think. So it's trying to hit external kube for something that is openshift specific.

[paralin]

So reading through the code:

E0812 00:23:49.586940       1 reflector.go:216] github.com/openshift/origin/pkg/controller/shared/shared_informer.go:87: Failed to list *api.SecurityContextConstraints: t
he server could not find the requested resource
I0812 00:23:49.886093       1 ensure.go:193] Ignoring default security context constraints when running on external Kubernetes.

So it should not use security context constraints.

E0812 00:23:58.859631       1 reflector.go:216] github.com/openshift/origin/pkg/controller/shared/shared_informer.go:87: Failed to list *api.SecurityContextConstraints: the server could not find the requested resource

I think something that should be ignoring the constraints is not.

[paralin]

@pweil- @danmcp Any updates on this? I can't deploy OpenShift because of this.

[paralin]

Updating this issue, now after updating to 1.4.0-alpha.0:

E1009 21:25:45.788504       1 reflector.go:214] github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/admission/storageclass/default/admission.go:62: Failed to 
list *storage.StorageClass: the server could not find the requested resource
E1009 21:25:45.804624       1 reflector.go:214] github.com/openshift/origin/pkg/build/controller/factory/factory.go:246: Failed to list *api.Pod: the server could not fin
d the requested resource (get builds)
E1009 21:25:45.805077       1 reflector.go:214] github.com/openshift/origin/pkg/build/controller/factory/factory.go:285: Failed to list *api.ImageStream: the server could
 not find the requested resource (get imageStreams)
E1009 21:25:45.808053       1 reflector.go:214] github.com/openshift/origin/pkg/build/controller/factory/factory.go:191: Failed to list *api.Build: the server could not f
ind the requested resource (get builds)
E1009 21:25:45.808676       1 reflector.go:214] github.com/openshift/origin/pkg/build/controller/factory/factory.go:324: Failed to list *api.BuildConfig: the server could
 not find the requested resource (get buildConfigs)
E1009 21:25:45.811129       1 reflector.go:214] github.com/openshift/origin/pkg/build/controller/factory/factory.go:288: Failed to list *api.BuildConfig: the server could
 not find the requested resource (get buildConfigs)

[paralin]

Here's my master-config.yaml:

admissionConfig:
  pluginConfig: null
apiLevels:
- v1
apiVersion: v1
assetConfig:
  extensionDevelopment: false
  extensionProperties: null
  extensionScripts: null
  extensionStylesheets: null
  extensions: null
  loggingPublicURL: ''
  logoutURL: ''
  masterPublicURL: https://[redacted]:443
  metricsPublicURL: ""
  publicURL: https://[redacted]:443/console/
  servingInfo:
    bindAddress: 0.0.0.0:443
    bindNetwork: tcp4
    certFile: master.server.crt
    clientCA: ''
    keyFile: master.server.key
    maxRequestsInFlight: 0
    namedCertificates: null
    requestTimeoutSeconds: 0
auditConfig:
  enabled: false
controllerConfig:
  serviceServingCert:
    signer:
      certFile: service-signer.crt
      keyFile: service-signer.key
controllerLeaseTTL: 0
controllers: '*'
corsAllowedOrigins:
- 127.0.0.1
- localhost
- [redacted]:443
disabledFeatures: null
dnsConfig:
  allowRecursiveQueries: true
  bindAddress: 0.0.0.0:8053
  bindNetwork: tcp4
etcdClientInfo:
  ca: etcd.server.crt
  certFile: master.etcd-client.crt
  keyFile: master.etcd-client.key
  urls:
  - http://etcd:4001
etcdConfig: null
etcdStorageConfig:
  kubernetesStoragePrefix: kubernetes.io
  kubernetesStorageVersion: v1
  openShiftStoragePrefix: openshift.io
  openShiftStorageVersion: v1
imageConfig:
  format: openshift/origin-${component}:${version}
  latest: false
imagePolicyConfig:
  disableScheduledImport: false
  maxImagesBulkImportedPerRepository: 5
  maxScheduledImageImportsPerMinute: 60
  scheduledImageImportMinimumIntervalSeconds: 900
jenkinsPipelineConfig:
  enabled: true
  parameters: null
  serviceName: jenkins
  templateName: jenkins
  templateNamespace: openshift
kind: MasterConfig
kubeletClientInfo:
  ca: ''
  certFile: master.kubelet-client.crt
  keyFile: master.kubelet-client.key
  port: 10250
kubernetesMasterConfig: null
masterClients:
  externalKubernetesClientConnectionOverrides:
    acceptContentTypes: application/vnd.kubernetes.protobuf,application/json
    burst: 400
    contentType: application/vnd.kubernetes.protobuf
    qps: 200
  externalKubernetesKubeConfig: external-master.kubeconfig
  openshiftLoopbackClientConnectionOverrides:
    acceptContentTypes: application/vnd.kubernetes.protobuf,application/json
    burst: 600
    contentType: application/vnd.kubernetes.protobuf
    qps: 300
  openshiftLoopbackKubeConfig: openshift-master.kubeconfig
masterPublicURL: https://[redacted]:443
networkConfig:
  clusterNetworkCIDR: 10.128.0.0/14
  externalIPNetworkCIDRs: null
  hostSubnetLength: 9
  networkPluginName: ''
  serviceNetworkCIDR: 172.30.0.0/16
oauthConfig:
  alwaysShowProviderSelection: false
  assetPublicURL: https://[redacted]:443/console/
  grantConfig:
    method: auto
    serviceAccountMethod: prompt
  identityProviders:
  - name: github
    challenge: false
    login: true
    mappingMethod: claim
    provider:
      apiVersion: v1
      kind: GitHubIdentityProvider
      clientID: [redacted]
      clientSecret: [redacted]
      organizations:
      - fuserobotics
  masterCA: ca-bundle.crt
  masterPublicURL: https://[redacted]:443
  masterURL: https://10.0.196.249:443
  sessionConfig:
    sessionMaxAgeSeconds: 300
    sessionName: ssn
    sessionSecretsFile: ''
  templates: null
  tokenConfig:
    accessTokenMaxAgeSeconds: 86400
    authorizeTokenMaxAgeSeconds: 300
pauseControllers: false
policyConfig:
  bootstrapPolicyFile: policy.json
  openshiftInfrastructureNamespace: openshift-infra
  openshiftSharedResourcesNamespace: openshift
  userAgentMatchingConfig:
    defaultRejectionMessage: ''
    deniedClients: null
    requiredClients: null
projectConfig:
  defaultNodeSelector: ''
  projectRequestMessage: ''
  projectRequestTemplate: ''
  securityAllocator:
    mcsAllocatorRange: s0:/2
    mcsLabelsPerProject: 5
    uidAllocatorRange: 1000000000-1999999999/10000
routingConfig:
  subdomain: router.default.svc.cluster.local
serviceAccountConfig:
  limitSecretReferences: false
  managedNames:
  - builder
  - deployer
  masterCA: ''
  privateKeyFile: ''
  publicKeyFiles:
  - serviceaccounts.public.key
servingInfo:
  bindAddress: 0.0.0.0:443
  bindNetwork: tcp4
  certFile: master.server.crt
  clientCA: ca.crt
  keyFile: master.server.key
  maxRequestsInFlight: 500
  namedCertificates: null
  requestTimeoutSeconds: 3600
volumeConfig:
  dynamicProvisioningEnabled: true

[paralin]

cc @liggitt

[liggitt]

More fundamental work is required to support that deployment model. That work is ongoing, but won't be completed this release.

[paralin]

@liggitt It worked in previous releases - what exactly broke it? Is there some kind of hack I could do to get this working in the meantime?

[liggitt]

API groups were added upstream without supporting code to federate groups from different servers. That work is still ongoing upstream and will be needed to unify a server running with API groups from Kube and from OpenShift

[paralin]

Great, thanks. I'll track upstream on that.