Not able to reinstall the gloo gateway(enterprise version) on openshift 4.2

[arunprasadlv]

I was running gloo enterprise 0.18.31 version and I am trying to upgrade to 0.20.x version.
Instead of upgrading I was suggested to uninstall and reinstall the gloo.

Uninstall was successful but i am seeing getting following error during the install -

C:\Arun\cmder
λ oc version
oc v3.9.33
kubernetes v1.9.1+a0ce1bc657
features: Basic-Auth

Server https://api.ocpdemo.tent.trt.csaa.pri:6443
kubernetes v1.14.6+b136e8b

C:\Arun\cmder
λ glooctl install gateway
Starting Gloo installation...
Installing CRDs...

Gloo failed to install! Detailed logs available at C:\Users\gz2lodh.gloo\debug.log.
Error: installing gloo in gateway mode: running kubectl apply on manifest: exit status 1

debug.log

NOTE: I have a admin permission at OC Project namespace level.I have attached the glooctl clusterrole definitions.

[arunprasadlv]

clusterrole-glooctl-clusterrole.txt

[marcogschmidt]

Hi @arunprasadlv, in the error log I see two types of errors:

• Some SchemaErrors
• Errors related to permissions (RBAC)

The Schema errors should be related to the version of the oc client you are using. The client is using the v1.9.1 version of the kubernetes client, while the server is running kuberentes 1.14. Could you upgrade oc to a more recent version (ideally 1.14.x) and see if the error still shows up?

The other errors (no RBAC policy matched) are caused by your user not being associated with the required RBAC Roles/ClusterRoles. I have put together an updated set of permissions for your user as well as some installation instructions and will share them with you shortly.

While investigating I also stumbled upon a bug that will prevent you from installing Gloo unless your user has the permission to create/update CRDs. To be able to get it to work you will need these two additional lines:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: glooctl-clusterrole
rules:
...
- verbs:
  - get
  - create # currently needed because of the bug
  - update # currently needed because of the bug
  apiGroups:
  - apiextensions.k8s.io
  resources:
  - customresourcedefinitions
...

Could you check if you would be able to get these permissions for your user? If not, you will need to wait for the next Gloo release, which will be published by the end of the week (probably sooner).

I'll post the install instructions soon.

[marcogschmidt]

Hi @arunprasadlv, as mentioned in my previous comment, here are the actions you need to perform to get Gloo Enterprise installed in your OpenShift project.

Update glooctl-clusterrole
Ask your admin to grant you the following set of privileges (see comments in the YAML for an explanation of the additions to your current ones):

glooctl-clusterrole

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: glooctl-clusterrole
rules:
- verbs:
  - get
  - list
  - watch
  apiGroups:
  - extensions
  - apps
  resources:
  - deployments
  - podsecuritypolicies
- verbs:
  - get
  - list
  - watch
  - create
  apiGroups:
  - ''
  resources:
  - pods
  - services
  - configmaps
  - namespaces
  - secrets
- verbs:
  - '*'
  apiGroups:
  - ''
  resources:
  - pods/portforward
  - pods/log
- verbs:
  - get
  - create # currently needed because of this bug: https://github.com/solo-io/gloo/issues/1491
  - update # currently needed because of this bug: https://github.com/solo-io/gloo/issues/1491
  apiGroups:
  - apiextensions.k8s.io
  resources:
  - customresourcedefinitions
- verbs:
  - '*' # Allow all actions on these Gloo resources
  apiGroups:
  - gloo.solo.io
  - enterprise.gloo.solo.io
  resources:
  - artifacts
  - upstreams
  - upstreamgroups
  - proxies
  - secrets
  - settings
  - authconfigs
- verbs:
  - '*' # Allow all actions on these Gloo resources
  apiGroups:
  - gateway.solo.io.v2
  resources:
  - gateways
- verbs:
  - '*' # Allow all actions on these Gloo resources
  apiGroups:
  - gateway.solo.io
  resources:
  - virtualservices
  - routetables
  - gateways
- verbs:
  - create
  - update
  - delete
  apiGroups:
  - ''
  resources:
  - secrets
- verbs:
  - get
  apiGroups:
  - ''
  resources:
  - nodes # You will need this to use some handy commands like 'glooctl proxy url'
- verbs:
  - get
  - create
  apiGroups:
  - ''
  resources:
  - serviceaccounts
- verbs:
  - get
  - create
  apiGroups:
  - 'rbac.authorization.k8s.io'
  resources:
  - roles # You will need this to create namespace-scoped RBAC resources
  - rolebindings

Create Gloo CRDs
As I understand, you already have the Gloo CRDs created. If this is the case you can skip this step. If not, ask your admin to create them for you. You can easily get the CRDs by running:

glooctl install gateway enterprise --dry-run > gloo-manifest.yaml

This writes the Gloo installation YAML manifest to a file. You can then just select the CustomResourceDefinitions and ask the admin to create them.

Create a Helm value file
Save the following YAML to a file (I'll assume it's named values-enterprise.yaml in the next steps).

values-enterprise.yaml

# These are values for the 'gloo' sub-chart (which is a dependency for the Gloo Enterprise chart)
gloo:
  crds:
    create: true
  namespace:
    # Do not create the installation namespace/project
    create: false
  gateway:
    # Turn off the gateway validating admission webhook,
    # as this currently requires creating cluster scoped RBAC resources
    validation: null
    deployment:
      # Allow the cluster to dynamically assign a user ID
      floatingUserId: true
  gatewayProxies:
    gatewayProxyV2:
      podTemplate:
        # Don't add the NET_BIND_SERVICE capability to the pod. This means that
        # the gateway proxy will not be able to bind to ports below 1024 (needed for OpenShift)
        disableNetBind: true
        # Allow the cluster to dynamically assign a user ID (needed for OpenShift)
        floatingUserId: true
  gloo:
    deployment:
      # Allow the cluster to dynamically assign a user ID (needed for OpenShift)
      floatingUserId: true
  discovery:
    deployment:
      # Allow the cluster to dynamically assign a user ID (needed for OpenShift)
      floatingUserId: true
global:
  glooRbac:
    # Create the RBAC resources
    create: true
    # Create Roles/RoleBindings instead of ClusterRoles/CLusterRoleBindings
    namespaced: true
settings:
  create: true
  # Enable to use install namespace as WatchNamespace and WriteNamespace
  singleNamespace: true
prometheus:
  # Don't install prometheus (this requires the creation of cluster scoped RBAC resources)
  enabled: false
grafana:
  # Don't install grafana (this requires the creation of cluster scoped RBAC resources)
  defaultInstallationEnabled: false

These values will be used by glooctl to customize your installation, mainly so that you do not need to create cluster-scoped RBAC resource. See the comments for an explanation of what each value does.

Install Gloo Enterprise
Once you are logged in to OpenShift with our user, switch to the project you want to install Gloo into (for the purpose of this guide I'll assume it's my-project) and run:

glooctl install gateway enterprise --namespace my-project --values values-enterprise.yaml --license-key $GLOO_LICENSE_KEY

where GLOO_LICENSE_KEY is your trial license key.

Now if you run kubectl get pods -n my-project you should see that the Gloo Enterprise pods are running. I just ran all the above steps myself, this is what the output should look like:

Please let me know if this works for you.

[marcogschmidt]

Closing due to inactivity. Can reopen if issue is still relevant.