Make every workspace containers and init containers configurable

[l0rd]

Is your enhancement related to a problem? Please describe

The spec of these workspace pod containers is not configurable:

• gateway container
• project-clone container

This can be annoying if we want to remove the CPULimit of these containers for example.

Describe the solution you'd like

Add the fields gatewayContainer and projectCloneContainer under CheCluster devEnvironments:

spec:
  devEnvironments:
    gatewayContainer:
        (...container settings...)
    projectCloneContainer:
        (...container settings...)

• The container API is already defined in the CheCluster API.

• Setting a container field empty should result in the property not being set. For example, to unset the gateway CPU limit (and keep the other container properties):

  spec:
    devEnvironments:
      gatewayContainer:
        resources:
          limits:
            cpu: '0'

Additional context

• Downstream issue

[l0rd]

The gateway changes should be Team A (as that's configured by Che operator), the projectClone changes should be Team B (as that's configured by DW operator).

[l0rd]

@mkuznyetsov working on this issue, gateway side, may be a good task for DevTools Week (cc @ibuziuk @tolusha)

[tolusha]

@l0rd

Setting a container field empty should result in the property not being set

What is the reason behind this? It is recommended to set some limits into a container.
I suggest to set defaults if property is not set in CheCluster CR.

[nickboldt]

PLEASE don't make this another change in CheCluster we have to then remove like in https://issues.redhat.com/browse/CRW-4077

Let's keep those out of the CR so that updates between versions don't keep breaking as we add/remove/change defaults in the CheCluster.

[l0rd]

What is the reason behind this? It is recommended to set some limits into a container. I suggest to set defaults if property is not set in CheCluster CR.

We want to keep the default as it is today. But we want to allow admins to specify that the workspace containers will have no limits/requests. This is considered a good practice in some cases.

[tolusha]

We want to keep the default as it is today. But we want to allow admins to specify that the workspace containers will have no limits/requests. This is considered a good practice in some cases.

I suggest doing this in the following way:

1. if limits/requests are set in CheCluster CR, then use it
2. if they are not set (nil), then use defaults
3. if they are set to "0" (empty can't be used), then don't set limits/requests

[mkuznyetsov]

gatewayContainer related PR is merged, associated Docs PR will be merged by the docs team

[l0rd]

To close this issue we are waiting for devfile/devworkspace-operator#1080 that should be addressed in DWO 0.21.0. After that we need a correspondent issue/PR on che-operator side.

[cgruver]

@l0rd

Setting a container field empty should result in the property not being set

What is the reason behind this? It is recommended to set some limits into a container. I suggest to set defaults if property is not set in CheCluster CR.

@tolusha Setting CPU Limits is not always a good practice. We crashed into the the default CPU limit of 500m recently while trying to debug why a workspace pod did not seem to be able to burst much CPU.

We also discovered that if a devfile component has a request of 500m (or greater), but no limit set, then the workspace fails to create.

IMO, we should allow for the Che admins to turn off the auto-injection of a CPU Limit.

Here's a really well written explaination for not setting CPU limits: https://home.robusta.dev/blog/stop-using-cpu-limits