CPU/Memory Limits and Resources are not being set in namespaces

[jamiebuxxx]

Hello,

While doing some finalizing of our Rancher2 terraform stack, we discovered the CPU/Memory resources and limits are not being set in the namespaces. Here's an example of namespace terraform resource.

resource "rancher2_namespace" "namespace" {
  name = "${var.namespace}"
  project_id = "${var.project_id}"
  resource_quota {
    limit {
      requests_cpu = "500m"
      requests_memory = "2048Mi"
      limits_cpu = "${var.cpu_limit}"
      limits_memory = "${var.memory_limit}"
      requests_storage = "${var.storage_request}"
    }
  }
}

When we then describe the namespace we see that previous values are only store under the field.cattle.io/resourceQuota annotations, but not under the Resources Limits or Resources Quota

kubectl describe ns eng-bro
Name:         eng-bro
Labels:       cattle.io/creator=norman
              field.cattle.io/projectId=p-zxsmg
Annotations:  cattle.io/status:
                {"Conditions":[{"Type":"InitialRolesPopulated","Status":"True","Message":"","LastUpdateTime":"2019-08-09T20:51:47Z"},{"Type":"ResourceQuot...
              field.cattle.io/creatorId: user-6jgnv
              field.cattle.io/description:
              field.cattle.io/projectId: c-fxthd:p-zxsmg
              field.cattle.io/resourceQuota:
                {"limit":{"limitsCpu":"2000m","limitsMemory":"2048Mi","requestsCpu":"500m","requestsMemory":"2048Mi","requestsStorage":"250Gi"}}
              lifecycle.cattle.io/create.namespace-auth: true
Status:       Active

No resource quota.

No resource limits.

Also, these changes are not show up in the Rancher2 UI either.

I can confirm that setting the resource quotas and limits using the normal kubectl derivatives works as expected.

[rawmind0]

@jamiebuxxx , the rancher project p-zxsmg where the namespace is assined has CPU/Memory
resources and limits defined??

[jamiebuxxx]

No, the projects do not have CPU/Memory resources and limits defined. And I don't see anything assigned in the GUI. Those are only defined in the namespace using the Terraform plugin.

This would be the preferred method as different namespaces can have different resource needs but still be under the same project.

[rawmind0]

Project CPU/Memory resources and limits should be defined before you can define CPU/Memory resources and limits for namespaces assigned to that project. Once project CPU/Memory resources and limits are defined you'll see namespace CPU/Memory resources and limits on the rancher UI.

Something lik that,

resource "rancher2_project" "foo" {
  name = "foo"
  cluster_id = "<clusted_id>"
  resource_quota {
    project_limit {
      limits_cpu = "2000m"
      limits_memory = "2000Mi"
      requests_storage = "2Gi"
    }
    namespace_default_limit {
      limits_cpu = "500m"
      limits_memory = "500Mi"
      requests_storage = "1Gi"
    }
  }
}

resource "rancher2_namespace" "foo" {
  name = "foo"
  project_id = "${rancher2_project.foo.id}"
  resource_quota {
    limit {
      limits_cpu = "100m"
      limits_memory = "100Mi"
      requests_storage = "1Gi"
    }
  }
}

[jamiebuxxx]

Interesting. I remember discussing this issue before. Cause whatever you allocate to the project resource/limits, then the namespace can not allocate over that value. I can see the logic of this concept, but this won't be ideal for our setup.

It will be difficult for us to allocate resources/limits to a project without know how many namespaces will be under a project. This could cause a newly created namespaces to not have limits because the project budget ran out. We saw this a while back with initial Rancher 2 testing.

So then this leads me to another question, why can set the resource/limits of a namespace under a project from the Web UI, but not from this provider? Considering that its extending the Rancher2 API

kubectl describe ns jenkins-sandbox
Name:         jenkins-sandbox
Labels:       cattle.io/creator=norman
              field.cattle.io/projectId=p-h64wj
Annotations:  cattle.io/status:
                {"Conditions":[{"Type":"InitialRolesPopulated","Status":"True","Message":"","LastUpdateTime":"2019-07-30T22:21:50Z"},{"Type":"ResourceQuot...
              field.cattle.io/containerDefaultResourceLimit: {"limitsCpu":"2000m","requestsCpu":"500m","requestsMemory":"256Mi"}
              field.cattle.io/creatorId: u-j46t4zp7gm
              field.cattle.io/projectId: c-8skfr:p-h64wj
              lifecycle.cattle.io/create.namespace-auth: true
Status:       Active

No resource quota.

Resource Limits
 Type       Resource  Min  Max  Default Request  Default Limit  Max Limit/Request Ratio
 ----       --------  ---  ---  ---------------  -------------  -----------------------
 Container  cpu       -    -    500m             2              -
 Container  memory    -    -    256Mi            -              -

[rawmind0]

Interesting. I remember discussing this issue before. Cause whatever you allocate to the project resource/limits, then the namespace can not allocate over that value. I can see the logic of this concept, but this won't be ideal for our setup.

A rancher project is a group of namespaces, then no namespace within a project can't allocate more that project resource/limits.

It will be difficult for us to allocate resources/limits to a project without know how many namespaces will be under a project. This could cause a newly created namespaces to not have limits because the project budget ran out. We saw this a while back with initial Rancher 2 testing.

I got your point, but you'd have same problem on k8s cluster sizing. If you set resource limits on your namespace (good practice), this could cause that newly created namespaces could not get them because k8s cluster don't have enough free resources. I guess that at some point you are going to scale the k8s cluster, you could do the same with the project resource/limits. When you set project resouce/limits and default namespaces resouce/limits, you have an estimation of how many namespaces may fit on it.

So then this leads me to another question, why can set the resource/limits of a namespace under a project from the Web UI, but not from this provider? Considering that its extending the Rancher2 API

You can't set namespace resource/limits from the UI as you can't from this provider. Your screenshot is showing other feature container default resource limit. These limits are applied to containers within the namespace and are managed by a distinct argument, namespace container_resource_limit

[jamiebuxxx]

Ok. Going to let this soak for bit. I would like to avoid breaking away to far from Kubernetes specific derivatives like request, limits, max/min limits. Adding the project resource element is how to think about our clusters.

I think we will need to come up with use cases to try and different ways to handle namespaces and projects for our uses.

[rawmind0]

Yeah, it's depending of the offering use case, clusters or projects in shared clusters.

Please, reopen issue if needed.