Rancher/ONAP collaboration: increase --max-pods kublet config via rancher agent config - via automated docker param

[obriensystems]

Rancher versions:
rancher/server or rancher/rancher:
1.6.14
rancher/agent or rancher/rancher-agent:

Infrastructure Stack versions:
healthcheck:
ipsec:
network-services:
scheduler:
kubernetes (if applicable):
1.8.10

Docker version: (docker version,docker info preferred)
17.03
Operating system and kernel: (cat /etc/os-release, uname -r preferred)
ubuntu 16.04
Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)
any (openstack, aws, azure)
Setup details: (single node rancher vs. HA rancher, internal DB vs. external DB)
single node
https://git.onap.org/logging-analytics/tree/deploy/rancher/oom_rancher_setup.sh

Environment Template: (Cattle/Kubernetes/Swarm/Mesos)
kubernetes
Steps to Reproduce:
attempt to deploy more than 110 pods
Results:
init state on remaining pods

see details - we need an automated way to pass in the max-pods parameter so we can retrofit the rancher installation script to handle a single node deployment of ONAP (currently over 200 pods)
https://jira.onap.org/browse/OOM-1138

Note: this is after a discussion with your C* suite executives on 20180612 on increasing onap/rancher collaboration
https://jira.onap.org/browse/OOM-1138
We hit the 110 kubernetes default pod limit a couple months ago - although we recommend clustering we should have an option to run a cluster node with more than 110 pods - we currently have 210 - when we reach 300+ on a 3 node system we will need this.

Work with Rancher labs to enable the --max-pods kubelet config for when we reach 300+ per 3 node cluster

The setting is on the kublet - I spend about 10 min researching how to add this as a config option to the rancher-agent docker - but could not find a way to pass in the parameter
https://lists.onap.org/pipermail/onap-discuss/2018-June/010246.html
kubernetes/kubernetes#23349

[obriensystems]

https://lists.onap.org/pipermail/onap-discuss/2018-June/010282.html

[obriensystems]

{noformat}
kubectl describe node

Capacity:
cpu: 16
memory: 125827328Ki
pods: 110
Allocatable:
cpu: 16
memory: 125724928Ki
pods: 110

{noformat}

[superseb]

Is it purely automating the setup of an environment? You can use the following command to create a environment template with the additional kubelet arg --max-pods=200. This is a captured API call when you configure it in the UI, so if you need more modifications, that's the way to do it. After, you can create an environment based on this environment template and add nodes.

curl 'http://your_rancher_server:8080/v2-beta/projecttemplate' --data-binary '{"type":"projectTemplate","baseType":"projectTemplate","name":"k8smaxpodstemplate","state":"active","accountId":null,"created":"2018-06-13T19:03:59Z","createdTS":1528916639000,"data":{"fields":{"stacks":[{"name":"network-services","templateId":"library:infra*network-services"},{"name":"ipsec","templateId":"library:infra*ipsec"},{"name":"scheduler","templateId":"library:infra*scheduler"},{"name":"healthcheck","templateId":"library:infra*healthcheck"}]}},"description":null,"externalId":"catalog://library:project*cattle:0","isPublic":false,"kind":"projectTemplate","removeTime":null,"removed":null,"stacks":[{"type":"catalogTemplate","name":"healthcheck","templateId":"library:infra*healthcheck"},{"type":"catalogTemplate","name":"kubernetes","answers":{"CONSTRAINT_TYPE":"none","CLOUD_PROVIDER":"rancher","AZURE_CLOUD":"AzurePublicCloud","AZURE_TENANT_ID":"","AZURE_CLIENT_ID":"","AZURE_CLIENT_SECRET":"","AZURE_SEC_GROUP":"","RBAC":false,"REGISTRY":"","BASE_IMAGE_NAMESPACE":"","POD_INFRA_CONTAINER_IMAGE":"rancher/pause-amd64:3.0","HTTP_PROXY":"","NO_PROXY":"rancher.internal,cluster.local,rancher-metadata,rancher-kubernetes-auth,kubernetes,169.254.169.254,169.254.169.250,10.42.0.0/16,10.43.0.0/16","ENABLE_ADDONS":true,"ENABLE_RANCHER_INGRESS_CONTROLLER":true,"RANCHER_LB_SEPARATOR":"rancherlb","DNS_REPLICAS":"1","ADDITIONAL_KUBELET_FLAGS":"--max-pods=200","FAIL_ON_SWAP":"false","ADDONS_LOG_VERBOSITY_LEVEL":"2","AUDIT_LOGS":false,"ADMISSION_CONTROLLERS":"NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,DefaultTolerationSeconds,ResourceQuota","SERVICE_CLUSTER_CIDR":"10.43.0.0/16","DNS_CLUSTER_IP":"10.43.0.10","KUBEAPI_CLUSTER_IP":"10.43.0.1","KUBERNETES_CIPHER_SUITES":"TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305","DASHBOARD_CPU_LIMIT":"100m","DASHBOARD_MEMORY_LIMIT":"300Mi","INFLUXDB_HOST_PATH":"","EMBEDDED_BACKUPS":true,"BACKUP_PERIOD":"15m0s","BACKUP_RETENTION":"24h","ETCD_HEARTBEAT_INTERVAL":"500","ETCD_ELECTION_TIMEOUT":"5000"},"templateId":"library:infra*k8s"},{"type":"catalogTemplate","name":"network-services","templateId":"library:infra*network-services"},{"type":"catalogTemplate","name":"ipsec","templateId":"library:infra*ipsec"}],"transitioning":"no","transitioningMessage":null,"transitioningProgress":null,"uuid":null}' --compressed

> kubectl get node -o json | jq '.items[].status | .allocatable,.capacity'
{
  "cpu": "2",
  "ephemeral-storage": "74761043435",
  "hugepages-1Gi": "0",
  "hugepages-2Mi": "0",
  "memory": "3944016Ki",
  "pods": "200"
}
{
  "cpu": "2",
  "ephemeral-storage": "81120924Ki",
  "hugepages-1Gi": "0",
  "hugepages-2Mi": "0",
  "memory": "4046416Ki",
  "pods": "200"
}

[superseb]

Anything else needed on this?

[obriensystems]

(+) procedure
https://wiki.onap.org/display/DW/Cloud+Native+Deployment#CloudNativeDeployment-Changemax-podsfromdefault110podlimit
Manual procedure: change the kubernetes template (1pt2) before using it to create an environment (1a7)

add --max-pods=500 to the "Additional Kubelet Flags" box on the v1.10.13 version of the kubernetes template from the "Manage Environments" dropdown on the left of the 8880 rancher console.