Adding RBAC support in kubernetes config (V2) #694

simon-mo · 2019-05-14T19:09:59Z

This build on top of #605

Enable Kubernetes metric test

AmplabJenkins · 2019-05-14T20:18:35Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1962/
Test FAILed.

rkooo567 · 2019-05-14T20:42:03Z

@simon-mo Tests fails with mxnet. I think the mxnet branch we merged is a little unstable? I will run a test again in case.

rkooo567 · 2019-05-14T20:42:06Z

Jenkins test this please

AmplabJenkins · 2019-05-14T21:21:22Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1964/
Test PASSed.

rkooo567 · 2019-05-14T21:27:44Z

We received this error logs

Traceback (most recent call last):
  File "kubernetes_integration_test.py", line 158, in <module>
    new_name=cluster_name)
  File "/home/travis/build/ucbrise/clipper/integration-tests/test_utils.py", line 189, in create_kubernetes_connection
    cl.start_clipper(num_frontend_replicas=num_frontend_replicas)
  File "/home/travis/build/ucbrise/clipper/clipper_admin/clipper_admin/clipper_admin.py", line 139, in start_clipper
    qf_http_timeout_content, num_frontend_replicas)
  File "/home/travis/build/ucbrise/clipper/clipper_admin/clipper_admin/kubernetes/kubernetes_container_manager.py", line 190, in start_clipper
    self._config_rbac()
  File "/home/travis/build/ucbrise/clipper/clipper_admin/clipper_admin/kubernetes/kubernetes_container_manager.py", line 324, in _config_rbac
    body=clusterrole_data, namespace=self.k8s_namespace)
  File "/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/kubernetes/client/apis/rbac_authorization_v1alpha1_api.py", line 59, in create_cluster_role
    (data) = self.create_cluster_role_with_http_info(body, **kwargs)
  File "/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/kubernetes/client/apis/rbac_authorization_v1alpha1_api.py", line 91, in create_cluster_role_with_http_info
    " to method create_cluster_role" % key
TypeError: Got an unexpected keyword argument 'namespace' to method create_cluster_role

rkooo567 · 2019-05-14T21:29:19Z

clipper_admin/clipper_admin/kubernetes/rbac_service_account.yaml

@@ -0,0 +1,5 @@
+apiVersion: v1


Seems like we should bind our new ServiceAccount to prom_deployment? https://stackoverflow.com/questions/44505461/how-to-configure-a-non-default-serviceaccount-on-a-deployment

@simon-mo So, I understand yaml file is correct, but don't we need to add serviceAccount to our pod -> https://github.com/ucbrise/clipper/blob/develop/clipper_admin/clipper_admin/kubernetes/prom_deployment.yaml?

I found this note from https://github.com/coreos/prometheus-operator/blob/master/Documentation/rbac.md,

Note: A cluster admin is required to create this ClusterRole and create a ClusterRoleBinding or RoleBinding to the ServiceAccount used by the Prometheus Pods. The ServiceAccount used by the Prometheus Pods can be specified in the Prometheus object.

Is it okay not to specify serviceAccount in our metric pod?

You are correct, we need to use this serviceAccount in our prometheus deployment

simon-mo · 2019-05-14T21:37:13Z

There's nothing wrong the yaml file. but we do need to modify the version api call.

AmplabJenkins · 2019-05-14T22:25:34Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1965/
Test PASSed.

rkooo567 · 2019-05-14T22:52:17Z

clipper_admin/clipper_admin/kubernetes/kubernetes_container_manager.py

@@ -163,6 +168,8 @@ def __init__(self,
        configuration.assert_hostname = False
        self._k8s_v1 = client.CoreV1Api()
        self._k8s_beta = client.ExtensionsV1beta1Api()
+        self._k8s_rbac=client.RbacAuthorizationV1beta1Api()


Nothing important, but we need spaces around = for this line.

Thanks! I'll make a lint PR formatting the entire code base.

AmplabJenkins · 2019-05-14T22:55:27Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1966/
Test PASSed.

AmplabJenkins · 2019-05-14T22:58:16Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1967/
Test PASSed.

AmplabJenkins · 2019-05-14T23:06:52Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1968/
Test PASSed.

AmplabJenkins · 2019-05-15T01:16:22Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1970/
Test PASSed.

This reverts commit 50dbf75.

AmplabJenkins · 2019-05-15T02:38:24Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/1971/
Test FAILed.

rkooo567 · 2019-05-15T03:46:04Z

So, #687 merged this PR locally and still has an issue accessing the metric service through kube proxy.

I think we should enable https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/, admission controller's ServiceAccount.

-> This didn't resolve the issue. I am not sure if it is essential.

jacekwachowiak · 2019-05-15T04:06:49Z

Full output from the metrics pod for your information:


[cloud-user@node1 ~]$ kubectl logs metrics-at-default-cluster-5bc5bcb66f-rqb7b
level=info ts=2019-05-15T02:30:54.082508507Z caller=main.go:225 msg="Starting Prometheus" version="(version=2.1.0, branch=HEAD, revision=85f23d82a045d103ea7f3c89a91fba4a93e6367a)"
level=info ts=2019-05-15T02:30:54.082615602Z caller=main.go:226 build_context="(go=go1.9.2, user=root@6e784304d3ff, date=20180119-12:01:23)"
level=info ts=2019-05-15T02:30:54.082635418Z caller=main.go:227 host_details="(Linux 3.10.0-693.el7.x86_64 #1 SMP Thu Jul 6 19:56:57 EDT 2017 x86_64 metrics-at-default-cluster-5bc5bcb66f-rqb7b (none))"
level=info ts=2019-05-15T02:30:54.082649724Z caller=main.go:228 fd_limits="(soft=1048576, hard=1048576)"
level=info ts=2019-05-15T02:30:54.08554498Z caller=main.go:499 msg="Starting TSDB ..."
level=info ts=2019-05-15T02:30:54.086248343Z caller=web.go:383 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2019-05-15T02:30:54.095947666Z caller=main.go:509 msg="TSDB started"
level=info ts=2019-05-15T02:30:54.096015563Z caller=main.go:585 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
level=error ts=2019-05-15T02:30:54.097265516Z caller=manager.go:214 component="discovery manager scrape" msg="Cannot create Kubernetes discovery" err="open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory"
level=info ts=2019-05-15T02:30:54.097327523Z caller=main.go:486 msg="Server is ready to receive web requests."
level=info ts=2019-05-15T02:30:54.097355883Z caller=manager.go:59 component="scrape manager" msg="Starting scrape manager..."

As you can see there is just one error for now. The connection through the browser still fails with:

Error: 'read tcp 10.233.90.0:57818->10.233.96.27:9090: read: connection reset by peer'
Trying to reach: 'http://10.233.96.27:9090/graph'

I'll let you know if I get anything new.

UPDATE:
Changing both of automountServiceAccountToken: false to true in /home/cloud-user/.local/lib/python3.6/site-packages/clipper_admin/kubernetes/prom_deployment.yaml makes the Prometheus pod log error disappear

rkooo567 · 2019-05-21T05:33:32Z

@simon-mo If you are busy, I will debug this tomorrow!

Also, this post #687 can be useful for future debug.

AmplabJenkins · 2019-05-21T09:09:45Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/2010/
Test PASSed.

rkooo567 · 2019-05-28T05:45:59Z

Please refer to this issue. #687

AmplabJenkins · 2019-05-29T09:27:01Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/2038/
Test PASSed.

rkooo567 · 2019-05-29T15:23:05Z

@simon-mo I bound default service account to prometheus cluster role, and it seems like it works fine. I think we can start from this and allow users to change settings later.

simon-mo · 2019-05-29T19:48:18Z

@rkooo567 good work! since I authored this PR. I'll leave you to approve and merge. Thanks!

rkooo567 · 2019-05-29T20:28:39Z

@simon-mo Awesome. I will merge it tonight after cleaning up. Let's release the next version after that.

@withsmilo @RehanSD Can you guys review the PR in them meantime?

AmplabJenkins · 2019-05-30T08:21:24Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/2045/
Test PASSed.

AmplabJenkins · 2019-05-30T10:41:32Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/2046/
Test PASSed.

withsmilo · 2019-05-30T12:33:01Z

clipper_admin/clipper_admin/kubernetes/kubernetes_container_manager.py

@@ -370,6 +377,7 @@ def _start_prometheus(self):
                CONFIG_FILES['metric']['deployment'],
                version=PROM_VERSION,
                cluster_name=self.cluster_name,
+                service_account_name=self.cluster_name+"-prometheus"


I can't find service_account_name in prom_deployment.yaml.

So, I decided to use default as prom_deployment service account because

it works

There's no reason to setup specific service account for users. We can just leave it to users.

I will delete this line.

AmplabJenkins · 2019-05-30T16:36:27Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/2051/
Test PASSed.

AmplabJenkins · 2019-05-31T04:44:47Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Clipper-PRB/2052/
Test PASSed.

rkooo567 · 2019-05-31T05:49:35Z

@simon-mo @withsmilo I will merge if one of you approve it

withsmilo

LGTM!

simon-mo mentioned this pull request May 14, 2019

Adding RBAC support in kubernetes config #605

Closed

rkooo567 mentioned this pull request May 14, 2019

[Metrics] Kubernetes Prometheus metrics address #687

Closed

rkooo567 reviewed May 14, 2019

View reviewed changes

yogesh and others added 3 commits May 14, 2019 14:37

Adding RBAC support in kubernetes config

349cf70

Enable metric test in k8s

2f0c2a5

Pass namespace to yaml templates, update API call

18ebf02

simon-mo force-pushed the k8s/prom-rbac branch from dd20a5f to 18ebf02 Compare May 14, 2019 21:44

simon-mo added 2 commits May 14, 2019 15:10

Use RbacAuthorizationV1beta1Api

d3fea27

Merge branch 'develop' into k8s/prom-rbac

964d522

withsmilo added status: in progress type: enhancement labels May 14, 2019

Fix python k8s api

0f89976

rkooo567 reviewed May 14, 2019

View reviewed changes

simon-mo added 2 commits May 14, 2019 17:43

Pass in svc account to deployment

11b2394

Use latest minikube

50dbf75

Revert "Use latest minikube"

a29ae25

This reverts commit 50dbf75.

Merge branch 'develop' into k8s/prom-rbac

3be80de

simon-mo self-assigned this May 27, 2019

rkooo567 added this to backlog in Tasks May 28, 2019

rkooo567 removed this from backlog in Tasks May 28, 2019

rkooo567 mentioned this pull request May 29, 2019

Monitoring is not working on google cloud #564

Closed

use default service account for prom

c0d1c8b

rkooo567 added 3 commits May 29, 2019 22:59

Merge branch 'develop' into k8s/prom-rbac

6c98955

clean up

b50a88e

Deleting cluster role instead of namespaced role

36d1740

withsmilo self-requested a review May 30, 2019 10:47

withsmilo reviewed May 30, 2019

View reviewed changes

removed service_account_name from start_prom

fd27d40

Merge branch 'develop' into k8s/prom-rbac

b7d675a

withsmilo self-requested a review May 31, 2019 05:51

withsmilo approved these changes May 31, 2019

View reviewed changes

rkooo567 merged commit 2599acd into ucbrise:develop May 31, 2019

withsmilo added status: accepted and removed status: in progress labels May 31, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding RBAC support in kubernetes config (V2) #694

Adding RBAC support in kubernetes config (V2) #694

simon-mo commented May 14, 2019

AmplabJenkins commented May 14, 2019

rkooo567 commented May 14, 2019

rkooo567 commented May 14, 2019

AmplabJenkins commented May 14, 2019

rkooo567 commented May 14, 2019

rkooo567 May 14, 2019

rkooo567 May 14, 2019

simon-mo May 14, 2019

simon-mo commented May 14, 2019

AmplabJenkins commented May 14, 2019

rkooo567 May 14, 2019 •

edited

simon-mo May 14, 2019

AmplabJenkins commented May 14, 2019

AmplabJenkins commented May 14, 2019

AmplabJenkins commented May 14, 2019

AmplabJenkins commented May 15, 2019

AmplabJenkins commented May 15, 2019

rkooo567 commented May 15, 2019 •

edited

jacekwachowiak commented May 15, 2019 •

edited

rkooo567 commented May 21, 2019 •

edited

AmplabJenkins commented May 21, 2019

rkooo567 commented May 28, 2019

AmplabJenkins commented May 29, 2019

rkooo567 commented May 29, 2019

simon-mo commented May 29, 2019

rkooo567 commented May 29, 2019

AmplabJenkins commented May 30, 2019

AmplabJenkins commented May 30, 2019

withsmilo May 30, 2019

rkooo567 May 30, 2019 •

edited

AmplabJenkins commented May 30, 2019

AmplabJenkins commented May 31, 2019

rkooo567 commented May 31, 2019

withsmilo left a comment

Adding RBAC support in kubernetes config (V2) #694

Adding RBAC support in kubernetes config (V2) #694

Conversation

simon-mo commented May 14, 2019

AmplabJenkins commented May 14, 2019

rkooo567 commented May 14, 2019

rkooo567 commented May 14, 2019

AmplabJenkins commented May 14, 2019

rkooo567 commented May 14, 2019

rkooo567 May 14, 2019

Choose a reason for hiding this comment

rkooo567 May 14, 2019

Choose a reason for hiding this comment

simon-mo May 14, 2019

Choose a reason for hiding this comment

simon-mo commented May 14, 2019

AmplabJenkins commented May 14, 2019

rkooo567 May 14, 2019 • edited

Choose a reason for hiding this comment

simon-mo May 14, 2019

Choose a reason for hiding this comment

AmplabJenkins commented May 14, 2019

AmplabJenkins commented May 14, 2019

AmplabJenkins commented May 14, 2019

AmplabJenkins commented May 15, 2019

AmplabJenkins commented May 15, 2019

rkooo567 commented May 15, 2019 • edited

jacekwachowiak commented May 15, 2019 • edited

rkooo567 commented May 21, 2019 • edited

AmplabJenkins commented May 21, 2019

rkooo567 commented May 28, 2019

AmplabJenkins commented May 29, 2019

rkooo567 commented May 29, 2019

simon-mo commented May 29, 2019

rkooo567 commented May 29, 2019

AmplabJenkins commented May 30, 2019

AmplabJenkins commented May 30, 2019

withsmilo May 30, 2019

Choose a reason for hiding this comment

rkooo567 May 30, 2019 • edited

Choose a reason for hiding this comment

AmplabJenkins commented May 30, 2019

AmplabJenkins commented May 31, 2019

rkooo567 commented May 31, 2019

withsmilo left a comment

Choose a reason for hiding this comment

rkooo567 May 14, 2019 •

edited

rkooo567 commented May 15, 2019 •

edited

jacekwachowiak commented May 15, 2019 •

edited

rkooo567 commented May 21, 2019 •

edited

rkooo567 May 30, 2019 •

edited