Added eksengine to engines/init.py #315

mikezuniga · 2018-06-06T15:40:24Z

Added boto3 to setup.py requirements
Added eks engine
Added eks kubeconfig template
Docs added
TODO added

current issue related with kqueen-k8s-api methods:

api_1           | HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'Date': 'Sun, 10 Jun 2018 10:12:10 GMT', 'Content-Length': '234'})
api_1           | HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"services is forbidden: User \"system:anonymous\" cannot list services at the cluster scope","reason":"Forbidden","details":{"kind":"services"},"code":403}
api_1           |
api_1           | Traceback (most recent call last):
api_1           |   File "/code/kqueen/models.py", line 124, in status
api_1           |     'addons': kubernetes.list_services(filter_addons=True),
api_1           |   File "/code/kqueen/kubeapi.py", line 238, in list_services
api_1           |     include_uninitialized=include_uninitialized
api_1           |   File "/usr/local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 14358, in list_service_for_all_namespaces
api_1           |     (data) = self.list_service_for_all_namespaces_with_http_info(**kwargs)
api_1           |   File "/usr/local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 14455, in list_service_for_all_namespaces_with_http_info
api_1           |     collection_formats=collection_formats)
api_1           |   File "/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 321, in call_api
api_1           |     _return_http_data_only, collection_formats, _preload_content, _request_timeout)
api_1           |   File "/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 155, in __call_api
api_1           |     _request_timeout=_request_timeout)
api_1           |   File "/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 342, in request
api_1           |     headers=headers)
api_1           |   File "/usr/local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 231, in GET
api_1           |     query_params=query_params)
api_1           |   File "/usr/local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 222, in request
api_1           |     raise ApiException(http_resp=r)
api_1           | kubernetes.client.rest.ApiException: (403)

faced when try to parse k8s config body

Added boto3 to setup.py requirements Added eks engine Added eks kubeconfig template

ekhomyakova · 2018-06-06T15:46:15Z

kqueen/engines/eks.py

+    'ACTIVE': config.get('CLUSTER_OK_STATE'),
+    'DELETING': config.get('CLUSTER_DEPROVISIONING_STATE'),
+    'FAILED': config.get('CLUSTER_ERROR_STATE'),
+    'UPDATED': config.get('CLUSTER_UPDATING_STATE')


'UPDATING' ?

@ekhomyakova I suppose those names comes from EKS, but I don't see updating in the list at all, so may be we can just remove updated from the list

for what i know the engine requires an updating state, so either we leave it or we need to put it to unknown

ekhomyakova · 2018-06-06T15:49:10Z

kqueen/engines/eks.py

+                    'number': True
+                }
+            },
+            'roleArn': {


It's better to use snake_case everywhere

ekhomyakova · 2018-06-06T15:49:26Z

kqueen/engines/eks.py

+                    'required': True
+                }
+            },
+            'subnetid': {


subnet_id

ekhomyakova · 2018-06-06T15:49:49Z

kqueen/engines/eks.py

+                    'required': True
+                }
+            },
+            'securitygroupid': {


security_group_id or securitygroup_id

katyafervent · 2018-06-06T16:20:31Z

kqueen/engines/eks.py

+        except Exception as e:
+            msg = 'Creating cluster {} failed with the following reason:'.format(self.cluster.id)
+            logger.exception(msg)
+            return False, msg


Please, return only "e" instead of msg, since message will be huge and looks like:

Error occurred with the following reason: Creating cluster failed with the following reason:'

katyafervent · 2018-06-06T16:21:49Z

kqueen/engines/eks.py

+    def engine_status(cls, **kwargs):
+        try:
+            aws_access_key = kwargs.get('aws_access_key', '')
+            aws_secret_key = kwargs.get('aws_secret_key', '')


did you mean 'aws_secret_access_key'?

katyafervent · 2018-06-06T16:26:48Z

kqueen/engines/eks.py

+    'ACTIVE': config.get('CLUSTER_OK_STATE'),
+    'DELETING': config.get('CLUSTER_DEPROVISIONING_STATE'),
+    'FAILED': config.get('CLUSTER_ERROR_STATE'),
+    'UPDATED': config.get('CLUSTER_UPDATING_STATE')


@ekhomyakova I suppose those names comes from EKS, but I don't see updating in the list at all, so may be we can just remove updated from the list

katyafervent · 2018-06-06T16:27:18Z

kqueen/engines/eks.py

+        """
+        Implementation of :func:`~kqueen.engines.base.BaseEngine.provision`
+        """
+        # self.name = kwargs.get('name', 'noname')


katyafervent · 2018-06-06T16:28:06Z

kqueen/engines/eks.py

+            return {}
+        state = STATE_MAP.get(response['cluster']['status'], config.get('CLUSTER_UNKNOWN_STATE'))
+
+        key = 'cluster-{}-{}'.format(response['cluster']['name'], response['cluster']['name'])


why names are used two times?

one of them should be the id, but EKS uses the name as id at the moment, the only other option is to use the ARN which is extremely long arn:aws:eks:us-west-2:012345678910:cluster/prod

katyafervent · 2018-06-06T16:29:34Z

kqueen/engines/eks.py

+        try:
+            response = self.client.describe_cluster(self.cluster.metadata['id'])
+        except Exception as e:
+            msg = 'Fetching data from backend for cluster {} failed with the following reason:'.format(self.cluster.metadata['heat_cluster_id'])


self.cluster.metadata['id']

katyafervent · 2018-06-06T17:11:26Z

kqueen/engines/eks.py

+            },
+            'subnetid': {
+                'type': 'text',
+                'label': 'Subnet Id',


Subnet/s Id - since more than one can be chosen. Ideally we need to get list of them and provide ability to pick subnets on the fly, but for now we need to parse string to the list at least (on the UI part)

katyafervent · 2018-06-06T17:54:59Z

kqueen/engines/eks.py

+            aws_secret_access_key=aws_secret_key
+        )
+        try:
+            list(client.list_clusters())


can we used more lightweight request here? (since this request is sending for each cluster, even for list all clusters operation)

at least add maxResults parameter=1

or use another request maybe generate_presigned_url?

seems, No , coz afaik only list_clusters can provide valid check on all required User IAM roles/policies and etc.. I understand that its huge request, but we are locked by small api

- Template structure defined - Kubeconfig download fixed - Cluster states fixed - Engine status fixed - TODO added current issue related with kqueen-k8s-api methods: ``` api_1 | Traceback (most recent call last): api_1 | File "/code/kqueen/models.py", line 124, in status api_1 | 'addons': kubernetes.list_services(filter_addons=True), api_1 | File "/code/kqueen/kubeapi.py", line 238, in list_services api_1 | include_uninitialized=include_uninitialized api_1 | File "/usr/local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 14358, in list_service_for_all_namespaces api_1 | (data) = self.list_service_for_all_namespaces_with_http_info(**kwargs) api_1 | File "/usr/local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 14455, in list_service_for_all_namespaces_with_http_info api_1 | collection_formats=collection_formats) api_1 | File "/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 321, in call_api api_1 | _return_http_data_only, collection_formats, _preload_content, _request_timeout) api_1 | File "/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 155, in __call_api api_1 | _request_timeout=_request_timeout) api_1 | File "/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 342, in request api_1 | headers=headers) api_1 | File "/usr/local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 231, in GET api_1 | query_params=query_params) api_1 | File "/usr/local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 222, in request api_1 | raise ApiException(http_resp=r) api_1 | kubernetes.client.rest.ApiException: (403) ``` faced when try to parse k8s config body

Refactor EKS

- Typo fixes in EKS engine

Docs added

katyafervent

Please, add new pipfile.lock

katyafervent · 2018-06-13T12:55:50Z

kqueen/engines/eks.py

+        },
+        'cluster': {
+            'node_count': {
+            # TODO unsupported, need to define template k8s input


I suggest to comment node_count then

katyafervent · 2018-06-13T12:57:31Z

kqueen/engines/resources/aws/template.yaml

+        - "-i"
+        - "<cluster-name>"
+        # - "-r"
+        # - "<role-arn>"


is it suppose to be commented?

for now yes

Heptio Auth configuration

naumvd95 · 2018-07-02T05:44:08Z

related issue kubernetes-client/python#514

naumvd95

lets say I'm ok with that, but in fact I just want to hide this PR from my notifications tab :)

naumvd95

lets say I'm ok with that, but in fact I just want to hide this PR from my notifications tab :)

Added eksengine to engines/__init__.py

f16ed10

Added boto3 to setup.py requirements Added eks engine Added eks kubeconfig template

mikezuniga requested review from naumvd95 and katyafervent June 6, 2018 15:40

ekhomyakova reviewed Jun 6, 2018

View reviewed changes

katyafervent reviewed Jun 6, 2018

View reviewed changes

naumvd95 and others added 4 commits June 10, 2018 11:35

Merge pull request #317 from Mirantis/eks-refactor

391f91c

Refactor EKS

Docs added

625c540

- Typo fixes in EKS engine

Merge pull request #318 from Mirantis/kqueen-eks

b016a1d

Docs added

naumvd95 requested review from katyafervent and ekhomyakova June 13, 2018 06:42

katyafervent reviewed Jun 13, 2018

View reviewed changes

naumvd95 and others added 2 commits June 20, 2018 15:40

Heptio Auth configuration

6144bcf

Merge pull request #321 from Mirantis/kqueen-eks

0c0df12

Heptio Auth configuration

naumvd95 added the TBD label Jul 2, 2018

naumvd95 approved these changes May 23, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added eksengine to engines/init.py #315

Added eksengine to engines/init.py #315

mikezuniga commented Jun 6, 2018 •

edited by naumvd95

Loading

ekhomyakova Jun 6, 2018

katyafervent Jun 6, 2018

mikezuniga Jun 6, 2018

ekhomyakova Jun 6, 2018

ekhomyakova Jun 6, 2018

ekhomyakova Jun 6, 2018

katyafervent Jun 6, 2018

katyafervent Jun 6, 2018

katyafervent Jun 6, 2018

katyafervent Jun 6, 2018

katyafervent Jun 6, 2018

mikezuniga Jun 6, 2018

katyafervent Jun 6, 2018

katyafervent Jun 6, 2018

katyafervent Jun 6, 2018

naumvd95 Jun 10, 2018

katyafervent left a comment

katyafervent Jun 13, 2018

katyafervent Jun 13, 2018

naumvd95 Jun 13, 2018

naumvd95 commented Jul 2, 2018

naumvd95 left a comment

naumvd95 left a comment

Added eksengine to engines/__init__.py #315

Are you sure you want to change the base?

Added eksengine to engines/__init__.py #315

Conversation

mikezuniga commented Jun 6, 2018 • edited by naumvd95 Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

katyafervent left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

naumvd95 commented Jul 2, 2018

naumvd95 left a comment

Choose a reason for hiding this comment

naumvd95 left a comment

Choose a reason for hiding this comment

Added eksengine to engines/init.py #315

Added eksengine to engines/init.py #315

mikezuniga commented Jun 6, 2018 •

edited by naumvd95

Loading