Add kubernetes module for aws #236

tei-k · 2020-11-12T06:37:04Z

Description

https://scalar-labs.atlassian.net/browse/DLT-7400

Done

Add Kubernetes module for AWS
Only support node group

Confirm

Terraform apply

$ cd example/aws/kubernetes
$ terraform apply -var-file=example.tfvars

Copy configuration files (scalar-k8/conf)
- inventory.ini
- kube_config (Copy from ${eks_cluster_name}_kubeconfig)
Fix scalar-k8/conf/scalardl-custom-values.yaml

-      service.beta.kubernetes.io/azure-load-balancer-internal: "true"
-      service.beta.kubernetes.io/azure-load-balancer-internal-subnet: "k8s_ingress"
+      service.beta.kubernetes.io/aws-load-balancer-type: clb
+      service.beta.kubernetes.io/aws-load-balancer-internal: "true"
+      # service.beta.kubernetes.io/azure-load-balancer-internal: "true"
+      # service.beta.kubernetes.io/azure-load-balancer-internal-subnet: "k8s_ingress"

ansible-playbook (scalar-k8/)

$ ansible-playbook -i ${SCALAR_K8S_CONFIG_DIR}/inventory.ini playbooks/playbook-install-tools.yml
$ ansible-playbook -i ${SCALAR_K8S_CONFIG_DIR}/inventory.ini playbooks/playbook-deploy-scalardl.yml
$ ansible-playbook -i ${SCALAR_K8S_CONFIG_DIR}/inventory.ini playbooks/playbook-deploy-prometheus.yml

Get resources

$ k get pod,svc -o wide

Ref

This reverts commit 3d452bb.

feeblefakie

Overall looking good. Left some suggestions and comments.

feeblefakie · 2020-11-24T01:57:33Z

modules/aws/kubernetes/vars.tf

+variable "kubernetes_cluster" {
+  type        = map
+  default     = {}
+  description = "Custom definition kubernetes properties that include name of the cluster, kubernetes version, etc.."


Suggested change

description = "Custom definition kubernetes properties that include name of the cluster, kubernetes version, etc.."

description = "Custom definition kubernetes properties that include the name of the cluster, kubernetes version, etc.."

feeblefakie · 2020-11-24T01:58:02Z

modules/aws/kubernetes/vars.tf

+}
+
+variable "kubernetes_node_groups" {
+  description = "Map of map of node groups to create"


Is it really a map of map? or typo?

Yes, It's a map of maps.

feeblefakie · 2020-11-24T02:03:09Z

modules/aws/kubernetes/iam_cluster.tf

@@ -0,0 +1,96 @@
+resource "aws_iam_role" "eks_cluster" {


Do we need to add a subsection for privileges in EKS deployment?
https://github.com/scalar-labs/scalar-terraform/blob/master/docs/CloudPrivileges.md

Oh, We need some privileges like below.
https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/docs/iam-permissions.md

ymorimo · 2020-11-25T03:40:44Z

examples/aws/kubernetes/example.tfvars

@@ -0,0 +1,45 @@
+# Optional


As with the other examples, the default region should be here at the top of example.tfvars.
(I didn't use the default ap-northeast-1 region and I got lost when terraform apply couldn't find the VPC that the network module created.)

Suggested change

# Optional

region = "ap-northeast-1"

# Optional

Good point! 👍
Also added backend.tf.s3 and remote.tf.s3 in 41f7f45

ymorimo · 2020-11-25T07:21:59Z

@tei-k Question.

I got the following kube_config output after terraform apply,

clusters:
- cluster:
    server: https://D0BE5C2FAD77ED66ED29817523BB7182.yl4.us-west-1.eks.amazonaws.com
    certificate-authority-data: LS0tLS1CRUdJTiBDRV...

but the server host in that file resolves to private IP addresses. Because of this kubectl fails to communicate with the endpoint.

$ host d0be5c2fad77ed66ed29817523bb7182.yl4.us-west-1.eks.amazonaws.com
d0be5c2fad77ed66ed29817523bb7182.yl4.us-west-1.eks.amazonaws.com has address 10.42.1.144
d0be5c2fad77ed66ed29817523bb7182.yl4.us-west-1.eks.amazonaws.com has address 10.42.41.117

Can you tell what is happening?

tei-k · 2020-11-25T07:56:49Z

@ymorimo

but the server host in that file resolves to private IP addresses. Because of this kubectl fails to communicate with the endpoint.

Good point! 👍 Sorry for missing of explanation on this point.

Because cluster_endpoint_public_access is false in the current default setting, So cannot access this endpoint from local.
To operate from the bastion we need to set up authentication manually.

Can you first create environment with cluster_endpoint_public_access = "true" ?

example.tfvars

# cluster_endpoint_private_access      = "true"
cluster_endpoint_public_access       = "true"

feeblefakie

LGTM!

Please create a PR for the update in scalardl-custom-values.yaml once this PR is merged.
Also, please write a getting started doc once Yusuke's doc reorganization is completed.

ymorimo

It worked. Thanks. LGTM!

Add kubernetes module for aws

a070820

tei-k self-assigned this Nov 12, 2020

tei-k changed the title ~~[]Add kubernetes module for aws~~ [WIP] Add kubernetes module for aws Nov 12, 2020

tei-k added 3 commits November 12, 2020 15:53

Delete inventory.ini

1978784

Delete public_key path

e22caf2

Add README

17b6577

tei-k force-pushed the add-kubenertes-for-aws branch from 1abce85 to 17b6577 Compare November 12, 2020 07:46

tei-k added 10 commits November 13, 2020 11:06

Add fargate

dfceacd

Add monitoring fargate

877f551

Fix kubernetes labels

8a78189

Add create flag for fargate

8a1d225

Add create flag for node groups

7e1d0c0

Add kube-system fargate

17d1af4

Seperate fargate.tfvars

5475a0d

Refactoring fargate vars

87392b5

Refactoring node group vars

c6d550d

Add fargate.tfvars

a0f49e5

tei-k force-pushed the add-kubenertes-for-aws branch from 78e95f4 to a0f49e5 Compare November 16, 2020 05:17

tei-k added 2 commits November 16, 2020 17:47

Fix coredns

27df599

Delete kubeconfig

15ac1f9

tei-k force-pushed the add-kubenertes-for-aws branch from 60afe87 to 15ac1f9 Compare November 16, 2020 23:51

tei-k added 5 commits November 17, 2020 17:58

Add timeouts for fargate resources

3bba575

Fix auth

a68d3a5

Remove fargate sub module

99897c8

Add cluster_log_kms_key_id

bbf879d

Fix README

b52e800

tei-k changed the title ~~[WIP] Add kubernetes module for aws~~ Add kubernetes module for aws Nov 20, 2020

Delete unnecessary variables

3d452bb

tei-k requested review from feeblefakie and ymorimo November 20, 2020 07:35

tei-k added 2 commits November 20, 2020 18:38

Revert "Delete unnecessary variables"

e521405

This reverts commit 3d452bb.

Delete unnecessary variables

13ab7c7

feeblefakie reviewed Nov 24, 2020

View reviewed changes

tei-k added 2 commits November 24, 2020 11:57

Fix readme

8132790

Fix kubernetes_labels

4537792

ymorimo reviewed Nov 25, 2020

View reviewed changes

Add region and s3 file

41f7f45

Add k8s permission to doc

43c1c57

tei-k added 2 commits November 26, 2020 09:56

Fix to true for cluster_endpoint_public_access by default

0a7492c

Add manage_aws_auth

d5cee92

tei-k mentioned this pull request Nov 26, 2020

Add awscli role for bastion scalar-labs/scalar-kubernetes#85

Merged

feeblefakie approved these changes Nov 30, 2020

View reviewed changes

ymorimo approved these changes Nov 30, 2020

View reviewed changes

feeblefakie merged commit ca97c22 into master Nov 30, 2020

feeblefakie deleted the add-kubenertes-for-aws branch November 30, 2020 07:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add kubernetes module for aws #236

Add kubernetes module for aws #236

Uh oh!

tei-k commented Nov 12, 2020 •

edited

Loading

Uh oh!

feeblefakie left a comment

Uh oh!

feeblefakie Nov 24, 2020

Uh oh!

feeblefakie Nov 24, 2020

Uh oh!

tei-k Nov 24, 2020

Uh oh!

feeblefakie Nov 24, 2020

Uh oh!

tei-k Nov 24, 2020

Uh oh!

ymorimo Nov 25, 2020

Uh oh!

tei-k Nov 25, 2020

Uh oh!

ymorimo commented Nov 25, 2020

Uh oh!

tei-k commented Nov 25, 2020

Uh oh!

feeblefakie left a comment

Uh oh!

ymorimo left a comment

Uh oh!

Uh oh!

	description = "Custom definition kubernetes properties that include name of the cluster, kubernetes version, etc.."
	description = "Custom definition kubernetes properties that include the name of the cluster, kubernetes version, etc.."

Add kubernetes module for aws #236

Add kubernetes module for aws #236

Uh oh!

Conversation

tei-k commented Nov 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Done

Confirm

Ref

Uh oh!

feeblefakie left a comment

Choose a reason for hiding this comment

Uh oh!

feeblefakie Nov 24, 2020

Choose a reason for hiding this comment

Uh oh!

feeblefakie Nov 24, 2020

Choose a reason for hiding this comment

Uh oh!

tei-k Nov 24, 2020

Choose a reason for hiding this comment

Uh oh!

feeblefakie Nov 24, 2020

Choose a reason for hiding this comment

Uh oh!

tei-k Nov 24, 2020

Choose a reason for hiding this comment

Uh oh!

ymorimo Nov 25, 2020

Choose a reason for hiding this comment

Uh oh!

tei-k Nov 25, 2020

Choose a reason for hiding this comment

Uh oh!

ymorimo commented Nov 25, 2020

Uh oh!

tei-k commented Nov 25, 2020

Uh oh!

feeblefakie left a comment

Choose a reason for hiding this comment

Uh oh!

ymorimo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tei-k commented Nov 12, 2020 •

edited

Loading