K8s multiregion eks #7782

taroface · 2020-07-23T23:22:12Z

Fixes #4314.
Relates to cockroachdb/cockroach#51775, which adds cockroachdb-statefulset-secure-eks.yaml, dnb-lb-eks.yaml, and configmap.yaml to the multiregion repo.
Also addresses cockroachdb/cockroach#51471.

Draft EKS multi-region deployment (manual steps; no script)
Under Create inbound rules, I'm not sure if the "Inter-node and load balancer-node communication" still fits for this rule on multi-region. It seems the load balancer points at CoreDNS on port 53, and CoreDNS provides the load balancer-node communication. Is this right?
Since add guidance for CPU requirements on single-cluster k8s deployment #5922 appears to still be unresolved, there is no guidance here on setting a CPU request and/or limit.
Bob's PR implied (to me) that --cache and --max-sql-memory are now dynamically set in the cockroach start command, so I removed this from the multi- and single-region guides.
- Also, I am not sure if the line "To avoid running out of memory when CockroachDB is not the only pod on a Kubernetes node, you must set memory limits explicitly. This is because CockroachDB does not detect the amount of memory allocated to its pod when run in Kubernetes." is still accurate after that PR. I have an instruction here to add a memory request & limit, but am not clear on whether there is still a risk of OOM, after Bob's changes, if you don't set limits.

cockroach-teamcity · 2020-07-23T23:22:19Z

This change is

cockroach-teamcity · 2020-07-23T23:26:53Z

Online preview: http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/ad2f8bf640d3966962ce408e5ffe1597a078d860/

Edited pages:

cockroach-teamcity · 2020-07-23T23:28:31Z

Online preview: http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/62fdb86c26c6f0cf9a7e0ecc8657e80927bf0114/

Edited pages:

taroface · 2020-07-29T15:41:01Z

@DuskEagle friendly ping on review!

DuskEagle

It seems the load balancer points at CoreDNS on port 53, and CoreDNS provides the load balancer-node communication
CoreDNS is only providing hostname->IP address translation in the multiregion setup. All actual application traffic is still flowing directly from CRDB node A to CRDB node B.

Bob's PR implied (to me) that --cache and --max-sql-memory are now dynamically set in the cockroach start command, so I removed this from the multi- and single-region guides.
Yep, Bob's PR updated our example K8s manifests to set max-sql-memory and cache automatically.

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @DuskEagle and @taroface)

_includes/v20.1/prod-deployment/aws-inbound-rules.md, line 24 at r1 (raw file):

You can set your network IP by selecting "My IP" in the Source field.

#### Load balancer-health check communication

Whether or not this is needed depends on how you're deploying CRDB on AWS. If you're running the binary directly on the VMs, or running it in a container which is using host networking, then you will need this. If instead you're running on Kubernetes, and you configure a K8s service of type LoadBalancer, then you don't need this rule, as the AWS load balancer will not be directly sending health checks to the AdminUI.

v20.1/orchestrate-cockroachdb-with-kubernetes-multi-cluster.md, line 227 at r1 (raw file):

    ~~~

    Each command creates three EKS instances in a region, one for each CockroachDB node you will deploy. Note that the Kubernetes scheduler automatically assigns each instance to a different availability zone in the region.

This isn't quite accurate. It's not the Kubernetes scheduler that creates the instances in 3 different availability zones. The eksctl script creates an autoscaling group which spans the different availability zones. The autoscaling group provisions EC2 instances across its AZs. So in a three-node cluster where each VM is in a separate AZ already, the K8s scheduler is not involved in assigning different AZs to each node.

v20.1/orchestrate-cockroachdb-with-kubernetes-multi-cluster.md, line 379 at r1 (raw file):

           cache 10
           forward . ip1 ip2 ip3 {      # <---- Modify
                force_tcp            # <---- Modify

A user following these instructions will not need to modify the force_tcp line. They'll only need to modify this line if they configured their Network Load Balancer to accept UDP traffic (which isn't covered in this doc, and probably doesn't have to be).

taroface

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @DuskEagle)

_includes/v20.1/prod-deployment/aws-inbound-rules.md, line 24 at r1 (raw file):

Previously, DuskEagle (Joel Kenny) wrote…

Whether or not this is needed depends on how you're deploying CRDB on AWS. If you're running the binary directly on the VMs, or running it in a container which is using host networking, then you will need this. If instead you're running on Kubernetes, and you configure a K8s service of type LoadBalancer, then you don't need this rule, as the AWS load balancer will not be directly sending health checks to the AdminUI.

Done. Thank you for this context.

v20.1/orchestrate-cockroachdb-with-kubernetes-multi-cluster.md, line 227 at r1 (raw file):

Previously, DuskEagle (Joel Kenny) wrote…

This isn't quite accurate. It's not the Kubernetes scheduler that creates the instances in 3 different availability zones. The eksctl script creates an autoscaling group which spans the different availability zones. The autoscaling group provisions EC2 instances across its AZs. So in a three-node cluster where each VM is in a separate AZ already, the K8s scheduler is not involved in assigning different AZs to each node.

Done.

v20.1/orchestrate-cockroachdb-with-kubernetes-multi-cluster.md, line 379 at r1 (raw file):

Previously, DuskEagle (Joel Kenny) wrote…

A user following these instructions will not need to modify the force_tcp line. They'll only need to modify this line if they configured their Network Load Balancer to accept UDP traffic (which isn't covered in this doc, and probably doesn't have to be).

Done.

taroface · 2020-08-20T15:37:53Z

TFTR @DuskEagle !

cockroach-teamcity · 2020-08-20T15:44:25Z

Online preview: http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/4118fb7de0274e2d5d027199b151973451ff9f1b/

Edited pages:

cockroach-teamcity · 2020-08-20T16:16:06Z

Online preview: http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/b053df54444baed416cbf03d8c6442bb2f79e9e5/

Edited pages:

Amruta-Ranade · 2020-08-26T13:21:16Z

_includes/v20.1/prod-deployment/insecure-test-load-balancing.md

@@ -1,8 +1,12 @@
 CockroachDB offers a pre-built `workload` binary for Linux that includes several load generators for simulating client traffic against your cluster. This step features CockroachDB's version of the [TPC-C](http://www.tpc.org/tpcc/) workload.

+{{site.data.alerts.callout_info}}
+Be sure that you have [set up an inbound rule](deploy-cockroachdb-on-aws-insecure.html#step-2-configure-your-network) that allows traffic from the application to the load balancer. In this case, you will run the sample workload on one of your instances. The traffic source for your inbound rule should therefore be the **internal (private)** IP address of that instance. To find this, open the Instances section of the Amazon EC2 console and click on the instance.


nit: Should "Instances" be "Instances" since we are referring to a UI element? If yes, need to update other places where it appears.

jseldess

Going to take me a while to get through this, so I'm submitting things as I find them.

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @DuskEagle, @jseldess, and @taroface)

v20.1/orchestrate-cockroachdb-with-kubernetes-multi-cluster.md, line 199 at r3 (raw file):

    --nodes 3 \
    --node-ami auto \
    --region <aws-region-1>

Need to add \ to the end of the line; otherwise, pasting into the terminal doesn't work as expected.

v20.1/orchestrate-cockroachdb-with-kubernetes-multi-cluster.md, line 211 at r3 (raw file):

    --nodes 3 \
    --node-ami auto \
    --region <aws-region-2>

Need to add \ to the end of the line; otherwise, pasting into the terminal doesn't work as expected.

v20.1/orchestrate-cockroachdb-with-kubernetes-multi-cluster.md, line 223 at r3 (raw file):

    --nodes 3 \
    --node-ami auto \
    --region <aws-region-3>

Need to add \ to the end of the line; otherwise, pasting into the terminal doesn't work as expected.

v20.1/orchestrate-cockroachdb-with-kubernetes-multi-cluster.md, line 955 at r3 (raw file):

<section class="filter-content" markdown="1" data-scope="eks">
## Step 6. Access the Admin UI

This is the wrong language for this step.

v20.2/orchestrate-cockroachdb-with-kubernetes-multi-cluster.md, line 199 at r3 (raw file):

    --nodes 3 \
    --node-ami auto \
    --region <aws-region-1>

Need to add \ to the end of the line; otherwise, pasting into the terminal doesn't work as expected.

v20.2/orchestrate-cockroachdb-with-kubernetes-multi-cluster.md, line 211 at r3 (raw file):

    --nodes 3 \
    --node-ami auto \
    --region <aws-region-2>

Need to add \ to the end of the line; otherwise, pasting into the terminal doesn't work as expected.

v20.2/orchestrate-cockroachdb-with-kubernetes-multi-cluster.md, line 223 at r3 (raw file):

    --nodes 3 \
    --node-ami auto \
    --region <aws-region-3>

Need to add \ to the end of the line; otherwise, pasting into the terminal doesn't work as expected.

v20.2/orchestrate-cockroachdb-with-kubernetes-multi-cluster.md, line 955 at r3 (raw file):

<section class="filter-content" markdown="1" data-scope="eks">
## Step 6. Access the Admin UI

This is the wrong language for this step.

jseldess

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @DuskEagle, @jseldess, and @taroface)

v20.1/orchestrate-cockroachdb-with-kubernetes-multi-cluster.md, line 200 at r3 (raw file):

    --node-ami auto \
    --region <aws-region-1>
    --vpc-cidr=<ip-range-1>

We don't use = for the other flags, so we should remove it here, too.

v20.1/orchestrate-cockroachdb-with-kubernetes-multi-cluster.md, line 212 at r3 (raw file):

    --node-ami auto \
    --region <aws-region-2>
    --vpc-cidr=<ip-range-2>

We don't use = for the other flags, so we should remove it here, too.

v20.1/orchestrate-cockroachdb-with-kubernetes-multi-cluster.md, line 224 at r3 (raw file):

    --node-ami auto \
    --region <aws-region-3>
    --vpc-cidr=<ip-range-3>

We don't use = for the other flags, so we should remove it here, too.

v20.2/orchestrate-cockroachdb-with-kubernetes-multi-cluster.md, line 200 at r3 (raw file):

    --node-ami auto \
    --region <aws-region-1>
    --vpc-cidr=<ip-range-1>

We don't use = for the other flags, so we should remove it here, too.

v20.2/orchestrate-cockroachdb-with-kubernetes-multi-cluster.md, line 212 at r3 (raw file):

    --node-ami auto \
    --region <aws-region-2>
    --vpc-cidr=<ip-range-2>

We don't use = for the other flags, so we should remove it here, too.

v20.2/orchestrate-cockroachdb-with-kubernetes-multi-cluster.md, line 224 at r3 (raw file):

    --node-ami auto \
    --region <aws-region-3>
    --vpc-cidr=<ip-range-3>

We don't use = for the other flags, so we should remove it here, too.

jseldess · 2020-08-27T15:20:15Z

v20.2/orchestrate-cockroachdb-with-kubernetes-multi-cluster.md, line 261 at r3 (raw file):

    {% include copy-clipboard.html %}
    ~~~ shell
    kubectl create namespace <cluster-namespace> --context=<cluster-context>

We use = for this flag but not for namespace. This is a nit, but I'd prefer we be consistent about this throughout.

jseldess · 2020-08-27T16:02:31Z

v20.1/orchestrate-cockroachdb-with-kubernetes-multi-cluster.md, line 296 at r3 (raw file):

### Create inbound rules

For each region, navigate to the Security Groups section of the [Amazon EC2 console](https://console.aws.amazon.com/ec2/) and locate the security group that enables communication between nodes int he cluster. It should have a name like `ClusterSharedNodeSecurityGroup`. [Add Custom TCP inbound rules](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security.html#adding-security-group-rule) to this security group to allow TCP communication on two ports:

int he > in the

jseldess · 2020-08-27T16:02:40Z

v20.2/orchestrate-cockroachdb-with-kubernetes-multi-cluster.md, line 296 at r3 (raw file):

### Create inbound rules

For each region, navigate to the Security Groups section of the [Amazon EC2 console](https://console.aws.amazon.com/ec2/) and locate the security group that enables communication between nodes int he cluster. It should have a name like `ClusterSharedNodeSecurityGroup`. [Add Custom TCP inbound rules](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security.html#adding-security-group-rule) to this security group to allow TCP communication on two ports:

int he > in the

jseldess

Unfortunately, I don't have the time to run through the entire tutorial. I made it up to configuring coredns. It's an incredibly complex process, and the documentation is stellar. I'm going to say

despite not testing the entirely. Ryan has run through this many times, and I'd rather get this live and in front of users. When we get feedback, we'll address it.

Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @DuskEagle, @jseldess, and @taroface)

v20.2/orchestrate-cockroachdb-with-kubernetes-multi-cluster.md, line 241 at r3 (raw file):

    {% include copy-clipboard.html %}
    ~~~ shell
    $ kubectl config get-contexts

This is a gke command.

cockroach-teamcity · 2020-08-27T21:41:36Z

Online preview: http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/532c21f6b3a40b307ae6b4363e74d4d33c589b99/

Edited pages:

taroface

Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @DuskEagle and @jseldess)

_includes/v20.1/prod-deployment/insecure-test-load-balancing.md, line 4 at r3 (raw file):

Previously, Amruta-Ranade (Amruta Ranade) wrote…

nit: Should "Instances" be "Instances" since we are referring to a UI element? If yes, need to update other places where it appears.

Instances is more a general term, not specific to UI, so this should be safe to leave without formatting.

v20.1/orchestrate-cockroachdb-with-kubernetes-multi-cluster.md, line 199 at r3 (raw file):