kubernetes: Evaluate local persistent volume support in v1.10

[a-robinson]

v1.10 is expected to be released on March 21, and it notably includes the promotion of LocalPersistentVolume to beta, meaning it will be available by default on all new kubernetes clusters. It's a good opportunity to eliminate our dependency on network-attached volumes and run better on bare metal clusters, but would need thorough testing given how new it is.

cc @mberhault @bobvawter @kannanlakshmi

Jira issue: CRDB-5809

[msau42]

BTW I'm the lead for the feature in Kubernetes, would be happy to help answer questions and always looking for feedback!

[a-robinson]

Thanks @msau42! We'll let you know if anything comes up once we get started.

[victorggsimida]

@a-robinson hi i also want to use k8s with local storage to deploy cockroach db.

I have some questions

1. the cockroach db statefulset secure yaml file is suitable for k8s with local storage ?

2. do i need to rewrite the deploy yaml before create -f

3. any bugs or problem or performance descrese with k8s local storage?

[a-robinson]

1. Yes, although you'll have to make the few changes necessary in order to tell Kubernetes to use local volumes - https://kubernetes.io/docs/concepts/storage/volumes/#local
2. Yes, the changes outlined in the docs linked above.
3. Not that I know of, but as you can probably tell from the lack of updates on this issue, we haven't tried to vet them yet. If you try them, we (and the upstream Kubernetes folks) would love to hear about your experiences with them.

[victorggsimida]

@a-robinson
appreciate your feedback.~~

I will try it and if has any result , I will report here

[jnicholls]

Coming from little knowledge on the CockroachDB's Helm setup (I need to dig into it next), I am interested in the ability for CockroachDB to properly run on any specified storage class, including one that is backed with local persistent volumes. I'm curious if CockroachDB already is architected to handle the case when an instance/pod is rescheduled to a fresh node with an empty data volume (as would be the case with instance storage/local volumes). CockroachDB would begin replicating data to the new instance? Does the cluster just consider this to be a new identity while the old instance is considered permanently dead, given its state is lost?

I'll dig into the code a bit, but if the answer to my questions are "yes" then local persistent volumes ought to work just fine for the cases where the number of nodes that have lost their persistent state and/or are not fully in-sync remains at N - 1, where N is the replication factor. I'm interested in local persistent volume support for deploying onto a bare metal k8s cluster without using network-attached storage or distributed file systems such as Ceph/Rook, but instead using direct-attached NVMe/SSD drives.

[github-actions]

We have marked this issue as stale because it has been inactive for
18 months. If this issue is still relevant, removing the stale label
or adding a comment will keep it active. Otherwise, we'll close it in
10 days to keep the issue queue tidy. Thank you for your contribution
to CockroachDB!

[Bessonov]

adding a comment will keep it active

[Bessonov]

Very interested in this one. But until now the most challenging part is the disk encryption.

[github-actions]

We have marked this issue as stale because it has been inactive for
18 months. If this issue is still relevant, removing the stale label
or adding a comment will keep it active. Otherwise, we'll close it in
10 days to keep the issue queue tidy. Thank you for your contribution
to CockroachDB!

[Bessonov]

a comment will keep it active