Kubernetes cluster - use of EC2 instance storage for pods

[davidopp]

This seems like a pretty good question, but I don't know the answer.

http://stackoverflow.com/questions/32915059/kubernetes-cluster-use-of-ec2-instance-storage-for-pods

@justinsb @thockin

[justinsb]

The problem here is that I allocated a very small partition to the 'kubernetes' mount vs the 'docker' mount. That was my bad, and should be fixed by #13803 and cherry-picked in #14272 (although I just noticed the cherry-pick has failed tests so I kicked shippable...)

I should have time to answer the SO question tomorrow...

[rroopreddy]

Does this make all of instance storage available to pods for "emptyDir" type volumes. I remember seeing somewhere that there will not be any separate partitioning of instance storage between Docker and Kubernetes. When do you think this will be out?

We badly in need of instance storage for persistence etc. Today all of the instance storage not usable and we had to Create EBS volumes for this purpose. EBS volumes are slow and expansive compared to free instance storage that comes with the larger EC2 instances

Appreciate it!

[rroopreddy]

Additionally, AWS/EBS volumes can not be used with replication controller. This is because each EBS volume need to be attached to one pod at a time compared to one replication controller spec with say 'scale = 3'.

For any real use 'emptyDir' volumes from local instance storage are required .

In the meantime, in there a pattern to have single EBS volume shared across multiple pods that spawned through replication controller scale instruction?

[thockin]

EBS is like most block-devices - single writer.

On Fri, Oct 9, 2015 at 4:58 PM, Ravindar Reddy notifications@github.com
wrote:

Additionally, AWS/EBS volumes can not be used with replication controller.
This is because each EBS volume need to be attached to one pod at a time
compared to one replication controller spec with say 'scale = 3'.

For any real use 'emptyDir' volumes from local instance storage are
required .

In the meantime, in there a pattern to have single EBS volume shared
across multiple pods that spawned through replication controller scale
instruction?

—
Reply to this email directly or view it on GitHub
#15055 (comment)
.

[geoah]

What would be the "kubernetes way" to deploy a production cassandra or other database cluster?
Where should the data be stored? Is the minion volume an acceptable place or is there a better practice?

[rroopreddy]

1. Cassandra has a set of nodes in (maps to pod in kubernetes) a cluster.
2. Each node is interchangeable and has a data directory for DB store/folder with a run-time.
3. A node run-time can die and brought back-up - it just need to point to last disconnected DB store/folder

Given Above, here is the typical way Cassandra is deployed with data storage

1. Host machine (kubernetes minions) has large /fast instance memory that is used for node DB store/folder
2. Cassandra node run-time can be brought up and link to a DB store/folder on a host
3. E.g: If I have 3-cassandra nodes, all I need is 3-run times need to be linked against 3-DB store/folders. Which-to-which does not matter
4. This work fine as long as host machine die, in which case the DB store/folder gone. This is not a issue as the data is backed up to a permanent store and restore on machine failures

Kubernates way could be

1. Define named volumes from host instance storage that can be mounted/used in pods
2. If a pod dies, you can bring it up and link to above defined volume on the instance

[geoah]

Hey @rroopreddy and thanks for taking the time to reply, you are absolutely right on all points.

My only concern is that by using named volumes you can no longer use a replication controller to scale your service as you'd need custom pod configs for each cassandra node. Or am I missing something?

Using your proposed "kubernetes way" for each cassandra node I would need the following:

• Persistent Volume (and also create the EBS in AWS)
• Replication controller with a size of 1 with pod template pointing to the above PV
  The RC is just to make sure the pod keeps being alive and well.
  And a service to group them all together I guess.

This way in order to add a new cassandra node I would need to 1) create the EBS 2) create a new VP 3) create a new RC.
Kinda seems like an overkill. -- Am I missing something here?

[rroopreddy]

Currently we are using EBS volume and has the issue you mentioned, i.e: replication controller can not be used. This i.e. because EBS volume can be mounted only on one cassandra node (pod in this case). So each node YAML has a pod definition with specific volume id.

Host volumes can differ a bit. Say you defined a pool of host volume disks available with some capacity (e.g: 30GB of 3-minions). And then, say carve out 15GB for each node (similar to persistentVolumeClaim). Only thing here is that previously allocated space has to be remembered and linked back while new one has to created. This will work for replication controller scale up models well

[rroopreddy]

Hi @geoah thanks for chasing this; lot of people have been asking for this for different use case scenarios. Just to clarify, here is a difference between EBS volume based model (persistent forever) and Host instance based model (persistent until machine goes down). We have been using EBS volume model for testing until the host instance approach is figured out.

As I mentioned above, could not figure out a way to use replication controller with current EBS volumes approach. I love to hear if there is a way to use replication controller with EBS volumes

[justinsb]

We now create one big volume for ephemeral storage:

> lvdisplay
  --- Logical volume ---
  LV Path                /dev/vg-ephemeral/ephemeral
  LV Name                ephemeral
  VG Name                vg-ephemeral
  LV UUID                KE2f7l-JeDm-ccMA-vp24-fAVf-kh69-SEEEeB
  LV Write Access        read/write
  LV Creation host, time ip-172-20-0-9, 2016-02-12 14:32:25 +0000
  LV Status              available
  # open                 1
  LV Size                3.99 GiB
  Current LE             1022
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           252:0

I think the issue about RCs and persistent volumes is important, but if there are remaining questions perhaps we an open new issues (I think issues already exist)