New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support creating a blank VHD in a storage account #655

Closed
colemickens opened this Issue Aug 7, 2016 · 21 comments

Comments

Projects
None yet
9 participants
@colemickens
Contributor

colemickens commented Aug 7, 2016

There are Kubernetes scenarios where users need to be able to provision blank VHDs in a storage account so they can make them available in their cluster.

Today, basically there are only two options as far as I can tell:

  1. Attach a new blank disk to a VM from the Portal or CLI. Immediately detach the disk. At least it's a valid VHD.
  2. Create the VHD on your local system. (qemu-img, etc)

There ought to be a way to do this from CLI (or better yet, if this were something that Storage could do via an API).

@rootfs

This comment has been minimized.

Show comment
Hide comment
@rootfs

rootfs Aug 7, 2016

Though it is possible to use a combination of PutPageBlob and PutPageb to create a blank page blob and then append a VHD label to the blob, an Azure API support is ultimate the best idea for the following reason:

  • it is atomic operation
  • Even if VHD format changes, the API won't change.

rootfs commented Aug 7, 2016

Though it is possible to use a combination of PutPageBlob and PutPageb to create a blank page blob and then append a VHD label to the blob, an Azure API support is ultimate the best idea for the following reason:

  • it is atomic operation
  • Even if VHD format changes, the API won't change.
@ahmetb

This comment has been minimized.

Show comment
Hide comment
@ahmetb

ahmetb Aug 8, 2016

Contributor

Here's a sample piece of code: https://gist.github.com/sedouard/0468d1b2d0153ca9a652

But ideally this problem should be resolved with upcoming new Disks APIs.

Contributor

ahmetb commented Aug 8, 2016

Here's a sample piece of code: https://gist.github.com/sedouard/0468d1b2d0153ca9a652

But ideally this problem should be resolved with upcoming new Disks APIs.

@jpoon

This comment has been minimized.

Show comment
Hide comment
@jpoon

jpoon Oct 14, 2016

One such user scenario is Pachyderm which requires a data-disk to be created and mounted to be used for their RethinkDB (pachyderm/pachyderm#960) and having an API to create a blank data-disk would be da 💣 or...at least put Azure on par with AWS and GCE:

AWS: aws ec2 create-volume ...
GCE: gcloud compute disks create ...

jpoon commented Oct 14, 2016

One such user scenario is Pachyderm which requires a data-disk to be created and mounted to be used for their RethinkDB (pachyderm/pachyderm#960) and having an API to create a blank data-disk would be da 💣 or...at least put Azure on par with AWS and GCE:

AWS: aws ec2 create-volume ...
GCE: gcloud compute disks create ...

@jpoon

This comment has been minimized.

Show comment
Hide comment
@jpoon

jpoon Oct 18, 2016

I ended up wrapping @sedouard's workaround into a "helper" Docker image (https://github.com/jpoon/azure-create-vhd). Still would be good to have this in the Azure CLI.

jpoon commented Oct 18, 2016

I ended up wrapping @sedouard's workaround into a "helper" Docker image (https://github.com/jpoon/azure-create-vhd). Still would be good to have this in the Azure CLI.

@colemickens

This comment has been minimized.

Show comment
Hide comment
@colemickens

colemickens Oct 18, 2016

Contributor

I had already done that as well, for the azure-kubernetes-demo repo that shows how to use data disks: https://github.com/colemickens/azure-tools

Mine can preformat as well, and create storage account/containers if they're missing.

Contributor

colemickens commented Oct 18, 2016

I had already done that as well, for the azure-kubernetes-demo repo that shows how to use data disks: https://github.com/colemickens/azure-tools

Mine can preformat as well, and create storage account/containers if they're missing.

@dbalaouras

This comment has been minimized.

Show comment
Hide comment
@dbalaouras

dbalaouras Feb 14, 2017

Is there any update on this? @colemickens docker image works good, but enabling this operation via the Azure CLI is really the way to go.

dbalaouras commented Feb 14, 2017

Is there any update on this? @colemickens docker image works good, but enabling this operation via the Azure CLI is really the way to go.

@colemickens

This comment has been minimized.

Show comment
Hide comment
@colemickens

colemickens Feb 14, 2017

Contributor

No update here. When everything moves to managed disks, there will be an easy way to do this, including CLI support. For now, the non-managed disks implementation simply has no API for disk creation.

For now, it's much easier to just use dynamic provisioning... you just create a PVC for whatever size you want and Kubernetes will make the PV for you.

Contributor

colemickens commented Feb 14, 2017

No update here. When everything moves to managed disks, there will be an easy way to do this, including CLI support. For now, the non-managed disks implementation simply has no API for disk creation.

For now, it's much easier to just use dynamic provisioning... you just create a PVC for whatever size you want and Kubernetes will make the PV for you.

@dbalaouras

This comment has been minimized.

Show comment
Hide comment
@dbalaouras

dbalaouras Mar 14, 2017

@colemickens I managed to use dynamic provisioning via the kubernetes.io/azure-disk Provisioner but I'm hitting the very low limit (i.e. 4) of mounted disks on my A2 VM.

Is there a way to do the same dynamic provisioning with azure-file? I.e., dynamically create files in my Storage Account and mount them on my Pods.

ta.

dbalaouras commented Mar 14, 2017

@colemickens I managed to use dynamic provisioning via the kubernetes.io/azure-disk Provisioner but I'm hitting the very low limit (i.e. 4) of mounted disks on my A2 VM.

Is there a way to do the same dynamic provisioning with azure-file? I.e., dynamically create files in my Storage Account and mount them on my Pods.

ta.

@colemickens

This comment has been minimized.

Show comment
Hide comment
@colemickens

colemickens Mar 14, 2017

Contributor

Dynamic provisioning for Azure Files should be in 1.7, and probably cherry-picked to 1.6.1, I think: kubernetes/kubernetes#42170

But you probably shouldn't use it for anything serious at all. The perf is definitely not good enough for a DB or anything.

Contributor

colemickens commented Mar 14, 2017

Dynamic provisioning for Azure Files should be in 1.7, and probably cherry-picked to 1.6.1, I think: kubernetes/kubernetes#42170

But you probably shouldn't use it for anything serious at all. The perf is definitely not good enough for a DB or anything.

@dbalaouras

This comment has been minimized.

Show comment
Hide comment
@dbalaouras

dbalaouras Mar 14, 2017

@colemickens thx for the prompt response. And yeah, I recon azure-files are SMB drives and not suitable for DBs. Wondering if it could work for a small SQLite though..

In any case, do you happen to know why is there a limitation on the number of vhds I can mount on a VM and why is it so low?

{
    "maxDataDiskCount": 4,
    "memoryInMb": 3584,
    "name": "Standard_A2",
    "numberOfCores": 2,
    "osDiskSizeInMb": 1047552,
    "resourceDiskSizeInMb": 138240
  }

dbalaouras commented Mar 14, 2017

@colemickens thx for the prompt response. And yeah, I recon azure-files are SMB drives and not suitable for DBs. Wondering if it could work for a small SQLite though..

In any case, do you happen to know why is there a limitation on the number of vhds I can mount on a VM and why is it so low?

{
    "maxDataDiskCount": 4,
    "memoryInMb": 3584,
    "name": "Standard_A2",
    "numberOfCores": 2,
    "osDiskSizeInMb": 1047552,
    "resourceDiskSizeInMb": 138240
  }
@colemickens

This comment has been minimized.

Show comment
Hide comment
@colemickens

colemickens Mar 14, 2017

Contributor

That's just how Azure sells their SKUs. I'd suggest contacting support to ask them and express your interest in different SKUs with lower compute and more disks.

Contributor

colemickens commented Mar 14, 2017

That's just how Azure sells their SKUs. I'd suggest contacting support to ask them and express your interest in different SKUs with lower compute and more disks.

@dbalaouras

This comment has been minimized.

Show comment
Hide comment
@dbalaouras

dbalaouras commented Mar 14, 2017

@jpoon

This comment has been minimized.

Show comment
Hide comment
@jpoon

jpoon Mar 14, 2017

@dbalaouras here's a table that shows the various SKUs and the number of data disks that you can attach. But please do reach out to Support expressing your need for MOAR disks.

jpoon commented Mar 14, 2017

@dbalaouras here's a table that shows the various SKUs and the number of data disks that you can attach. But please do reach out to Support expressing your need for MOAR disks.

@dbalaouras

This comment has been minimized.

Show comment
Hide comment
@dbalaouras

dbalaouras Mar 14, 2017

@jpoon thx. My concern is that in best case, the max number of disks is double the number of cores according to that table. That's not super useful when using Dynamic Provisioning, thus (at least) 1 vhd per Pod, cause you are then limited to 3 Pods per Node (when using A2 VMs) or 2 x cores - 1 :-/

dbalaouras commented Mar 14, 2017

@jpoon thx. My concern is that in best case, the max number of disks is double the number of cores according to that table. That's not super useful when using Dynamic Provisioning, thus (at least) 1 vhd per Pod, cause you are then limited to 3 Pods per Node (when using A2 VMs) or 2 x cores - 1 :-/

@rutsky

This comment has been minimized.

Show comment
Hide comment
@rutsky

rutsky Mar 14, 2017

@dbalaouras

And yeah, I recon azure-files are SMB drives and not suitable for DBs. Wondering if it could work for a small SQLite though..

I tried to use Azure File for SQLite database storage for Grafana which failed with errors about "locked" database. Adding "nobrl" to CIFS share mount options workarounds this issue, but I doubt this somehow makes SQLite correctly work on SMB share, I think data corruptions can be expected.

rutsky commented Mar 14, 2017

@dbalaouras

And yeah, I recon azure-files are SMB drives and not suitable for DBs. Wondering if it could work for a small SQLite though..

I tried to use Azure File for SQLite database storage for Grafana which failed with errors about "locked" database. Adding "nobrl" to CIFS share mount options workarounds this issue, but I doubt this somehow makes SQLite correctly work on SMB share, I think data corruptions can be expected.

@yugangw-msft

This comment has been minimized.

Show comment
Hide comment
@yugangw-msft

yugangw-msft Apr 3, 2017

Contributor

When everything moves to managed disks, there will be an easy way to do this, including CLI support.

Yep, try az disk create

Contributor

yugangw-msft commented Apr 3, 2017

When everything moves to managed disks, there will be an easy way to do this, including CLI support.

Yep, try az disk create

@writeameer

This comment has been minimized.

Show comment
Hide comment
@writeameer

writeameer May 2, 2017

Contributor

Trying to use

az disk create

with Kubernetes like this example from @colemickens (https://github.com/colemickens/azure-kubernetes-demo/blob/master/test-azure-disk.yaml) , where I need to specify VHD url. Managed disks don't seem to have the notion of VHD URLs ? How can I use managed disks in the yaml for Kubernetes ?

Contributor

writeameer commented May 2, 2017

Trying to use

az disk create

with Kubernetes like this example from @colemickens (https://github.com/colemickens/azure-kubernetes-demo/blob/master/test-azure-disk.yaml) , where I need to specify VHD url. Managed disks don't seem to have the notion of VHD URLs ? How can I use managed disks in the yaml for Kubernetes ?

@yugangw-msft

This comment has been minimized.

Show comment
Hide comment
@yugangw-msft

yugangw-msft May 2, 2017

Contributor

I need to specify VHD url. Managed disks don't seem to have the notion of VHD URLs

I don't really know the e2e flow, but for this specific question, try az disk grant-access

Contributor

yugangw-msft commented May 2, 2017

I need to specify VHD url. Managed disks don't seem to have the notion of VHD URLs

I don't really know the e2e flow, but for this specific question, try az disk grant-access

@jpoon

This comment has been minimized.

Show comment
Hide comment
@jpoon

jpoon May 2, 2017

@writeameer managed disk support in k8s is coming and, I think, is slated for 1.7 (kubernetes/kubernetes#41950)

jpoon commented May 2, 2017

@writeameer managed disk support in k8s is coming and, I think, is slated for 1.7 (kubernetes/kubernetes#41950)

@writeameer

This comment has been minimized.

Show comment
Hide comment
@writeameer

writeameer May 3, 2017

Contributor

@yugangw-msft Thank you.

@jpoon Got it, thank you for the link to the issue - great insight - much appreciated.

Contributor

writeameer commented May 3, 2017

@yugangw-msft Thank you.

@jpoon Got it, thank you for the link to the issue - great insight - much appreciated.

@yugangw-msft

This comment has been minimized.

Show comment
Hide comment
@yugangw-msft

yugangw-msft Aug 9, 2018

Contributor

Hi folks, can this issue be closed now since managed disk support has been integrated with azure/k8s? Let me know if I missed anything else, or there are other improvements I should made.

Contributor

yugangw-msft commented Aug 9, 2018

Hi folks, can this issue be closed now since managed disk support has been integrated with azure/k8s? Let me know if I missed anything else, or there are other improvements I should made.

@colemickens colemickens closed this Aug 9, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment