Azure storage driver #372

Andrey-mp · 2016-12-26T14:40:07Z

This patchset adds support of Azure Cloud.
There are several operations implemented in patchset:
create, attach, detach, remove (all base operations
for volumes).

CLAassistant · 2016-12-26T14:40:08Z

All committers have signed the CLA.

houndci-bot · 2016-12-26T14:40:14Z

drivers/storage/azure/azure.go

+)
+
+const (
+	ConfigAZURE                    = Name


exported const ConfigAZURE should have comment (or a comment on this block) or be unexported

houndci-bot · 2016-12-26T14:40:14Z

drivers/storage/azure/azure.go

+	ClientSecretKey = "clientSecret"
+	CertPathKey     = "certPath"
+
+	// Storage auth keys


comment on exported const StorageAccount should be of the form "StorageAccount ..."

houndci-bot · 2016-12-26T14:40:14Z

drivers/storage/azure/azure.go

+	DefaultMaxRetries = 10
+	DefaultUseHTTPS   = true
+
+	// Config keys:


comment on exported const TenantIDKey should be of the form "TenantIDKey ..."

houndci-bot · 2016-12-26T14:40:14Z

drivers/storage/azure/azure.go

+
+	// DefaultMaxRetries is the max number of times to retry failed operations
+	DefaultMaxRetries = 10
+	DefaultUseHTTPS   = true


exported const DefaultUseHTTPS should have comment (or a comment on this block) or be unexported

houndci-bot · 2016-12-26T15:19:40Z

drivers/storage/azure/azure.go

+	ContainerKey      = "container"
+	// Use https or not for making Azure URI's
+	UseHTTPS          = "useHTTPS"
+	// Tag


comment on exported const TagKey should be of the form "TagKey ..."

houndci-bot · 2016-12-26T15:19:41Z

drivers/storage/azure/azure.go

+	ResourceGroupKey  = "resourceGroup"
+	// Name of container in the storage account ('vhds' by default)
+	ContainerKey      = "container"
+	// Use https or not for making Azure URI's


comment on exported const UseHTTPS should be of the form "UseHTTPS ..."

houndci-bot · 2016-12-26T15:19:41Z

drivers/storage/azure/azure.go

+	SubscriptionIDKey = "subscriptionID"
+	// Name of resource group
+	ResourceGroupKey  = "resourceGroup"
+	// Name of container in the storage account ('vhds' by default)


comment on exported const ContainerKey should be of the form "ContainerKey ..."

houndci-bot · 2016-12-26T15:19:41Z

drivers/storage/azure/azure.go

+
+	// ID of subscription
+	SubscriptionIDKey = "subscriptionID"
+	// Name of resource group


comment on exported const ResourceGroupKey should be of the form "ResourceGroupKey ..."

houndci-bot · 2016-12-26T15:19:41Z

drivers/storage/azure/azure.go

+	StorageAccessKey = "storageAccessKey"
+	// TODO: add option to pass StorageURI
+
+	// ID of subscription


comment on exported const SubscriptionIDKey should be of the form "SubscriptionIDKey ..."

houndci-bot · 2016-12-26T15:19:41Z

drivers/storage/azure/azure.go

+
+	// Name of storage account
+	StorageAccount   = "storageAccount"
+	// Access key of storage account


comment on exported const StorageAccessKey should be of the form "StorageAccessKey ..."

houndci-bot · 2016-12-26T15:19:41Z

drivers/storage/azure/azure.go

+	// Path to application certificate in case of authorization via certificate
+	CertPathKey     = "certPath"
+
+	// Name of storage account


comment on exported const StorageAccount should be of the form "StorageAccount ..."

houndci-bot · 2016-12-26T15:19:41Z

drivers/storage/azure/azure.go

+	ClientIDKey     = "clientID"
+	// Secret of this application
+	ClientSecretKey = "clientSecret"
+	// Path to application certificate in case of authorization via certificate


comment on exported const CertPathKey should be of the form "CertPathKey ..."

houndci-bot · 2016-12-26T15:19:42Z

drivers/storage/azure/azure.go

+	TenantIDKey     = "tenantID"
+	// Application ID
+	ClientIDKey     = "clientID"
+	// Secret of this application


comment on exported const ClientSecretKey should be of the form "ClientSecretKey ..."

houndci-bot · 2016-12-26T15:19:42Z

drivers/storage/azure/azure.go

+	// Config keys:
+	// Directory ID
+	TenantIDKey     = "tenantID"
+	// Application ID


comment on exported const ClientIDKey should be of the form "ClientIDKey ..."

houndci-bot · 2016-12-26T15:19:42Z

drivers/storage/azure/azure.go

+	// Use https prefix by default or not for Azure URI's
+	DefaultUseHTTPS   = true
+
+	// Config keys:


comment on exported const TenantIDKey should be of the form "TenantIDKey ..."

houndci-bot · 2016-12-26T15:19:42Z

drivers/storage/azure/azure.go

+	// TagDelimiter separates tags from volume or snapshot names
+	TagDelimiter = "/"
+
+	// Use https prefix by default or not for Azure URI's


comment on exported const DefaultUseHTTPS should be of the form "DefaultUseHTTPS ..."

houndci-bot · 2016-12-26T15:27:53Z

drivers/storage/azure/azure.go

+	ContainerKey      = "container"
+	// UseHTTPS is a flag about use https or not for making Azure URI's
+	UseHTTPS          = "useHTTPS"
+	// TagKey


comment on exported const TagKey should be of the form "TagKey ..."

houndci-bot · 2016-12-26T15:27:53Z

drivers/storage/azure/azure.go

+	ContainerKey      = "container"
+	// UseHTTPS is a flag about use https or not for making Azure URI's
+	UseHTTPS          = "useHTTPS"
+	// TagKey


comment on exported const TagKey should be of the form "TagKey ..."

codecov-io · 2016-12-26T20:27:37Z

Codecov Report

Merging #372 into master will increase coverage by -0.36%.

@@            Coverage Diff             @@
##           master     #372      +/-   ##
==========================================
- Coverage   30.83%   30.47%   -0.36%     
==========================================
  Files          29       29              
  Lines        1719     1739      +20     
==========================================
  Hits          530      530              
- Misses       1131     1151      +20     
  Partials       58       58

Impacted Files	Coverage Δ
drivers/storage/vfs/executor/vfs_executor.go	`2.1% <ø> (-0.15%)`	❌
api/types/types_drivers_executor.go	`0% <ø> (ø)`	✅

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1abc39c...c1e6e4a. Read the comment docs.

akutz · 2017-01-18T17:32:16Z

Hi @Andrey-mp,

Do you mind rebasing this brand onto master? Thank you!

akutz

Hi @Andrey-mp,

This is a great PR -- thank you so very much the obvious time and effort you spent on this. I have some suggestions/questions to which I hope you're able to respond below.

One question I had regarding the entire PR is whether or not you used the GoMetalinter or at least go-fmt to ensure the files are consistent in their syntax/style?

Also, if possible can you try and limit the width of lines to 80 chars. I may be pedantic, but I find it makes things easier to read.

Thank you again!

akutz · 2017-01-18T17:33:08Z

Makefile

@@ -847,6 +847,7 @@ $1:
 	BUILD_TAGS="$$(BUILD_TAGS)" GOOS=$2 GOARCH=amd64 $$(MAKE) $$@
 $1-clean:
 	rm -f $1
+	rm -f $(EXECUTORS_GENERATED)


Good catch!

akutz · 2017-01-18T17:38:23Z

drivers/storage/azure/azure.go

+
+const (
+	// ConfigAZURE is a config key
+	ConfigAZURE                    = Name


Hi @Andrey-mp,

This is the only place I will mention this in my review, but why did you make AZURE capitalized throughout your use of it in constant/variable names? I checked, and it seems Microsoft doesn't use all-caps to write Azure, so I'm just wondering why you did?

Will be renamed.

akutz · 2017-01-18T17:48:52Z

drivers/storage/azure/executor/azure_executor.go

+	ctx types.Context,
+	opts types.Store) (string, error) {
+	// All possible device paths on Linux instances are /dev/sd[c-p]
+	letters := []string{


Since Go will keep malloc'd memory on the stack if it does not leak it probably makes more sense to keep this variable local to this function, but since mallocs are expensive in Go, I'm wondering if it makes sense to turn this into a package-scoped field that is allocated once on the heap.

akutz · 2017-01-18T17:53:31Z

drivers/storage/azure/executor/azure_executor.go

+	localDeviceMapping := localDevices.DeviceMap
+
+	for localDevice := range localDeviceMapping {
+		re, _ := regexp.Compile(`^/dev/` +


RegEx compilations are pretty expensive. The pieces of this regex are static, so why not use regexp.MustCompile once and assign the result to a package-level field. Compiling the regex in a loop seems pretty expensive.

akutz · 2017-01-18T18:15:24Z

drivers/storage/azure/storage/azure_storage.go

+
+	disks := *vm.StorageProfile.DataDisks
+	for i, disk := range disks {
+		if disk.Name != nil && *disk.Name == volumeID {


Hi @Andrey-mp,

I wasn't sure if you intended to do a case-sensitive comparison or not. If not then perhaps see strings.EqualFold(string, string) bool.

akutz · 2017-01-18T19:05:24Z

drivers/storage/azure/storage/azure_storage.go

+	return name
+}
+
+// Retrieve config arguments


Hi @Andrey-mp,

I'm not going to make this comment on each of the below functions, but this remark does apply to all of the code below related to retrieving configuration properties.

There is no need to check the environment variables directly. Please see this example for the EBS driver (ignore the backwards-compatability checks). The gofig package uses @spf13's Viper, which in turn enables the assignment of environment variables to defined configuration properties.

Let's look at line #87 in your azure.go:

r.Key(gofig.String, "", "", "", ConfigAZURESubscriptionIDKey)

In addition to notifying Gofig about the property azure.subscriptionID, the above function call instructs Viper to bind the property azure.subscriptionID to the CLI flag --azureSubscriptionID and the environment variable AZURE_SUBSCRIPTIONID. You can actually use Gofig to overwrite which CLI flag or environment variable the property is bound via the Key function's trailing variadic parameter.

For example, if we wanted to still use the config property azure.subscriptionID but change the CLI flag to azureSubID and the environment variable to AZURE_SUB_ID we could rewrite the above line like so:

r.Key(gofig.String, "", "", "", ConfigAZURESubscriptionIDKey, "azureSubID", "AZURE_SUB_ID")

Once the configuration properties are bound to a CLI flag or environment variable, invoking config.GetString(azure.ConfigAZURESubscriptionIDKey) checks first the environment variable, CLI flag argument, and finally a configuration property in that order.

akutz · 2017-01-18T19:06:04Z

drivers/storage/azure/storage/azure_storage.go

+	for _, blob := range *blobs {
+		volumeSD, error := d.toTypeVolume(ctx, &blob, attachments)
+		if error != nil {
+			ctx.WithError(error).Error("Failed to convert volume")


Hi @Andrey-mp,

Do you intend to have this error logged but not returned? If so perhaps log it as a warning instead?

codenrhoden

Echo'ing what @akutz said, thanks for the very thorough and complete PR! Hurray for tests.

Most of my comments are minor. My main feedback is that I don't think the filtering in toTypeVolume is necessary, and i also think that handling of the attachmentState can be handled by the framework instead of the driver.

codenrhoden · 2017-01-18T19:35:50Z

drivers/storage/azure/azure.go

+	// ContainerKey is a name of container in the storage account ('vhds' by default)
+	ContainerKey      = "container"
+	// UseHTTPS is a flag about use https or not for making Azure URI's
+	UseHTTPS          = "useHTTPS"


Looks like both UseHTTPS and StorageAccount are used only when setting up config keys. To be consistent with all the other keys used in this manner, I feel like these should be renamed to UseHTTPSKey and StorageAccountKey.

codenrhoden · 2017-01-18T19:44:09Z

drivers/storage/azure/executor/azure_executor.go

+
+const procPartitions = "/proc/partitions"
+
+var devRX = regexp.MustCompile(`^sd[a-z]$`)


This is looking for /dev/sd[a-z], but the comment in NextDevice says it can only be /dev/sd[c-p]. Are these both true?

If newly attached devices can only go to p, do we need to include all the way up to z?

Functionally I'm sure this works, it just jumps out to me as an inconsistency.

This looks the same to me. availLetters is still defining [c-z], the regex is looking for [a-z], and the first comment inside of NextDevice says that it can be [c-p].

codenrhoden · 2017-01-18T19:53:26Z

drivers/storage/azure/storage/azure_storage.go

+	}
+
+	writeHkey(hkey, &d.subscriptionID)
+	writeHkey(hkey, &d.resourceGroup)


resourceGroup doesn't seem to be a part of uniquely identifying a session. I don't think it needs to be a part of the hash key. It's never used in this function again.

codenrhoden · 2017-01-18T19:56:07Z

drivers/storage/azure/storage/azure_storage.go

+	iid := context.MustInstanceID(ctx)
+	return &types.Instance{
+		Name: iid.ID,
+		//Region:       iid.Fields[azure.InstanceIDFieldRegion],


Seems like this can be removed for now?

codenrhoden · 2017-01-18T19:59:15Z

drivers/storage/azure/storage/azure_storage.go

+		"size_in_bytes": size,
+	}
+
+	volume, _ := d.getVolume(ctx, volumeName, types.VolAttNone)


should an error here be ignored? What if the API failed and the check wasn't able to be performed?

codenrhoden · 2017-01-18T20:14:31Z

drivers/storage/azure/storage/azure_storage.go

+				ID:     blob.Metadata["microsoftazurecompute_vmname"],
+				Driver: d.name,
+			}
+			if attachments.Mine() {


Is it necessary for the Azure driver to do this filtering itself? I don't think that it is, and I don't think that it needs to handle setting attachState either. The framework handles this for you, here: https://github.com/codedellemc/libstorage/blob/release/0.4.0/api/server/router/volume/volume_routes.go#L131-L209

If a driver really does need to calculate attachState itself, it can do so. That will override the framework's default behavior of saying the volume is available if there are no attachments returned, it is attached if it is attached to the IID of the caller, or unavailable if attached to a different IID.

It is the intended behavior that attachState is not set if attachments.Requested() is false.

codenrhoden · 2017-01-18T20:17:51Z

drivers/storage/azure/utils/utils.go

+}
+
+// IsAzureInstance returns a flag indicating whether the executing host is an Azure
+// instance based on whether or not the metadata URL can be accessed.


The comment here does not match how the function is actually implemented. It does not use metadata that I can see.

codenrhoden · 2017-01-18T20:20:15Z

drivers/storage/azure/utils/utils.go

+		// UUID can be obtained as descried in https://azure.microsoft.com/en-us/blog/accessing-and-using-azure-vm-unique-id/
+		// but this code will use hostname as ID
+
+		hostname, err = os.Hostname()


How likely is this to happen? I'm guess that AZURE_INSTANCE_ID should normally be set. But falling back to a hostname seems iffy, unless Asure has requirements on whether hostnames have to be unique?

May be a non-issue, just curious.

Azure doesn't set AZURE_INSTANCE_ID inside VM-s. This variable was implemented for unit tests only.
Azure set hostname inside of VM equals to name of VM in Azure engine. And this name is unique in the engine.
So because there is no other way to obtain instance identifier (that is suitable for REST API) then we get it from the hostname.

houndci-bot · 2017-01-23T12:14:15Z

drivers/storage/azure/azure.go

+	// ContainerKey is a name of container in the storage account
+	// ('vhds' by default)
+	ContainerKey = "container"
+	// UseHTTPS is a flag about use https or not for making Azure URI's


comment on exported const UseHTTPSKey should be of the form "UseHTTPSKey ..."

This patchset adds support of Azure Cloud. There are several operations implemented in patchset: create, attach, detach, remove (all base operations for volumes).

Andrey-mp · 2017-01-24T10:22:22Z

Hi all.

We've thought that we fixed all your comments :)
Also I run gometalinter and it doesn't show me valuable errors.
We are waiting for next review.

codenrhoden

Just have the one remaining comment on device names. I'll start playing with this on Azure!

I'd like to see a first attempt made at adding the driver into the docs, at .docs/user-guide/storage-providers.md.

Should be interesting, as I haven't used Azure before.

codenrhoden · 2017-01-31T17:53:36Z

drivers/storage/azure/executor/azure_executor.go

+
+const procPartitions = "/proc/partitions"
+
+var devRX = regexp.MustCompile(`^sd[a-z]$`)


This looks the same to me. availLetters is still defining [c-z], the regex is looking for [a-z], and the first comment inside of NextDevice says that it can be [c-p].

Andrey-mp · 2017-01-31T20:25:30Z

Hi Travis,
We've got this code from the EBS driver that is very similar to Azure.
And in it code I see the same - NextDevice and LocalDevices has different letters...
And other executors have different behaviour for these functions also.

codenrhoden · 2017-02-17T19:50:44Z

Closing in deference to #421

houndci-bot reviewed Dec 26, 2016

View reviewed changes

akutz self-assigned this Jan 10, 2017

akutz added enhancement azure labels Jan 10, 2017

akutz requested changes Jan 18, 2017

View reviewed changes

akutz requested a review from codenrhoden January 18, 2017 19:11

codenrhoden requested changes Jan 18, 2017

View reviewed changes

Andrey-mp force-pushed the master-2 branch from f4bf380 to 517e7b4 Compare January 23, 2017 12:14

houndci-bot reviewed Jan 23, 2017

View reviewed changes

Support Azure for libstorage

99f709f

This patchset adds support of Azure Cloud. There are several operations implemented in patchset: create, attach, detach, remove (all base operations for volumes).

Andrey-mp force-pushed the master-2 branch from 517e7b4 to 99f709f Compare January 23, 2017 12:37

akutz changed the title ~~Support Azure for libstorage~~ Azure storage driver Jan 26, 2017

akutz mentioned this pull request Jan 26, 2017

Azure storage driver rexray/rexray#528

Closed

akutz added work-in-progress storagedrivers and removed enhancement labels Jan 26, 2017

codenrhoden requested changes Jan 31, 2017

View reviewed changes

fix comment in NextDevice

c1e6e4a

cduchesne mentioned this pull request Feb 6, 2017

Add Azure storage driver rexray/rexray#711

Closed

codenrhoden mentioned this pull request Feb 17, 2017

Azure Unmanaged Disk support #421

Merged

codenrhoden closed this Feb 17, 2017


		const procPartitions = "/proc/partitions"

		var devRX = regexp.MustCompile(`^sd[a-z]$`)

Azure storage driver #372

Azure storage driver #372

Conversation

Andrey-mp commented Dec 26, 2016

CLAassistant commented Dec 26, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-io commented Dec 26, 2016 • edited Loading

Codecov Report

akutz commented Jan 18, 2017 • edited Loading

akutz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akutz Jan 18, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codenrhoden left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Andrey-mp commented Jan 24, 2017

codenrhoden left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Andrey-mp commented Jan 31, 2017

codenrhoden commented Feb 17, 2017

CLAassistant commented Dec 26, 2016 •

edited

Loading

codecov-io commented Dec 26, 2016 •

edited

Loading

akutz commented Jan 18, 2017 •

edited

Loading

akutz Jan 18, 2017 •

edited

Loading