Copy VM page blob from one Azure subscription to another subscription

[jonbullock]

I'm trying to use the Azure Storage SDK for Java to copy the page blob of an Azure VM (that is Stopped and Deallocated) from one Azure subscription to another.

Here's the code I'm using:

public class BlobCopyExampleClean {

    public static final String sourceStorageConnectionString =
            "DefaultEndpointsProtocol=https;"
            + "AccountName=sourceStorageAccount;"
            + "AccountKey=key123";

    public static final String destinationStorageConnectionString =
            "DefaultEndpointsProtocol=https;"
            + "AccountName=destinationStorageAccount;"
            + "AccountKey=key321";

    public static void main(String[] args) {

        try {
            CloudStorageAccount srcAccount = CloudStorageAccount.parse(sourceStorageConnectionString);
            CloudBlobClient srcSrvClient = srcAccount.createCloudBlobClient();
            CloudBlobContainer srcContainer = srcSrvClient.getContainerReference("vhds");

            CloudStorageAccount destAccount = CloudStorageAccount.parse(destinationStorageConnectionString);
            CloudBlobClient destSrvClient = destAccount.createCloudBlobClient();
            CloudBlobContainer destContainer = destSrvClient.getContainerReference("vhds");

            boolean result = destContainer.createIfNotExists();

            CloudBlob srcBlob = srcContainer.getPageBlobReference("testvm-2015-11-06.vhd");
            if (srcBlob.exists()) {
                CloudBlob destBlob = destContainer.getPageBlobReference("testvm-2015-11-06-copied.vhd");
                System.out.println("Starting blob copy...");
                String copyJobId = destBlob.startCopyFromBlob(srcBlob);

                CopyState copyState = destBlob.getCopyState();

                while (copyState.getStatus().equals(CopyStatus.PENDING)) {
                    System.out.println("... copying ...");
                    Thread.sleep(30000);
            }

                System.out.println("Copy complete, status was: " + copyState.getStatus() + "!");
            } else {
                System.out.println("Source blob does not exist!");
            }
        } catch (InvalidKeyException e) {
            e.printStackTrace();
        } catch (URISyntaxException e) {
            e.printStackTrace();
        } catch (StorageException e) {
            e.printStackTrace();
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }
}

No matter what I try I always get the following error returned to me:

Starting blob copy...
com.microsoft.azure.storage.StorageException: The specified resource does not exist.
    at com.microsoft.azure.storage.StorageException.translateException(StorageException.java:89)
    at com.microsoft.azure.storage.core.StorageRequest.materializeException(StorageRequest.java:305)
    at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:175)
    at com.microsoft.azure.storage.blob.CloudBlob.startCopy(CloudBlob.java:883)
    at com.microsoft.azure.storage.blob.CloudBlob.startCopyFromBlob(CloudBlob.java:788)
    at com.company.azure.storage.BlobCopyExampleClean.main(BlobCopyExampleClean.java:44)

I've tried using v1.3.1, v3.1.0 & v4.0.0 of the SDK library and get the same error using both blob.startCopyFromBlob() (v1.3.1 & v3.1.0) and blob.startCopy() (v4.0.0).

The page blob in question has no lease, as the Azure Management Portal says the lease status is "Unlocked" and it does exist, even the Azure API confirms this by entering the code block at line 42.

I've tried copying the blob to another storage account within the same subscription and that gives the same error too.

Looking at the exception in more detail the error code is "CannotVerifyCopySource".

What am I doing wrong?

[mirobers]

To copy a blob across accounts, you need to use a SAS token for the source or mark the source container for public access. See the "Authorization" section of the following page:
https://msdn.microsoft.com/en-us/library/azure/dd894037.aspx

So for example your code could generate a SAS token granting read access to the source blob, append it to the source blob URL's query string, and then start the copy from the URL.

[emgerner-msft]

mirobers response is spot on. If you're looking for the specific APIs, check out generateSharedAccessSignature on the blob object. You can use the token returned by this to either make a new CloudStorageAccount and follow the code flow above to get a blob reference from that, or append it to the blob URL and use the CloudBlockBlob(URL) constructor to directly get a blob reference.

Your code has one additional problem in that getCopyState does not make a service call. This is just getting the blob copy state previously set by startCopy. Inside your while loop you should try using downloadAttributes instead which will actually do a service call to get the updated copy information from the blob.

[jonbullock]

Thanks, using the following code works:

                SharedAccessBlobPolicy policy = new SharedAccessBlobPolicy();
                GregorianCalendar calendar = new GregorianCalendar(TimeZone.getTimeZone("UTC"));
                calendar.setTime(new Date());
                policy.setSharedAccessStartTime(calendar.getTime());
                calendar.add(Calendar.HOUR, 1);
                policy.setSharedAccessExpiryTime(calendar.getTime());
                policy.setPermissions(EnumSet.of(SharedAccessBlobPermissions.READ));
                String sas = srcBlob.generateSharedAccessSignature(policy, null);
//              System.out.println("SAS = " + sas);
                URI newUri = new URI(srcBlob.getUri() + "?" + sas);
//              System.out.println("URI = " + newUri.toString());
                CloudBlob srcBlob2 = new CloudPageBlob(newUri);

                System.out.println("Does blob exist? = " + srcBlob2.exists());
                String copyJobId = destBlob.startCopy(newUri);

However this approach doesn't seem to like being used from behind a proxy (that doesn't require authentication).

When going via a proxy (using -Dhttps.proxyHost & -Dhttps.proxyPort) I get the following response on line System.out.println("Does blob exist? = " + srcBlob2.exists()); and the same for the startCopy() method call if I comment out the exist check:

com.microsoft.azure.storage.StorageException: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
    at com.microsoft.azure.storage.StorageException.translateFromHttpStatus(StorageException.java:175)
    at com.microsoft.azure.storage.StorageException.translateException(StorageException.java:94)
    at com.microsoft.azure.storage.core.StorageRequest.materializeException(StorageRequest.java:305)
    at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:175)
    at com.microsoft.azure.storage.blob.CloudBlob.exists(CloudBlob.java:1795)
    at com.microsoft.azure.storage.blob.CloudBlob.exists(CloudBlob.java:1782)
    at com.microsoft.azure.storage.blob.CloudBlob.exists(CloudBlob.java:1757)
    at com.company.azure.storage.BlobCopyExampleCleanv400.main(BlobCopyExampleCleanv400.java:78)

I tried setting the proxy option on OperationContext instead but the code hung on the first service call (eventually it returns a Connection timed out StorageException), which in my case was creating the destination container if it didn't already exist. Is this supported via a proxy? Or should I raise a separate issue for this?

[emgerner-msft]

Proxy on OperationContext and proxy via JVM should work, and we've tested it works with a particular proxy. If you're getting auth failures I'd expect that your proxy is adding or removing headers which affects our authentication (hence 'Server failed to authenticate'). Check if this is the case with your proxy and which particular headers it's adding/removing and we can maybe try to help you find a work around.

[jonbullock]

Thanks Emily, I suspected the proxy would get in the way somehow. Don't suppose you have any suggestion on how I could inspect my request after it's passed through the proxy in question? Unfortunately I've got zero control over the proxy.

[emgerner-msft]

There's a variety of tooling available for this on the internet. Wireshark is probably the most generic across platforms and has the most flexibility. I'm not sure exactly how it interacts with proxies but given how low-level it is I'd say it's a good shot.

[jonbullock]

Thanks for the suggestion Emily, unfortunately I cannot intercept my request using Wireshark once it's passed through the proxy. If I get any joy finding out what's happening with the proxy I'll let you know.