Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migration from Classic to Basic/Standard/Premium takes a long time and/or fails #67

Closed
SteveLas opened this issue Nov 3, 2017 · 5 comments

Comments

@SteveLas
Copy link
Contributor

SteveLas commented Nov 3, 2017

We've been working to transition from Classic Azure Container Registries, which use a storage account in the customers subscription, to a fully managed Azure Container Registry.
For more info on the different SKUs and why you'd want to migrate to the managed Basic, Standard and Premium SKUs, please see http://aka.ms/acr/skus

We've found customers with large or older registries are either taking a long time, 24+ hours, or they fail.
We've identified throttling and bugs related to the blob copying of storage accounts between a customer's subscription and the managed ACR storage accounts.

We're working with the storage team to resolve 2 underlying bugs. Unfortunately, since Storage is a core service of Azure, rolling out a change takes time to adhere to safe deployment practices.

While SKU migration is available in the az CLI, we have not yet enabled migration in the Azure Portal. Once we have more customer validations and confidence, we'll enable the portal.

The bugs impact the speed and reliability of the copy, but do not impact the source storage account. While migration is running, docker push is blocked. However, all docker login and pull operations continue to function. Since migration is a copy of the Classic storage account, there is no risk of data loss. If the migration fails, your classic registry will be fully intact.

If you wish to take advantage of the managed registry features, and your registry has fewer than 500 image/tags, you can initiate
az acr update --name standard. Once in the managed registry, you can easily move between basic, standard and classic.

If migration takes longer than 3 hours, it's likely the migration will not succeed. Please contact support and request cancelling the migration. Please provide the login url: yourregistry.azurecr.io

We realize the critical importance of providing a scalable, reliable container registry and we're working hard to provide the best possible experience. We apologize for any inconvenience and appreciate our patient, early adopter customers that have been impacted.

We'll provide further updates as we have the fix rolled out.
Steve

@SteveLas
Copy link
Contributor Author

A quick update:
Just prior to Thanksgiving, we had a fix we were rolling out. The fix is still a temporary workaround until the permanent storage fix is deployed. However, as customers depend on Azure for their holiday shopping, we had an Azure wide blackout for deployments for Black Friday and the week of cyber monday. We plan to start the deployment December 4th, which will take the week to complete as we follow "Safe Deployment Practices", where we gradually roll out to concentric rings of regions. While it's possible your region will be available sooner than others, we need to wait until all regions are complete before we can enable the portal migration, through the overview tab, clicking update.
We appreciate your patience, and hope you understand we're working to "balance the needs of the many, over the needs of the few."
Steve

@SteveLas
Copy link
Contributor Author

Another quick update.
As we tested the upgrade with several customers, we found the workaround had a higher CPU utilization. To minimize the impact, we're completing our VM Scaleset work to auto-scale the registry backend if multiple customers are upgrading per region and putting a larger tax on the system.
We hope to have this completed by weeks end.
As noted above, you can use the az acr update command prior to the portal update. We just expect the Portal update to trigger far more users as we also promote the migration capability.
Thanks,
Steve

@DonMartin76
Copy link

For a approx. 200 GB upgrade from classic to Premium, what's the typical duration of the upgrade process? I just triggered it, and I am beginning to think that it was perhaps a bad idea, if it would take 24h+.

@sajayantony
Copy link
Contributor

@DonMartin76 - Did you face any issue during the migration?

@SteveLas
Copy link
Contributor Author

SteveLas commented Dec 15, 2017

I'm happy to report the portal update has been rolled out. Customers with classic registries will be invited to upgrade to a managed registry. Upgrading through the portal is treated a bit special, when compared to the CLI. Using the az acr update CLI, you specify the target SKU. If your storage account doesn't fit, it will fail. You can, of course, upgrade to premium, then scale back down.
Using the portal, we will automatically pick the smallest SKU, based on the amount of storage you have.
Unless @DonMartin76 or others provide additional feedback we haven't seen, I'm hoping this is now resolved.
Thanks for your patience, and the customers that helped find and validated the fixes.
Steve

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants