Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Network/VPNGateway] deployment operation failed due to an intermittent error #21164

Open
wuxu92 opened this issue Oct 18, 2022 · 43 comments
Open
Labels
Network - VPN Gateway Service Attention Workflow: This issue is responsible by Azure service team.

Comments

@wuxu92
Copy link

wuxu92 commented Oct 18, 2022

Brief

VPN gateway deployment keeps failing with helpless information

Description

When creating a virtual network gateway for VPN Type with route-based policy, the create operation returns with server-side error without debugging/helpful information to investigate. I have tried with different SKU types and modified the customeRoute prefix value, which makes no different and all failed.

related API:

example

gateway request payload

{
    "location": "eastus",
    "properties": {
        "activeActive": false,
        "customRoutes": {
            "addressPrefixes": [
                "101.168.0.6/32"
            ]
        },
        "enableBgp": false,
        "enablePrivateIpAddress": true,
        "gatewayType": "Vpn",
        "ipConfigurations": [
            {
                "name": "vnetGatewayConfig",
                "properties": {
                    "privateIPAllocationMethod": "Dynamic",
                    "publicIPAddress": {
                        "id": "/subscriptions/xxx-xxxx/resourceGroups/xxx-rg/providers/Microsoft.Network/publicIPAddresses/vpnpubip001"
                    },
                    "subnet": {
                        "id": "/subscriptions/xxx-xxxx/resourceGroups/xxx-rg/providers/Microsoft.Network/virtualNetworks/vpngw001/subnets/GatewaySubnet"
                    }
                }
            }
        ],
        "sku": {
            "name": "VpnGw3AZ",
            "tier": "VpnGw3AZ"
        },
        "vpnType": "RouteBased"
    }
}

reponse error message:

{
    "error": {
        "code": "VmssGatewayDeploymentFailed",
        "details": [],
        "message": "The gateway deployment operation failed due to an intermittent error. Please try again."
    },
    "status": "Failed"
}
@ghost ghost added the needs-triage Workflow: This is a new issue that needs to be triaged to the appropriate team. label Oct 18, 2022
@JackTn JackTn added Network - VPN Gateway Service Attention Workflow: This issue is responsible by Azure service team. labels Oct 20, 2022
@ghost ghost removed the needs-triage Workflow: This is a new issue that needs to be triaged to the appropriate team. label Oct 20, 2022
@ghost
Copy link

ghost commented Oct 20, 2022

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @vpngwsuppgithub.

Issue Details

Brief

VPN gateway deployment keeps failing with helpless information

Description

When creating a virtual network gateway for VPN Type with route-based policy, the create operation returns with server-side error without debugging/helpful information to investigate. I have tried with different SKU types and modified the customeRoute prefix value, which makes no different and all failed.

related API:

example

gateway request payload

{
    "location": "eastus",
    "properties": {
        "activeActive": false,
        "customRoutes": {
            "addressPrefixes": [
                "101.168.0.6/32"
            ]
        },
        "enableBgp": false,
        "enablePrivateIpAddress": true,
        "gatewayType": "Vpn",
        "ipConfigurations": [
            {
                "name": "vnetGatewayConfig",
                "properties": {
                    "privateIPAllocationMethod": "Dynamic",
                    "publicIPAddress": {
                        "id": "/subscriptions/xxx-xxxx/resourceGroups/xxx-rg/providers/Microsoft.Network/publicIPAddresses/vpnpubip001"
                    },
                    "subnet": {
                        "id": "/subscriptions/xxx-xxxx/resourceGroups/xxx-rg/providers/Microsoft.Network/virtualNetworks/vpngw001/subnets/GatewaySubnet"
                    }
                }
            }
        ],
        "sku": {
            "name": "VpnGw3AZ",
            "tier": "VpnGw3AZ"
        },
        "vpnType": "RouteBased"
    }
}

reponse error message:

{
    "error": {
        "code": "VmssGatewayDeploymentFailed",
        "details": [],
        "message": "The gateway deployment operation failed due to an intermittent error. Please try again."
    },
    "status": "Failed"
}
Author: wuxu92
Assignees: -
Labels:

Network - VPN Gateway, Service Attention, needs-triage

Milestone: -

@mayank-reynencourt
Copy link

Hi,

i'm also trying to create VNG and face same issue , any update on this ?

@el-memer
Copy link

el-memer commented Nov 2, 2022

Hi,
I'm also facing same issue here (Region West Europe), I've opened a support ticket and will send updates here if I have some.

@birdnathan
Copy link

Same here - West Europe. Please fix!

@FletchAD
Copy link

FletchAD commented Nov 2, 2022

Same here - UKSouth and UKWest, tried with VpnGw1 & VpnGw1AZ, interestingly the basic SKU seems to work ok, but that's not what I need.

@slaffka-vlasov
Copy link

The same is for uswest3 with VpnGw1 & VpnGw2. Basic doesn't work as well

@FrankMormino
Copy link

Same thing here - for west Europe as well.

@FrankMormino
Copy link

Hi, I'm also facing same issue here (Region West Europe), I've opened a support ticket and will send updates here if I have some.

Same here - trying the same in West Europe - any luck on the support ticket side?

@ghost
Copy link

ghost commented Nov 2, 2022

Same issue here. Happening in West Europe trying to deploy a VpnGw2AZ. Tried deployment via Terraform and the Portal.

@BevanSin
Copy link

BevanSin commented Nov 2, 2022

Note there is an issue currently with Azure Key Vault that may be causing this problem - if you look in the Service Health page there is an alert there that is affecting downstream Azure services like VPN Gateway.

@birdnathan
Copy link

From Azure support:
“Hi Nathan

Thank you for contacting Microsoft Azure Networking support.

Please know that we are currently investigating an ongoing global issue regarding Virtual Network Gateway (VPN Gateway) deployments and we will get back to you with updates once this has been mitigated.”

Seems the key vault issue is wiping out network services. The status website need to reflect this and not just list key vault as impacted

@haciz
Copy link

haciz commented Nov 2, 2022

Hello,

Indeed joining the list to report an issue, neither the empty PUT request is helping

Set-AzVirtualNetworkGateway: Long running operation failed with status 'Failed'. Additional Info:'The gateway deployment operation failed due to an intermittent error. Please try again.' StatusCode: 200 ReasonPhrase: OK Status: Failed ErrorCode: VmssGatewayDeploymentFailed ErrorMessage: The gateway deployment operation failed due to an intermittent error. Please try again.

@imkevinjones
Copy link

Same issue here... This has been happening all day.

@shanyuen
Copy link

shanyuen commented Nov 3, 2022

Yesterday I bought a subscription and created a fresh virtual network in southeast asia.
But still error.

Status: Conflict
{"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.","details":[{"code":"VmssGatewayDeploymentFailed","message":"The gateway deployment operation failed due to an intermittent error. Please try again."}]}
SKU: VpnGw1
Correlation id: 2f5880ba-a8b2-4184-801e-a5b8cea6f3b4

@FletchAD
Copy link

FletchAD commented Nov 3, 2022

After this confirmation at 3rd Nov @ 05:52 UTC https://status.azure.com/en-us/status/history/ I was able to provision VpnGw1 in UKSouth this morning.

@ghost
Copy link

ghost commented Nov 3, 2022

Seems to be mitigated. Can also deploy the VGW now.

@el-memer
Copy link

el-memer commented Nov 3, 2022

Got an answer from the support team.

I apologize for the inconvenience that was caused but as we checked, there is a service disruption in all regions, which impacts the deployments of the VPN Gateway resources. We hope that it might be resolved by the end of the day. Our internal team are actively investigating and working on a solution, however, currently we do not have a specific resolution date or time.

EDIT: My deployment succeeded this morning in Region West Europe for a VpnGw1 Route based !

I'll let you know if I have some updates from support, but indeed this seems to be mitigated.

@haciz
Copy link

haciz commented Nov 3, 2022

Indeed it works on Express route gateways as well West Europe.

@slaffka-vlasov
Copy link

Confirmed creation of VpnGw1 in west us 3

@laglergruener
Copy link

laglergruener commented Dec 11, 2023

Hi,
we‘ve currently the same issue, mentioned above in WestEurope for AZ gateways.
Is the problem still available?
Thanks,
Hannes

@rik-v
Copy link

rik-v commented Jan 11, 2024

This issue seems to be current again. We're currently unable to deploy any VPN gateway to any subscription or any tenant.
Tried 3 different tenants, 4 different subscriptions, 2 types of SKU (VpnGw1 and VpnGw2), all in West Europe.
We noticed this starting january 9, 2024, however, it is not unlikely this issue persisted before that date.

No mentions of this on the service health pages. So, curious as to what's causing it this time.

@soufianerabi
Copy link

Yeah, for the past three days, we've been facing the same issue in our organization. The deployment has failed multiple times, even after many retries. Unfortunately, Microsoft support wasn't helpful. We're still trying to deploy this resource, but it continues to fail.

@tanarchytan
Copy link

tanarchytan commented Jan 15, 2024

Also a problem on my side. tried it since 09 of January. Region West Europe with SKU VPNGw2

{
"code": "DeploymentFailed",
"target": "/subscriptions/[...]/resourceGroups/rg-net-hub/providers/Microsoft.Resources/deployments/Microsoft.Template-20240115105923",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "ResourceDeploymentFailure",
"target": "/subscriptions/[...]/resourceGroups/rg-net-hub/providers/Microsoft.Network/virtualNetworkGateways/vpng-[...]-prod",
"message": "The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'."
}
]
}

@rik-v
Copy link

rik-v commented Jan 15, 2024

@soufianerabi @tanarchytan, I tried deploying in north europe last friday (jan. 12th, 2024), which completes successfully, which makes me believe Microsoft simply has some serious resource issues in the west europe region.

It's not the first time (or sign of) this is happening, a few months back we were unable to roll out other resources (VM's, VMSS'es (for AKS)) because apparently the resource was unavailable in a specific zone in west europe.
We ended up deploying to "just" 2 zones instead of 3 (still ok, but we wanted 3 :)).
To be clear: we haven't checked if those resources have become available again in all zones since then, so perhaps things improved (although I strongly doubt that considering the fact we can't deploy VPN to west europe)

Would be helpful if Microsoft would simply clarify the current state of west europe, if there's a capacity issue, that's annoying, but I'd rather have them tell us then having to find out this way...

@TGosselink
Copy link

Some here, VPN deploy in West Europe:
The gateway deployment operation failed due to an intermittent error. Please try again. (Code: VmssGatewayDeploymentFailed)

@lgriffithsdoherty
Copy link

Me too

@foxmeyson
Copy link

Microsoft is not good, there are problems all the time. That's horrible

@rbnmk
Copy link

rbnmk commented Jan 16, 2024

Currently facing same issue in West Europe for VpnGw1 and VpnGw1AZ

@mtc3net
Copy link

mtc3net commented Jan 16, 2024

Me too

@lgriffithsdoherty
Copy link

lgriffithsdoherty commented Jan 16, 2024

Screenshot 2024-01-16 143451

Microsoft have called me to say that they are not sure what the issue is and it could be months before it is resolved. The attached states capacity constraints but still no resolution in site just a warning set up to say capacity constraints wont allow you to build gateways in west europe for the foreseeable.

@rik-v
Copy link

rik-v commented Jan 16, 2024

@lgriffithsdoherty , thank you for sharing this! :)
This very unfortunate news, especially for new customers :(
Their proposed (temporary) solution, although understandable, is unlikely to help much though as you can only deploy a gateway to a VNet in the same region. Which means you'll have to move your virtual network too, or you have to start managing 2. Also, this might incur outbound network traffic cost (because resources might be in different regions).

I hope Microsoft makes quick work of expanding their capacity in west europe :/

@atovivan
Copy link

atovivan commented Jan 16, 2024

This is a serious blocking issue for us as well, that will possibly postponed our go-live date. I tried to move only the vnet and vpn gateway to a different region but without success, possible we will need to move all network related resources or the entire infrastructure to a different region. For now we are using a bastion VM but that is not a cheap solution as on one VM only two people can work simultaneously.

Looks like Microsoft doesn't care much about this problem.

@rik-v
Copy link

rik-v commented Jan 16, 2024

@atovivan Instead of moving the VNet, you might want to try creating a second VNet for just the VPN, then peer that VNet with the 'original' VNet, that should work? (otherwise, perhaps a custom solution with a VM + custom VPN software (like wireguard) could work for your needs? However crappy that is, it might be 'a way out')

@efcorpa
Copy link

efcorpa commented Jan 17, 2024

Cannot believe this is happening and even that Microsoft says it could be months before it is resolved.
Can anybody confirm it works with a given SKU?
Is there any workaround other than creating a VNet in another region and peer it with the one at WE?

Another question is: is this an intermitent error and perhaps if I'm lucky I will get my VPN deployed, or does it fail every time?

@redoz
Copy link

redoz commented Jan 18, 2024

@efcorpa FWIW I tried deploying every SKU multiple times yesterday and it failed every single time.

@jordyvpaassen
Copy link

@efcorpa I've the same issue by different customers, the only one option is to make a resource group with a vpn gateway and a vnet in a other region and make a peer between your current vnet and the one that is deployed in a other region. If you have questions let me know

@mestief
Copy link

mestief commented Jan 24, 2024

The problem unfortunately still exists 👎
Guess I'll have to use North Europe for now.

@Agger1995
Copy link

I have been retrying every day for a few weeks at this point, and only just today was I able to succesfully deploy a VPN Gateway to westeurope, with the SKU VpnGW1.
Something has been fixed, but I can't find any official information on the incident anywhere.

@redoz
Copy link

redoz commented Jan 29, 2024

I have been retrying every day for a few weeks at this point, and only just today was I able to succesfully deploy a VPN Gateway to westeurope, with the SKU VpnGW1. Something has been fixed, but I can't find any official information on the incident anywhere.

Probably just enough people gave up on westeurope and picked a different region. We moved everything to Sweden central, not being able to reliably deploy things is not an risk we're willing to take.

@MarkTallentire
Copy link

Over a year later and this issue still persists. At a minimum it'd be nice if it was capacity issues that this is reflected in the error message.

@wuxu92
Copy link
Author

wuxu92 commented Mar 19, 2024

Hi @MarkTallentire, are you still experiencing this issue in West Europe region? the capacity issue should have been resolved already.

@MarkTallentire
Copy link

MarkTallentire commented Mar 19, 2024 via email

@JacopoNicolosiAlten
Copy link

same problem in West Europe for VPNGw2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Network - VPN Gateway Service Attention Workflow: This issue is responsible by Azure service team.
Projects
None yet
Development

No branches or pull requests