Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Once a private endpoint has been assigned to a subnet; it cannot be removed #39634

Closed
tonyt73 opened this issue Sep 27, 2019 · 27 comments
Closed

Comments

@tonyt73
Copy link

tonyt73 commented Sep 27, 2019

https://github.com/MicrosoftDocs/azure-docs/blob/master/articles/private-link/create-private-endpoint-portal.md

When you create a new subnet and assign a private endpoint to it.
If you then remove the private endpoint, private link and the association to the vnet subnet and then try to add service endpoint definitions, you will get the following error.
Set-AzVirtualNetwork : Private endpoint network policies cannot be enabled on private endpoint subnet
Even when the subnet clearly has no private endpoints and the private endpoint and link are deleted.
Also trying to change the PrivateEndpointNetworkPolicies flag back to Enabled will also result in the same error.


Document Details

Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

@PRMerger9 PRMerger9 added the Pri3 label Sep 27, 2019
@tonyt73
Copy link
Author

tonyt73 commented Sep 27, 2019

Temporary solution:

  • Create a temp subnet
  • Move all the nic's (ipconfigurations) from the affected subnet to the temp subnet
  • Delete the affected subnet
  • Recreate the affect subnet with the service endpoints and nsg
  • Move all the nic's (ipconfigurations) back to the recreated subnet

@msrini-MSFT
Copy link
Contributor

@tonyt73 , Can you post the output of the below PS command?

$VNET= Get-AzVirtualNetwork -Name "Name of your Vnet" -ResourceGroupName "RG Name"

Get-AzVirtualNetworkSubnetConfig -Name "Name of your Subnet" -VirtualNetwork $VNET

@msrini-MSFT
Copy link
Contributor

@tonyt73 , do you have any update on this issue?

@tonyt73
Copy link
Author

tonyt73 commented Sep 28, 2019

You'll notice that the Internal subnet does not have any private endpoints defined.
From either with the virtual network object or the virtual network subnet config object.

Yet, Azure portal and the PS Set-AzVirtualNetwork commands seem to think it does.

Also worth mentioning that you can't change the policy items either

"PrivateEndpointNetworkPolicies": "Disabled",
"PrivateLinkServiceNetworkPolicies": "Enabled"

Trying to change Disabled to Enabled with also fail.


Name                   : myvnet
ResourceGroupName      : myvnetresourcegroup
Location               : northcentralus
Id                     : /subscriptions/{subid}/resourceGroups/myvnetresourcegroup/providers/Microsoft.Network/virtualNetworks/myvnet
ResourceGuid           : guid
ProvisioningState      : Succeeded
AddressSpace           : {
                           "AddressPrefixes": [
                             "10.112.0.0/16"
                           ]
                         }
DhcpOptions            : {
                           "DnsServers": [
                             "10.112.2.4",
                             "10.112.2.5"
                           ]
                         }
Subnets                : [
                           {
                             "Delegations": [],
                             "Name": "GatewaySubnet",
                             "Id": "/subscriptions/{subid}/resourceGroups/myvnetresourcegroup/providers/Microsoft.Network/virtualNetworks/myvnet/subnets/GatewaySubnet",
                             "AddressPrefix": [
                               "10.112.100.224/27"
                             ],
                             "IpConfigurations": [
                               {
                                 "Id": "/subscriptions/{subid}/resourceGroups/myvnet/providers/Microsoft.Network/virtualNetworkGateways/gateway/ipConfigurations/default"
                               }
                             ],
                             "ServiceAssociationLinks": [],
                             "ResourceNavigationLinks": [],
                             "ServiceEndpoints": [],
                             "ServiceEndpointPolicies": [],
                             "PrivateEndpoints": [],
                             "ProvisioningState": "Succeeded",
                             "PrivateEndpointNetworkPolicies": "Enabled",
                             "PrivateLinkServiceNetworkPolicies": "Enabled"
                           },
                           {
                             "Delegations": [],
                             "Name": "ServiceHosts",
                             "Id": "/subscriptions/{subid}/resourceGroups/myvnetresourcegroup/providers/Microsoft.Network/virtualNetworks/myvnet/subnets/ServiceHosts",
                             "AddressPrefix": [
                               "10.112.100.0/29"
                             ],
                             "IpConfigurations": [
                               {
                                 "Id": "/subscriptions/{subid}/resourceGroups/myvm/providers/Microsoft.Network/networkInterfaces/myvmnic/ipConfigurations/ipconfig1"
                               }
                             ],
                             "ServiceAssociationLinks": [],
                             "ResourceNavigationLinks": [],
                             "NetworkSecurityGroup": {
                               "Id": "/subscriptions/{subid}/resourceGroups/myvnetresourcegroup/providers/Microsoft.Network/networkSecurityGroups/mynsg"
                             },
                             "ServiceEndpoints": [],
                             "ServiceEndpointPolicies": [],
                             "PrivateEndpoints": [],
                             "ProvisioningState": "Succeeded",
                             "PrivateEndpointNetworkPolicies": "Enabled",
                             "PrivateLinkServiceNetworkPolicies": "Enabled"
                           },
                           {
                             "Delegations": [],
                             "Name": "External",
                             "Id": "/subscriptions/{subid}/resourceGroups/myvnetresourcegroup/providers/Microsoft.Network/virtualNetworks/myvnet/subnets/External",
                             "AddressPrefix": [
                               "10.112.1.0/24"
                             ],
                             "IpConfigurations": [
                               {
                                 "Id": "/subscriptions/{subid}/resourceGroups/mywebvm/providers/Microsoft.Network/networkInterfaces/mywebvmnic/ipConfigurations/ipconfig1"
                               },
                               {
                                 "Id": "/subscriptions/{subid}/resourceGroups/myvm2/providers/Microsoft.Network/networkInterfaces/myvm2nic/ipConfigurations/ipconfig1"
                               },
                             ],
                             "ServiceAssociationLinks": [],
                             "ResourceNavigationLinks": [],
                             "NetworkSecurityGroup": {
                               "Id": "/subscriptions/{subid}/resourceGroups/myvnetresourcegroup/providers/Microsoft.Network/networkSecurityGroups/NCU_NSG_External"
                             },
                             "ServiceEndpoints": [
                               {
                                 "ProvisioningState": "Succeeded",
                                 "Service": "Microsoft.AzureCosmosDB",
                                 "Locations": [
                                   "*"
                                 ]
                               },
                               {
                                 "ProvisioningState": "Succeeded",
                                 "Service": "Microsoft.Sql",
                                 "Locations": [
                                   "northcentralus"
                                 ]
                               },
                               {
                                 "ProvisioningState": "Succeeded",
                                 "Service": "Microsoft.ServiceBus",
                                 "Locations": [
                                   "*"
                                 ]
                               }
                             ],
                             "ServiceEndpointPolicies": [],
                             "PrivateEndpoints": [],
                             "ProvisioningState": "Succeeded",
                             "PrivateEndpointNetworkPolicies": "Disabled",
                             "PrivateLinkServiceNetworkPolicies": "Disabled"
                           },
                           {
                             "Delegations": [],
                             "Name": "Internal",
                             "Id": "/subscriptions/{subid}/resourceGroups/myvnetresourcegroup/providers/Microsoft.Network/virtualNetworks/myvnet/subnets/Internal",
                             "AddressPrefix": [
                               "10.112.2.0/24"
                             ],
                             "IpConfigurations": [
                               {
                                 "Id": "/subscriptions/{subid}/resourceGroups/NCU_RG_DNS/p roviders/Microsoft.Network/networkInterfaces/dnsvm/ipConfigurations/ipconfig1"
                               },
                             ],
                             "ServiceAssociationLinks": [],
                             "ResourceNavigationLinks": [],
                             "NetworkSecurityGroup": {
                               "Id": "/subscriptions/{subid}/resourceGroups/myvnetresourcegroup/providers/Microsoft.Network/networkSecurityGroups/NCU_NSG_Internal"
                             },
                             "ServiceEndpoints": [],
                             "ServiceEndpointPolicies": [],
                             "PrivateEndpoints": [],
                             "ProvisioningState": "Succeeded",
                             "PrivateEndpointNetworkPolicies": "Disabled",
                             "PrivateLinkServiceNetworkPolicies": "Disabled"
                           }
                         ]
EnableDdosProtection   : false
DdosProtectionPlan     : null



> Get-AzVirtualNetworkSubnetConfig -name Internal -VirtualNetwork $vn

Name                              : Internal
Id                                : /subscriptions/{subid}/resourceGroups/myvnetresourcegroup/providers/Microsoft.Network/virtualNetworks/myvnet/subnets/Internal
ProvisioningState                 : Succeeded
AddressPrefix                     : {10.112.2.0/24}
IpConfigurations                  : [
                                      {
                                        "Id": "/subscriptions/{subid}/resourceGroups/NCU_RG_DNS/providers/Microsoft.Network/networkInterfaces/dnsvm/ipConfigurations/ipconfig1"
                                      }
                                    ]
ResourceNavigationLinks           : []
ServiceAssociationLinks           : []
NetworkSecurityGroup              : {
                                      "Id": "/subscriptions/{subid}/resourceGroups/NCU_RG_Network/providers/Microsoft.Network/networkSecurityGroups/NCU_NSG_Internal"
                                    }
RouteTable                        : null
NatGateway                        : null
ServiceEndpoints                  : []
ServiceEndpointPolicies           : []
PrivateEndpoints                  : []
PrivateEndpointNetworkPolicies    : Disabled
PrivateLinkServiceNetworkPolicies : Disabled

@msrini-MSFT
Copy link
Contributor

@KumudD , Can you take a look at this issue and update the doc with the alternative ?

@asudbring
Copy link
Contributor

#assign:@malopMSFT

@malopMSFT
Copy link
Contributor

Hi tonyt73

We had this into our bug fixes list, do you mind trying to replicate the issue and see if it's still active
It should be solved by now with one of our previous releases

@rtaylor72
Copy link

I just got this error as part of my build process. I can no longer run "Set-AzVirtualNetwork" on a loaded vnet config (Get-AzVirtualNetwork) if that vnet has a subnet with a private endpoint attached to it. I have also narrowed it down to when I run "Set-AzVirtualNetworkSubnetConfig" with the "-PrivateLinkServiceNetworkPoliciesFlag 'Enabled'" setting causes the error.

@rtaylor72
Copy link

Circling back around to this... a little confusing, but I came across something stating PrivateLinkServiceNetworkPoliciesFlag has to be 'Disabled' in order to add a private end point... seems not intuitive.

@CloudPlumber
Copy link

Hey everybody,

I ran into the same issue, when I deployed a private endpoint into the subnet via pipeline and set this setting in a template. After deleting it in the template the error stayed and now I always get the error when I execute the pipeline:

"PrivateEndpointNetworkPoliciesCannotBeEnabledOnPrivateEndpointSubnet","message":"Private endpoint network policies cannot be enabled on private endpoint subnet /subscriptions/%subscriptionID%/resourceGroups/%RG%/providers/Microsoft.Network/virtualNetworks/%VNETNAME%/subnets/%SUBNETNAME%.

@kevinwedwards
Copy link

We're also seeing this issue with our vnet arm template config, after deploying private link for sql server via another arm template. Feels similar to the error above.

@malopMSFT
Copy link
Contributor

Hi guys,

Appreciate if you can raise a support ticket, it's hard t diagnose multiple issues without looking at the configuration.

I'm reading multiple issues listed here with PrivateEndpointNetworkPoliciesCannotBeEnabledOnPrivateEndpointSubnet and PrivateLinkServiceNetworkPoliciesFlag as 2 separate independent controls.

@markmcdon7
Copy link

Did anyone ever raise a support ticket and get a fix from them?

@mdehaas
Copy link

mdehaas commented Feb 2, 2021

We did but it did not lead to a satisfying result. Somewhere along the process also we decided this scenario does not fit into our Modern Workplace offering. We are only using SMB3 shares in Azure as part of an Application Resource Group for storing the application data (as was once the intended purpose from Microsoft). The only landing zones for data which has to be accessed from end-user devices are OneDrive and Sharepoint Online. We have not seen a case from our customers which forced us into another route (Customers just need some convincing that shifting a share to Azure is not 'modern')

@gowrisankar22
Copy link

gowrisankar22 commented Feb 19, 2021

@msrini-MSFT @SaiKishor-MSFT I am also facing the same issue. I Unable to change the PrivateEndpointNetworkPolicies

For SubnetProperties I want to disable the value forPrivateEndpointNetworkPolicies and PrivateLinkServiceNetworkPolicies but the below sample code is not disabling that. Can you help ?


const (
	PrivateEndpointNetworkPolicies string = "Disabled"
)


var penp = PrivateEndpointNetworkPolicies

SubnetPropertiesFormat: &network.SubnetPropertiesFormat{
       AddressPrefix:                  to.StringPtr(cidr),
	PrivateEndpointNetworkPolicies: &penp,
	PrivateLinkServiceNetworkPolicies: &penp,
}

@asudbring
Copy link
Contributor

Sending to PM for review

#assign:@ivapplyr

@reborn1867
Copy link

Hi @malopMSFT, Any update on this? I'm facing the same issue and I cannot understand why this issue kept being ignored? It's pretty straight forward and there must be a better approach to handle it.

@malopMSFT
Copy link
Contributor

Please raise a support ticket for proper troubleshooting and diagnosis of the issue, is not clear on this open item what is the issue with out documentation that describes those properties and behavior.

For active issues on deployments, support ticket is the right method as many times requires specific investigation onto a particular customer deployment.

@reborn1867
Copy link

@malopMSFT Honestly, I don't think this issue has anything to do with "particular customer deployment", the symptom is easy to reproduce:

  • create a subnet in a vnet
  • disable setting privateLinkServiceNetworkPolicies for the subnet as required by azure doc
  • assign a private endpoint to the subnet
  • try updating the subnet to enable privateLinkServiceNetworkPolicies with following command
    az network vnet subnet update \
    --disable-private-endpoint-network-policies false \
    --name test-subnet \
    --resource-group test-rg \
    --vnet-name test-vnet
    
  • The error message indicates it's not allowed to update private endpoint subnet
    (PrivateEndpointNetworkPoliciesCannotBeEnabledOnPrivateEndpointSubnet) Private endpoint network policies cannot be enabled on private endpoint subnet
    

Do you think this is as expected?

@malopMSFT
Copy link
Contributor

Is expected, Users must get an error when deploying private endpoints and the property is enabled.

The sequence can be any of the following:

  1. subnet with PEPolicies=Enables then deploy a PE => Error
  2. Subnet has PE (because PEPolicies are disabled) then updating the subnet to PEPolicies=Enabled => Error

The property is mainly designed to provide a toggle once NSG is supported, currently in public preview.

For preview subscriptions, this error does not show and instead, NSG will start to apply to the private endpoints

Details can be found here:
https://docs.microsoft.com/en-us/azure/private-link/disable-private-endpoint-network-policy

@apeschel
Copy link

Confirming this bug is still active as of Dec 21, 2021.

@prithvy
Copy link

prithvy commented Jan 7, 2022

Confirming this bug is still active as of JAN 7, 2022.

@ivapplyr
Copy link
Member

ivapplyr commented Jan 7, 2022

Hello Folks,

For those who are receiving this error message, it is per design. Please read documentation below to have access to the Public Preview of Private Link NSG & UDR :)

https://azure.microsoft.com/en-us/updates/private-link-nsg-support-extended-regional-availability/
https://azure.microsoft.com/en-us/updates/extended-regional-availability-for-private-link-udr-support/

@ivapplyr
Copy link
Member

ivapplyr commented Jan 7, 2022

@tonyt73 would we be able to close this as there are multi thread of issues prior to 2021

@IPvPho
Copy link

IPvPho commented Jun 30, 2022

When trying to disable the Private Link Service POlicies via Az CLI, we are now getting this error: "however, it does not have permission to perform action 'Microsoft.Network/routeTables/join/action' on the linked scope(s)"

We understand the related error about the policies being enabled when trying to add a PLS, but we are not sure why this is trying to read/write to the route table.

@asudbring
Copy link
Contributor

@IPvPho Can you provide more information on the steps and environment when getting this error? Is this a new deployment of private endpoint to an existing subnet with the policies applied? Are there any permissions or policies deployed in your environment that would prevent your account from updating route tables? Can you provide the full error you are receiving and possibly a screenshot?

Thanks!

@ivapplyr do you know why it would error out on updating the route table?

@asudbring
Copy link
Contributor

asudbring commented Mar 15, 2023

Thank you for you dedication to our documentation.

Unfortunately, we have been unable to review this issue in a timely manner. We sincerely apologize for the delayed response. We are closing this issue. If you feel that the problem persists, please respond to this issue with additional information.

Please continue to provide feedback about the documentation. We appreciate your contributions to our community.

#please-close

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests