Add-AzureRmServiceFabricNodeType fails with error: The Resource 'Microsoft.Network/virtualNetworks/VNETX' under resource group 'RG01' was not found

[asetiMSFT]

Cmdlet(s)
Add-AzureRmServiceFabricNodeType

PowerShell Version
5.1.17763.134

Module Version
AzureRM 6.12.0

OS Version
10.0.17763.253

Description
When attempting to add new node type to an existing cluster that has VMSS that uses existing VNet from other resource group that's different than resource group of the cluster, the command failed with error:

Add-AzureRmServiceFabricNodeType : The Resource 'Microsoft.Network/virtualNetworks/VNETX' under resource group 'RG01' was not found.
At line:1 char:1
+ Add-AzureRmServiceFabricNodeType -ResourceGroupName 'RG01 ...
    + CategoryInfo          : CloseError: (:) [Add-AzureRmServiceFabricNodeType], CloudException
    + FullyQualifiedErrorId : Microsoft.Azure.Commands.ServiceFabric.Commands.AddAzureRmServiceFabricNodeType

Cluster is healthy and was deployed based on this doc to use existing vnet from different resource group:
https://blogs.msdn.microsoft.com/kwill/2016/10/05/azure-service-fabric-common-networking-scenarios/#ExistingVnet

Perhaps using fully qualified resource id (https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-template-functions-resource#resourceid ) from within the code (https://github.com/Azure/azure-powershell/blob/master/src/ServiceFabric/ServiceFabric/Commands/AddAzureRmServiceFabricNodeType.cs ) may fix this problem? Or add new parameter to allow user to specify different resource group for certain resource (in this case, it's virtual network that's in different resource group than the cluster).

Please note that I'm submitting this issue on behalf of our customer. This seems to be a supported scenario and can be mitigated via ARM template update, however until this issue is fixed, user can't use this cmdlet for this scenario.

Script/Steps for Reproduction
Have an existing cluster with all resources in resource group RG01 except for the virtual network resource is in resource group RG02 (all under the same subscription).
Than run command:

$pwd = ConvertTo-SecureString -String 'password' -AsPlainText -Force

Add-AzureRmServiceFabricNodeType -ResourceGroupName 'RG01' -Name 'Cluster' -NodeType 'NT02' -Capacity 3 -VmUserName 'clusteradmin' -VmPassword $pwd

[markcowl]

@asetiMSFT Can you try running Resolve-AzureRmError after this error occurs - this should print additional information about the message returned from the services.

Also, please gather additional debugging info by setting

$DebugPreference='Continue' and then running the failing command

[asetiMSFT]

Hi Mark, I will ask my customer (they are in East coast and I'm here in West coast (Redmond/Sammamish area)). They'll get back Monday and may be available over the weekend. At any rates, I'll forward the requests to them. My shift is almost over but I can monitor their email during weekend and post anything new here. I'm still working on other case at the moment. Thanks for your help!

[asetiMSFT]

Hi Mark, got a reply from cx as follows (sensitive data was omitted). If I missed something, pls let me know. Thanks!

PS C:\WINDOWS\system32> 
 $DebugPreference='Continue'

PS C:\WINDOWS\system32> Resolve-AzureRmError -Last
DEBUG: 7:15:53 AM - ResolveError begin processing with ParameterSet 'LastErrorParameterSet'.
DEBUG: 7:15:53 AM - using account id '<OMITTED>'...

   HistoryId: 60


RequestId      : <OMITTED>
Message        : The Resource 'Microsoft.Network/virtualNetworks/VNETX' under resource group 'RG01' was not found.
ServerMessage  : ResourceNotFound: The Resource 'Microsoft.Network/virtualNetworks/VNETX' under resource group 'RG01' was not found. (System.Collections.Generic.List`1[Microsoft.Rest.Azure.CloudError])
ServerResponse : {NotFound}
RequestMessage : {GET https://management.azure.com/subscriptions/<OMITTED>/resourceGroups/RG1/providers/Microsoft.Network/virtualNetworks/VNETX/subnets/<OMITTED>-Subnet-0?api-version=2017-10-01}
InvocationInfo : {Add-AzureRmServiceFabricNodeType}
Line           : Add-AzureRmServiceFabricNodeType
Position       : At line:1 char:1
                 + Add-AzureRmServiceFabricNodeType
                 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
StackTrace     :    at Microsoft.Azure.Management.Internal.Network.Version2017_10_01.SubnetsOperations.<GetWithHttpMessagesAsync>d__6.
                 MoveNext()
                 --- End of stack trace from previous location where exception was thrown ---
                    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                    at 
                 Microsoft.Azure.Management.Internal.Network.Version2017_10_01.SubnetsOperationsExtensions.<GetAsync>d__3.MoveNext()
                 --- End of stack trace from previous location where exception was thrown ---
                    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                    at 
                 Microsoft.Azure.Management.Internal.Network.Version2017_10_01.SubnetsOperationsExtensions.Get(ISubnetsOperations 
                 operations, String resourceGroupName, String virtualNetworkName, String subnetName, String expand)
                    at Microsoft.Azure.Commands.ServiceFabric.Commands.AddAzureRmServiceFabricNodeType.GetSubnetAddressPrefix(VirtualM
                 achineScaleSetIPConfiguration ipConfiguration)
                    at Microsoft.Azure.Commands.ServiceFabric.Commands.AddAzureRmServiceFabricNodeType.CreateNetworkResource(VirtualMa
                 chineScaleSetNetworkConfiguration existingNetworkConfig)
                    at Microsoft.Azure.Commands.ServiceFabric.Commands.AddAzureRmServiceFabricNodeType.GetProfiles(VirtualMachineScale
                 SetExtensionProfile& vmExtProfile, VirtualMachineScaleSetOSProfile& osProfile, VirtualMachineScaleSetStorageProfile& 
                 storageProfile, VirtualMachineScaleSetNetworkProfile& networkProfile)
                    at Microsoft.Azure.Commands.ServiceFabric.Commands.AddAzureRmServiceFabricNodeType.CreateVmss()
                    at Microsoft.Azure.Commands.ServiceFabric.Commands.AddAzureRmServiceFabricNodeType.ExecuteCmdlet()
                    at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ProcessRecord()
HistoryId      : 60

DEBUG: AzureQoSEvent: CommandName - Resolve-AzureRmError; IsSuccess - True; Duration - 00:00:00.0351621; Exception - ;
DEBUG: Finish sending metric.
DEBUG: 7:15:55 AM - ResolveError end processing.
DEBUG: 7:15:55 AM - ResolveError end processing.

[asetiMSFT]

I'll post the debug info for the failing command tomorrow as it's getting late here and I needed to ensure I don't post any sensitive info here.

[markcowl]

@asetiMSFT Were you able to get the debugging information?

[asetiMSFT]

I was but my team is super swamped right now and didn't get a chance to spend time on removing sensitive info. ETA: EOD today.

[asetiMSFT]

Hi Mark, I was out for few days due to snow and outage. It took me quite a while trying to ensure there's no sensitive info from the debugging information. Hopefully I didn't remove too much. Note that the names and some values are not real names but they should be consistent/doesn't change the context (hopefully). Let me know if I'm missing anything.

`PS C:\WINDOWS\system32>
$DebugPreference='Continue'

$pwd = ConvertTo-SecureString -String 'xxxxxxxxx' -AsPlainText -Force

PS C:\WINDOWS\system32> Add-AzureRmServiceFabricNodeType
DEBUG: 7:00:23 AM - GetAzureRMContextCommand begin processing with ParameterSet 'GetSingleContext'.
DEBUG: 7:00:23 AM - using account id ''...
DEBUG: AzureQoSEvent: CommandName - Get-AzureRmContext; IsSuccess - True; Duration - 00:00:00.0087877; Exception - ;
DEBUG: Finish sending metric.
DEBUG: 7:00:23 AM - GetAzureRMContextCommand end processing.
DEBUG: 7:00:23 AM - GetAzureRMContextCommand end processing.
DEBUG: 7:00:23 AM - GetAzureRMContextCommand begin processing with ParameterSet 'GetSingleContext'.
DEBUG: 7:00:23 AM - using account id ''...
DEBUG: AzureQoSEvent: CommandName - Get-AzureRmContext; IsSuccess - True; Duration - 00:00:00.0031047; Exception - ;
DEBUG: Finish sending metric.
DEBUG: 7:00:24 AM - GetAzureRMContextCommand end processing.
DEBUG: 7:00:24 AM - GetAzureRMContextCommand end processing.
DEBUG: 7:00:24 AM - GetAzureRMContextCommand begin processing with ParameterSet 'GetSingleContext'.
DEBUG: 7:00:24 AM - using account id ''...
DEBUG: AzureQoSEvent: CommandName - Get-AzureRmContext; IsSuccess - True; Duration - 00:00:00.0038541; Exception - ;
DEBUG: Finish sending metric.
DEBUG: 7:00:25 AM - GetAzureRMContextCommand end processing.
DEBUG: 7:00:25 AM - GetAzureRMContextCommand end processing.
cmdlet Add-AzureRmServiceFabricNodeType at command pipeline position 1
Supply values for the following parameters:
(Type !? for Help.)
ResourceGroupName: RG01
Name: Cluster
Capacity: 3
VmUserName:
NodeType: onboard
DEBUG: 7:01:58 AM - AddAzureRmServiceFabricNodeType begin processing with ParameterSet '__AllParameterSets'.
DEBUG: 7:01:58 AM - using account id ''...
DEBUG: [Common.Authentication]: Authenticating using Account: '', environment: 'Az
ureCloud', tenant: ''
DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Information: 2 :
DEBUG: 01/28/2019 12:01:58: - AcquireTokenHandlerBase: === Token Acquisition started:
Authority: https://login.microsoftonline.com//
Resource: https://management.core.windows.net/
ClientId:
CacheType: Microsoft.Azure.Commands.Common.Authentication.ProtectedFileTokenCache (2 items)
Authentication Target: User

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Information: 2 :
DEBUG: 01/28/2019 12:01:58: - TokenCache: Deserialized 2 items to token cache.

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Verbose: 1 :
DEBUG: 01/28/2019 12:01:58: - TokenCache: Looking up cache for a token...

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Information: 2 :
DEBUG: 01/28/2019 12:01:58: - TokenCache: An item matching the requested resource was found in the
cache

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Verbose: 1 :
DEBUG: 01/28/2019 12:01:58: - TokenCache: 56.4911298516667 minutes left until token in cache expir
es

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Information: 2 :
DEBUG: 01/28/2019 12:01:58: - TokenCache: A matching item (access token or refresh token or both)
was found in the cache

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Information: 2 :
DEBUG: 01/28/2019 12:01:58: - AcquireTokenHandlerBase: === Token Acquisition finished successfully
. An access token was retuned:
Access Token Hash:
Refresh Token Hash:
Expiration Time: 01/28/2019 12:58:27 +00:00
User Hash:

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Information: 2 :
DEBUG: 01/28/2019 12:01:58: - AcquireTokenHandlerBase: === Token Acquisition started:
Authority: https://login.microsoftonline.com//
Resource: https://management.core.windows.net/
ClientId:
CacheType: Microsoft.Azure.Commands.Common.Authentication.ProtectedFileTokenCache (2 items)
Authentication Target: User

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Information: 2 :
DEBUG: 01/28/2019 12:01:58: - TokenCache: Deserialized 2 items to token cache.

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Verbose: 1 :
DEBUG: 01/28/2019 12:01:58: - TokenCache: Looking up cache for a token...

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Information: 2 :
DEBUG: 01/28/2019 12:01:58: - TokenCache: An item matching the requested resource was found in the
cache

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Verbose: 1 :
DEBUG: 01/28/2019 12:01:58: - TokenCache: 56.491079815 minutes left until token in cache expires

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Information: 2 :
DEBUG: 01/28/2019 12:01:58: - TokenCache: A matching item (access token or refresh token or both)
was found in the cache

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Information: 2 :
DEBUG: 01/28/2019 12:01:58: - AcquireTokenHandlerBase: === Token Acquisition finished successfully
. An access token was retuned:
Access Token Hash:
Refresh Token Hash:
Expiration Time: 01/28/2019 12:58:27 +00:00
User Hash:

DEBUG: ============================ HTTP REQUEST ============================

HTTP Method:
GET

Absolute Uri:
https://management.azure.com/subscriptions//resourceGroups/RG01/prov
iders/Microsoft.ServiceFabric/clusters/Cluster?api-version=2016-09-01

Headers:
x-ms-client-request-id :
accept-language : en-US

Body:

DEBUG: ============================ HTTP RESPONSE ============================

Status Code:
OK

Headers:
Pragma : no-cache
Strict-Transport-Security : max-age=31536000; includeSubDomains
X-Content-Type-Options : nosniff
Cache-Control : no-cache
Server : Microsoft-HTTPAPI/2.0,Microsoft-HTTPAPI/2.0
x-ms-ratelimit-remaining-subscription-reads: 11999
x-ms-request-id :
x-ms-correlation-request-id :
x-ms-routing-request-id : SOUTHCENTRALUS:20190128T120159Z:
Date : Mon, 28 Jan 2019 12:01:58 GMT

Body:
{
"type": "Microsoft.ServiceFabric/clusters",
"location": "East US",
"id": "/subscriptions//resourcegroups/RG01/providers/Microsoft.Ser
viceFabric/clusters/Cluster",
"name": "Cluster",
"tags": {
"resourceType": "Service Fabric",
"clusterName": "Cluster",
"organization": "",
"subscription": ""
},
"etag": "W/"636808060962721642"",
"properties": {
"provisioningState": "Succeeded",
"clusterId": "",
"clusterCodeVersion": "6.4.622.9590",
"clusterState": "Ready",
"managementEndpoint": "https://10.16.20.4:19080",
"clusterEndpoint": "https://eastus.servicefabric.azure.com/runtime/clusters/",
"certificate": {
"thumbprint": "",
"x509StoreName": "My"
},
"clientCertificateThumbprints": [],
"clientCertificateCommonNames": [],
"fabricSettings": [
{
"name": "Security",
"parameters": [
{
"name": "ClusterProtectionLevel",
"value": "EncryptAndSign"
}
]
}
],
"diagnosticsStorageAccountConfig": {
"storageAccountName": "",
"protectedAccountKeyName": "StorageAccountKey1",
"blobEndpoint": "https://.blob.core.windows.net/",
"queueEndpoint": "https://.queue.core.windows.net/",
"tableEndpoint": "https://.table.core.windows.net/"
},
"vmImage": "Windows",
"reliabilityLevel": "Bronze",
"nodeTypes": [
{
"name": "ss01",
"vmInstanceCount": 3,
"clientConnectionEndpointPort": ,
"httpGatewayEndpointPort": ,
"applicationPorts": {
"startPort": ,
"endPort":
},
"ephemeralPorts": {
"startPort": ,
"endPort":
},
"isPrimary": true,
"reverseProxyEndpointPort": ,
"durabilityLevel": "Bronze"
},
{
"name": "ss02",
"vmInstanceCount": 3,
"clientConnectionEndpointPort": ,
"httpGatewayEndpointPort": ,
"applicationPorts": {
"startPort": ,
"endPort":
},
"ephemeralPorts": {
"startPort": ,
"endPort":
},
"isPrimary": false,
"reverseProxyEndpointPort": ,
"durabilityLevel": "Bronze"
},
{
"name": "ss03",
"vmInstanceCount": 3,
"clientConnectionEndpointPort": ,
"httpGatewayEndpointPort": ,
"applicationPorts": {
"startPort": ,
"endPort":
},
"ephemeralPorts": {
"startPort": ,
"endPort":
},
"isPrimary": false,
"reverseProxyEndpointPort": ,
"durabilityLevel": "Bronze"
}
],
"upgradeMode": "Automatic",
"availableClusterVersions": [
{
"codeVersion": "6.4.617.9590",
"supportExpiryUtc": "9999-12-31T23:59:59.9999999",
"environment": "Windows"
},
{
"codeVersion": "6.4.622.9590",
"supportExpiryUtc": "9999-12-31T23:59:59.9999999",
"environment": "Windows"
}
]
}
}

DEBUG: [Common.Authentication]: Authenticating using Account: '', environment: 'Az
ureCloud', tenant: ''
DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Information: 2 :
DEBUG: 01/28/2019 12:01:59: - AcquireTokenHandlerBase: === Token Acquisition started:
Authority: https://login.microsoftonline.com//
Resource: https://management.core.windows.net/
ClientId:
CacheType: Microsoft.Azure.Commands.Common.Authentication.ProtectedFileTokenCache (2 items)
Authentication Target: User

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Information: 2 :
DEBUG: 01/28/2019 12:01:59: - TokenCache: Deserialized 2 items to token cache.

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Verbose: 1 :
DEBUG: 01/28/2019 12:01:59: - TokenCache: Looking up cache for a token...

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Information: 2 :
DEBUG: 01/28/2019 12:01:59: - TokenCache: An item matching the requested resource was found in the
cache

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Verbose: 1 :
DEBUG: 01/28/2019 12:01:59: - TokenCache: 56.4629202016667 minutes left until token in cache expir
es

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Information: 2 :
DEBUG: 01/28/2019 12:01:59: - TokenCache: A matching item (access token or refresh token or both)
was found in the cache

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Information: 2 :
DEBUG: 01/28/2019 12:01:59: - AcquireTokenHandlerBase: === Token Acquisition finished successfully
. An access token was retuned:
Access Token Hash:
Refresh Token Hash:
Expiration Time: 01/28/2019 12:58:27 +00:00
User Hash:

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Information: 2 :
DEBUG: 01/28/2019 12:01:59: - AcquireTokenHandlerBase: === Token Acquisition started:
Authority: https://login.microsoftonline.com//
Resource: https://management.core.windows.net/
ClientId:
CacheType: Microsoft.Azure.Commands.Common.Authentication.ProtectedFileTokenCache (2 items)
Authentication Target: User

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Information: 2 :
DEBUG: 01/28/2019 12:01:59: - TokenCache: Deserialized 2 items to token cache.

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Verbose: 1 :
DEBUG: 01/28/2019 12:01:59: - TokenCache: Looking up cache for a token...

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Information: 2 :
DEBUG: 01/28/2019 12:01:59: - TokenCache: An item matching the requested resource was found in the
cache

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Verbose: 1 :
DEBUG: 01/28/2019 12:01:59: - TokenCache: 56.462820255 minutes left until token in cache expires

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Information: 2 :
DEBUG: 01/28/2019 12:01:59: - TokenCache: A matching item (access token or refresh token or both)
was found in the cache

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Information: 2 :
DEBUG: 01/28/2019 12:01:59: - AcquireTokenHandlerBase: === Token Acquisition finished successfully
. An access token was retuned:
Access Token Hash:
Refresh Token Hash:
Expiration Time: 01/28/2019 12:58:27 +00:00
User Hash:

DEBUG: ============================ HTTP REQUEST ============================

HTTP Method:
GET

Absolute Uri:
https://management.azure.com/subscriptions//resourceGroups/RG01/prov
iders/Microsoft.Compute/virtualMachineScaleSets?api-version=2017-12-01

Headers:
x-ms-client-request-id :
accept-language : en-US

Body:

DEBUG: ============================ HTTP RESPONSE ============================

Status Code:
OK

Headers:
Pragma : no-cache
x-ms-ratelimit-remaining-resource: Microsoft.Compute/HighCostGetVMScaleSet3Min;179,Microsoft.Compute/HighCostGetVMScaleSet30Min;899
Strict-Transport-Security : max-age=31536000; includeSubDomains
x-ms-request-id :
Cache-Control : no-cache
Server : Microsoft-HTTPAPI/2.0,Microsoft-HTTPAPI/2.0
x-ms-ratelimit-remaining-subscription-reads: 11998
x-ms-correlation-request-id :
x-ms-routing-request-id : SOUTHCENTRALUS:20190128T120200Z:
X-Content-Type-Options : nosniff
Date : Mon, 28 Jan 2019 12:01:59 GMT

Body:
{
"value": [
{
"sku": {
"name": "Standard_F2s_v2",
"tier": "Standard",
"capacity": 3
},
"properties": {
"singlePlacementGroup": true,
"upgradePolicy": {
"mode": "Automatic",
"automaticOSUpgrade": false
},
"virtualMachineProfile": {
"osProfile": {
"computerNamePrefix": "core",
"adminUsername": "",
"windowsConfiguration": {
"provisionVMAgent": true,
"enableAutomaticUpdates": true
},
"secrets": [
{
"sourceVault": {
"id": "/subscriptions//resourceGroups/RG01/provide
rs/Microsoft.KeyVault/vaults/"
},
"vaultCertificates": [
{
"certificateUrl": "https://.vault.azure.net/secrets//",
"certificateStore": "My"
}
]
}
]
},
"storageProfile": {
"osDisk": {
"createOption": "FromImage",
"caching": "ReadOnly",
"managedDisk": {
"storageAccountType": "Standard_LRS"
}
},
"imageReference": {
"publisher": "MicrosoftWindowsServer",
"offer": "WindowsServer",
"sku": "2012-R2-Datacenter",
"version": "latest"
}
},
"networkProfile": {
"networkInterfaceConfigurations": [
{
"name": "NIC-0",
"properties": {
"primary": true,
"enableAcceleratedNetworking": false,
"dnsSettings": {
"dnsServers": []
},
"ipConfigurations": [
{
"name": "NIC-0",
"properties": {
"subnet": {
"id": "/subscriptions//resourceGroups/RG01/providers
/Microsoft.Network/virtualNetworks/VNETX/subnets/-Subnet-0"
},
"privateIPAddressVersion": "IPv4",
"loadBalancerBackendAddressPools": [
{
"id": "/subscriptions//resourceGroups/RG01/providers/Microsoft.Network/loadBalancers/LB01/backendAddressPools/"
}
],
"loadBalancerInboundNatPools": [
{
"id": "/subscriptions//resourceGroups/RG01/providers/Microsoft.Network/loadBalancers/LB01/inboundNatPools/"
}
]
}
}
]
}
}
]
},
"extensionProfile": {
"extensions": [
{
"properties": {
"autoUpgradeMinorVersion": true,
"settings": {
"clusterEndpoint": "https://eastus.servicefabric.azure.com/runtime/clusters/",
"nodeTypeRef": "ss01",
"dataPath": "D:\\SvcFab",
"durabilityLevel": "Bronze",
"enableParallelJobs": true,
"nicPrefixOverride": "10.16.20.0/24",
"certificate": {
"thumbprint": "",
"x509StoreName": "My"
}
},
"publisher": "Microsoft.Azure.ServiceFabric",
"type": "ServiceFabricNode",
"typeHandlerVersion": "1.0"
},
"name": "core_ServiceFabricNode"
},
{
"properties": {
"autoUpgradeMinorVersion": true,
"settings": {
"WadCfg": {
"DiagnosticMonitorConfiguration": {
"overallQuotaInMB": "50000",
"EtwProviders": {
"EtwEventSourceProviderConfiguration": [
{
"provider": "Microsoft-ServiceFabric-Actors",
"scheduledTransferKeywordFilter": "1",
"scheduledTransferPeriod": "PT5M",
"DefaultEvents": {
"eventDestination": "ServiceFabricReliableActorEventTable"
}
},
{
"provider": "Microsoft-ServiceFabric-Services",
"scheduledTransferPeriod": "PT5M",
"DefaultEvents": {
"eventDestination": "ServiceFabricReliableServiceEventTable"
}
}
],
"EtwManifestProviderConfiguration": [
{
"provider": "cbd93bc2-71e5-4566-b3a7-595d8eeca6e8",
"scheduledTransferLogLevelFilter": "Information",
"scheduledTransferKeywordFilter": "4611686018427387904",
"scheduledTransferPeriod": "PT5M",
"DefaultEvents": {
"eventDestination": "ServiceFabricSystemEventTable"
}
}
]
}
}
},
"StorageAccount": ""
},
"publisher": "Microsoft.Azure.Diagnostics",
"type": "IaaSDiagnostics",
"typeHandlerVersion": "1.5"
},
"name": "VMDiagnosticsVmExt_vmNodeType0Name"
}
]
}
},
"provisioningState": "Succeeded",
"overprovision": false,
"uniqueId": ""
},
"type": "Microsoft.Compute/virtualMachineScaleSets",
"location": "eastus",
"tags": {
"resourceType": "Service Fabric",
"clusterName": "Cluster",
"organization": "",
"subscription": ""
},
"id": "/subscriptions//resourceGroups/RG01/providers/Microsoft
.Compute/virtualMachineScaleSets/ss01",
"name": "ss01"
},
{
"sku": {
"name": "Standard_F2s_v2",
"tier": "Standard",
"capacity": 3
},
"properties": {
"singlePlacementGroup": true,
"upgradePolicy": {
"mode": "Automatic",
"automaticOSUpgrade": false
},
"virtualMachineProfile": {
"osProfile": {
"computerNamePrefix": "ss03",
"adminUsername": "",
"windowsConfiguration": {
"provisionVMAgent": true,
"enableAutomaticUpdates": true
},
"secrets": [
{
"sourceVault": {
"id": "/subscriptions//resourceGroups/RG01/provide
rs/Microsoft.KeyVault/vaults/"
},
"vaultCertificates": [
{
"certificateUrl": "https://.vault.azure.net/secrets//",
"certificateStore": "My"
}
]
}
]
},
"storageProfile": {
"osDisk": {
"createOption": "FromImage",
"caching": "ReadOnly",
"managedDisk": {
"storageAccountType": "Standard_LRS"
}
},
"imageReference": {
"publisher": "MicrosoftWindowsServer",
"offer": "WindowsServer",
"sku": "2012-R2-Datacenter",
"version": "latest"
}
},
"networkProfile": {
"networkInterfaceConfigurations": [
{
"name": "NIC-2",
"properties": {
"primary": true,
"enableAcceleratedNetworking": false,
"dnsSettings": {
"dnsServers": []
},
"ipConfigurations": [
{
"name": "NIC-2",
"properties": {
"subnet": {
"id": "/subscriptions//resourceGroups/RG01/providers
/Microsoft.Network/virtualNetworks/VNETX/subnets/-Subnet-2"
},
"privateIPAddressVersion": "IPv4",
"loadBalancerBackendAddressPools": [
{
"id": "/subscriptions//resourceGroups/RG01/providers/Microsoft.Network/loadBalancers/LB-Cluster-ss03/backendAddressPools/"
}
],
"loadBalancerInboundNatPools": [
{
"id": "/subscriptions//resourceGroups/RG01/providers/Microsoft.Network/loadBalancers/LB-Cluster-ss03/inboundNatPools/"
}
]
}
}
]
}
}
]
},
"extensionProfile": {
"extensions": [
{
"properties": {
"autoUpgradeMinorVersion": true,
"settings": {
"clusterEndpoint": "https://eastus.servicefabric.azure.com/runtime/clusters/",
"nodeTypeRef": "ss03",
"dataPath": "D:\\SvcFab",
"durabilityLevel": "Bronze",
"enableParallelJobs": true,
"nicPrefixOverride": "10.16.22.0/24",
"certificate": {
"thumbprint": "",
"x509StoreName": "My"
}
},
"publisher": "Microsoft.Azure.ServiceFabric",
"type": "ServiceFabricNode",
"typeHandlerVersion": "1.0"
},
"name": "ss03_ServiceFabricNode"
},
{
"properties": {
"autoUpgradeMinorVersion": true,
"settings": {
"WadCfg": {
"DiagnosticMonitorConfiguration": {
"overallQuotaInMB": "50000",
"EtwProviders": {
"EtwEventSourceProviderConfiguration": [
{
"provider": "Microsoft-ServiceFabric-Actors",
"scheduledTransferKeywordFilter": "1",
"scheduledTransferPeriod": "PT5M",
"DefaultEvents": {
"eventDestination": "ServiceFabricReliableActorEventTable"
}
},
{
"provider": "Microsoft-ServiceFabric-Services",
"scheduledTransferPeriod": "PT5M",
"DefaultEvents": {
"eventDestination": "ServiceFabricReliableServiceEventTable"
}
}
],
"EtwManifestProviderConfiguration": [
{
"provider": "cbd93bc2-71e5-4566-b3a7-595d8eeca6e8",
"scheduledTransferLogLevelFilter": "Information",
"scheduledTransferKeywordFilter": "4611686018427387904",
"scheduledTransferPeriod": "PT5M",
"DefaultEvents": {
"eventDestination": "ServiceFabricSystemEventTable"
}
}
]
}
}
},
"StorageAccount": ""
},
"publisher": "Microsoft.Azure.Diagnostics",
"type": "IaaSDiagnostics",
"typeHandlerVersion": "1.5"
},
"name": "VMDiagnosticsVmExt_vmNodeType2Name"
}
]
}
},
"provisioningState": "Succeeded",
"overprovision": false,
"uniqueId": ""
},
"type": "Microsoft.Compute/virtualMachineScaleSets",
"location": "eastus",
"tags": {
"resourceType": "Service Fabric",
"clusterName": "Cluster",
"organization": "",
"subscription": ""
},
"id": "/subscriptions//resourceGroups/RG01/providers/Microsoft
.Compute/virtualMachineScaleSets/ss03",
"name": "ss03"
},
{
"sku": {
"name": "Standard_F2s_v2",
"tier": "Standard",
"capacity": 3
},
"properties": {
"singlePlacementGroup": true,
"upgradePolicy": {
"mode": "Automatic",
"automaticOSUpgrade": false
},
"virtualMachineProfile": {
"osProfile": {
"computerNamePrefix": "ss02",
"adminUsername": "",
"windowsConfiguration": {
"provisionVMAgent": true,
"enableAutomaticUpdates": true
},
"secrets": [
{
"sourceVault": {
"id": "/subscriptions//resourceGroups/RG01/provide
rs/Microsoft.KeyVault/vaults/"
},
"vaultCertificates": [
{
"certificateUrl": "https://.vault.azure.net/secrets//",
"certificateStore": "My"
}
]
}
]
},
"storageProfile": {
"osDisk": {
"createOption": "FromImage",
"caching": "ReadOnly",
"managedDisk": {
"storageAccountType": "Standard_LRS"
}
},
"imageReference": {
"publisher": "MicrosoftWindowsServer",
"offer": "WindowsServer",
"sku": "2012-R2-Datacenter",
"version": "latest"
}
},
"networkProfile": {
"networkInterfaceConfigurations": [
{
"name": "NIC-1",
"properties": {
"primary": true,
"enableAcceleratedNetworking": false,
"dnsSettings": {
"dnsServers": []
},
"ipConfigurations": [
{
"name": "NIC-1",
"properties": {
"subnet": {
"id": "/subscriptions//resourceGroups/RG01/providers
/Microsoft.Network/virtualNetworks/VNETX/subnets/-Subnet-1"
},
"privateIPAddressVersion": "IPv4",
"loadBalancerBackendAddressPools": [
{
"id": "/subscriptions//resourceGroups/RG01/providers/Microsoft.Network/loadBalancers/LB-Cluster-ss02/backendAddressPools/"
}
],
"loadBalancerInboundNatPools": [
{
"id": "/subscriptions//resourceGroups/RG01/providers/Microsoft.Network/loadBalancers/LB-Cluster-ss02/inboundNatPools/"
}
]
}
}
]
}
}
]
},
"extensionProfile": {
"extensions": [
{
"properties": {
"autoUpgradeMinorVersion": true,
"settings": {
"clusterEndpoint": "https://eastus.servicefabric.azure.com/runtime/clusters/",
"nodeTypeRef": "ss02",
"dataPath": "D:\\SvcFab",
"durabilityLevel": "Bronze",
"enableParallelJobs": true,
"nicPrefixOverride": "10.16.21.0/24",
"certificate": {
"thumbprint": "",
"x509StoreName": "My"
}
},
"publisher": "Microsoft.Azure.ServiceFabric",
"type": "ServiceFabricNode",
"typeHandlerVersion": "1.0"
},
"name": "ss02_ServiceFabricNode"
},
{
"properties": {
"autoUpgradeMinorVersion": true,
"settings": {
"WadCfg": {
"DiagnosticMonitorConfiguration": {
"overallQuotaInMB": "50000",
"EtwProviders": {
"EtwEventSourceProviderConfiguration": [
{
"provider": "Microsoft-ServiceFabric-Actors",
"scheduledTransferKeywordFilter": "1",
"scheduledTransferPeriod": "PT5M",
"DefaultEvents": {
"eventDestination": "ServiceFabricReliableActorEventTable"
}
},
{
"provider": "Microsoft-ServiceFabric-Services",
"scheduledTransferPeriod": "PT5M",
"DefaultEvents": {
"eventDestination": "ServiceFabricReliableServiceEventTable"
}
}
],
"EtwManifestProviderConfiguration": [
{
"provider": "cbd93bc2-71e5-4566-b3a7-595d8eeca6e8",
"scheduledTransferLogLevelFilter": "Information",
"scheduledTransferKeywordFilter": "4611686018427387904",
"scheduledTransferPeriod": "PT5M",
"DefaultEvents": {
"eventDestination": "ServiceFabricSystemEventTable"
}
}
]
}
}
},
"StorageAccount": ""
},
"publisher": "Microsoft.Azure.Diagnostics",
"type": "IaaSDiagnostics",
"typeHandlerVersion": "1.5"
},
"name": "VMDiagnosticsVmExt_vmNodeType1Name"
}
]
}
},
"provisioningState": "Succeeded",
"overprovision": false,
"uniqueId": ""
},
"type": "Microsoft.Compute/virtualMachineScaleSets",
"location": "eastus",
"tags": {
"resourceType": "Service Fabric",
"clusterName": "Cluster",
"organization": "",
"subscription": ""
},
"id": "/subscriptions//resourceGroups/RG01/providers/Microsoft
.Compute/virtualMachineScaleSets/ss02",
"name": "ss02"
}
]
}

DEBUG: [Common.Authentication]: Authenticating using Account: '', environment: 'Az
ureCloud', tenant: ''
DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Information: 2 :
DEBUG: 01/28/2019 12:02:00: - AcquireTokenHandlerBase: === Token Acquisition started:
Authority: https://login.microsoftonline.com//
Resource: https://management.core.windows.net/
ClientId:
CacheType: Microsoft.Azure.Commands.Common.Authentication.ProtectedFileTokenCache (2 items)
Authentication Target: User

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Information: 2 :
DEBUG: 01/28/2019 12:02:00: - TokenCache: Deserialized 2 items to token cache.

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Verbose: 1 :
DEBUG: 01/28/2019 12:02:00: - TokenCache: Looking up cache for a token...

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Information: 2 :
DEBUG: 01/28/2019 12:02:00: - TokenCache: An item matching the requested resource was found in the
cache

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Verbose: 1 :
DEBUG: 01/28/2019 12:02:00: - TokenCache: 56.450010525 minutes left until token in cache expires

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Information: 2 :
DEBUG: 01/28/2019 12:02:00: - TokenCache: A matching item (access token or refresh token or both)
was found in the cache

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Information: 2 :
DEBUG: 01/28/2019 12:02:00: - AcquireTokenHandlerBase: === Token Acquisition finished successfully
. An access token was retuned:
Access Token Hash:
Refresh Token Hash:
Expiration Time: 01/28/2019 12:58:27 +00:00
User Hash:

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Information: 2 :
DEBUG: 01/28/2019 12:02:00: - AcquireTokenHandlerBase: === Token Acquisition started:
Authority: https://login.microsoftonline.com//
Resource: https://management.core.windows.net/
ClientId:
CacheType: Microsoft.Azure.Commands.Common.Authentication.ProtectedFileTokenCache (2 items)
Authentication Target: User

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Information: 2 :
DEBUG: 01/28/2019 12:02:00: - TokenCache: Deserialized 2 items to token cache.

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Verbose: 1 :
DEBUG: 01/28/2019 12:02:00: - TokenCache: Looking up cache for a token...

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Information: 2 :
DEBUG: 01/28/2019 12:02:00: - TokenCache: An item matching the requested resource was found in the
cache

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Verbose: 1 :
DEBUG: 01/28/2019 12:02:00: - TokenCache: 56.44997726 minutes left until token in cache expires

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Information: 2 :
DEBUG: 01/28/2019 12:02:00: - TokenCache: A matching item (access token or refresh token or both)
was found in the cache

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Information: 2 :
DEBUG: 01/28/2019 12:02:00: - AcquireTokenHandlerBase: === Token Acquisition finished successfully
. An access token was retuned:
Access Token Hash:
Refresh Token Hash:
Expiration Time: 01/28/2019 12:58:27 +00:00
User Hash:

DEBUG: ============================ HTTP REQUEST ============================

HTTP Method:
GET

Absolute Uri:
https://management.azure.com/subscriptions//resourceGroups/RG01/prov
iders/Microsoft.Network/virtualNetworks/VNETX/subnets/-Subnet-0?api-version=2017-10-01

Headers:
x-ms-client-request-id :
accept-language : en-US

Body:

DEBUG: ============================ HTTP RESPONSE ============================

Status Code:
NotFound

Headers:
Pragma : no-cache
x-ms-failure-cause : gateway
x-ms-request-id :
x-ms-correlation-request-id :
x-ms-routing-request-id : SOUTHCENTRALUS:20190128T120201Z:
Strict-Transport-Security : max-age=31536000; includeSubDomains
X-Content-Type-Options : nosniff
Cache-Control : no-cache
Date : Mon, 28 Jan 2019 12:02:00 GMT

Body:
{
"error": {
"code": "ResourceNotFound",
"message": "The Resource 'Microsoft.Network/virtualNetworks/VNETX' under resource group 'RG02' was not found."
}
}

Add-AzureRmServiceFabricNodeType : The Resource 'Microsoft.Network/virtualNetworks/VNETX' under resource group
'RG01' was not found.
At line:1 char:1

• Add-AzureRmServiceFabricNodeType

•   + CategoryInfo          : CloseError: (:) [Add-AzureRmServiceFabricNodeType], CloudException
    + FullyQualifiedErrorId : Microsoft.Azure.Commands.ServiceFabric.Commands.AddAzureRmServiceFabricNodeType
  
  

DEBUG: AzureQoSEvent: CommandName - Add-AzureRmServiceFabricNodeType; IsSuccess - False; Duration - 00:00:03.2237103; Exception - Micro
soft.Rest.Azure.CloudException: The Resource 'Microsoft.Network/virtualNetworks/VNETX' under resource group 'RG02' was not found.
at Microsoft.Azure.Management.Internal.Network.Version2017_10_01.SubnetsOperations.d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.Management.Internal.Network.Version2017_10_01.SubnetsOperationsExtensions.d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.Management.Internal.Network.Version2017_10_01.SubnetsOperationsExtensions.Get(ISubnetsOperations operations, Stri
ng resourceGroupName, String virtualNetworkName, String subnetName, String expand)
at Microsoft.Azure.Commands.ServiceFabric.Commands.AddAzureRmServiceFabricNodeType.GetSubnetAddressPrefix(VirtualMachineScaleSetIPCo
nfiguration ipConfiguration)
at Microsoft.Azure.Commands.ServiceFabric.Commands.AddAzureRmServiceFabricNodeType.CreateNetworkResource(VirtualMachineScaleSetNetwo
rkConfiguration existingNetworkConfig)
at Microsoft.Azure.Commands.ServiceFabric.Commands.AddAzureRmServiceFabricNodeType.GetProfiles(VirtualMachineScaleSetExtensionProfil
e& vmExtProfile, VirtualMachineScaleSetOSProfile& osProfile, VirtualMachineScaleSetStorageProfile& storageProfile, VirtualMachineScaleS
etNetworkProfile& networkProfile)
at Microsoft.Azure.Commands.ServiceFabric.Commands.AddAzureRmServiceFabricNodeType.CreateVmss()
at Microsoft.Azure.Commands.ServiceFabric.Commands.AddAzureRmServiceFabricNodeType.ExecuteCmdlet()
at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ProcessRecord();
DEBUG: Finish sending metric.
DEBUG: 7:02:01 AM - AddAzureRmServiceFabricNodeType end processing.
DEBUG: 7:02:01 AM - AddAzureRmServiceFabricNodeType end processing.

PS C:\WINDOWS\system32>

`

[maddieclayton]

@vaishnavk @juhacket Can you take a look at this issue?

[a-santamaria]

@asetiMSFT thanks for reporting. I'll file a bug to fix this scenario, in the cmdlet logic we are assuming all resources are under the same resource group. As you pointed out this can be mitigated via ARM template deployment, was the customer unblocked doing this?

[asetiMSFT]

Hi Alfredo, thanks for the follow up. The customer said they were unblocked by doing ARM template deployment but they mentioned they preferred to do it via cmdlet previously but since they have no choice, they had to do it anyway. I'll follow up with customer again and let him know about the progress so far. Thanks for your help! Please expect delays in my responses for the next days/week since my area has been pummeled by snow storm, power outages, etc.