Skip to content

Commit

Permalink
Added SlbDiagnostics script
Browse files Browse the repository at this point in the history
Added NC PowerShell examples
Added VMM script examples
  • Loading branch information
JMesser81 committed Oct 4, 2016
1 parent b2d9ec4 commit aa98a5f
Show file tree
Hide file tree
Showing 6 changed files with 253 additions and 64 deletions.
49 changes: 49 additions & 0 deletions Diagnostics/DumpSlbRestState.ps1
@@ -0,0 +1,49 @@
[CmdletBinding()]
param(
[string][parameter(Mandatory=$false, HelpMessage="Network controller Base REST URI e.g. https://192.168.0.4")]$NcURI
= 'https://sa18n30nc.sa18.nttest.microsoft.com'
)

$headers = @{"Accept"="application/json"}
$content = "application/json; charset=UTF-8"
$network = "$NCURI/Networking/v1"
$slbStateRetry = 30
$maxRetryCount = 20

$method = "Put"
$uri = "$network/diagnostics/slbstate"

$body = '{"properties": { }}'

try
{

$result = Invoke-WebRequest -Headers $headers -ContentType $content -Method $method -Uri $uri -Body $body -DisableKeepAlive -UseBasicParsing -UseDefaultCredentials
$result.Content
$resultObject = ConvertFrom-Json $result.Content
$resultsUri = $network + $resultObject.properties.slbStateResult.resourceRef

$totalWait=0

do
{
$totalWait += $slbStateRetry
Write-Host ">>> Sleeping ... for $slbStateRetry seconds ..."
Start-Sleep -Seconds $slbStateRetry
Write-Host ">>> Polling ... $resultsUri"
$tempResult = Invoke-WebRequest -Headers $headers -Method GET -Uri $resultsUri -UseBasicParsing -UseDefaultCredentials
$tempResultObject = ConvertFrom-Json $tempResult.Content
Write-Host ">>> $(Get-Date -Format G) Current State: $($tempResultObject.properties.provisioningState)"
}
until (($tempResultObject.properties.provisioningState) -ne "Updating" -or $totalWait -gt $slbStateRetry * $maxRetryCount)

$fileName = "stateOp_" + [System.Math]::Truncate((Get-Date -UFormat %s)) + ".txt"
$tempResult.Content > $fileName
Write-Host "Success output written to $fileName" -ForegroundColor Green
}

catch
{
Write-Error "Failed $_"
throw
}
69 changes: 5 additions & 64 deletions Diagnostics/Troubleshooting.md
@@ -1,76 +1,17 @@
# Troubleshooting SDN

Deploying the Microsoft Windows SDN Stack may require some troubleshooting of problems that arise during fabric and tenant deployment. The instructions provided below is for collecting a set of data which will aid in the troubleshooting and triage process. Please look at the [SDN Troubleshooting](https://technet.microsoft.com/en-us/library/mt715794.aspx) TechNet article for more information on individual commands and triage.
Deploying the Microsoft Windows SDN Stack may require some troubleshooting of problems that arise during fabric and tenant deployment. Please reference the [SDN Troubleshooting Topic](https://technet.microsoft.com/en-us/library/mt715794.aspx) for more details.

Make sure you have the most recent diagnostic KBs (download location forthcoming) installed on all of your NC nodes and Hyper-V Hosts. Also, make sure the tools have been installed on the Hyper-V Hosts:
```none
PS> Add-WindowsFeature RSAT-NetworkController –IncludeManagementTools
PS> Import-Module NetworkControllerDiagnostics
```
### Triage and Data Collection
1. Validate that Network Controller is up and running correctly (Executed from one of the NC Nodes):
```none
PS> Debug-WinFabNodeStatus
```
Check that ReplicaStatus is Ready and HealthState is Ok (if any nodes are not in Ready/Ok state, note which one is unhealthy in the bug)

```none
PS> Get-NetworkControllerReplica
```

Check that the Replica Status is Ready for each service (if any service is not in Ready state, note which service is unhealthy and on which node it is running in the bug)

2. Validate the NC Host Agents have made connections to the Network Controller (Execute on each Hyper-V host)
```none
C:\> netstat -anp tcp |findstr 6640
```

There should be three ESTABLISHED connections and one LISTENING socket
- Listening on Hyper-V hosts IP on port 6640
- Two established connections to Hyper-V host IP on port 6640 from NC node(s) on ephemeral ports (> 32000) Connection established bet
- One established connection from Hyper-V host IP to REST IP on port 6640

3. Check the Network Controller’s configuration state (Executed from any Hyper-V host)
```none
PS> Debug-NetworkControllerConfigurationState -NcIpAddress <Enter FQDN or IP – based on cert subject name configured>
```

Look for any resources which have status Warning or Failure
_Caveat: If you deployed using VMM, please use the VMM variant of the script available on GitHub [Debug-NetworkControllerConfigurationStateVmm](https://github.com/Microsoft/SDN/blob/master/Diagnostics/Debug-NetworkControllerConfigurationVMM.ps1)_

4. Check the SLB Configuration State (Executed from an NC node)
```none
PS > Debug-SlbConfigState
```
Output location should be indicated – default is C:\SDNDiagnostics\NetworkControllerState\SlbConfigState.txt
_Caveat: This script does not work for VMM-based deployments_

5. Check policies in Host Agent
```none
C:\> ovsdb-client.exe dump tcp:127.0.0.1:6641 ms_vtep
```
The key table in this output is the ucast_macs_remote table which lists the tenant VM NIC IP and MAC address. Check to see if policy is missing for any given tenant VM IP address.

6. Look for HNV Provider Addresses (PA IPs) on the host
```none
PS > Get-ProviderAddress
```

Attach the full output of all of these commands to the bug.

### Collecting Logs and Traces
Next step will probably be log collection. In order to proceed in an investigation, we need both the Host ID and the Port Profile IDs of any VM NICs for which there is no policy available in the Host Agent’s OVSDB ms_vtep database.
If you aren't able to troubleshoot the issue on you're own, the next step will be to collect logs. In order to proceed in an investigation, we need both the Host ID and the Port Profile IDs of any VM NICs for which there is no policy available in the Host Agent’s OVSDB ms_vtep database.


1. Collect most recent ETL log files under C:\SDNDiagnostics\Logs directory on all NC nodes and Hyper-V host in question (Zip)
2. Execute this script to get the Host ID
1. Execute this script to get the Host ID
```none
PS > Get-ItemProperty "hklm:\system\currentcontrolset\services\nchostagent\parameters" -Name HostId |fl HostId
```
3. Execute this script (download from GitHub – [Get-AllPortProfiles](https://github.com/Microsoft/SDN/blob/master/Diagnostics/Get-AllPortProfiles.ps1) ) to get the Port Profile IDs for each VM (indicate which VM NIC does not have policies)

Attach this information to the bug as well.
2. Execute this script (download from GitHub – [Get-AllPortProfiles](https://github.com/Microsoft/SDN/blob/master/Diagnostics/Get-AllPortProfiles.ps1) ) to get the Port Profile IDs for each VM (indicate which VM NIC does not have policies)

Lastly, make a note of what was happening before the degradation of service or error occurred.

### Gateways Troubleshooting

Expand Down
32 changes: 32 additions & 0 deletions NCRESTExamples/OutboundNAT.ps1
@@ -0,0 +1,32 @@
$uri = "https://10.127.132.211"

#EXAMPLE2: Outbound NAT

$vipip = "10.127.132.34"
$lbid = "5290ff9c-b854-4d3d-a747-c220dd396181_10.127.132.34"
$publicvipln = get-networkcontrollerlogicalnetwork -ConnectionUri $uri -resourceid "5290ff9c-b854-4d3d-a747-c220dd396181"

$lb = Get-NetworkControllerLoadBalancer -Connectionuri $uri -ResourceId $lbid
$fe = Get-NetworkControllerLoadBalancerFrontendIPConfiguration -ConnectionUri $uri -LoadBalancerId $lbid
$backend = Get-NetworkControllerLoadBalancerBackendAddressPool -ConnectionUri $uri -LoadBalancerId $lbid

$onat = @{}
$onat.ResourceId = "onat1"
$onat.properties = @{}
$onat.properties.frontendipconfigurations = @()
$onat.properties.frontendipconfigurations += $fe
$onat.properties.backendaddresspool = $backend
$onat.properties.protocol = "ALL"
$lb.properties.OutboundNatRules += $onat

New-NetworkControllerLoadBalancer -ConnectionUri $uri -ResourceId $lbid -Properties $lb.properties

#Give a network interface outbound NAT access. Repeat for each NIC.
$lb = Get-NetworkControllerLoadBalancer -Connectionuri $uri -ResourceId $lbid

$nic = get-networkcontrollernetworkinterface -connectionuri $uri -resourceid "f5539fc6-d389-42ab-a450-74c738019cba"
$nic.properties.IpConfigurations[0].properties.LoadBalancerBackendAddressPools += $lb.properties.backendaddresspools[0]

new-networkcontrollernetworkinterface -connectionuri $uri -resourceid $nic.resourceid -properties $nic.properties -force


49 changes: 49 additions & 0 deletions NCRESTExamples/WebTierAcl.ps1
@@ -0,0 +1,49 @@
$uri = "https://10.127.132.211"

$ruleproperties = new-object Microsoft.Windows.NetworkController.AclRuleProperties
$ruleproperties.Protocol = "TCP"
$ruleproperties.SourcePortRange = "0-65535"
$ruleproperties.DestinationPortRange = "80"
$ruleproperties.Action = "Allow"
$ruleproperties.SourceAddressPrefix = "10.1.1.0/24"
$ruleproperties.DestinationAddressPrefix = "10.1.1.0/24"
$ruleproperties.Priority = "100"
$ruleproperties.Type = "Inbound"
$ruleproperties.Logging = "Enabled"

$aclrule1 = new-object Microsoft.Windows.NetworkController.AclRule
$aclrule1.Properties = $ruleproperties
$aclrule1.ResourceId = "AllowTCP80_Inbound"

$ruleproperties = new-object Microsoft.Windows.NetworkController.AclRuleProperties
$ruleproperties.Protocol = "All"
$ruleproperties.SourcePortRange = "0-65535"
$ruleproperties.DestinationPortRange = "0-65535"
$ruleproperties.Action = "Deny"
$ruleproperties.SourceAddressPrefix = "*"
$ruleproperties.DestinationAddressPrefix = "*"
$ruleproperties.Priority = "110"
$ruleproperties.Type = "Inbound"
$ruleproperties.Logging = "Enabled"

$aclrule2 = new-object Microsoft.Windows.NetworkController.AclRule
$aclrule2.Properties = $ruleproperties
$aclrule2.ResourceId = "BlockAll_Inbound"

$acllistproperties = new-object Microsoft.Windows.NetworkController.AccessControlListProperties
$acllistproperties.AclRules = @($aclrule1, $aclrule2)
$acl = New-NetworkControllerAccessControlList -ResourceId "VNet1_Subnet1_ACL" -Properties $acllistproperties -ConnectionUri $uri

$vnet = Get-NetworkControllerVirtualNetwork -ConnectionUri $uri | where { $_.properties.AddressSpace.AddressPrefixes[0] -eq "10.1.1.0/24" }
$vmsubnet = Get-NetworkControllerVirtualSubnet -ConnectionUri $uri -VirtualNetworkId $vnet.ResourceId | where { $_.Properties.AddressPrefix -eq "10.1.1.0/24" }

$vmsubnet.properties.AccessControlList = $acl

New-NetworkControllerVirtualSubnet -ConnectionUri $uri -VirtualNetworkId $vnet.ResourceId -ResourceId $vmsubnet.ResourceId -Properties $vmsubnet.properties



#$nic = get-networkcontrollernetworkinterface -ConnectionUri $uri -ResourceId "MyVM_Ethernet1"
#$acl = get-networkcontrolleraccesscontrollist -ConnectionUri $uri -resourceid "AllowAllACL"
#$nic.properties.ipconfigurations[0].properties.AccessControlList = $acl
#new-networkcontrollernetworkinterface -ConnectionUri $uri -Properties $nic.properties -ResourceId $nic.resourceid
52 changes: 52 additions & 0 deletions VMM/scripts/AddVNets.ps1
@@ -0,0 +1,52 @@
$logicalNetwork = Get-SCLogicalNetwork -ID "9c89d0d7-8e08-4022-823b-4924b7207847" # HNV Provider


$max_vnets = 25
$max_subnets = 5

foreach ($i in 1..$max_vnets)
{
# Create VM Network
$vnetname = "vnet$i"
$vmNetwork = New-SCVMNetwork -Name $vnetname -LogicalNetwork $logicalNetwork -IsolationType "WindowsNetworkVirtualization" -CAIPAddressPoolType "IPV4" -PAIPAddressPoolType "IPV4"
#Write-Output $vmNetwork

foreach ($j in 1..$max_subnets)
{
# Create VM Subnet
$prefix = "10.$i.$j.0/24"
$subnet = New-SCSubnetVLan -Subnet $prefix
$vmsubnet = New-SCVMSubnet -Name "Vnet$i-Subnet_$prefix" -VMNetwork $vmNetwork -SubnetVLan $subnet
#Write-Output $vmsubnet

# Create VM Subnet IP Pool
$allGateways = @()
$allDnsServer = @()
$allDnsSuffixes = @()
$allWinsServers = @()

New-SCStaticIPAddressPool -Name "Vnet$i-Subnet_$prefix IP Pool" -VMSubnet $vmSubnet -Subnet $prefix -IPAddressRangeStart "10.$i.$j.4" -IPAddressRangeEnd "10.$i.$j.254" -DefaultGateway $allGateways -DNSServer $allDnsServer -DNSSuffix "" -DNSSearchSuffix $allDnsSuffixes -RunAsynchronously
}
}

sleep 5

# Remove VNets
foreach ($i in 1..$max_vnets)
{
$vmnetwork = Get-SCVMNetwork -Name "vnet$i"

foreach ($j in 1..$max_subnets)
{
$prefix = "10.$i.$j.0/24"
$vmsubnet = Get-SCVMSubnet -Name "Vnet$i-Subnet_$prefix" -VMNetwork $vmnetwork
$ippool = Get-SCStaticIPAddressPool -Name "Vnet$i-Subnet_$prefix IP Pool" -VMSubnet $vmsubnet
Remove-SCStaticIPAddressPool -StaticIPAddressPool $ippool

# $vmsubnet = Get-SCVMSubnet -Name "Subnet_$prefix"
Remove-SCVMSubnet $vmsubnet
}

# $vmnetwork = Get-SCVMNetwork -Name "vnet$i"
Remove-SCVMNetwork $vmnetwork
}
66 changes: 66 additions & 0 deletions VMM/scripts/CreateTenantVIP.ps1
@@ -0,0 +1,66 @@
param(

[Parameter(Mandatory=$false)]
# Name of the Network Controller Network Service
# This value should be the name you gave the Network Controller service
# when you on-boarded the Network Controller to VMM
$LBServiceName = "NC",

[Parameter(Mandatory=$false)]
# Name of the VM instances to which you want to assign the VIP
$VipMemberVMNames = @("Web_VM1", "Web_VM2"),

[Parameter(Mandatory=$false)]
# VIP address you want to assign from the VIP pool.
# Pick any VIP that falls within your VIP IP Pool range.
$VipAddress = "10.127.132.34",

[Parameter(Mandatory=$false)]
# Name of the VIP VM Network
$VipNetworkName = "Public VIP",

[Parameter(Mandatory=$false)]
# The name of the VIP template you created via the VMM Console.
$VipTemplateName = "Web",

[Parameter(Mandatory=$false)]
# Arbitrary but good to match the VIP you're using.
$VipName = "TenantVIPWebTest"

)

Import-Module virtualmachinemanager

$lb = Get-scLoadBalancer | where { $_.Service.Name -eq $LBServiceName};
$vipNetwork = get-scvmnetwork -Name $VipNetworkName;

$vipMemberNics = @();
foreach ($vmName in $VipMemberVMNames)
{
$vm = get-scvirtualmachine -Name $vmName;
# if ($vm.VirtualNetworkAdapters[0].VMNetwork.ID -ne $vipNetwork.ID)
# {
# $vm.VirtualNetworkAdapters[0] | set-scvirtualnetworkadapter -VMNetwork $vipNetwork;
# }

$vipMemberNics += $vm.VirtualNetworkAdapters[0];
}

$existingVip = get-scloadbalancervip -Name $VipName
if ($existingVip -ne $null)
{
# foreach ($mem in $existingVip.VipMembers)
# {
# $mem | remove-scloadbalancervipmember;
# }

$existingVip | remove-scloadbalancervip;
}

$vipt = get-scloadbalancerviptemplate -Name $VipTemplateName;

$vip = New-SCLoadBalancerVIP -Name $VipName -LoadBalancer $lb -IPAddress $VipAddress -LoadBalancerVIPTemplate $vipt -FrontEndVMNetwork $vipNetwork -BackEndVirtualNetworkAdapters $vipMemberNics;
Write-Output "Created VIP " $vip;

$vip = get-scloadbalancervip -Name $VipName;
Write-Output "VIP with members " $vip;

0 comments on commit aa98a5f

Please sign in to comment.