Dude, Where’s My Network!? – A Case of The Unexplained

[itoleck]

When writing a script to check Windows Domain and SQL connectivity I encountered an error. My SQL server could not talk to the domain; strange I thought. The SQL server is in Azure and I only have so many credits so it stays off most of the time. Maybe the domain trust was broken, but I was sure I changed my domain not to expire machine passwords (Domain member Maximum machine account password age (Windows 10) – Windows security | Microsoft Docs). Let’s check.

Looks okay to me. I decided to take a look at the domain controller, which is the only domain controller for my home network, as I only run 1. I may be looney (Looney Tunes | WB Kids GO! – DC Kids – WB Parents | WB Kids GO) but I’m not crazy.

The first thing is to check the network. I did do some work on the disks last week and maybe I connected the wrong network port. The SuperMicro board has (2) 10GBE NICs. I was surprised when I was remoted into my domain controller to see this.

Since I am connected to the server there must be a network. It’s just that the server doesn’t know it’s connected to a network. At this point I have a question. What is responsible for Windows to know that it is connected to a network?

This question can be answered many ways, but I chose the most old-fashioned, the good ‘ol Control Panel->Windows Firewall. I will miss you Control Panel when you are finally completely removed from Windows 10.

Now that I know that the server thinks that it’s on a public network the communication problems make more sense. Now to change the server to domain profile. First, check the network profile from PowerShell.

I take a wild guess and do not even try to use Set-NetConnectionProfile. Let’s try to find out why the profiles seem to be missing. This brings us to question 2, which I didn’t have at the beginning. Who is responsible for my network profiles? There is a lot of discussion online about how this works. It turns out there are 2 Windows Services that work together to make the magic happen; Network Location Awareness (NLA) (The Role of NLA – Win32 apps | Microsoft Docs) and Network List Service (Netprofm) (Network List Manager policies (Windows 10) – Windows security | Microsoft Docs). The later does not have much publicly available information on how it works, but the simple job is to take what NLA says about how the machine is connected to it’s networks and create a profile that will either be Domain, Private or a Public/Guest network.

I took a detour and verified in WMI that there were no network profiles on the machine, not that I don’t trust PowerShell. This is how I verified with wbemtest.exe. This is a good profile. If the instances are empty that’s bad.

At this point there is a fork in the road. Do I go down the NLA path and look at the IP network settings to double check my default gateway and such or do I look at the Network List Service? The simple answer is that if you are on a machine who’s IP information does not change much; as in this case where the machine is a domain controller with a hard coded IP address, subnet mask and default gateway you can verify but it’s not likely that there is an issue. On machines that are mobile or use Wifi, NLA is much more important as there are many checks that go into guessing what kind of network the machine is connected to like IP address, default gateway but also if the Wifi connection is open or encrypted.

I must dig into the Network List Service and find out why it is misbehaving. When troubleshooting Windows Services it is a good practice to make sure the service is running in it’s own process. This can be done with sc.exe.

For some weird reason there is a space after the equals sign (type= own). Now restart the service to get it in it’s own process.

sc.exe stop netprofm

sc.exe start netprofm

Strange that the Services.msc control panel has a restart button but sc.exe does not have a command.

You can check the service is running by itself using Task Manager.

Now time to, “when in doubt use Procmon” (Process Monitor) (Windows Sysinternals – Windows Sysinternals | Microsoft Docs). When running Procmon it is good to know which process ID you want to trace. That way you can create a filter on the PID. You can also drop all other events so the output is just the meaty bits.

I captured the Procmon while starting the Network List service and stopped capturing when the service said “started”; which was only about 1 second and ended up with about 1,000 events to go through.

A favorite technique of mine is to filter the remaining events by the result of the event. Basically, if the event was successful or if there was an error code. To do this go to the Tools menu->Count Occurrences.

The event results have some scary sounding names like; Buffer Overflow, Name not Found, Path not Found and others but not all of them are errors. It quite depends on the situation. In this case though, Access Denied does look like an issue, and it usually is.

By double-clicking on the result, Procmon will add a filter to your events. Now I am looking at just the netprofm service’s permissions errors. If you double-click the event you will get more information on the event.

The details that are important are the registry path HKLM\Software\Microsoft\Windows NT\CurrentVersion\NetworkList\Profiles{31BAC97E-BBAA-4A8B-94BE-EFEA442DAA2F}, result (Access Denied), User (NT AUTHORITY\LOCAL SERVICE) and the registry operation (RegCreateKey). Looks like the netprofm service’s login account is having issues writing the network profile information to the registry.

Now the fix. The best way to approach this would be to look at a known good Windows Server 2016 server and compare the permissions of the registry and add or remove what is needed to get the NT AUTHORITY\LOCAL SERVICE account access. In my case though since this is my single home domain; I just changed the netprofm service to login using the localsystem account which does have access.

Next time maybe we’ll research ways to track down the why, but for now back to work.