DNSService UDP listener is unable to start

[vitalybibikov]

After update from 6.0 to 6.1 there is an issue with DNSService:

Unhealthy event: SourceId='System.FabricDnsService', Property='Socket', HealthState='Warning', ConsiderWarningAsError=false.

DnsService UDP listener is unable to start. Please make sure there are no processes listening on the DNS port 53.

List of processes listening on the DNS port:
UDP 0.0.0.0:53 *:* 6612

And 6612 is blocked by svchost.exe: which is used by Docker service.

[nrandell]

Exactly the same problem here as well. Mine is blocked by svchost.exe
c:\windows\system32\svchost.exe -k netsvcs -p -s SharedAccess
procexp says it is for Internet Connection Sharing

[nrandell]

Even though I have this problem, I'm still able to use my development cluster - so your problem may be something else

[asaheed]

same problem here, as soon as I kill a process which is using port 53 the next process is grabbing it ... in the end I am unable to free the port (svchost, searchUI, chrome ... this port is loved by anyone)

[milvuko]

You have to stop the ICS service which is causing the issue. Go to Services (services.msc) and find the Internet Connection Sharing (ICS) service, then stop the service.
This can be done from the admin cmd window as well by executing the command:
sc stop sharedaccess

More information on ICS service is available at:
https://blogs.technet.microsoft.com/virtualization/2017/07/26/hyper-v-virtual-machine-gallery-and-networking-improvements/

Note that ICS is running by default on Windows RS3 Client OS.

[nrandell]

I've noticed that you need to kill ICS every reboot. I tried disabling it in services, but it appeared to be re-enabled and manually started by another process.

[tanwarsatya]

Exact same issues, disabled the ICS and its working. Any long term solution for this, and why this issues never came in earlier versions.

[lucamazzanti]

Same issue, now i will create a node without the default Service Fabric Local Cluster Manager preset, but with the samples available here, in this way i can disable the DNS Service.

The issue comes out from the service ICS, that uses 0.0.0.0:53.
Who is launching it manually?
I disabled it with services, regedit, but invain.
I think it is Hyper-V. I didnt had this issue months ago.
Please find a way tho configure this bootstrapper or leave each processes alive.
Docker CE for Windows is not an issue, it uses the port 53 on another network dedicated, please ask Hyper-V do the same if viable.

[nunocenteno]

I've seen the same issue. In my scenario, I've a Windows 10 dev machine, with Docker for Windows installed, Visual Studio and Service Fabric. I noticed that when the error of Dns is happening in Service Fabric, docker is working just fine, if I restart the ICS service, Service Fabric just works fine, but docker stops working. Somehow ICS is essential for Docker for Windows, or Hyper-V as someone already stated.

The solution that seems to be just working for me, is to stop ICS and restart Docker for Windows. Until the restart Docker keeps in a failing state.

[zaphide]

Noticed same issue (and killing ICS service does seem to clear the issue).

This also resulted in a build up of excessive CPU usage for both FabricDnsService and FabricGatewayService that would build gradually over time.

[pksorensen]

I disabled the dns service in my dev service fabric cluster instead.

irst select Remove local cluster from the little SF tray icon.

Then go to C:\Program Files\Microsoft SDKs\Service Fabric\ClusterSetup\NonSecure\OneNode (or FiveNode for a local 5 node cluster) and update ClusterManifestTemplate.json.

Look for:

"addOnFeatures": [
  "DnsService"
]

And just remove DnsService from that list.

Then start up the cluster again.

[mikkelhegn]

This should be fixed in the 6.2 runtime release

[lkts]

Hi, I am seeing this problem using 6.2 runtime.

[ninzavivek]

Could you provide information about your environment ?

• Is this a SFRP cluster on One box environment?
• Is the OS Windows or Linux?
• What is the error are you getting, we do list error message on SF explorer page.
• What is the exact SF version?

[lkts]

@ninzavivek Hi, this seems sporadic, i will get back to you if i am able to catch it.

[lkts]

@ninzavivek

1. Local 1 Node cluster
2. Windows 10 with latest updates

Unhealthy event: SourceId='System.FabricDnsService', Property='Socket', HealthState='Warning', ConsiderWarningAsError=false. DnsService UDP listener is unable to start. Please make sure there are no processes listening on the DNS port 53. List of processes listening on the DNS port:   
UDP    0.0.0.0:53             *:*                      7664    
UDP    127.0.0.1:53           *:*                    8924    
UDP    172.20.208.1:53        *:*                 5888
--

Those processes are Internet Connection Sharing (ICS), Cisco DNS Proxy, dockerd.
4. Version listed for services is 6.2.269.9494

[ninzavivek]

Could you provide more information on how the cluster was created (Are you using own cluster manifest file or default from the SDK or this is an upgraded cluster)?

Also, please confirm the value of "AllowMultipleListeners" parameter in DNS Service section of cluster manifest - http://localhost:19080/Explorer/index.html#/tab/manifest

  <Parameter Name="InstanceCount" Value="-1" />

  <Parameter Name="IsEnabled" Value="True" />

  **<Parameter Name="AllowMultipleListeners" Value="true" />**

</Section>

[lkts]

Default cluster created by SDK, no modifications.
AllowMultipleListeners is true.

[ninzavivek]

I am unable to reproduce this issue on RS3, RS4 machines and we haven't seen any other reports of it.

I will need logs to see what is happening on your instance. When the issue happens flush the logs as follows and share it with us.
o Pefrmon->Data Collector Sets->Event Trace Sessions->FabricTrace Session->left click->(Stop)
o Logs location can be found from the cluster manifest ( By default it is : C:\SFdevCluster\log\Traces)

[benmartens]

By "no other reports" I assume you mean no reports other than every else on this thread. I worked around this by using pksorensen's most recent post (and I assume all the other thumbs up did as well.)

[lkts]

@ninzavivek I am not sure i can share logs with you, warnings from DNS that i see are DNS UdpListener, failed to bind socket, error 10013

[suhuruli]

Any updates on this?

[anousham]

I am seeting the same issue.

[mikkelhegn]

To summarize some of the things on this thread. Original problem with the DNS Service not being able to share port with other services was fixed in 6.2.

There are other Cluster setup issues that some might run in to, this for instance (although title is "DNS Issue"): microsoft/service-fabric-cli#110

I'm closing this issue, as the original problem is solved, please open new issues for specific problems you may run in to. Thanks.

[ibuildit]

@mikkelhegn , I'm running the august 20 release from web installer and I have the same issue. I'm unable to deactivate ICS, in a matter of minutes it's back up again.

[ibuildit]

I'm trying to get it to work with Mesh on localhost, which in turn needs Docker...which uses the port.

It seems to me that what ever was attempted fixed in 6.2 is still there.

[ibuildit]

If I disable Hyper-V, the problem is gone and DNS service is OK, but of course, that makes Service Fabric very disappointed...

[joshmouch]

I get a similar issue with the latest Service Fabric on a local cluster with the default SDK installation. In the windows event log, I see:
DnsRemoteQueryOp activityId {4e50cbf3-0f56-4aa9-8dcd-de6837fea69e}, failed to start the UDP listener

I tried disabling the Internet Connection Sharing services. I tried closing Docker for Windows. I haven't tried disabling Hyper-V yet.

[kseager]

Are you sure there is nothing outstanding that is listening on UDP port 53? Check the output from: netstat -nao | find ":53" | find "UDP" to see if there are any processes listed

[festivus]

I have the same issue with latest sdk, single node cluster built using defaults. i have docker installed. ICS is using port 53.

[joshlang]

Same problem. Fresh install of service fabric & docker.

The instant I stop Internet Connection Sharing, the problem goes away.

[mchudinov]

Same problem with Fabric v 7.0.457 DNSService does not start. And the whole local cluster does not work.
I have docker for Windows installed. Stop of the ICS service does not help.

[RaulRG]

I am having the same problems. Docker will not start and the problem is that the port 53 is in use. The culprit is a service from Microsoft

c:\windows\system32\svchost.exe -k netsvcs -p -s SharedAccess

@nrandell already posted about this in 2018. Is there a way of configuring Docker to use another port? Killing the service does not make me feel very good and as I am using Docker for Windows it should coexist with it. As it seems, the service is configured to auto start.