dns.json missing in homestead and homestead-prov

[mdavis-xyz]

Symptoms

When I spin up a kubernetes deployment, homestead and homestead-prov fail to come up properly.

kubectl get pods

NAME                              READY     STATUS             RESTARTS   AGE
cassandra-7777c47568-ddxnj        1/1       Running            0          4h
homestead-568c574587-d8h8r        1/2       CrashLoopBackOff   41         4h
homestead-prov-7f9d9bc455-lmbhl   0/1       CrashLoopBackOff   41         4h
...

kubectl logs homestead-568c574587-d8h8r homestead

2018-05-15 07:01:33,214 INFO exited: homestead (exit status 0; not expected)
2018-05-15 07:01:47,228 INFO spawned: 'homestead' with pid 2685
2018-05-15 07:01:47,809 INFO exited: homestead (exit status 0; not expected)
2018-05-15 07:02:02,824 INFO spawned: 'homestead' with pid 2862
2018-05-15 07:02:03,400 INFO exited: homestead (exit status 0; not expected)
2018-05-15 07:02:19,417 INFO spawned: 'homestead' with pid 3039
2018-05-15 07:02:19,996 INFO exited: homestead (exit status 0; not expected)
2018-05-15 07:02:37,018 INFO spawned: 'homestead' with pid 3216
2018-05-15 07:02:37,632 INFO exited: homestead (exit status 0; not expected)
2018-05-15 07:02:55,650 INFO spawned: 'homestead' with pid 3393
2018-05-15 07:02:56,225 INFO exited: homestead (exit status 0; not expected)
2018-05-15 07:03:15,245 INFO spawned: 'homestead' with pid 3580
2018-05-15 07:03:15,821 INFO exited: homestead (exit status 0; not expected)
2018-05-15 07:03:21,188 WARN received SIGTERM indicating exit request
2018-05-15 07:03:21,188 INFO waiting for nginx, homestead, socket-factory-sig, socket-factory-mgmt to die
2018-05-15 07:03:21,189 INFO stopped: socket-factory-mgmt (terminated by SIGTERM)
2018-05-15 07:03:22,190 INFO stopped: socket-factory-sig (terminated by SIGTERM)
2018-05-15 07:03:22,195 INFO stopped: nginx (exit status 0)

When I docker exec into the container (hard to do because it keeps being restarted by k8s) and look in /var/log/homestead/homestead_current.txt, I see:

15-05-2018 07:05:40.126 UTC [7f6379c8e7c0] Status dnscachedresolver.cpp:123: Creating Cached Resolver using servers:
15-05-2018 07:05:40.126 UTC [7f6379c8e7c0] Status dnscachedresolver.cpp:133:     10.0.0.10
15-05-2018 07:05:40.126 UTC [7f6379c8e7c0] Error static_dns_cache.cpp:89: DNS config file /etc/clearwater/dns.json mis
sing
15-05-2018 07:05:40.126 UTC [7f6379c8e7c0] Status a_record_resolver.cpp:29: Created ARecordResolver
15-05-2018 07:05:40.126 UTC [7f6379c8e7c0] Status main.cpp:653: Using local impu store: astaire.2991678f-f9dd-4098-8fe
8-1395c572a307.westeurope.aksapp.io
15-05-2018 07:05:40.129 UTC [7f6379c8e7c0] Status http_connection_pool.cpp:37: Connection pool will use calculated res
ponse timeout of 550ms
15-05-2018 07:05:40.130 UTC [7f6379c8e7c0] Status httpconnection.h:35: Configuring HTTP Connection
15-05-2018 07:05:40.130 UTC [7f6379c8e7c0] Status httpconnection.h:36:   Connection created for server sprout.2991678f
-f9dd-4098-8fe8-1395c572a307.westeurope.aksapp.io:9888
15-05-2018 07:05:40.130 UTC [7f6379c8e7c0] Status main.cpp:1013: No HSS configured - using Homestead-prov
15-05-2018 07:05:40.130 UTC [7f6379c8e7c0] Status a_record_resolver.cpp:29: Created ARecordResolver
15-05-2018 07:05:40.130 UTC [7f6379c8e7c0] Status cassandra_store.cpp:266: Configuring store connection
15-05-2018 07:05:40.130 UTC [7f6379c8e7c0] Status cassandra_store.cpp:267:   Hostname:  cassandra.2991678f-f9dd-4098-8
fe8-1395c572a307.westeurope.aksapp.io
15-05-2018 07:05:40.130 UTC [7f6379c8e7c0] Status cassandra_store.cpp:268:   Port:      9160
15-05-2018 07:05:40.130 UTC [7f6379c8e7c0] Status cassandra_store.cpp:296: Configuring store worker pool
15-05-2018 07:05:40.130 UTC [7f6379c8e7c0] Status cassandra_store.cpp:297:   Threads:   10
15-05-2018 07:05:40.130 UTC [7f6379c8e7c0] Status cassandra_store.cpp:298:   Max Queue: 0
15-05-2018 07:05:40.131 UTC [7f6366ffd700] Status alarm.cpp:244: Reraising all alarms with a known state
15-05-2018 07:05:40.636 UTC [7f6379c8e7c0] Error main.cpp:1056: Failed to initialize the Cassandra store with error co
de 3.
15-05-2018 07:05:40.636 UTC [7f6379c8e7c0] Status main.cpp:1057: Homestead is shutting down

I think the crucial error message is DNS config file /etc/clearwater/dns.json missing

What is that file? Where does it come from?

Impact

Severe: It is not possible to deploy clearwater on kubernetes.

homestead and homestead-prov do not instantiate sucessfully.

Release and environment

Using Azure Kubernetes Service (AKS).

I've enabled HTTP Application Routing, which means that Azure manages DNS for me. The only unusual step is that Azure requires that I add ingresses to do so. The DNS records point to each service (bono, ellis, sprout etc).

Steps to reproduce

Apply pull requests #88 and #89. Then do a kubernetes deployment, on AKS.

I did not do this optional step because I do not think it is necessary for this case.

[mdavis-xyz]

I checked /etc/clearwater/dns.json on an all-in-one installation. The file exists, the contents is just:

{
  "hostnames": [
  ]
}

[johadalin]

This conversation is ongoing in parallel on our mailing list, so i won't duplicate all that here. Latest update is at http://lists.projectclearwater.org/mailman/private/clearwater_lists.projectclearwater.org/2018-May/003948.html

I think there are a fair few red-herrings in all of this. The dns.json not being present is currently just a benign artifact of how our processes run in containers. As in the thread, i think the real issue is net connectivity into your cassandra pods on port 9160, which is causing our processes to abort startup, and so get rebooted by kubernetes.
Hopefully the thread has enough info in it for you to continue debugging this problem 👍