/v1/HealthCheck dials default EtcdAdvertiseAddress even though only kubernetess is configured.

[nipak]

http://localhost:65111/v1/HealthCheck

{"error":"all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:81: connect: connection refused"","message":"all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:81: connect: connection refused"","code":14}

My env configuration

Environment:
  K8S_NODE_NAME:                  (v1:spec.nodeName)
  K8S_POD_NAME:                   (v1:metadata.name)
  K8S_POD_ID:                     (v1:metadata.uid)
  K8S_NAMESPACE:                  (v1:metadata.namespace)
  K8S_LABELS_APP:                 (v1:metadata.labels['app'])
  K8S_CONTAINER_NAME:            gubernator
  GUBER_DEBUG:                   true
  GUBER_HTTP_ADDRESS:            0.0.0.0:65111
  GUBER_GRPC_ADDRESS:            0.0.0.0:65110
  GUBER_CACHE_SIZE:              50000
  GUBER_BATCH_TIMEOUT:           50ms
  GUBER_BATCH_LIMIT:             1000
  GUBER_BATCH_WAIT:              500ns
  GUBER_K8S_NAMESPACE:            (v1:metadata.namespace)
  GUBER_K8S_POD_IP:               (v1:status.podIP)
  GUBER_K8S_POD_PORT:            65110
  GUBER_K8S_ENDPOINTS_SELECTOR:  app=gubernator

[nipak]

debug logs

time="2020-05-06T21:02:32Z" level=debug msg="Debug enabled"
time="2020-05-06T21:02:32Z" level=debug msg="K8s peer pool config found"
(main.ServerConfig) {
GRPCListenAddress: (string) (len=13) "0.0.0.0:65110",
EtcdAdvertiseAddress: (string) (len=12) "127.0.0.1:81",
HTTPListenAddress: (string) (len=13) "0.0.0.0:65111",
EtcdKeyPrefix: (string) (len=17) "/gubernator-peers",
CacheSize: (int) 50000,
EtcdConf: (clientv3.Config) {
Endpoints: ([]string) (len=1 cap=1) {
(string) (len=14) "localhost:2379"
},
AutoSyncInterval: (time.Duration) 0s,
DialTimeout: (time.Duration) 5s,
DialKeepAliveTime: (time.Duration) 0s,
DialKeepAliveTimeout: (time.Duration) 0s,
MaxCallSendMsgSize: (int) 0,
MaxCallRecvMsgSize: (int) 0,
TLS: (*tls.Config)(),
Username: (string) "",
Password: (string) "",
RejectOldCluster: (bool) false,
DialOptions: ([]grpc.DialOption) ,
Context: (context.Context)
},
Behaviors: (gubernator.BehaviorConfig) {
BatchTimeout: (time.Duration) 50ms,
BatchWait: (time.Duration) 500ns,
BatchLimit: (int) 1000,
GlobalSyncWait: (time.Duration) 0s,
GlobalTimeout: (time.Duration) 0s,
GlobalBatchLimit: (int) 0
},
K8PoolConf: (gubernator.K8sPoolConfig) {
OnUpdate: (gubernator.UpdateFunc) ,
Namespace: (string) (len=10) "xxx",
Selector: (string) (len=14) "app=gubernator",
PodIP: (string) (len=14) "192.168.12.112",
PodPort: (string) (len=5) "65110",
Enabled: (bool) true
}
}

W0506 21:20:15.159510 1 reflector.go:302] pkg/mod/k8s.io/client-go@v0.0.0-20190620085101-78d2af792bab/tools/cache/reflector.go:98: watch of *v1.Endpoints ended with: too old resource version: 96310397 (96311433)
W0506 21:34:34.291840 1 reflector.go:302] pkg/mod/k8s.io/client-go@v0.0.0-20190620085101-78d2af792bab/tools/cache/reflector.go:98: watch of *v1.Endpoints ended with: too old resource version: 96313784 (96314011)

[thrawn01]

Refactoring the functional tests to include GRPC Gateway so issues like this do not fall through the cracks.

[kakkoyun]

@thrawn01 I think I have got bitten by this as well. It would be nice to get #55 merged :)

[thrawn01]

Release https://github.com/mailgun/gubernator/releases/tag/v1.0.0-rc.1 should resolve this issue.