ERROR: could not watch for Kubernetes service changes

[DipeshMitthalal]

Hi Team,
We noticed that Ambassador manytime complains about could not watch for Kubernetes service changes

We are runnign Ambassador as daemonset and i have attached logs of two ambassador isntances.
I would need your assistance in finding out if everything is ok? We also noticed that Ambassador has restarted few times.
ambassador2.log
ambassador.log

[MattCollinge]

We are seeing this too. It seems to be causing an error in K8s API server by passing a python None as the resource version: GET /api/v1/services?resourceVersion=None&watch=True
Logs from K8s API Server:

I1111 14:27:39.066553       1 get.go:238] Starting watch for /api/v1/services, rv=None labels= fields= timeout=49m56.934516242s
E1111 14:27:39.066690       1 status.go:64] apiserver received an error that is not an metav1.Status: resourceVersion: Invalid value: "None": strconv.ParseUint: parsing "None": invalid syntax
I1111 14:27:39.066960       1 wrap.go:42] GET /api/v1/services?resourceVersion=None&watch=True: (542.851µs) 500

@richarddli I think this is a cause of this issue: #854

[MattCollinge]

I tracked it down to the version of the K8s python library that you are using in Ambassador.
Version in Ambassador: https://github.com/kubernetes-client/python-base/blob/789de6a60dea9442ee47601a788d92c653ea69e9/watch/watch.py

Commit that I think fixes this issue: kubernetes-client/python-base@38cc265

The critical part being:

# For custom objects that we don't have model defined, json
            # deserialization results in dictionary
            elif (isinstance(js['object'], dict) and 'metadata' in js['object']
                  and 'resourceVersion' in js['object']['metadata']):
                self.resource_version = js['object']['metadata'][
                    'resourceVersion']

I believe the fix is to bump the version of kubernetes from 6.0.0 to 8.0.0 here: https://github.com/datawire/ambassador/blob/master/ambassador/requirements.txt

Just to add to the above this is the Ambassador log:

Traceback (most recent call last):
  File "/ambassador/kubewatch.py", line 447, in main
    watch_loop(restarter)
  File "/ambassador/kubewatch.py", line 345, in watch_loop
    for evt in watched:
  File "/usr/lib/python3.6/site-packages/kubernetes/watch/watch.py", line 122, in stream
    resp = func(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 14358, in list_service_for_all_namespaces
    (data) = self.list_service_for_all_namespaces_with_http_info(**kwargs)
  File "/usr/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 14455, in list_service_for_all_namespaces_with_http_info
    collection_formats=collection_formats)
  File "/usr/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 321, in call_api
    _return_http_data_only, collection_formats, _preload_content, _request_timeout)
  File "/usr/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 155, in __call_api
    _request_timeout=_request_timeout)
  File "/usr/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 342, in request
    headers=headers)
  File "/usr/lib/python3.6/site-packages/kubernetes/client/rest.py", line 231, in GET
    query_params=query_params)
  File "/usr/lib/python3.6/site-packages/kubernetes/client/rest.py", line 222, in request
    raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (500)
Reason: Internal Server Error
HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'Date': 'Tue, 13 Nov 2018 17:36:38 GMT', 'Content-Length': '186'})
HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"resourceVersion: Invalid value: \\"None\\": strconv.ParseUint: parsing \\"None\\": invalid syntax","code":500}\n'```

[kflynn]

I've switched Ambassador to kube-client 8.0.0 in commit e5dcd66, which will be included in 0.50.0-ea7. Leaving this open for verification that that helps.

[kflynn]

Closing this issue; ongoing Azure work is tracked in #1039.