K8S: add retries for job and pod status fetching

[oavdeev]

When running our integration test suite, for longer running tests I'd regularly run into issues when K8S API sometimes returns intermittent 500 errors, like the one below, or a variation of it with GRPC "Transport closed" error, and that would cause tasks to fail. This PR adds retries to K8S API calls that are made most frequently, and that fixes the issue for integration tests.

We could extend this to other K8S API calls we make too (but it could be another PR if that looks good).

2021-10-12 20:51:09.614 [10748/foreach_join_x/197086 (pid 52067)] Internal error
--
  | 2021-10-12 20:51:09.623 [10748/foreach_join_x/197086 (pid 52067)] Traceback (most recent call last):
  | 2021-10-12 20:51:09.623 [10748/foreach_join_x/197086 (pid 52067)] File "/var/lib/buildkite-agent/builds/buildkite-i-0455616781cbc5b41-1/outerbounds/integration-test-suite-eks/metaflow/cli.py", line 1008, in main
  | 2021-10-12 20:51:09.623 [10748/foreach_join_x/197086 (pid 52067)] start(auto_envvar_prefix='METAFLOW', obj=state)
  | 2021-10-12 20:51:09.624 [10748/foreach_join_x/197086 (pid 52067)] File "/var/lib/buildkite-agent/.local/lib/python3.7/site-packages/click/core.py", line 1128, in __call__
  | 2021-10-12 20:51:09.624 [10748/foreach_join_x/197086 (pid 52067)] return self.main(args, kwargs)
  | 2021-10-12 20:51:10.201 [10748/foreach_join_x/197086 (pid 52067)] File "/var/lib/buildkite-agent/.local/lib/python3.7/site-packages/click/core.py", line 1053, in main
  | 2021-10-12 20:51:10.201 [10748/foreach_join_x/197086 (pid 52067)] rv = self.invoke(ctx)
  | 2021-10-12 20:51:10.201 [10748/foreach_join_x/197086 (pid 52067)] File "/var/lib/buildkite-agent/.local/lib/python3.7/site-packages/click/core.py", line 1659, in invoke
  | 2021-10-12 20:51:10.201 [10748/foreach_join_x/197086 (pid 52067)] return _process_result(sub_ctx.command.invoke(sub_ctx))
  | 2021-10-12 20:51:10.201 [10748/foreach_join_x/197086 (pid 52067)] File "/var/lib/buildkite-agent/.local/lib/python3.7/site-packages/click/core.py", line 1659, in invoke
  | 2021-10-12 20:51:10.202 [10748/foreach_join_x/197086 (pid 52067)] return _process_result(sub_ctx.command.invoke(sub_ctx))
  | 2021-10-12 20:51:10.202 [10748/foreach_join_x/197086 (pid 52067)] File "/var/lib/buildkite-agent/.local/lib/python3.7/site-packages/click/core.py", line 1395, in invoke
  | 2021-10-12 20:51:10.202 [10748/foreach_join_x/197086 (pid 52067)] return ctx.invoke(self.callback, ctx.params)
  | 2021-10-12 20:51:10.202 [10748/foreach_join_x/197086 (pid 52067)] File "/var/lib/buildkite-agent/.local/lib/python3.7/site-packages/click/core.py", line 754, in invoke
  | 2021-10-12 20:51:10.202 [10748/foreach_join_x/197086 (pid 52067)] return __callback(args, kwargs)
  | 2021-10-12 20:51:10.202 [10748/foreach_join_x/197086 (pid 52067)] File "/var/lib/buildkite-agent/.local/lib/python3.7/site-packages/click/decorators.py", line 26, in new_func
  | 2021-10-12 20:51:10.203 [10748/foreach_join_x/197086 (pid 52067)] return f(get_current_context(), args, kwargs)
  | 2021-10-12 20:51:10.203 [10748/foreach_join_x/197086 (pid 52067)] File "/var/lib/buildkite-agent/builds/buildkite-i-0455616781cbc5b41-1/outerbounds/integration-test-suite-eks/metaflow/plugins/aws/eks/kubernetes_cli.py", line 238, in step
  | 2021-10-12 20:51:10.203 [10748/foreach_join_x/197086 (pid 52067)] kubernetes.wait(echo=echo)
  | 2021-10-12 20:51:10.203 [10748/foreach_join_x/197086 (pid 52067)] File "/var/lib/buildkite-agent/builds/buildkite-i-0455616781cbc5b41-1/outerbounds/integration-test-suite-eks/metaflow/plugins/aws/eks/kubernetes.py", line 355, in wait
  | 2021-10-12 20:51:10.203 [10748/foreach_join_x/197086 (pid 52067)] wait_for_launch(self._job)
  | 2021-10-12 20:51:10.204 [10748/foreach_join_x/197086 (pid 52067)] File "/var/lib/buildkite-agent/builds/buildkite-i-0455616781cbc5b41-1/outerbounds/integration-test-suite-eks/metaflow/plugins/aws/eks/kubernetes.py", line 321, in wait_for_launch
  | 2021-10-12 20:51:10.204 [10748/foreach_join_x/197086 (pid 52067)] new_status = job.status
  | 2021-10-12 20:51:10.204 [10748/foreach_join_x/197086 (pid 52067)] File "/var/lib/buildkite-agent/builds/buildkite-i-0455616781cbc5b41-1/outerbounds/integration-test-suite-eks/metaflow/plugins/aws/eks/kubernetes_client.py", line 682, in status
  | 2021-10-12 20:51:10.204 [10748/foreach_join_x/197086 (pid 52067)] return self._get_status()
  | 2021-10-12 20:51:10.204 [10748/foreach_join_x/197086 (pid 52067)] File "/var/lib/buildkite-agent/builds/buildkite-i-0455616781cbc5b41-1/outerbounds/integration-test-suite-eks/metaflow/plugins/aws/eks/kubernetes_client.py", line 546, in _get_status
  | 2021-10-12 20:51:10.204 [10748/foreach_join_x/197086 (pid 52067)] self._pod = self._fetch_pod()
  | 2021-10-12 20:51:10.205 [10748/foreach_join_x/197086 (pid 52067)] File "/var/lib/buildkite-agent/builds/buildkite-i-0455616781cbc5b41-1/outerbounds/integration-test-suite-eks/metaflow/plugins/aws/eks/kubernetes_client.py", line 423, in _fetch_pod
  | 2021-10-12 20:51:10.205 [10748/foreach_join_x/197086 (pid 52067)] label_selector="job-name={}".format(self._name),
  | 2021-10-12 20:51:10.205 [10748/foreach_join_x/197086 (pid 52067)] File "/var/lib/buildkite-agent/.local/lib/python3.7/site-packages/kubernetes/client/api/core_v1_api.py", line 15302, in list_namespaced_pod
  | 2021-10-12 20:51:10.205 [10748/foreach_join_x/197086 (pid 52067)] return self.list_namespaced_pod_with_http_info(namespace, kwargs)  # noqa: E501
  | 2021-10-12 20:51:10.205 [10748/foreach_join_x/197086 (pid 52067)] File "/var/lib/buildkite-agent/.local/lib/python3.7/site-packages/kubernetes/client/api/core_v1_api.py", line 15427, in list_namespaced_pod_with_http_info
  | 2021-10-12 20:51:10.206 [10748/foreach_join_x/197086 (pid 52067)] collection_formats=collection_formats)
  | 2021-10-12 20:51:10.206 [10748/foreach_join_x/197086 (pid 52067)] File "/var/lib/buildkite-agent/.local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 353, in call_api
  | 2021-10-12 20:51:10.206 [10748/foreach_join_x/197086 (pid 52067)] _preload_content, _request_timeout, _host)
  | 2021-10-12 20:51:10.206 [10748/foreach_join_x/197086 (pid 52067)] File "/var/lib/buildkite-agent/.local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 184, in __call_api
  | 2021-10-12 20:51:10.206 [10748/foreach_join_x/197086 (pid 52067)] _request_timeout=_request_timeout)
  | 2021-10-12 20:51:10.206 [10748/foreach_join_x/197086 (pid 52067)] File "/var/lib/buildkite-agent/.local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 377, in request
  | 2021-10-12 20:51:10.207 [10748/foreach_join_x/197086 (pid 52067)] headers=headers)
  | 2021-10-12 20:51:10.207 [10748/foreach_join_x/197086 (pid 52067)] File "/var/lib/buildkite-agent/.local/lib/python3.7/site-packages/kubernetes/client/rest.py", line 243, in GET
  | 2021-10-12 20:51:10.207 [10748/foreach_join_x/197086 (pid 52067)] query_params=query_params)
  | 2021-10-12 20:51:10.207 [10748/foreach_join_x/197086 (pid 52067)] File "/var/lib/buildkite-agent/.local/lib/python3.7/site-packages/kubernetes/client/rest.py", line 233, in request
  | 2021-10-12 20:51:10.207 [10748/foreach_join_x/197086 (pid 52067)] raise ApiException(http_resp=r)
  | 2021-10-12 20:51:10.207 [10748/foreach_join_x/197086 (pid 52067)] kubernetes.client.exceptions.ApiException: (500)
  | 2021-10-12 20:51:10.208 [10748/foreach_join_x/197086 (pid 52067)] Reason: Internal Server Error
  | 2021-10-12 20:51:10.208 [10748/foreach_join_x/197086 (pid 52067)] HTTP response headers: HTTPHeaderDict({'Audit-Id': '5e724cff-c10d-4ceb-895d-cfd8628552e9', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': 'cd0ea542-87b8-4eff-a88e-08f6aa6e3a68', 'X-Kubernetes-Pf-Prioritylevel-Uid': '8d7e6e31-ea70-4ae2-b89f-1ea3e1738e07', 'Date': 'Tue, 12 Oct 2021 20:51:09 GMT', 'Content-Length': '122'})
  | 2021-10-12 20:51:10.208 [10748/foreach_join_x/197086 (pid 52067)] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"etcdserver: request timed out","code":500}
  | 2021-10-12 20:51:10.208 [10748/foreach_join_x/197086 (pid 52067)]
  | 2021-10-12 20:51:10.208 [10748/foreach_join_x/197086 (pid 52067)]
  | 2021-10-12 20:51:10.209 [10748/foreach_join_x/197086 (pid 52067)]
  | 2021-10-12 20:51:10.277 [10748/foreach_join_x/197086 (pid 52067)] Task failed.