k8s python library cannot connect to k8s cluster from pod through internal k8s api endpoint http://localhost:8080 despite the fact that kubectl works with same config

[rradecki-migo]

What happened (please include outputs or screenshots):

$ cat ~/.kube/myconfig
---

apiVersion: v1
clusters:
- cluster:
    server: http://localhost:8080
  name: arn:aws:eks:eu-central-1:MY_AWS_ACCOUNT_ID:cluster/staging
contexts:
- context:
    cluster: arn:aws:eks:eu-central-1:MY_AWS_ACCOUNT_ID:cluster/staging
    user: arn:aws:eks:eu-central-1:MY_AWS_ACCOUNT_ID:cluster/staging
  name: arn:aws:eks:eu-central-1:MY_AWS_ACCOUNT_ID:cluster/staging
current-context: arn:aws:eks:eu-central-1:MY_AWS_ACCOUNT_ID:cluster/staging
kind: Config
preferences: {}

$ kubectl --kubeconfig /root/.kube/myconfig get ns
NAME                         STATUS   AGE
default                      Active   2y11d
...

$ pip list | grep kubernetes
kubernetes          20.13.0

$ pip list
Package             Version
------------------- ---------
ansible             5.0.1
ansible-core        2.12.1
awscli              1.22.21
boto                2.49.0
boto3               1.20.21
botocore            1.23.21
cachetools          4.2.4
certifi             2021.10.8
cffi                1.15.0
charset-normalizer  2.0.7
colorama            0.4.3
cryptography        35.0.0
docutils            0.15.2
google-auth         2.3.3
idna                3.3
Jinja2              3.0.3
jmespath            0.10.0
kubernetes          20.13.0
MarkupSafe          2.0.1
netaddr             0.8.0
oauthlib            3.1.1
packaging           21.2
passlib             1.7.4
pip                 21.3.1
pyasn1              0.4.8
pyasn1-modules      0.2.8
pycparser           2.21
pyparsing           2.4.7
python-dateutil     2.8.2
python-string-utils 1.0.0
PyYAML              5.4.1
requests            2.26.0
requests-oauthlib   1.3.0
resolvelib          0.5.4
rsa                 4.7.2
ruamel.yaml         0.17.17
ruamel.yaml.clib    0.2.6
s3transfer          0.5.0
setuptools          59.1.1
six                 1.16.0
urllib3             1.26.7
websocket-client    1.2.1
wheel               0.37.0

$ python
Python 3.8.5 (default, Aug  5 2020, 08:22:02)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from kubernetes import client, config
>>> config.load_kube_config('/root/.kube/migo-staging.config')
>>> core_v1 = client.CoreV1Api()
>>> core_v1.list_namespace()
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/local/lib/python3.8/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.8/http/client.py", line 950, in send
    self.connect()
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f8de520b4f0>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 14447, in list_namespace
    return self.list_namespace_with_http_info(**kwargs)  # noqa: E501
  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 14554, in list_namespace_with_http_info
    return self.api_client.call_api(
  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 373, in request
    return self.rest_client.GET(url,
  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/rest.py", line 240, in GET
    return self.request("GET", url,
  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/rest.py", line 213, in request
    r = self.pool_manager.request(method, url,
  File "/usr/local/lib/python3.8/site-packages/urllib3/request.py", line 74, in request
    return self.request_encode_url(
  File "/usr/local/lib/python3.8/site-packages/urllib3/request.py", line 96, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "/usr/local/lib/python3.8/site-packages/urllib3/poolmanager.py", line 375, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 783, in urlopen
    return self.urlopen(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 783, in urlopen
    return self.urlopen(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 783, in urlopen
    return self.urlopen(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /api/v1/namespaces (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f8de520b4f0>: Failed to establish a new connection: [Errno 111] Connection refused'))

What you expected to happen:
I should get a response object with all available namespaces. Kubectl config with http://localhost:8080 endpoint works well with kubectl but is problematic when loaded from this library.

How to reproduce it (as minimally and precisely as possible):
Shown above.

Anything else we need to know?:

Environment:

• Kubernetes setup: EKS on AWS, pod which invokes k8s api commands is configured with a service account which uses IRSA (IAM roles for service accounts) mechanism to provide access to AWS api
• Kubernetes version (kubectl version):

kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.3", GitCommit:"c92036820499fedefec0f847e2054d824aea6cd1", GitTreeState:"clean", BuildDate:"2021-10-27T18:41:28Z", GoVersion:"go1.16.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.7-eks-d88609", GitCommit:"d886092805d5cc3a47ed5cf0c43de38ce442dfcb", GitTreeState:"clean", BuildDate:"2021-07-31T00:29:12Z", GoVersion:"go1.15.12", Compiler:"gc", Platform:"linux/amd64"}

• OS (e.g., MacOS 10.13.6):
  debian container

cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

• Python version (python --version)

python --version
Python 3.8.5

• Python client version (pip list | grep kubernetes)
  Provided above.

[roycaihw]

/assign @yliaog

[yliaog]

does it work if replacing localhost:8080 with 127.0.0.1:8080, or with a remote IP?

[rradecki-migo]

hi @yliaog
sadly it does not

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.