We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
I am trying to use gcsfs to access gcs from the NASA pleaides supercomputer.
I tried to initialize a gcsfs.mapping before having my credentials set up properly (although I thought my default credentials were valid.)
I was able to get past this error by setting GOOGLE_APPLICATION_CREDENTIALS environment variable as described in the google cloud storage docs.
GOOGLE_APPLICATION_CREDENTIALS
However, the info I got from gcsfs was now very helpful.
In [1]: import logging ...: logging.basicConfig( ...: format="%(created)0.3f %(levelname)s %(name)s %(message)s", ...: level=logging.INFO) ...: logging.getLogger("gcsfs.core").setLevel(logging.DEBUG) ...: logging.getLogger("gcsfs.gcsfs").setLevel(logging.DEBUG) In [2]: import gcsfs In [3]: fs = gcsfs.GCSFileSystem(project='pangeo-181919') 1519678908.044 INFO google.auth.compute_engine._metadata Compute Engine Metadata server unavailable. 1519678908.044 DEBUG gcsfs.core Connection with method "google_default" failed In [4]: gcsmap = gcsfs.mapping.GCSMap('pangeo-data', gcs=fs) 1519679002.503 DEBUG gcsfs.core exists(args=('pangeo-data',), kwargs={}) 1519679002.503 DEBUG gcsfs.core _list_buckets(args=(), kwargs={}) 1519679002.503 DEBUG gcsfs.core _call(args=('get', 'b/'), kwargs={'project': 'pangeo-181919'}) 1519679002.519 ERROR gcsfs.core _call exception: HTTPConnectionPool(host='metadata.google.internal', port=80): Max retries exceeded with url: /computeMetadata/v1/instance/service-accounts/default/?recursive=true (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fffcca57c18>: Failed to establish a new connection: [Errno -2] Name or service not known',)) Traceback (most recent call last): File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/urllib3/connection.py", line 141, in _new_conn (self.host, self.port), self.timeout, **extra_kw) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/urllib3/util/connection.py", line 60, in create_connection for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/socket.py", line 745, in getaddrinfo for res in _socket.getaddrinfo(host, port, family, type, proto, flags): socket.gaierror: [Errno -2] Name or service not known During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen chunked=chunked) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/urllib3/connectionpool.py", line 357, in _make_request conn.request(method, url, **httplib_request_kw) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/http/client.py", line 1239, in request self._send_request(method, url, body, headers, encode_chunked) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/http/client.py", line 1285, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/http/client.py", line 1234, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/http/client.py", line 1026, in _send_output self.send(msg) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/http/client.py", line 964, in send self.connect() File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/urllib3/connection.py", line 166, in connect conn = self._new_conn() File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/urllib3/connection.py", line 150, in _new_conn self, "Failed to establish a new connection: %s" % e) urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fffcca57c18>: Failed to establish a new connection: [Errno -2] Name or service not known During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/requests/adapters.py", line 440, in send timeout=timeout File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen _stacktrace=sys.exc_info()[2]) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/urllib3/util/retry.py", line 388, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='metadata.google.internal', port=80): Max retries exceeded with url: /computeMetadata/v1/instance/service-accounts/default/?recursive=true (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fffcca57c18>: Failed to establish a new connection: [Errno -2] Name or service not known',)) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/transport/requests.py", line 120, in __call__ **kwargs) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/requests/sessions.py", line 508, in request resp = self.send(prep, **send_kwargs) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/requests/sessions.py", line 618, in send r = adapter.send(request, **kwargs) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/requests/adapters.py", line 508, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='metadata.google.internal', port=80): Max retries exceeded with url: /computeMetadata/v1/instance/service-accounts/default/?recursive=true (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fffcca57c18>: Failed to establish a new connection: [Errno -2] Name or service not known',)) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/compute_engine/credentials.py", line 90, in refresh self._retrieve_info(request) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/compute_engine/credentials.py", line 72, in _retrieve_info service_account=self._service_account_email) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/compute_engine/_metadata.py", line 179, in get_service_account_info recursive=True) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/compute_engine/_metadata.py", line 115, in get response = request(url=url, method='GET', headers=_METADATA_HEADERS) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/transport/requests.py", line 124, in __call__ six.raise_from(new_exc, caught_exc) File "<string>", line 3, in raise_from google.auth.exceptions.TransportError: HTTPConnectionPool(host='metadata.google.internal', port=80): Max retries exceeded with url: /computeMetadata/v1/instance/service-accounts/default/?recursive=true (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fffcca57c18>: Failed to establish a new connection: [Errno -2] Name or service not known',)) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/gcsfs/core.py", line 431, in _call r = meth(self.base + path, params=kwargs, json=json) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/requests/sessions.py", line 521, in get return self.request('GET', url, **kwargs) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/transport/requests.py", line 198, in request self._auth_request, method, url, request_headers) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/credentials.py", line 121, in before_request self.refresh(request) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/compute_engine/credentials.py", line 96, in refresh six.raise_from(new_exc, caught_exc) File "<string>", line 3, in raise_from google.auth.exceptions.RefreshError: HTTPConnectionPool(host='metadata.google.internal', port=80): Max retries exceeded with url: /computeMetadata/v1/instance/service-accounts/default/?recursive=true (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fffcca57c18>: Failed to establish a new connection: [Errno -2] Name or service not known',)) 1519679004.039 ERROR gcsfs.core _call exception: HTTPConnectionPool(host='metadata.google.internal', port=80): Max retries exceeded with url: /computeMetadata/v1/instance/service-accounts/default/?recursive=true (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fffcca57d30>: Failed to establish a new connection: [Errno -2] Name or service not known',)) Traceback (most recent call last): File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/urllib3/connection.py", line 141, in _new_conn (self.host, self.port), self.timeout, **extra_kw) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/urllib3/util/connection.py", line 60, in create_connection for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/socket.py", line 745, in getaddrinfo for res in _socket.getaddrinfo(host, port, family, type, proto, flags): socket.gaierror: [Errno -2] Name or service not known During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen chunked=chunked) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/urllib3/connectionpool.py", line 357, in _make_request conn.request(method, url, **httplib_request_kw) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/http/client.py", line 1239, in request self._send_request(method, url, body, headers, encode_chunked) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/http/client.py", line 1285, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/http/client.py", line 1234, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/http/client.py", line 1026, in _send_output self.send(msg) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/http/client.py", line 964, in send self.connect() File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/urllib3/connection.py", line 166, in connect conn = self._new_conn() File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/urllib3/connection.py", line 150, in _new_conn self, "Failed to establish a new connection: %s" % e) urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fffcca57d30>: Failed to establish a new connection: [Errno -2] Name or service not known During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/requests/adapters.py", line 440, in send timeout=timeout File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen _stacktrace=sys.exc_info()[2]) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/urllib3/util/retry.py", line 388, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='metadata.google.internal', port=80): Max retries exceeded with url: /computeMetadata/v1/instance/service-accounts/default/?recursive=true (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fffcca57d30>: Failed to establish a new connection: [Errno -2] Name or service not known',)) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/transport/requests.py", line 120, in __call__ **kwargs) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/requests/sessions.py", line 508, in request resp = self.send(prep, **send_kwargs) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/requests/sessions.py", line 618, in send r = adapter.send(request, **kwargs) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/requests/adapters.py", line 508, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='metadata.google.internal', port=80): Max retries exceeded with url: /computeMetadata/v1/instance/service-accounts/default/?recursive=true (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fffcca57d30>: Failed to establish a new connection: [Errno -2] Name or service not known',)) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/compute_engine/credentials.py", line 90, in refresh self._retrieve_info(request) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/compute_engine/credentials.py", line 72, in _retrieve_info service_account=self._service_account_email) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/compute_engine/_metadata.py", line 179, in get_service_account_info recursive=True) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/compute_engine/_metadata.py", line 115, in get response = request(url=url, method='GET', headers=_METADATA_HEADERS) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/transport/requests.py", line 124, in __call__ six.raise_from(new_exc, caught_exc) File "<string>", line 3, in raise_from google.auth.exceptions.TransportError: HTTPConnectionPool(host='metadata.google.internal', port=80): Max retries exceeded with url: /computeMetadata/v1/instance/service-accounts/default/?recursive=true (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fffcca57d30>: Failed to establish a new connection: [Errno -2] Name or service not known',)) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/gcsfs/core.py", line 431, in _call r = meth(self.base + path, params=kwargs, json=json) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/requests/sessions.py", line 521, in get return self.request('GET', url, **kwargs) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/transport/requests.py", line 198, in request self._auth_request, method, url, request_headers) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/credentials.py", line 121, in before_request self.refresh(request) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/compute_engine/credentials.py", line 96, in refresh six.raise_from(new_exc, caught_exc) File "<string>", line 3, in raise_from google.auth.exceptions.RefreshError: HTTPConnectionPool(host='metadata.google.internal', port=80): Max retries exceeded with url: /computeMetadata/v1/instance/service-accounts/default/?recursive=true (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fffcca57d30>: Failed to establish a new connection: [Errno -2] Name or service not known',)) 1519679007.052 ERROR gcsfs.core _call exception: HTTPConnectionPool(host='metadata.google.internal', port=80): Max retries exceeded with url: /computeMetadata/v1/instance/service-accounts/default/?recursive=true (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fffcca72f98>: Failed to establish a new connection: [Errno -2] Name or service not known',)) Traceback (most recent call last): File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/urllib3/connection.py", line 141, in _new_conn (self.host, self.port), self.timeout, **extra_kw) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/urllib3/util/connection.py", line 60, in create_connection for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/socket.py", line 745, in getaddrinfo for res in _socket.getaddrinfo(host, port, family, type, proto, flags): socket.gaierror: [Errno -2] Name or service not known During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen chunked=chunked) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/urllib3/connectionpool.py", line 357, in _make_request conn.request(method, url, **httplib_request_kw) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/http/client.py", line 1239, in request self._send_request(method, url, body, headers, encode_chunked) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/http/client.py", line 1285, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/http/client.py", line 1234, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/http/client.py", line 1026, in _send_output self.send(msg) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/http/client.py", line 964, in send self.connect() File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/urllib3/connection.py", line 166, in connect conn = self._new_conn() File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/urllib3/connection.py", line 150, in _new_conn self, "Failed to establish a new connection: %s" % e) urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fffcca72f98>: Failed to establish a new connection: [Errno -2] Name or service not known During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/requests/adapters.py", line 440, in send timeout=timeout File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen _stacktrace=sys.exc_info()[2]) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/urllib3/util/retry.py", line 388, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='metadata.google.internal', port=80): Max retries exceeded with url: /computeMetadata/v1/instance/service-accounts/default/?recursive=true (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fffcca72f98>: Failed to establish a new connection: [Errno -2] Name or service not known',)) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/transport/requests.py", line 120, in __call__ **kwargs) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/requests/sessions.py", line 508, in request resp = self.send(prep, **send_kwargs) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/requests/sessions.py", line 618, in send r = adapter.send(request, **kwargs) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/requests/adapters.py", line 508, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='metadata.google.internal', port=80): Max retries exceeded with url: /computeMetadata/v1/instance/service-accounts/default/?recursive=true (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fffcca72f98>: Failed to establish a new connection: [Errno -2] Name or service not known',)) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/compute_engine/credentials.py", line 90, in refresh self._retrieve_info(request) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/compute_engine/credentials.py", line 72, in _retrieve_info service_account=self._service_account_email) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/compute_engine/_metadata.py", line 179, in get_service_account_info recursive=True) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/compute_engine/_metadata.py", line 115, in get response = request(url=url, method='GET', headers=_METADATA_HEADERS) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/transport/requests.py", line 124, in __call__ six.raise_from(new_exc, caught_exc) File "<string>", line 3, in raise_from google.auth.exceptions.TransportError: HTTPConnectionPool(host='metadata.google.internal', port=80): Max retries exceeded with url: /computeMetadata/v1/instance/service-accounts/default/?recursive=true (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fffcca72f98>: Failed to establish a new connection: [Errno -2] Name or service not known',)) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/gcsfs/core.py", line 431, in _call r = meth(self.base + path, params=kwargs, json=json) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/requests/sessions.py", line 521, in get return self.request('GET', url, **kwargs) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/transport/requests.py", line 198, in request self._auth_request, method, url, request_headers) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/credentials.py", line 121, in before_request self.refresh(request) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/compute_engine/credentials.py", line 96, in refresh six.raise_from(new_exc, caught_exc) File "<string>", line 3, in raise_from google.auth.exceptions.RefreshError: HTTPConnectionPool(host='metadata.google.internal', port=80): Max retries exceeded with url: /computeMetadata/v1/instance/service-accounts/default/?recursive=true (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fffcca72f98>: Failed to establish a new connection: [Errno -2] Name or service not known',)) 1519679016.526 ERROR gcsfs.core _call exception: HTTPConnectionPool(host='metadata.google.internal', port=80): Max retries exceeded with url: /computeMetadata/v1/instance/service-accounts/default/?recursive=true (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fffcc963128>: Failed to establish a new connection: [Errno -2] Name or service not known',)) Traceback (most recent call last): File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/urllib3/connection.py", line 141, in _new_conn (self.host, self.port), self.timeout, **extra_kw) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/urllib3/util/connection.py", line 60, in create_connection for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/socket.py", line 745, in getaddrinfo for res in _socket.getaddrinfo(host, port, family, type, proto, flags): socket.gaierror: [Errno -2] Name or service not known During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen chunked=chunked) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/urllib3/connectionpool.py", line 357, in _make_request conn.request(method, url, **httplib_request_kw) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/http/client.py", line 1239, in request self._send_request(method, url, body, headers, encode_chunked) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/http/client.py", line 1285, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/http/client.py", line 1234, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/http/client.py", line 1026, in _send_output self.send(msg) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/http/client.py", line 964, in send self.connect() File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/urllib3/connection.py", line 166, in connect conn = self._new_conn() File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/urllib3/connection.py", line 150, in _new_conn self, "Failed to establish a new connection: %s" % e) urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fffcc963128>: Failed to establish a new connection: [Errno -2] Name or service not known During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/requests/adapters.py", line 440, in send timeout=timeout File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen _stacktrace=sys.exc_info()[2]) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/urllib3/util/retry.py", line 388, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='metadata.google.internal', port=80): Max retries exceeded with url: /computeMetadata/v1/instance/service-accounts/default/?recursive=true (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fffcc963128>: Failed to establish a new connection: [Errno -2] Name or service not known',)) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/transport/requests.py", line 120, in __call__ **kwargs) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/requests/sessions.py", line 508, in request resp = self.send(prep, **send_kwargs) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/requests/sessions.py", line 618, in send r = adapter.send(request, **kwargs) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/requests/adapters.py", line 508, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='metadata.google.internal', port=80): Max retries exceeded with url: /computeMetadata/v1/instance/service-accounts/default/?recursive=true (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fffcc963128>: Failed to establish a new connection: [Errno -2] Name or service not known',)) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/compute_engine/credentials.py", line 90, in refresh self._retrieve_info(request) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/compute_engine/credentials.py", line 72, in _retrieve_info service_account=self._service_account_email) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/compute_engine/_metadata.py", line 179, in get_service_account_info recursive=True) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/compute_engine/_metadata.py", line 115, in get response = request(url=url, method='GET', headers=_METADATA_HEADERS) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/transport/requests.py", line 124, in __call__ six.raise_from(new_exc, caught_exc) File "<string>", line 3, in raise_from google.auth.exceptions.TransportError: HTTPConnectionPool(host='metadata.google.internal', port=80): Max retries exceeded with url: /computeMetadata/v1/instance/service-accounts/default/?recursive=true (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fffcc963128>: Failed to establish a new connection: [Errno -2] Name or service not known',)) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/gcsfs/core.py", line 431, in _call r = meth(self.base + path, params=kwargs, json=json) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/requests/sessions.py", line 521, in get return self.request('GET', url, **kwargs) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/transport/requests.py", line 198, in request self._auth_request, method, url, request_headers) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/credentials.py", line 121, in before_request self.refresh(request) File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/compute_engine/credentials.py", line 96, in refresh six.raise_from(new_exc, caught_exc) File "<string>", line 3, in raise_from google.auth.exceptions.RefreshError: HTTPConnectionPool(host='metadata.google.internal', port=80): Max retries exceeded with url: /computeMetadata/v1/instance/service-accounts/default/?recursive=true (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fffcc963128>: Failed to establish a new connection: [Errno -2] Name or service not known',)) --------------------------------------------------------------------------- gaierror Traceback (most recent call last) /nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/urllib3/connection.py in _new_conn(self) 140 conn = connection.create_connection( --> 141 (self.host, self.port), self.timeout, **extra_kw) 142 /nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/urllib3/util/connection.py in create_connection(address, timeout, source_address, socket_options) 59 ---> 60 for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): 61 af, socktype, proto, canonname, sa = res /nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/socket.py in getaddrinfo(host, port, family, type, proto, flags) 744 addrlist = [] --> 745 for res in _socket.getaddrinfo(host, port, family, type, proto, flags): 746 af, socktype, proto, canonname, sa = res gaierror: [Errno -2] Name or service not known During handling of the above exception, another exception occurred: NewConnectionError Traceback (most recent call last) /nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw) 600 body=body, headers=headers, --> 601 chunked=chunked) 602 /nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/urllib3/connectionpool.py in _make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw) 356 else: --> 357 conn.request(method, url, **httplib_request_kw) 358 /nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/http/client.py in request(self, method, url, body, headers, encode_chunked) 1238 """Send a complete request to the server.""" -> 1239 self._send_request(method, url, body, headers, encode_chunked) 1240 /nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/http/client.py in _send_request(self, method, url, body, headers, encode_chunked) 1284 body = _encode(body, 'body') -> 1285 self.endheaders(body, encode_chunked=encode_chunked) 1286 /nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/http/client.py in endheaders(self, message_body, encode_chunked) 1233 raise CannotSendHeader() -> 1234 self._send_output(message_body, encode_chunked=encode_chunked) 1235 /nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/http/client.py in _send_output(self, message_body, encode_chunked) 1025 del self._buffer[:] -> 1026 self.send(msg) 1027 /nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/http/client.py in send(self, data) 963 if self.auto_open: --> 964 self.connect() 965 else: /nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/urllib3/connection.py in connect(self) 165 def connect(self): --> 166 conn = self._new_conn() 167 self._prepare_conn(conn) /nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/urllib3/connection.py in _new_conn(self) 149 raise NewConnectionError( --> 150 self, "Failed to establish a new connection: %s" % e) 151 NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fffcc963128>: Failed to establish a new connection: [Errno -2] Name or service not known During handling of the above exception, another exception occurred: MaxRetryError Traceback (most recent call last) /nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/requests/adapters.py in send(self, request, stream, timeout, verify, cert, proxies) 439 retries=self.max_retries, --> 440 timeout=timeout 441 ) /nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw) 638 retries = retries.increment(method, url, error=e, _pool=self, --> 639 _stacktrace=sys.exc_info()[2]) 640 retries.sleep() /nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/urllib3/util/retry.py in increment(self, method, url, response, error, _pool, _stacktrace) 387 if new_retry.is_exhausted(): --> 388 raise MaxRetryError(_pool, url, error or ResponseError(cause)) 389 MaxRetryError: HTTPConnectionPool(host='metadata.google.internal', port=80): Max retries exceeded with url: /computeMetadata/v1/instance/service-accounts/default/?recursive=true (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fffcc963128>: Failed to establish a new connection: [Errno -2] Name or service not known',)) During handling of the above exception, another exception occurred: ConnectionError Traceback (most recent call last) /nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/transport/requests.py in __call__(self, url, method, body, headers, timeout, **kwargs) 119 method, url, data=body, headers=headers, timeout=timeout, --> 120 **kwargs) 121 return _Response(response) /nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/requests/sessions.py in request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json) 507 send_kwargs.update(settings) --> 508 resp = self.send(prep, **send_kwargs) 509 /nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/requests/sessions.py in send(self, request, **kwargs) 617 # Send the request --> 618 r = adapter.send(request, **kwargs) 619 /nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/requests/adapters.py in send(self, request, stream, timeout, verify, cert, proxies) 507 --> 508 raise ConnectionError(e, request=request) 509 ConnectionError: HTTPConnectionPool(host='metadata.google.internal', port=80): Max retries exceeded with url: /computeMetadata/v1/instance/service-accounts/default/?recursive=true (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fffcc963128>: Failed to establish a new connection: [Errno -2] Name or service not known',)) The above exception was the direct cause of the following exception: TransportError Traceback (most recent call last) /nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/compute_engine/credentials.py in refresh(self, request) 89 try: ---> 90 self._retrieve_info(request) 91 self.token, self.expiry = _metadata.get_service_account_token( /nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/compute_engine/credentials.py in _retrieve_info(self, request) 71 request, ---> 72 service_account=self._service_account_email) 73 /nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/compute_engine/_metadata.py in get_service_account_info(request, service_account) 178 'instance/service-accounts/{0}/'.format(service_account), --> 179 recursive=True) 180 /nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/compute_engine/_metadata.py in get(request, path, root, recursive) 114 --> 115 response = request(url=url, method='GET', headers=_METADATA_HEADERS) 116 /nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/transport/requests.py in __call__(self, url, method, body, headers, timeout, **kwargs) 123 new_exc = exceptions.TransportError(caught_exc) --> 124 six.raise_from(new_exc, caught_exc) 125 /nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/six.py in raise_from(value, from_value) TransportError: HTTPConnectionPool(host='metadata.google.internal', port=80): Max retries exceeded with url: /computeMetadata/v1/instance/service-accounts/default/?recursive=true (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fffcc963128>: Failed to establish a new connection: [Errno -2] Name or service not known',)) The above exception was the direct cause of the following exception: RefreshError Traceback (most recent call last) <ipython-input-4-8e538214a4de> in <module>() ----> 1 gcsmap = gcsfs.mapping.GCSMap('pangeo-data', gcs=fs) /nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/gcsfs/mapping.py in __init__(self, root, gcs, check, create) 41 if create: 42 self.gcs.mkdir(bucket) ---> 43 elif not self.gcs.exists(bucket): 44 raise ValueError("Bucket %s does not exist." 45 " Create bucket with the ``create=True`` keyword" % <decorator-gen-131> in exists(self, path) /nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/gcsfs/core.py in _tracemethod(f, self, *args, **kwargs) 49 logger.log(logging.DEBUG - 1, tb_io.getvalue()) 50 ---> 51 return f(self, *args, **kwargs) 52 53 /nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/gcsfs/core.py in exists(self, path) 768 return bool(self.info(path)) 769 else: --> 770 if bucket in self.buckets: 771 return True 772 else: /nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/gcsfs/core.py in buckets(self) 449 def buckets(self): 450 """Return list of available project buckets.""" --> 451 return [b["name"] for b in self._list_buckets()["items"]] 452 453 @classmethod <decorator-gen-122> in _list_buckets(self) /nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/gcsfs/core.py in _tracemethod(f, self, *args, **kwargs) 49 logger.log(logging.DEBUG - 1, tb_io.getvalue()) 50 ---> 51 return f(self, *args, **kwargs) 52 53 /nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/gcsfs/core.py in _list_buckets(self) 568 items = [] 569 page = self._call( --> 570 'get', 'b/', project=self.project 571 ) 572 <decorator-gen-117> in _call(self, method, path, *args, **kwargs) /nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/gcsfs/core.py in _tracemethod(f, self, *args, **kwargs) 49 logger.log(logging.DEBUG - 1, tb_io.getvalue()) 50 ---> 51 return f(self, *args, **kwargs) 52 53 /nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/gcsfs/core.py in _call(self, method, path, *args, **kwargs) 435 logger.exception("_call exception: %s", e) 436 if retry == self.retries - 1: --> 437 raise e 438 if is_retriable(e): 439 # retry /nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/gcsfs/core.py in _call(self, method, path, *args, **kwargs) 429 try: 430 time.sleep(2**retry - 1) --> 431 r = meth(self.base + path, params=kwargs, json=json) 432 validate_response(r, path) 433 break /nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/requests/sessions.py in get(self, url, **kwargs) 519 520 kwargs.setdefault('allow_redirects', True) --> 521 return self.request('GET', url, **kwargs) 522 523 def options(self, url, **kwargs): /nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/transport/requests.py in request(self, method, url, data, headers, **kwargs) 196 197 self.credentials.before_request( --> 198 self._auth_request, method, url, request_headers) 199 200 response = super(AuthorizedSession, self).request( /nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/credentials.py in before_request(self, request, method, url, headers) 119 # the http request.) 120 if not self.valid: --> 121 self.refresh(request) 122 self.apply(headers) 123 /nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/compute_engine/credentials.py in refresh(self, request) 94 except exceptions.TransportError as caught_exc: 95 new_exc = exceptions.RefreshError(caught_exc) ---> 96 six.raise_from(new_exc, caught_exc) 97 98 @property /nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/six.py in raise_from(value, from_value) RefreshError: HTTPConnectionPool(host='metadata.google.internal', port=80): Max retries exceeded with url: /computeMetadata/v1/instance/service-accounts/default/?recursive=true (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fffcc963128>: Failed to establish a new connection: [Errno -2] Name or service not known',))
The text was updated successfully, but these errors were encountered:
No branches or pull requests
I am trying to use gcsfs to access gcs from the NASA pleaides supercomputer.
I tried to initialize a gcsfs.mapping before having my credentials set up properly (although I thought my default credentials were valid.)
I was able to get past this error by setting
GOOGLE_APPLICATION_CREDENTIALSenvironment variable as described in the google cloud storage docs.However, the info I got from gcsfs was now very helpful.
The text was updated successfully, but these errors were encountered: