Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

All of my gsutil commands cannot works and return socket.timeout: timed out #498

Closed
hhh151671 opened this issue Jan 29, 2018 · 18 comments
Closed

Comments

@hhh151671
Copy link

my version:

gcloud info
Google Cloud SDK [185.0.0]
...
Python Version: [2.7.12 (default, Nov 20 2017, 18:23:56)  [GCC 5.4.0 20160609]]
Python Location: [/usr/bin/python2]
Site Packages: [Disabled]

Installation Root: [/home/david/important/TS_P/google-cloud-sdk]
Installed Components:
  core: [2018.01.12]
  gsutil: [4.28]
  bq: [2.0.28]
...
Installation Properties: [/home/david/important/TS_P/google-cloud-sdk/properties]
User Config Directory: [/home/david/.config/gcloud]
Active Configuration Name: [default]
Active Configuration Path: [/home/david/.config/gcloud/configurations/config_default]
Logs Directory: [/home/david/.config/gcloud/logs]
Last Log File: [/home/david/.config/gcloud/logs/2018.01.29/08.58.57.179050.log]

git: [git version 2.7.4]
ssh: [OpenSSH_7.2p2 Ubuntu-4ubuntu2.4, OpenSSL 1.0.2g  1 Mar 2016]

I have a poxy(socks5) on my linux .And I can run gcloud init and connect successfully.But my all gsutil commands cannot works and have no request timeout,like these:

gsutil ls -d
INFO 0128 22:56:07.070065 util.py] Retrying request, attempt #1...
INFO 0128 22:57:09.526079 util.py] Retrying request, attempt #2...
INFO 0128 22:58:13.198094 util.py] Retrying request, attempt #3...
...
INFO 0128 23:26:39.959507 util.py] Retrying request, attempt #22...
Traceback (most recent call last):
  File "/home/david/important/TS_P/google-cloud-sdk/platform/gsutil/gsutil", line 22, in <module>
    gsutil.RunMain()
  File "/home/david/important/TS_P/google-cloud-sdk/platform/gsutil/gsutil.py", line 114, in RunMain
    sys.exit(gslib.__main__.main())
  File "/home/david/important/TS_P/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 383, in main
    perf_trace_token=perf_trace_token)
  File "/home/david/important/TS_P/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 577, in _RunNamedCommandAndHandleExceptions
    collect_analytics=True)
  File "/home/david/important/TS_P/google-cloud-sdk/platform/gsutil/gslib/command_runner.py", line 299, in RunNamedCommand
    return_code = command_inst.RunCommand()
  File "/home/david/important/TS_P/google-cloud-sdk/platform/gsutil/gslib/commands/ls.py", line 464, in RunCommand
    bucket_fields=bucket_fields):
  File "/home/david/important/TS_P/google-cloud-sdk/platform/gsutil/gslib/wildcard_iterator.py", line 456, in IterBuckets
    for blr in self._ExpandBucketWildcards(bucket_fields=bucket_fields):
  File "/home/david/important/TS_P/google-cloud-sdk/platform/gsutil/gslib/wildcard_iterator.py", line 375, in _ExpandBucketWildcards
    provider=self.wildcard_url.scheme):
  File "/home/david/important/TS_P/google-cloud-sdk/platform/gsutil/gslib/gcs_json_api.py", line 531, in ListBuckets
    global_params=global_params)
  File "/home/david/important/TS_P/google-cloud-sdk/platform/gsutil/gslib/third_party/storage_apitools/storage_v1_client.py", line 425, in List
    config, request, global_params=global_params)
  File "/home/david/important/TS_P/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/base_api.py", line 701, in _RunMethod
    http, http_request, **opts)
  File "/home/david/important/TS_P/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/http_wrapper.py", line 341, in MakeRequest
    check_response_func=check_response_func)
  File "/home/david/important/TS_P/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/http_wrapper.py", line 391, in _MakeRequestNoRetry
    redirections=redirections, connection_type=connection_type)
  File "/home/david/important/TS_P/google-cloud-sdk/platform/gsutil/third_party/oauth2client/oauth2client/client.py", line 616, in new_request
    self._refresh(request_orig)
  File "/home/david/important/TS_P/google-cloud-sdk/platform/gsutil/third_party/oauth2client/oauth2client/client.py", line 885, in _refresh
    self._do_refresh_request(http_request)
  File "/home/david/important/TS_P/google-cloud-sdk/platform/gsutil/third_party/oauth2client/oauth2client/client.py", line 905, in _do_refresh_request
    self.token_uri, method='POST', body=body, headers=headers)
  File "/home/david/important/TS_P/google-cloud-sdk/platform/gsutil/third_party/httplib2/python2/httplib2/__init__.py", line 1570, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/home/david/important/TS_P/google-cloud-sdk/platform/gsutil/third_party/httplib2/python2/httplib2/__init__.py", line 1317, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/home/david/important/TS_P/google-cloud-sdk/platform/gsutil/third_party/httplib2/python2/httplib2/__init__.py", line 1252, in _conn_request
    conn.connect()
  File "/home/david/important/TS_P/google-cloud-sdk/platform/gsutil/third_party/httplib2/python2/httplib2/__init__.py", line 1018, in connect
    sock.connect((self.host, self.port))
  File "/home/david/important/TS_P/google-cloud-sdk/platform/gsutil/third_party/httplib2/python2/httplib2/socks.py", line 424, in connect
    self.__negotiatehttp(destpair[0], destpair[1])
  File "/home/david/important/TS_P/google-cloud-sdk/platform/gsutil/third_party/httplib2/python2/httplib2/socks.py", line 376, in __negotiatehttp
    resp = resp + self.recv(1)
socket.timeout: timed out

How to solve it?

@houglum
Copy link
Collaborator

houglum commented Feb 1, 2018

This issue isn't specific to gsutil; we've seen a slightly-higher-than-usual number of socket timeouts to GCS in general over the past week or so. The GCS team is looking into it. For Google developers wanting more information, the issue tracker number is 72221304 (sorry to external users; the visibility for that issue is internal-only).

Given that this isn't a gsutil bug, I'll go ahead and close this GitHub issue.

@houglum houglum closed this as completed Feb 1, 2018
@kaizen1

This comment has been minimized.

@hollicksfeir
Copy link

I had the same problem, for me it was a firewall configuration issue. I had an egress deny on tcp 80,443, 8080, which explains the socket timeout. I changed this for my service account and now it works. Hope this helps

@jvargh81
Copy link

jvargh81 commented Mar 6, 2019

INFO 0306 14:44:04.069279 retry_util.py] Retrying request, attempt #6...

@den-run-ai
Copy link

Same issue, how do I even troubleshoot this?

gsutil ls
INFO 0417 16:07:04.280598 retry_util.py] Retrying request, attempt #4...
INFO 0417 16:07:16.602803 retry_util.py] Retrying request, attempt #5...
INFO 0417 16:07:44.378415 retry_util.py] Retrying request, attempt #6...

I was just following this tutorial:

https://cloud.google.com/vision/automl/docs/quickstart?refresh=1

@den-run-ai
Copy link

BTW, I'm using the cloud shell, so supposedly no firewall issues should apply to me?

@houglum
Copy link
Collaborator

houglum commented Apr 17, 2019

@denfromufa you can use the top-level -D flag to see the command being run, which request is failing, etc.

I can get this to happen as well -- I think this occurs because the "gsutil" executable on Cloud Shell instances points to the gcloud installation of gsutil, which assumes that if you're running on GCE, you're on an instance owned by your project. Normally, it will just use the GCE auth plugin to query the VM instance's metadata server and get credentials for your instance's default service account. However, since the instance is owned by the Cloud Shell project, it can't do that, and it fails when trying to fetch those credentials (I see a BadStatusLine error when running gsutil -D ls).

I confirmed this by creating a config file with my credentials at ~/.boto, then running the underlying gsutil executable (/google/google-cloud-sdk/platform/gsutil/gsutil) directly -- this works fine, whereas the gcloud gsutil executable fails because it proxies an extra option through to gsutil that causes it to try and fetch credentials from the GCE metadata server (the -D output shows the command as Command being run: /google/google-cloud-sdk/platform/gsutil/gsutil -o GSUtil:default_project_id=myproject -o GoogleCompute:service_account=default -D ls).

This is an issue with the Cloud SDK -- it should detect if users are running on the Cloud Shell, and if so, not pass the GoogleCompute:service_account=default option through to gsutil. Please file an issue with the Cloud SDK team by clicking the "Create new Cloud SDK issue" link at https://cloud.google.com/support/docs/issue-trackers.

@den-run-ai
Copy link

den-run-ai commented Apr 18, 2019

@houglum it works today without any changes ¯\_(ツ)_/¯

@houglum
Copy link
Collaborator

houglum commented Apr 18, 2019

Still fails for me today. I'd still guess that the problem comes from trying to obtain the initial access token, but I'd have to add some import pdb; pdb.set_trace() statements down in the depths of the system's httplib module and poke around to be sure.

@shihabuddinbuet
Copy link

shihabuddinbuet commented Nov 24, 2019

I'm facing the same issue from my compute engine. Yesterday the gsutil command was working fine. I didn't do any change over the night. If i run the command enabling debug mode, i'm getting following error :

DEBUG 1124 03:46:39.659342 http_wrapper.py] Caught server not found error, retrying: Unable to find the server at storage.googleapis.com
DEBUG 1124 03:46:39.659546 http_wrapper.py] Retrying request to url  after exception Unable to find the server at storage.googleapis.com

@shihabuddinbuet
Copy link

I'm facing the same issue from my compute engine. Yesterday the gsutil command was working fine. I didn't do any change over the night. If i run the command enabling debug mode, i'm getting following error :

DEBUG 1124 03:46:39.659342 http_wrapper.py] Caught server not found error, retrying: Unable to find the server at storage.googleapis.com
DEBUG 1124 03:46:39.659546 http_wrapper.py] Retrying request to url  after exception Unable to find the server at storage.googleapis.com

It's working now without any change. Don't know why. :(

@surak
Copy link

surak commented Aug 7, 2020

I had the same problem. It was ipv6.

What I tried was: gsutil -D blabla
It has shown me the url. I copied it and went to wget as
wget "URL" (with quotes, as the URL has some &)

It got lost trying to resolve ipv6 name - Disabled ipv6 and all is well in the world.

@ggabriel96
Copy link

ggabriel96 commented Oct 8, 2020

Been having this problem for the third day in a row. Disabling IPv6 did not help. I just cannot find out what is going on.

The weird part is that some other things break too and it looks like it's not deterministic... like sometimes go get, apk add, or apt-get install fail (mainly building Docker containers), sometimes docker pull... and everything from Cloud SDK that touches the internet. gcloud init takes forever to pass the networking test.

But nslookup, ping, dig are ALL fine (and working pretty fast). Browsing the internet via Firefox or Chrome too. I tried with DNS servers from Google, CloudFlare and my network provider (either set in my router or directly in my computer).

All of this happens in Fedora 32 (kernel Linux 5.8.13-200.fc32.x86_64) and Ubuntu 20.04 (didn't remember to take note of the kernel, but it was the latest available). Cloud SDK versions:

$ gcloud info
Google Cloud SDK [313.0.0]

Platform: [Linux, x86_64] uname_result(system='Linux', node='xps', release='5.8.13-200.fc32.x86_64', version='#1 SMP Thu Oct 1 21:49:42 UTC 2020', machine='x86_64', processor='x86_64')
Locale: ('en_GB', 'UTF-8')
Python Version: [3.8.5 (default, Sep  4 2020, 07:30:14)  [GCC 7.3.0]]
Python Location: [/home/gbrl/miniconda3/envs/gcloud/bin/python3]
Site Packages: [Disabled]

Installation Root: [/home/gbrl/google-cloud-sdk]
Installed Components:
  core: [2020.10.02]
  gsutil: [4.53]
  bq: [2.0.61]
System PATH: [/home/gbrl/google-cloud-sdk/bin:/home/gbrl/miniconda3/envs/gcloud/bin:/home/gbrl/miniconda3/condabin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/home/gbrl/bin]
Python PATH: [/home/gbrl/google-cloud-sdk/lib/third_party:/home/gbrl/google-cloud-sdk/lib:/home/gbrl/miniconda3/envs/gcloud/lib/python38.zip:/home/gbrl/miniconda3/envs/gcloud/lib/python3.8:/home/gbrl/miniconda3/envs/gcloud/lib/python3.8/lib-dynload]
Cloud SDK on PATH: [True]
Kubectl on PATH: [False]

Installation Properties: [/home/gbrl/google-cloud-sdk/properties]
User Config Directory: [/home/gbrl/.config/gcloud]
Active Configuration Name: [default]
Active Configuration Path: [/home/gbrl/.config/gcloud/configurations/config_default]

Account: [<my-email>]
Project: [<my-project>]

Current Properties:
  [core]
    account: [<my-email>]
    disable_usage_reporting: [True]
    project: [<my-project>]

Logs Directory: [/home/gbrl/.config/gcloud/logs]
Last Log File: [/home/gbrl/.config/gcloud/logs/2020.10.08/10.40.47.488518.log]

git: [git version 2.26.2]
ssh: [OpenSSH_8.3p1, OpenSSL 1.1.1g FIPS  21 Apr 2020]

Feel free to ping me if someone needs more information.

@ggabriel96
Copy link

Well... apparently the problems with Docker and Cloud SDK were unrelated, because all of a sudden gsutil and friends started working again for some reason that I can't explain. Then I could focus on Docker and fixed that later. I guess I'll never know what happened, because this behavior ocurred on 2 fresh-installs (Fedora 32 and Ubuntu 20.04). 🤷‍♂️

@terrynguyen255
Copy link

Nice debugging @surak.
Thank you for saving my day

@MatteoFelici
Copy link

Still having this issue, it's been from weeks. It works and it does not in casual way.

@Panesher
Copy link

I solve this issue by connecting with my phone. However, you can fix this by turn off "WAN Blocking" option.
Solution source

@rituraj-AU
Copy link

rituraj-AU commented Apr 5, 2022

Encountered this issue again, today. gsutil was working fine yesterday and woke up to find it doesn't work no matter what.

Nice to see solid support from Google over 4 years and even closing the issue for us!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests