local/install_deps.bash support http proxy on ubuntu18.04

[sanqingshan]

when run on ubuntu 18.04 with root user and http proxy,there is some error occur at the end:

Running: bazel run //local:create_gopath (cwd='src')
| INFO: Invocation ID: 637ff326-b2bb-4fb5-a4ca-270784072b85
| Loading:
| Loading: 0 packages loaded
| Analyzing: target //local:create_gopath (0 packages loaded, 0 targets configured)
| Analyzing: target //local:create_gopath (0 packages loaded, 0 targets configured)
| Analyzing: target //local:create_gopath (0 packages loaded, 0 targets configured)
| ERROR: /home/clusterfuzz/src/go/server/cron/BUILD.bazel:3:1: no such package '@com_google_cloud_go//datastore': failed to fetch com_google_cloud_go: 2019/02/19 15:13:43 unrecognized import path "cloud.google.com/go"
|  and referenced by '//go/server/cron:go_default_library'
| DEBUG: Rule 'org_golang_x_sys' modified arguments {"shallow_since": "1538141909 +0000"}
| DEBUG: Rule 'org_golang_x_tools' modified arguments {"sha256": "2384fa91351a7414b643c5230422ce45f5aa2be8a82727609afd4e64e6973a30"}
| ERROR: Analysis of target '//local:create_gopath' failed; build aborted: no such package '@com_google_cloud_go//datastore': failed to fetch com_google_cloud_go: 2019/02/19 15:13:43 unrecognized import path "cloud.google.com/go"
| INFO: Elapsed time: 35.316s
| INFO: 0 processes.
| FAILED: Build did NOT complete successfully (0 packages loaded, 0 targets configured)
| ERROR: Build failed. Not running target
| FAILED: Build did NOT complete successfully (0 packages loaded, 0 targets configured)
| Return code is non-zero (1).
| Exit.
(ENV) root@xxxx:/home/clusterfuzz

it is because can not get go packages,but i has config http proxy,so there was some other solution?

[YourButterfly]

hello， I get a similar error at ubuntu 16.04.

| Analyzing: target //local:create_gopath (4 packages loaded, 16 targets configured)
| INFO: SHA256 (https://codeload.github.com/golang/tools/zip/3e7aa9e59977626dc60433e9aeadf1bb63d28295) = 2384fa91351a7414b643c5230422ce45f5aa2be8a82727609afd4e64e6973a30
| DEBUG: Rule 'org_golang_x_tools' modified arguments {"sha256": "2384fa91351a7414b643c5230422ce45f5aa2be8a82727609afd4e64e6973a30"}
| DEBUG: Rule 'org_golang_x_sys' modified arguments {"shallow_since": "1538141909 +0000"}
| ERROR: /home/pwd/clusterfuzz/src/go/cloud/db/BUILD.bazel:3:1: no such package '@org_golang_google_api//iterator': failed to fetch org_golang_google_api: 2019/02/19 19:08:11 unrecognized import path "google.golang.org/api"
|  and referenced by '//go/cloud/db:go_default_library'
| ERROR: Analysis of target '//local:create_gopath' failed; build aborted: no such package '@org_golang_google_api//iterator': failed to fetch org_golang_google_api: 2019/02/19 19:08:11 unrecognized import path "google.golang.org/api"
| INFO: Elapsed time: 772.928s
| INFO: 0 processes.
| FAILED: Build did NOT complete successfully (42 packages loaded, 6834 targets configured)
| ERROR: Build failed. Not running target
| FAILED: Build did NOT complete successfully (42 packages loaded, 6834 targets configured)
| Return code is non-zero (1).
| Exit.

[Dor1s]

@sanqingshan does your proxy resolve cloud.google.com?

@YourButterfly the same question about google.golang.org for you?

[chenhb-zte]

I knew this issue. use HTTP(S)_PRXOY instead of http(s)_proxy. bazel only request the proxy with HTTP(S)_PROXY

[sanqingshan]

@sanqingshan does your proxy resolve cloud.google.com?

@YourButterfly the same question about google.golang.org for you?

yes,it can resolve

[Dor1s]

Thanks for confirming. Can you also check the suggestion by @chenhb-zte ? That sounds reasonable, though I haven't tried running the script with proxy myself.

[sanqingshan]

I knew this issue. use HTTP(S)_PRXOY instead of http(s)_proxy. bazel only request the proxy with HTTP(S)_PROXY

thanks for you suggestion,it is right,I have resolve it,here is some tips when use http proxy by root

export http_proxy=http://x.x.x.x
export https_proxy=${http_proxy}
export HTTP_PROXY=http://x.x.x.x
export HTTPS_PROXY=${HTTP_PROXY}
alias curl='curl -x x.x.x.x '
sed -i 's/sudo//g'  local/install_deps_linux.bash 

and replace bower install to bower install --allow-root

[sanqingshan]

I has get new error when start local server by

python butler.py run_server --bootstrap

(ENV) root@xxxx:/home/clusterfuzz# python butler.py run_server --bootstrap
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Running: pkill -KILL -f "dev_appserver.py"
| Return code is non-zero (-9).
Running: pkill -KILL -f "CloudDatastore.jar"
| Return code is non-zero (-9).
Running: pkill -KILL -f "pubsub-emulator"
| Return code is non-zero (-9).
Running: pip install -r src/requirements.txt --upgrade --target src/third_party
| DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
| Collecting backports.lzma==0.0.13 (from -r src/requirements.txt (line 1))
| Collecting google-api-python-client==1.7.4 (from -r src/requirements.txt (line 2))
| Collecting google-auth-oauthlib==0.2.0 (from -r src/requirements.txt (line 3))
|   Using cached https://files.pythonhosted.org/packages/46/df/d1f94ee2cffe5a83721f262efe51f3b2dcdd3b616caf007b8490e824c550/google_auth_oauthlib-0.2.0-py2.py3-none-any.whl
| Collecting google-cloud-datastore==1.7.0 (from -r src/requirements.txt (line 4))
|   Using cached https://files.pythonhosted.org/packages/d8/4b/aab0f1578eff1146e019670e079f5f939e46c5b004b65063b450c8d87af0/google_cloud_datastore-1.7.0-py2.py3-none-any.whl
| Collecting google-cloud-monitoring==0.30.1 (from -r src/requirements.txt (line 5))
|   Using cached https://files.pythonhosted.org/packages/1f/d3/9d8133efb90453a6a0747141aa78c64fa29618ecc713bea6e7b1bb9ff49d/google_cloud_monitoring-0.30.1-py2.py3-none-any.whl
| Collecting google-cloud-storage==1.13.2 (from -r src/requirements.txt (line 6))
|   Using cached https://files.pythonhosted.org/packages/92/da/115f23ba9aa56f4d5de3f02306a36eb6d0884561e0059987865cb0cce42e/google_cloud_storage-1.13.2-py2.py3-none-any.whl
| Collecting grpcio==1.15.0 (from -r src/requirements.txt (line 7))
|   Using cached https://files.pythonhosted.org/packages/3d/15/b34114198a2bc9c9bb73b21e2b559468a1a68bb28b373d21da6e51d6204f/grpcio-1.15.0-cp27-cp27mu-manylinux1_x86_64.whl
| Collecting html5lib==1.0.1 (from -r src/requirements.txt (line 8))
|   Using cached https://files.pythonhosted.org/packages/a5/62/bbd2be0e7943ec8504b517e62bab011b4946e1258842bc159e5dfde15b96/html5lib-1.0.1-py2.py3-none-any.whl
| Collecting httplib2==0.11.3 (from -r src/requirements.txt (line 9))
| Collecting mozprocess==0.26 (from -r src/requirements.txt (line 10))
|   Using cached https://files.pythonhosted.org/packages/2f/f8/25c6593b16eda2dc1bea23a0ec92c59158c095981c5d8393ffd2379034bb/mozprocess-0.26-py2-none-any.whl
| Collecting oauth2client==4.1.3 (from -r src/requirements.txt (line 11))
|   Using cached https://files.pythonhosted.org/packages/95/a9/4f25a14d23f0786b64875b91784607c2277eff25d48f915e39ff0cff505a/oauth2client-4.1.3-py2.py3-none-any.whl
| Collecting protobuf==3.6.1 (from -r src/requirements.txt (line 12))
|   Using cached https://files.pythonhosted.org/packages/b8/c2/b7f587c0aaf8bf2201405e8162323037fe8d17aa21d3c7dda811b8d01469/protobuf-3.6.1-cp27-cp27mu-manylinux1_x86_64.whl
| Collecting pytz==2018.5 (from -r src/requirements.txt (line 13))
|   Using cached https://files.pythonhosted.org/packages/30/4e/27c34b62430286c6d59177a0842ed90dc789ce5d1ed740887653b898779a/pytz-2018.5-py2.py3-none-any.whl
| Collecting PyYAML==4.2b4 (from -r src/requirements.txt (line 14))
| Collecting requests==2.21.0 (from -r src/requirements.txt (line 15))
|   Using cached https://files.pythonhosted.org/packages/7d/e3/20f3d364d6c8e5d2353c72a67778eb189176f08e873c9900e10c0287b84b/requests-2.21.0-py2.py3-none-any.whl
| Collecting urllib3==1.23 (from -r src/requirements.txt (line 16))
|   Using cached https://files.pythonhosted.org/packages/bd/c9/6fdd990019071a4a32a5e7cb78a1d92c53851ef4f56f62a3486e6a7d8ffb/urllib3-1.23-py2.py3-none-any.whl
| Collecting google-auth-httplib2>=0.0.3 (from google-api-python-client==1.7.4->-r src/requirements.txt (line 2))
|   Using cached https://files.pythonhosted.org/packages/33/49/c814d6d438b823441552198f096fcd0377fd6c88714dbed34f1d3c8c4389/google_auth_httplib2-0.0.3-py2.py3-none-any.whl
| Collecting google-auth>=1.4.1 (from google-api-python-client==1.7.4->-r src/requirements.txt (line 2))
|   Using cached https://files.pythonhosted.org/packages/c5/9b/ed0516cc1f7609fb0217e3057ff4f0f9f3e3ce79a369c6af4a6c5ca25664/google_auth-1.6.3-py2.py3-none-any.whl
| Collecting six<2dev,>=1.6.1 (from google-api-python-client==1.7.4->-r src/requirements.txt (line 2))
|   Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
| Collecting uritemplate<4dev,>=3.0.0 (from google-api-python-client==1.7.4->-r src/requirements.txt (line 2))
|   Using cached https://files.pythonhosted.org/packages/f6/25/66a49231b44409d7f07cfcf2506a8b070ce3c99fc47cc256bea833f24791/uritemplate-3.0.0-py2-none-any.whl
| Collecting requests-oauthlib>=0.7.0 (from google-auth-oauthlib==0.2.0->-r src/requirements.txt (line 3))
|   Using cached https://files.pythonhosted.org/packages/c2/e2/9fd03d55ffb70fe51f587f20bcf407a6927eb121de86928b34d162f0b1ac/requests_oauthlib-1.2.0-py2.py3-none-any.whl
| Collecting google-cloud-core<0.29dev,>=0.28.0 (from google-cloud-datastore==1.7.0->-r src/requirements.txt (line 4))
|   Using cached https://files.pythonhosted.org/packages/0f/41/ae2418b4003a14cf21c1c46d61d1b044bf02cf0f8f91598af572b9216515/google_cloud_core-0.28.1-py2.py3-none-any.whl
| Collecting google-api-core[grpc]<2.0.0dev,>=1.0.0 (from google-cloud-datastore==1.7.0->-r src/requirements.txt (line 4))
|   Using cached https://files.pythonhosted.org/packages/8b/01/13758ff9b970008ccf9e0dcc3b86d0e01937d7485b9a2c6142c9c2bdb4da/google_api_core-1.7.0-py2.py3-none-any.whl
| Collecting google-resumable-media>=0.3.1 (from google-cloud-storage==1.13.2->-r src/requirements.txt (line 6))
|   Using cached https://files.pythonhosted.org/packages/e2/5d/4bc5c28c252a62efe69ed1a1561da92bd5af8eca0cdcdf8e60354fae9b29/google_resumable_media-0.3.2-py2.py3-none-any.whl
| Collecting enum34>=1.0.4 (from grpcio==1.15.0->-r src/requirements.txt (line 7))
|   Using cached https://files.pythonhosted.org/packages/c5/db/e56e6b4bbac7c4a06de1c50de6fe1ef3810018ae11732a50f15f62c7d050/enum34-1.1.6-py2-none-any.whl
| Collecting futures>=2.2.0 (from grpcio==1.15.0->-r src/requirements.txt (line 7))
|   Using cached https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl
| Collecting webencodings (from html5lib==1.0.1->-r src/requirements.txt (line 8))
|   Using cached https://files.pythonhosted.org/packages/f4/24/2a3e3df732393fed8b3ebf2ec078f05546de641fe1b667ee316ec1dcf3b7/webencodings-0.5.1-py2.py3-none-any.whl
| Collecting mozinfo (from mozprocess==0.26->-r src/requirements.txt (line 10))
|   Using cached https://files.pythonhosted.org/packages/65/54/7cb1912fcf3f2176616fd17b9afa9665e82918e6a1fcc7b91b1407a602c9/mozinfo-1.0.0-py2.py3-none-any.whl
| Collecting rsa>=3.1.4 (from oauth2client==4.1.3->-r src/requirements.txt (line 11))
|   Using cached https://files.pythonhosted.org/packages/02/e5/38518af393f7c214357079ce67a317307936896e961e35450b70fad2a9cf/rsa-4.0-py2.py3-none-any.whl
| Collecting pyasn1>=0.1.7 (from oauth2client==4.1.3->-r src/requirements.txt (line 11))
|   Using cached https://files.pythonhosted.org/packages/7b/7c/c9386b82a25115cccf1903441bba3cbadcfae7b678a20167347fa8ded34c/pyasn1-0.4.5-py2.py3-none-any.whl
| Collecting pyasn1-modules>=0.0.5 (from oauth2client==4.1.3->-r src/requirements.txt (line 11))
|   Using cached https://files.pythonhosted.org/packages/da/98/8ddd9fa4d84065926832bcf2255a2b69f1d03330aa4d1c49cc7317ac888e/pyasn1_modules-0.2.4-py2.py3-none-any.whl
| Collecting setuptools (from protobuf==3.6.1->-r src/requirements.txt (line 12))
|   Using cached https://files.pythonhosted.org/packages/d1/6a/4b2fcefd2ea0868810e92d519dacac1ddc64a2e53ba9e3422c3b62b378a6/setuptools-40.8.0-py2.py3-none-any.whl
| Collecting certifi>=2017.4.17 (from requests==2.21.0->-r src/requirements.txt (line 15))
|   Using cached https://files.pythonhosted.org/packages/9f/e0/accfc1b56b57e9750eba272e24c4dddeac86852c2bebd1236674d7887e8a/certifi-2018.11.29-py2.py3-none-any.whl
| Collecting chardet<3.1.0,>=3.0.2 (from requests==2.21.0->-r src/requirements.txt (line 15))
|   Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
| Collecting idna<2.9,>=2.5 (from requests==2.21.0->-r src/requirements.txt (line 15))
|   Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
| Collecting cachetools>=2.0.0 (from google-auth>=1.4.1->google-api-python-client==1.7.4->-r src/requirements.txt (line 2))
|   Using cached https://files.pythonhosted.org/packages/39/2b/d87fc2369242bd743883232c463f28205902b8579cb68dcf5b11eee1652f/cachetools-3.1.0-py2.py3-none-any.whl
| Collecting oauthlib>=3.0.0 (from requests-oauthlib>=0.7.0->google-auth-oauthlib==0.2.0->-r src/requirements.txt (line 3))
|   Using cached https://files.pythonhosted.org/packages/16/95/699466b05b72b94a41f662dc9edf87fda4289e3602ecd42d27fcaddf7b56/oauthlib-3.0.1-py2.py3-none-any.whl
| Collecting googleapis-common-protos!=1.5.4,<2.0dev,>=1.5.3 (from google-api-core[grpc]<2.0.0dev,>=1.0.0->google-cloud-datastore==1.7.0->-r src/requirements.txt (line 4))
| Collecting mozfile>=0.12 (from mozinfo->mozprocess==0.26->-r src/requirements.txt (line 10))
|   Using cached https://files.pythonhosted.org/packages/85/ed/4f8bec5ac04373ef0c79428bdd50ce039e91e4b53660d08e9a45988b74dc/mozfile-2.0.0-py2.py3-none-any.whl
| google-cloud-storage 1.13.2 has requirement google-cloud-core<0.30dev,>=0.29.0, but you'll have google-cloud-core 0.28.1 which is incompatible.
| proto-google-cloud-datastore-v1 0.90.4 has requirement oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.3 which is incompatible.
| googledatastore 7.0.1 has requirement httplib2<0.10,>=0.9.1, but you'll have httplib2 0.11.3 which is incompatible.
| googledatastore 7.0.1 has requirement oauth2client<4.0.0,>=2.0.1, but you'll have oauth2client 4.1.3 which is incompatible.
| tensorboard 1.8.0 has requirement html5lib==0.9999999, but you'll have html5lib 1.0.1 which is incompatible.
| bleach 1.5.0 has requirement html5lib!=0.9999,!=0.99999,<0.99999999,>=0.999, but you'll have html5lib 1.0.1 which is incompatible.
| Installing collected packages: backports.lzma, httplib2, cachetools, six, pyasn1, pyasn1-modules, rsa, google-auth, google-auth-httplib2, uritemplate, google-api-python-client, oauthlib, urllib3, certifi, chardet, idna, requests, requests-oauthlib, google-auth-oauthlib, setuptools, protobuf, futures, pytz, googleapis-common-protos, enum34, grpcio, google-api-core, google-cloud-core, google-cloud-datastore, google-cloud-monitoring, google-resumable-media, google-cloud-storage, webencodings, html5lib, mozfile, mozinfo, mozprocess, oauth2client, PyYAML
| Successfully installed PyYAML-4.2b4 backports.lzma-0.0.13 cachetools-3.1.0 certifi-2018.11.29 chardet-3.0.4 enum34-1.1.6 futures-3.2.0 google-api-core-1.7.0 google-api-python-client-1.7.4 google-auth-1.6.3 google-auth-httplib2-0.0.3 google-auth-oauthlib-0.2.0 google-cloud-core-0.28.1 google-cloud-datastore-1.7.0 google-cloud-monitoring-0.30.1 google-cloud-storage-1.13.2 google-resumable-media-0.3.2 googleapis-common-protos-1.5.8 grpcio-1.15.0 html5lib-1.0.1 httplib2-0.11.3 idna-2.8 mozfile-2.0.0 mozinfo-1.0.0 mozprocess-0.26 oauth2client-4.1.3 oauthlib-3.0.1 protobuf-3.6.1 pyasn1-0.4.5 pyasn1-modules-0.2.4 pytz-2018.5 requests-2.21.0 requests-oauthlib-1.2.0 rsa-4.0 setuptools-40.8.0 six-1.12.0 uritemplate-3.0.0 urllib3-1.23 webencodings-0.5.1
Running: bower install --allow-root
| bower webcomponentsjs extra-resolution Unnecessary resolution: webcomponentsjs#^v1.0.2
| bower iron-location   extra-resolution Unnecessary resolution: iron-location#1 - 2
| bower mocha           extra-resolution Unnecessary resolution: mocha#^3.1.2
| bower iron-collapse   extra-resolution Unnecessary resolution: iron-collapse#^2.0.0
Created symlink: source: /home/clusterfuzz/configs/test, target /home/clusterfuzz/src/appengine/config.
Created symlink: source: /home/clusterfuzz/configs/test, target /home/clusterfuzz/src/go/server/config.
Created symlink: source: /home/clusterfuzz/src/protos, target /home/clusterfuzz/src/appengine/protos.
Created symlink: source: /home/clusterfuzz/src/python, target /home/clusterfuzz/src/appengine/python.
Created symlink: source: /home/clusterfuzz/src/third_party, target /home/clusterfuzz/src/appengine/third_party.
Running: bazel run //local:create_gopath (cwd='src')
| INFO: Invocation ID: 6537451b-498a-4226-b318-90db5874ed26
| Loading:
| Loading: 0 packages loaded
| Analyzing: target //local:create_gopath (0 packages loaded, 0 targets configured)
| DEBUG: Rule 'com_github_golang_protobuf' modified arguments {"shallow_since": "1534281267 -0700"}
| DEBUG: Rule 'go_googleapis' modified arguments {"sha256": "c65fa17dcc40408f0a6d9d1a309e60bf896201be9cd4d09043779fc525fc901c"}
| DEBUG: Rule 'org_golang_google_grpc' modified arguments {"shallow_since": "1536688131 -0700"}
| DEBUG: Rule 'org_golang_x_net' modified arguments {"shallow_since": "1537976840 +0000"}
| DEBUG: Rule 'org_golang_x_text' modified arguments {"shallow_since": "1513256923 +0000"}
| INFO: Analysed target //local:create_gopath (0 packages loaded, 0 targets configured).
| INFO: Found 1 target...
| [0 / 1] [-----] BazelWorkspaceStatusAction stable-status.txt
| Target //local:create_gopath up-to-date:
|   bazel-bin/local/create_gopath
| INFO: Elapsed time: 0.431s, Critical Path: 0.01s
| INFO: 0 processes.
| INFO: Build completed successfully, 1 total action
| INFO: Running command line: bazel-bin/local/create_gopath local/gopath
| INFO: Build completed successfully, 1 total action
| /root/.cache/bazel/_bazel_root/2ad9d25912904b4b1a3186f98d003514/execroot/__main__/bazel-out/k8-fastbuild/bin/local/gopath
Running: python polymer_bundler.py (cwd='local')
| Building templates for App Engine...
| App Engine templates built successfully.
Created symlink: source: /home/clusterfuzz/local/storage/local_gcs, target /home/clusterfuzz/src/appengine/local_gcs.
Traceback (most recent call last):
  File "butler.py", line 282, in <module>
    main()
  File "butler.py", line 256, in main
    command.execute(args)
  File "src/local/butler/run_server.py", line 162, in execute
    test_utils.setup_pubsub(constants.TEST_APP_ID)
  File "/home/clusterfuzz/src/python/tests/test_libs/test_utils.py", line 308, in setup_pubsub
    _create_pubsub_topic(client, project, queue['name'])
  File "/home/clusterfuzz/src/python/tests/test_libs/test_utils.py", line 284, in _create_pubsub_topic
    if client.get_topic(full_name):
  File "/home/clusterfuzz/src/python/google_cloud_utils/pubsub.py", line 193, in get_topic
    response = self._execute_with_retry(request)
  File "/home/clusterfuzz/src/python/base/retry.py", line 88, in _wrapper
    result = func(*args, **kwargs)
  File "/home/clusterfuzz/src/python/google_cloud_utils/pubsub.py", line 108, in _execute_with_retry
    return request.execute()
  File "/home/clusterfuzz/src/third_party/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/home/clusterfuzz/src/third_party/googleapiclient/http.py", line 837, in execute
    method=str(self.method), body=self.body, headers=self.headers)
  File "/home/clusterfuzz/src/third_party/googleapiclient/http.py", line 162, in _retry_request
    resp, content = http.request(uri, method, *args, **kwargs)
  File "/home/clusterfuzz/src/third_party/google_auth_httplib2.py", line 198, in request
    uri, method, body=body, headers=request_headers, **kwargs)
  File "/home/clusterfuzz/src/third_party/httplib2/__init__.py", line 1694, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/home/clusterfuzz/src/third_party/httplib2/__init__.py", line 1434, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/home/clusterfuzz/src/third_party/httplib2/__init__.py", line 1390, in _conn_request
    response = conn.getresponse()
  File "/usr/lib/python2.7/httplib.py", line 1121, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 438, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 394, in _read_status
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib/python2.7/socket.py", line 480, in readline
    data = self._sock.recv(self._rbufsize)
  File "/usr/lib/python2.7/socket.py", line 174, in _dummy
    raise error(EBADF, 'Bad file descriptor')
socket.error: [Errno 9] Bad file descriptor

has it relation to net proxy?

[chenhb-zte]

I did not meet this issue, but suggest to unset all proxy option after installing all deps.
you could use python butler.py run_server --skip-install-deps to launch the server.

[sanqingshan]

I did not meet this issue, but suggest to unset all proxy option after installing all deps.
you could use python butler.py run_server --skip-install-deps to launch the server.

it also occur error without proxy

Running: bower install --allow-root
| bower webcomponentsjs extra-resolution Unnecessary resolution: webcomponentsjs#^v1.0.2
| bower iron-location   extra-resolution Unnecessary resolution: iron-location#1 - 2
| bower mocha           extra-resolution Unnecessary resolution: mocha#^3.1.2
| bower iron-collapse   extra-resolution Unnecessary resolution: iron-collapse#^2.0.0
Traceback (most recent call last):
  File "butler.py", line 282, in <module>
    main()
  File "butler.py", line 256, in main
    command.execute(args)
  File "src/local/butler/run_server.py", line 132, in execute
    common.install_dependencies()
  File "src/local/butler/common.py", line 278, in install_dependencies
    _install_chromedriver()
  File "src/local/butler/common.py", line 185, in _install_chromedriver
    version_request = urllib2.urlopen(constants.CHROMEDRIVER_VERSION_URL)
  File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 429, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 447, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1241, in https_open
    context=self._context)
  File "/usr/lib/python2.7/urllib2.py", line 1198, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [Errno 110] Connection timed out>

[ghost]

Internet problem

[Dor1s]

Yes, it looks like some connection issues. Please make sure that your proxy supports both HTTP and HTTPS requests and you have a proper DNS server specified.

[sanqingshan]

I has resolve it,it is because it will access localhost port 9004 9003 9006 and so on when init,so the proxy server must on the same machine with clustfuzz server and igore localhost by proxy rules.

[aressu1985]

Hi sanqingshan:
Now I meet the similar problem,and have set no_proxy for the localhost,you mean,my proxy server must run on the machine which the clustfuzz is running on?@sanqingshan

The tracestack is as following:
INFO 2019-02-27 05:57:37,600 devappserver2.py:278] Skipping SDK update check.
| INFO 2019-02-27 05:57:37,693 datastore_emulator.py:155] Starting Cloud Datastore emulator at: http://localhost:9004
| WARNING 2019-02-27 05:57:37,747 simple_search_stub.py:1196] Could not read search indexes from local/storage/search_indexes
| INFO 2019-02-27 05:57:38,871 datastore_emulator.py:161] Cloud Datastore emulator responded after 1.178564 seconds
| INFO 2019-02-27 05:57:38,872 api_server.py:275] Starting API server at: http://localhost:38449
| INFO 2019-02-27 05:57:38,878 api_server.py:265] Starting gRPC API server at: http://localhost:34005
| INFO 2019-02-27 05:57:38,947 dispatcher.py:256] Starting module "default" running at: http://localhost:9000
| INFO 2019-02-27 05:57:39,019 dispatcher.py:256] Starting module "cron-service" running at: http://localhost:9001
| INFO 2019-02-27 05:57:39,019 admin_server.py:150] Starting admin server at: http://localhost:9002
Bootstrapping datastore...
Running: python butler.py run setup --non-dry-run --local --config-dir=configs/test
| DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
| Creating config
| Creating fuzzer afl
| Creating fuzzer libFuzzer
| Creating template afl
| Creating template libfuzzer
| Creating template engine_asan
| Creating template engine_ubsan
| Creating template engine_msan
| Done
| INFO 2019-02-27 05:58:37,733 instance.py:294] Instance PID: 22299
| HTTPError()
| HTTPError()
| Traceback (most recent call last):
| File "/usr/lib/google-cloud-sdk/platform/google_appengine/lib/cherrypy/cherrypy/wsgiserver/wsgiserver2.py", line 1302, in communicate
| req.respond()
| File "/usr/lib/google-cloud-sdk/platform/google_appengine/lib/cherrypy/cherrypy/wsgiserver/wsgiserver2.py", line 831, in respond
| self.server.gateway(self).respond()
| File "/usr/lib/google-cloud-sdk/platform/google_appengine/lib/cherrypy/cherrypy/wsgiserver/wsgiserver2.py", line 2115, in respond
| response = self.req.server.wsgi_app(self.env, self.start_response)
| File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/wsgi_server.py", line 292, in call
| INFO 2019-02-27 05:58:38,692 module.py:861] cron-service: "GET /_ah/start HTTP/1.1" 500 -
| return app(environ, start_response)
| File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/request_rewriter.py", line 314, in _rewriter_middleware
| response_body = iter(application(environ, wrapped_start_response))
| File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/runtime/request_handler.py", line 165, in call
| self._flush_logs(response.get('logs', []))
| File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/runtime/request_handler.py", line 308, in _flush_logs
| apiproxy_stub_map.MakeSyncCall('logservice', 'Flush', request, response)
| File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 97, in MakeSyncCall
| return stubmap.MakeSyncCall(service, call, request, response)
| File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 331, in MakeSyncCall
| rpc.CheckSuccess()
| File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/api/apiproxy_rpc.py", line 136, in CheckSuccess
| self._traceback)
| File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/api/apiproxy_rpc.py", line 161, in _WaitImpl
| self.request, self.response)
| File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 223, in MakeSyncCall
| self._MakeRealSyncCall(service, call, request, response)
| File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 248, in _MakeRealSyncCall
| encoded_response = self._server.Send(self._path, encoded_request)
| File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/tools/appengine_rpc.py", line 469, in Send
| f = self.opener.open(req)
| File "/usr/lib/python2.7/urllib2.py", line 435, in open
| response = meth(req, response)
| File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
| 'http', request, response, code, msg, hdrs)
| File "/usr/lib/python2.7/urllib2.py", line 473, in error
| return self._call_chain(*args)
| File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
| result = func(*args)
| File "/usr/lib/python2.7/urllib2.py", line 556, in http_error_default
| raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
| HTTPError: HTTP Error 504: Connection failed

[zlrs]

Hi sanqingshan:
Now I meet the similar problem,and have set no_proxy for the localhost,you mean,my proxy server must run on the machine which the clustfuzz is running on?@sanqingshan

The tracestack is as following:
INFO 2019-02-27 05:57:37,600 devappserver2.py:278] Skipping SDK update check.
| INFO 2019-02-27 05:57:37,693 datastore_emulator.py:155] Starting Cloud Datastore emulator at: http://localhost:9004
| WARNING 2019-02-27 05:57:37,747 simple_search_stub.py:1196] Could not read search indexes from local/storage/search_indexes
| INFO 2019-02-27 05:57:38,871 datastore_emulator.py:161] Cloud Datastore emulator responded after 1.178564 seconds
| INFO 2019-02-27 05:57:38,872 api_server.py:275] Starting API server at: http://localhost:38449
| INFO 2019-02-27 05:57:38,878 api_server.py:265] Starting gRPC API server at: http://localhost:34005
| INFO 2019-02-27 05:57:38,947 dispatcher.py:256] Starting module "default" running at: http://localhost:9000
| INFO 2019-02-27 05:57:39,019 dispatcher.py:256] Starting module "cron-service" running at: http://localhost:9001
| INFO 2019-02-27 05:57:39,019 admin_server.py:150] Starting admin server at: http://localhost:9002
Bootstrapping datastore...
Running: python butler.py run setup --non-dry-run --local --config-dir=configs/test
| DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
| Creating config
| Creating fuzzer afl
| Creating fuzzer libFuzzer
| Creating template afl
| Creating template libfuzzer
| Creating template engine_asan
| Creating template engine_ubsan
| Creating template engine_msan
| Done
| INFO 2019-02-27 05:58:37,733 instance.py:294] Instance PID: 22299
| HTTPError()
| HTTPError()
| Traceback (most recent call last):
| File "/usr/lib/google-cloud-sdk/platform/google_appengine/lib/cherrypy/cherrypy/wsgiserver/wsgiserver2.py", line 1302, in communicate
| req.respond()
| File "/usr/lib/google-cloud-sdk/platform/google_appengine/lib/cherrypy/cherrypy/wsgiserver/wsgiserver2.py", line 831, in respond
| self.server.gateway(self).respond()
| File "/usr/lib/google-cloud-sdk/platform/google_appengine/lib/cherrypy/cherrypy/wsgiserver/wsgiserver2.py", line 2115, in respond
| response = self.req.server.wsgi_app(self.env, self.start_response)
| File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/wsgi_server.py", line 292, in call
| INFO 2019-02-27 05:58:38,692 module.py:861] cron-service: "GET /_ah/start HTTP/1.1" 500 -
| return app(environ, start_response)
| File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/request_rewriter.py", line 314, in _rewriter_middleware
| response_body = iter(application(environ, wrapped_start_response))
| File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/runtime/request_handler.py", line 165, in call
| self._flush_logs(response.get('logs', []))
| File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/runtime/request_handler.py", line 308, in _flush_logs
| apiproxy_stub_map.MakeSyncCall('logservice', 'Flush', request, response)
| File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 97, in MakeSyncCall
| return stubmap.MakeSyncCall(service, call, request, response)
| File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 331, in MakeSyncCall
| rpc.CheckSuccess()
| File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/api/apiproxy_rpc.py", line 136, in CheckSuccess
| self._traceback)
| File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/api/apiproxy_rpc.py", line 161, in _WaitImpl
| self.request, self.response)
| File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 223, in MakeSyncCall
| self._MakeRealSyncCall(service, call, request, response)
| File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 248, in _MakeRealSyncCall
| encoded_response = self._server.Send(self._path, encoded_request)
| File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/tools/appengine_rpc.py", line 469, in Send
| f = self.opener.open(req)
| File "/usr/lib/python2.7/urllib2.py", line 435, in open
| response = meth(req, response)
| File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
| 'http', request, response, code, msg, hdrs)
| File "/usr/lib/python2.7/urllib2.py", line 473, in error
| return self._call_chain(*args)
| File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
| result = func(*args)
| File "/usr/lib/python2.7/urllib2.py", line 556, in http_error_default
| raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
| HTTPError: HTTP Error 504: Connection failed

I have met 500/504 error similar to yours. Network troubleshotting sucks...