Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

vcpkg install --head fails in some environments due to GitHub API rate limits; vcpkg should allow us to provide GitHub credentials for authenticated GitHub API requests #19177

Open
brycelelbach opened this issue Jul 28, 2021 · 5 comments
Open

vcpkg install --head fails in some environments due to GitHub API rate limits; vcpkg should allow us to provide GitHub credentials for authenticated GitHub API requests #19177

brycelelbach opened this issue Jul 28, 2021 · 5 comments
Assignees
@NancyLi1013
Labels
category:vcpkg-bug The issue is with the vcpkg system (including helper scripts in `scripts/cmake/`)

Comments

@brycelelbach
Copy link

brycelelbach commented Jul 28, 2021 •
edited

Describe the bug
We're running into issues when using vcpkg install --head anywhere within NVIDIA's corporate VPN, including personal developer workstations and our CI system.

The problem appears to be that vcpkg install --head will attempt to access api.github.com methods unauthenticated (e.g. without providing an authentication token/API key), which fails due to GitHub API rate limits, presumably because there's other things within our corporate VPN that are hammering GitHub APIs.

We recognize this is probably more of our problem than your problem, but we've got no idea how to workaround this.

It is highly unlikely that we'll be able to track down or stop the sources of all the GitHub API requests that are being made elsewhere at NVIDIA and are interfering with our usage of vcpkg.

GitHub's API supports authenticated requests. Authenticated requests not only get a higher API rate limit, but the rate limit is counted per-user, instead of per site/per IP. However, I couldn't find a way to pass a GitHub personal access tokens/etc to vcpkg. Adding a way to pass GitHub personal access tokens/etc to the vcpkg command line tool to be used for all GitHub API requests would solve this issue. I think more than just GitHub personal access tokens should be supported (like GitHub Enterprise credentials or repo credentials if that's a thing), but personal access tokens would be a good start.

$ curl https://api.github.com/repos/pantor/inja/git/refs/heads/master
{"message":"API rate limit exceeded for 216.228.112.21. (But here's the good
news: Authenticated requests get a higher rate limit. Check out the documentation
for more details.)",
"documentation_url":"https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting"}
$ curl -u brycelelbach:${GITHUB_AUTH_TOKEN} https://api.github.com/repos/pantor/inja/git/refs/heads/master
{
  "ref": "refs/heads/master",
  "node_id": "MDM6UmVmMTAwMTA5MzAwOnJlZnMvaGVhZHMvbWFzdGVy",
  "url": "https://api.github.com/repos/pantor/inja/git/refs/heads/master",
  "object": {
    "sha": "2ba5783705da8638683f7b7a8b6771a9738d7127",
    "type": "commit",
    "url": "https://api.github.com/repos/pantor/inja/git/commits/2ba5783705da8638683f7b7a8b6771a9738d7127"
  }
}

https://docs.github.com/en/developers/apps/building-github-apps/rate-limits-for-github-apps

https://docs.github.com/en/github/authenticating-to-github/keeping-your-account-and-data-secure/creating-a-personal-access-token

To Reproduce

docker run -i ubuntu:20.04 /bin/bash -s << EOF
apt-get -y update > /dev/null 2>&1
apt-get -y install git curl zip unzip tar g++ > /dev/null 2>&1
git clone https://github.com/microsoft/vcpkg /tmp/vcpkg 2>&1
/tmp/vcpkg/bootstrap-vcpkg.sh
/tmp/vcpkg/vcpkg --head install inja
EOF

Failure logs

$ docker run -i ubuntu:20.04 /bin/bash -s << EOF
apt-get -y update > /dev/null 2>&1
apt-get -y install git curl zip unzip tar g++ > /dev/null 2>&1
git clone https://github.com/microsoft/vcpkg /tmp/vcpkg 2>&1
/tmp/vcpkg/bootstrap-vcpkg.sh
/tmp/vcpkg/vcpkg --head install inja
EOF
Cloning into '/tmp/vcpkg'...
Downloading cmake...
Downloading cmake... done.
Extracting cmake...
Extracting cmake... done.
Downloading ninja...
Downloading ninja... done.
Extracting ninja...
Extracting ninja... done.
Downloading vcpkg tool sources
Building vcpkg-tool...
-- The CXX compiler identification is GNU 9.3.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/g++-9 - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting the C++ compiler in use
-- Detecting the C++ compiler in use - gcc
-- Detecting the C++ standard library
-- Detecting the C++ standard library - libstdc++
-- Detecting how to use the C++ filesystem library
-- Detecting how to use the C++ filesystem library - <filesystem>
-- Looking for C++ include pthread.h
-- Looking for C++ include pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Found Threads: TRUE  
-- Configuring done
-- Generating done
-- Build files have been written to: /tmp/vcpkg/buildtrees/_vcpkg/build
[0/2] Re-checking globbed directories...
[1/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/base/enums.cpp.o
[2/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/base/stringview.cpp.o
[3/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/base/checks.cpp.o
[4/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/base/machinetype.cpp.o
[5/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/base/system.cpp.o
[6/92] Building CXX object CMakeFiles/vcpkg.dir/src/vcpkg.cpp.o
[7/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/base/strings.cpp.o
[8/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/base/chrono.cpp.o
[9/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/base/parse.cpp.o
[10/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/base/downloads.cpp.o
[11/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/base/cofffilereader.cpp.o
[12/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/base/hash.cpp.o
[13/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/base/system.proxy.cpp.o
[14/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/base/system.print.cpp.o
[15/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/base/unicode.cpp.o
[16/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/base/system.process.cpp.o
[17/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/base/uint128.cpp.o
[18/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/archives.cpp.o
[19/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/base/xmlserializer.cpp.o
[20/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/buildenvironment.cpp.o
[21/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/commands.ciclean.cpp.o
[22/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/binaryparagraph.cpp.o
[23/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/commands.contact.cpp.o
[24/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/commands.buildexternal.cpp.o
[25/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/commands.cache.cpp.o
[26/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/commands.add-version.cpp.o
[27/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/cmakevars.cpp.o
[28/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/base/files.cpp.o
[29/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/commands.create.cpp.o
[30/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/commands.check-support.cpp.o
[31/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/base/json.cpp.o
[32/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/commands.cpp.o
[33/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/commands.fetch.cpp.o
[34/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/commands.hash.cpp.o
[35/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/commands.civerifyversions.cpp.o
[36/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/commands.env.cpp.o
[37/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/commands.edit.cpp.o
[38/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/commands.init-registry.cpp.o
[39/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/build.cpp.o
[40/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/commands.format-manifest.cpp.o
[41/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/commands.ci.cpp.o
[42/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/commands.info.cpp.o
[43/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/commands.owns.cpp.o
[44/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/commands.list.cpp.o
[45/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/commands.autocomplete.cpp.o
[46/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/commands.upload-metrics.cpp.o
[47/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/commands.search.cpp.o
[48/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/commands.portsdiff.cpp.o
[49/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/commands.porthistory.cpp.o
[50/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/commands.xvsinstances.cpp.o
[51/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/commands.version.cpp.o
[52/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/commands.xdownload.cpp.o
[53/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/configuration.cpp.o
[54/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/binarycaching.cpp.o
[55/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/globalstate.cpp.o
[56/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/commands.integrate.cpp.o
[57/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/commands.setinstalled.cpp.o
[58/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/commands.upgrade.cpp.o
[59/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/commands.dependinfo.cpp.o
[60/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/export.chocolatey.cpp.o
[61/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/input.cpp.o
[62/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/packagespec.cpp.o
[63/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/help.cpp.o
[64/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/metrics.cpp.o
[65/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/export.cpp.o
[66/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/platform-expression.cpp.o
[67/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/portfileprovider.cpp.o
[68/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/remove.cpp.o
[69/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/statusparagraph.cpp.o
[70/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/paragraphs.cpp.o
[71/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/statusparagraphs.cpp.o
[72/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/postbuildlint.buildtype.cpp.o
[73/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/export.ifw.cpp.o
[74/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/triplet.cpp.o
[75/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/vcpkgcmdarguments.cpp.o
[76/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/userconfig.cpp.o
[77/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/versiondeserializers.cpp.o
[78/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/update.cpp.o
[79/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/dependencies.cpp.o
[80/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/registries.cpp.o
[81/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/visualstudio.cpp.o
[82/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/sourceparagraph.cpp.o
[83/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/export.prefab.cpp.o
[84/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/versiont.cpp.o
[85/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/postbuildlint.cpp.o
[86/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/tools.cpp.o
[87/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/install.cpp.o
[88/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/vcpkgpaths.cpp.o
[89/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/versions.cpp.o
[90/92] Building CXX object CMakeFiles/vcpkglib.dir/src/vcpkg/vcpkglib.cpp.o
[91/92] Linking CXX static library libvcpkglib.a
[92/92] Linking CXX executable vcpkg
Telemetry
---------
vcpkg collects usage data in order to help us improve your experience.
The data collected by Microsoft is anonymous.
You can opt-out of telemetry by re-running the bootstrap-vcpkg script with -disableMetrics,
passing --disable-metrics to vcpkg on the command line,
or by setting the VCPKG_DISABLE_METRICS environment variable.

Read more about vcpkg telemetry at docs/about/privacy.md
Computing installation plan...
The following packages will be built and installed:
    inja[core]:x64-linux -> 3.2.0 (+HEAD)
  * nlohmann-json[core]:x64-linux -> 3.9.1
Additional packages (*) will be modified to complete this operation.
Detecting compiler hash for triplet x64-linux...
Could not locate cached archive: /root/.cache/vcpkg/archives/ef/ef8eae8cfe4cd361bbfa4442a297140dab61a8632529693b94219e95801c42e8.zip
Starting package 1/2: nlohmann-json:x64-linux
Building package nlohmann-json[core]:x64-linux...
-- Downloading https://github.com/nlohmann/json/raw/v3.9.1/CMakeLists.txt -> nlohmann-json-v3.9.1/CMakeLists.txt...
-- Downloading https://github.com/nlohmann/json/raw/v3.9.1/LICENSE.MIT -> nlohmann-json-v3.9.1/LICENSE.MIT...
-- Downloading https://github.com/nlohmann/json/raw/v3.9.1/nlohmann_json.natvis -> nlohmann-json-v3.9.1/nlohmann_json.natvis...
-- Downloading https://github.com/nlohmann/json/raw/v3.9.1/cmake/config.cmake.in -> nlohmann-json-v3.9.1/cmake/config.cmake.in...
-- Downloading https://github.com/nlohmann/json/raw/v3.9.1/cmake/pkg-config.pc.in -> nlohmann-json-v3.9.1/cmake/pkg-config.pc.in...
-- Downloading https://github.com/nlohmann/json/raw/v3.9.1/cmake/nlohmann_jsonConfigVersion.cmake.in -> nlohmann-json-v3.9.1/cmake/nlohmann_jsonConfigVersion.cmake.in...
-- Downloading https://github.com/nlohmann/json/releases/download/v3.9.1/include.zip -> nlohmann-json-v3.9.1-include.zip...
-- Extracting source /tmp/vcpkg/downloads/nlohmann-json-v3.9.1-include.zip
-- Using source at /tmp/vcpkg/buildtrees/nlohmann-json/3.9.1-74ced61d52.clean
-- Configuring x64-linux-dbg
-- Configuring x64-linux-rel
-- Building x64-linux-dbg
-- Building x64-linux-rel
-- Installing: /tmp/vcpkg/packages/nlohmann-json_x64-linux/share/nlohmann-json/copyright
-- Performing post-build validation
-- Performing post-build validation done
Stored binary cache: /root/.cache/vcpkg/archives/ef/ef8eae8cfe4cd361bbfa4442a297140dab61a8632529693b94219e95801c42e8.zip
Building package nlohmann-json[core]:x64-linux... done
Installing package nlohmann-json[core]:x64-linux...
Installing package nlohmann-json[core]:x64-linux... done
Elapsed time for package nlohmann-json:x64-linux: 6.994 s
Starting package 2/2: inja:x64-linux
Building package inja[core]:x64-linux from HEAD...
-- Downloading https://api.github.com/repos/pantor/inja/git/refs/heads/master -> pantor-inja-master.tar.gz.version...
-- Downloading https://api.github.com/repos/pantor/inja/git/refs/heads/master... Failed. Status: 22;"HTTP response code said error"
CMake Error at scripts/cmake/vcpkg_download_distfile.cmake:251 (message):
      
      Failed to download file.
      If you use a proxy, please check your proxy setting. Possible causes are:
      
      1. You are actually using an HTTP proxy, but setting HTTPS_PROXY variable
         to `https://address:port`. This is not correct, because `https://` prefix
         claims the proxy is an HTTPS proxy, while your proxy (v2ray, shadowsocksr
         , etc..) is an HTTP proxy. Try setting `http://address:port` to both
         HTTP_PROXY and HTTPS_PROXY instead.
      
      2. You are using Fiddler. Currently a bug (https://github.com/microsoft/vcpkg/issues/17752)
         will set HTTPS_PROXY to `https://fiddler_address:port` which lead to problem 1 above.
         Workaround is open Windows 10 Settings App, and search for Proxy Configuration page,
         Change `http=address:port;https=address:port` to `address`, and fill the port number.
      
      3. You proxy's remote server is out of service.
      
      In future vcpkg releases, if you are using Windows, you no longer need to set
      HTTP(S)_PROXY environment variables. Vcpkg will simply apply Windows IE Proxy
      Settings set by your proxy software. See (https://github.com/microsoft/vcpkg-tool/pull/49)
      and (https://github.com/microsoft/vcpkg-tool/pull/77)
      
      Otherwise, please submit an issue at https://github.com/Microsoft/vcpkg/issues

Call Stack (most recent call first):
  scripts/cmake/vcpkg_from_github.cmake:153 (vcpkg_download_distfile)
  ports/inja/portfile.cmake:1 (vcpkg_from_github)
  scripts/ports.cmake:141 (include)


Error: Building package inja:x64-linux failed with: BUILD_FAILED
Please ensure you're using the latest portfiles with `./vcpkg update`, then
submit an issue at https://github.com/Microsoft/vcpkg/issues including:
  Package: inja:x64-linux
  Vcpkg version: 2021-07-21-unknownhash

Additionally, attach any relevant sections from the log files above.
@BillyONeal
Copy link
Member

#19077 may help

@brycelelbach brycelelbach mentioned this issue Jul 28, 2021
Merged
@NancyLi1013
Copy link
Contributor

Duplicated to #19078.

Since the changes in #19077 have been merged to vcpkg master branch now.
Could you please try to update vcpkg to the latest commit via git pull to rebuild this? @brycelelbach

@NancyLi1013 NancyLi1013 added the category:vcpkg-bug The issue is with the vcpkg system (including helper scripts in `scripts/cmake/`) label Jul 28, 2021
@ras0219-msft
Copy link
Contributor

ras0219-msft commented Jul 28, 2021 •
edited

We could also change --head to not use any github APIs and instead either:

  1. Switch over to vcpkg_from_git()
  2. Just directly download master.zip instead of trying to identify+cache the precise commit

@ras0219-msft
Copy link
Contributor

#19077 doesn't really fix the primary issue, which is that public ports like inja will fail. #19077 would fix this issue for a private port that already has an auth token parameter in non-head mode.

@brycelelbach
Copy link
Author

I very much like the idea of not using GitHub APIs, that'd be much better from my end.

@brycelelbach brycelelbach mentioned this issue Feb 1, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@NancyLi1013 NancyLi1013

Labels
category:vcpkg-bug The issue is with the vcpkg system (including helper scripts in `scripts/cmake/`)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@brycelelbach @BillyONeal @ras0219-msft @NancyLi1013