Feature Request: allow private repository on vcpkg #17720

jasjuang · 2018-01-05T00:01:44Z

jasjuang
Jan 5, 2018

I have a private repo A that is dependent on some external libraries and private repo B. Currently external libraries are installed in vcpkg and private repo B is installed in the default C:/Program Files (x86). I attempted to write a portfile for private repo B, so everything will be from vcpkg but the problem is when I do vcpkg install repo_B I am expecting a git credential prompt in the terminal, but instead I get the error

Failed. Status: 22;"HTTP response code said error"

Is it possible to not fail directly and prompt for git credentials instead?

ras0219-msft · 2018-01-05T04:02:53Z

ras0219-msft
Jan 5, 2018
Maintainer

I'm not sure; could you sanitize the portfile for repo_B and post it along with some more details of the situation? (For example, I don't understand why/how you're using git to access C:/Program Files (x86))

0 replies

jasjuang · 2018-01-05T06:26:53Z

jasjuang
Jan 5, 2018
Author

@ras0219-msft The portfile is shown below,

include(vcpkg_common_functions)

vcpkg_from_github(
    OUT_SOURCE_PATH SOURCE_PATH
    REPO mycompany/repo_B
    REF 1.0
    SHA512 5a7e84ecc5a54320c74776c133bfdbeaf0d4496a7a7fdf2f4ccf89e66b3665a577a370a662ac97a350a2b1f717ce769cb0826057ebb3b13c9c2fee65f20ac7b4
    HEAD_REF master
)

vcpkg_configure_cmake(
    SOURCE_PATH ${SOURCE_PATH}
)

vcpkg_install_cmake()

vcpkg_fixup_cmake_targets(CONFIG_PATH "lib/cmake/repo_B")

vcpkg_copy_pdbs()

file(REMOVE_RECURSE ${CURRENT_PACKAGES_DIR}/debug/include)
file(REMOVE_RECURSE ${CURRENT_PACKAGES_DIR}/debug/share)

# Put the license file where vcpkg expects it
file(COPY ${SOURCE_PATH}/copying.txt DESTINATION ${CURRENT_PACKAGES_DIR}/share/repo_B/)
file(RENAME ${CURRENT_PACKAGES_DIR}/share/repo_B/copying.txt ${CURRENT_PACKAGES_DIR}/share/repo_B/copyright)

repo_B is private so when vcpkg install repo_B I am expecting a git credentials prompt.

C:/Program Files (x86) is the default install location of repo_B, for example,

git clone https://github.com/mycompany/repo_B
mkdir build && cd build
cmake -A x64 ..
msbuild repo_B.sln
msbuild INSTALL.vcxproj 
# headers, libs, cmake configs of repo_B will be installed to C:/Program Files (x86)

0 replies

ras0219-msft · 2018-01-13T01:44:27Z

ras0219-msft
Jan 13, 2018
Maintainer

Ok, I got a "proof of concept" to work:

Create a PAT on GitHub (under developer settings) with "repo" permissions
Edit scripts\cmake\vcpkg_download_distfile.cmake and add USERPWD <githubusername>:<PAT> to the parameters for all file(DOWNLOAD) commands

This allows me to download archives from private repos, however it certainly has huge problems (first of which, it will pass your PAT to all websites, not just GitHub).

In general though, I think an interaction-less approach will be better than constantly requiring the user to enter their password.

My thoughts are therefore to add it as an environment variable that you would set (VCPKG_GITHUB_USERPWD?) and it would only apply when pulling from GitHub. This should also (hopefully) be very friendly to CI machines, which often have ways to inject credentials through environment variables.

Would that work well for you?

0 replies

crackedmind · 2018-01-13T01:49:26Z

crackedmind
Jan 13, 2018

My thoughts are therefore to add it as an environment variable that you would set (VCPKG_GITHUB_USERPWD?) and it would only apply when pulling from GitHub.

but what about private repos on bitbucket/gitlab ?

0 replies

jasjuang · 2018-01-13T02:08:34Z

jasjuang
Jan 13, 2018
Author

@ras0219-msft I think it will be great to have both interaction and interaction-less. Interaction-less is great for CI purposes but at the same time there are situations where multiple people are accessing the same computer/server so storing github password through environment variables is not a good idea for those cases. In this case credential prompt would be more suitable.

0 replies

ras0219-msft · 2018-01-13T03:26:40Z

ras0219-msft
Jan 13, 2018
Maintainer

In the shared computer case, the idea would be that you could set this environment variable just for the shell you're using:

$env:VCPKG_GITHUB_USERPWD = "user:passwd"
vcpkg stuff

Once the shell closes, your password is no longer stored :) Though, it wouldn't work very well while giving a presentation.

@crackedmind You're definitely right that this only solves GitHub for now! I think a more general solution can be evolved out of this, but I want to make sure we solve @jasjuang's problem in the short term.

Some possible evolutions would be:

More environment variables for well known services like the ones you mentioned (this has obvious limits, but still does solve real problems for real people in a convenient way)
A user-wide config file that provides a mapping from hosts to user+password, kind of like how git's credential helpers work. We might even try querying git's credential helper directly, but I think this will take more time to investigate if it is reasonable.

A big problem with the general case is that every service can have completely different ways to expect authentication (Maybe they require OAuth tokens in the URL, maybe they use HTTP basic auth, maybe they want SSL client certs, etc). Therefore, we'll definitely need to collect lots of specific individual use cases!

0 replies

jasjuang · 2018-01-13T03:32:55Z

jasjuang
Jan 13, 2018
Author

@ras0219-msft Yes that makes sense, so I guess as long as the error message is set your environment variable VCPKG_GITHUB_USERPWD instead of Failed. Status: 22;"HTTP response code said error" that will be a good first version.

0 replies

MikeGitb · 2018-01-14T06:33:54Z

MikeGitb
Jan 14, 2018

@ras0219-msft: Git already has a credential manager, I don't understand why it is necessesary to reinvent it. Can't we just use it? And I'm not sure how I feel about storing my password as plaintext.

0 replies

jasjuang · 2018-03-24T01:12:51Z

jasjuang
Mar 24, 2018
Author

Just in case anyone is interested. I figured out a hack that will make it work temporarily. I downloaded the release from github manually and place it in the downloads folder, so when I do vcpkg install myprivaterepo it will actually use the "cached" version and proceed to install like usual.

0 replies

donny-dont · 2018-09-11T18:02:14Z

donny-dont
Sep 11, 2018

@ras0219-msft I'm hitting this as well but with vcpkg_download_distfile. Would it be possible to extend it to include headers? That way we could set the authorization header for a download.

0 replies

soroshsabz · 2019-01-10T07:25:54Z

soroshsabz
Jan 10, 2019

ITNOA

I think a good documentation in wiki for this scenario is a very good choice and needed.

0 replies

katusk · 2019-10-14T09:13:06Z

katusk
Oct 14, 2019

@jasjuang Your hack works for me as well, thanks. Have you tried vcpkg_from_git instead of vcpkg_from_github with an "https://" link? I had good experience with vcpkg_from_git instead of vcpkg_from_bitbucket lately in case of a private Bitbucket repository. I had to provide my credentials upon vcpkg install though (interaction).

Unfortunately, vcpkg_from_git seems to require a "https://" link, see CMake script below. I do not fully get the comment below the check there. @ras0219-msft Can you please provide more detail on this?

https://github.com/microsoft/vcpkg/blob/master/scripts/cmake/vcpkg_from_git.cmake#L56

Anyway, if I modify that message(FATAL_ERROR ...) to message(WARNING ...), and I pass an "ssh://" link to vcpkg_from_git, then it works with that too for my private Bitbucket repository. If you have your ssh keys properly set up, you do not even need to provide any credentials in this case (interaction-less). We need this for CI purposes.

NOTE: I used 40 characters of an SHA for my purposes, like so (WARNING: potentially bad command invocation; see above):

# WARNING. This one is a modified version of the vcpkg_from_git command.
vcpkg_from_git(
    OUT_SOURCE_PATH SOURCE_PATH
    URL ssh://bitbucket.org/org/proj.git
    REF d6ab0b626ea949c18d3167e6451df3b3961b848a
)

0 replies

daydream123 · 2020-05-21T00:25:42Z

daydream123
May 21, 2020

My thoughts are therefore to add it as an environment variable that you would set (VCPKG_GITHUB_USERPWD?) and it would only apply when pulling from GitHub.

but what about private repos on bitbucket/gitlab ?

I also encountered this problem，seems no place to input my username and password, did you find any other solution? this problem stoped me to use vcpkg in my company projects.

0 replies

klalumiere · 2020-10-14T18:29:15Z

klalumiere
Oct 14, 2020

It took me a while to figure this one out, even though the information is on the Internet (like always I guess) here. What we want to do is to "inject" our git credentials inside the script.

Here's a TL;DR/particular implementation that I use on Jenkins:

Create a script name e.g. .git-askpass.sh that contains

#!/usr/bin/env bash
echo username=$GIT_USERNAME
echo password=$GIT_PASSWORD

Run sh "git config --global credential.helper \"${pwd()}/.git-askpass.sh\""
Wrap the script that calls e.g. git clone https://github.com/myorg/privaterepo in the block

withCredentials([usernamePassword(credentialsId: 'github',
        usernameVariable: 'GIT_USERNAME',
        passwordVariable: 'GIT_PASSWORD')])
{}

e.g.

withCredentials([usernamePassword(credentialsId: 'github',
    usernameVariable: 'GIT_USERNAME',
    passwordVariable: 'GIT_PASSWORD')])
{
    sh "./scriptThatCallsGit.sh"
}

Of course this can be adapted for Windows with e.g. a PowerShell script.

From what I understand, I'd argue that this approach solves the need raised by this issue. The approach also provides a clear separation of concern between authentication and content retrieval, which makes vcpkg's code easier to maintain.

0 replies

PhoebeHui · 2021-05-27T08:54:47Z

PhoebeHui
May 27, 2021
Collaborator

For vcpkg_from_github, you can pass the token via 'AUTHORIZATION_TOKEN' to setup the private repo. See https://github.com/Microsoft/vcpkg/blob/master/scripts/cmake/vcpkg_from_github.cmake#L16

0 replies

ras0219-msft · 2021-06-08T23:41:17Z

ras0219-msft
Jun 8, 2021
Maintainer

See also the discussion #17272 for some solutions

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature Request: allow private repository on vcpkg #17720

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 16 comments

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Feature Request: allow private repository on vcpkg #17720

Replies: 16 comments

ras0219-msft Jan 5, 2018 Maintainer

jasjuang Jan 5, 2018 Author

ras0219-msft Jan 13, 2018 Maintainer

jasjuang Jan 13, 2018 Author

ras0219-msft Jan 13, 2018 Maintainer

jasjuang Jan 13, 2018 Author

jasjuang Mar 24, 2018 Author

PhoebeHui May 27, 2021 Collaborator

ras0219-msft Jun 8, 2021 Maintainer

ras0219-msft
Jan 5, 2018
Maintainer

jasjuang
Jan 5, 2018
Author

ras0219-msft
Jan 13, 2018
Maintainer

jasjuang
Jan 13, 2018
Author

ras0219-msft
Jan 13, 2018
Maintainer

jasjuang
Jan 13, 2018
Author

jasjuang
Mar 24, 2018
Author

PhoebeHui
May 27, 2021
Collaborator

ras0219-msft
Jun 8, 2021
Maintainer