FetchContent_MakeAvailable fails to clone private repositories

[jshellen]

TLDR:
CMake configuration step fails to FetchContent_MakeAvailable when trying to clone a private repository.

Background:
We have a C++ project which uses CMake and FetchContent to manage external dependencies. All external dependencies are contained in private repositories under our own organization. We are trying to automate unit testing using GitHub Actions but are unable to make this work due to the fact that FetchContent_MakeAvailable fails when trying to clone into private repositories.

cmake.yml

name: CMake
on: [push]
env:
  BUILD_TYPE: Release

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2

    - name: Install GoogleTest
      run: (...)
    
    # This step fails, see error log below
    - name: Configure
      run: |
        mkdir build
        cd build
        cmake ..

Error log:

Run mkdir build
-- The C compiler identification is GNU 9.3.0
-- The CXX compiler identification is GNU 9.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
Scanning dependencies of target common-populate
[ 11%] Creating directories for 'common-populate'
[ 22%] Performing download step (git clone) for 'common-populate'
-- Configuring incomplete, errors occurred!
See also "/home/runner/work/queue/queue/build/CMakeFiles/CMakeOutput.log".
Cloning into 'common-src'...
fatal: could not read Username for 'https://github.com': No such device or address
Cloning into 'common-src'...
fatal: could not read Username for 'https://github.com': No such device or address
Cloning into 'common-src'...
fatal: could not read Username for 'https://github.com': No such device or address
-- Had to git clone more than once:
          3 times.
CMake Error at common-subbuild/common-populate-prefix/tmp/common-populate-gitclone.cmake:31 (message):
  Failed to clone repository: 'https://github.com/ORGANIZATION_NAME/common'


make[2]: *** [CMakeFiles/common-populate.dir/build.make:115: common-populate-prefix/src/common-populate-stamp/common-populate-download] Error 1
make[1]: *** [CMakeFiles/Makefile2:95: CMakeFiles/common-populate.dir/all] Error 2
make: *** [Makefile:103: all] Error 2

CMake Error at /usr/local/share/cmake-3.19/Modules/FetchContent.cmake:989 (message):
  Build step for common failed: 2
Call Stack (most recent call first):
  /usr/local/share/cmake-3.19/Modules/FetchContent.cmake:1118:EVAL:2 (__FetchContent_directPopulate)
  /usr/local/share/cmake-3.19/Modules/FetchContent.cmake:1118 (cmake_language)
  /usr/local/share/cmake-3.19/Modules/FetchContent.cmake:1161 (FetchContent_Populate)
  CMakeLists.txt:11 (FetchContent_MakeAvailable)


Error: Process completed with exit code 1.

How can we fix this?

[joeyes86]

Hi! Have you managed to solve it? I faced with the same one.

[catpatris]

Likewise...

[catpatris]

This is how I did it... not sure if it's the best solution:

1. Make a Github Personal Access Token (PAT) as per https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token

2. In your .github/workflows/ yaml file:

env:
  GH_PAT: ${{ secrets.GH_PAT }}

jobs:
  build:
    runs-on: ubuntu-22.04
    steps:
      - name: Checkout this repo
        uses: actions/checkout@v3

3. In CMakeLists.txt:

if(DEFINED ENV{GITHUB_ACTION})
  set(GIT_REPO https://<username>:$ENV{GH_PAT}@github.com/MyOrg/other-repo)
else()
  set(GIT_REPO https://github.com/MyOrg/other-repo)
endif()

FetchContent_Declare(
  OtherRepo
  GIT_REPOSITORY  ${GIT_REPO}
  GIT_TAG         e35e6fe2402137b31e813d5290e2a9d123951d1e
)

FetchContent_MakeAvailable(OtherRepo)

Replace <username> with that used in generating the PAT.

The conditional in CMakeLists.txt allows local builds.

[joeyes86]

I found this solution (based on PAT as well):
git config --global credential.helper store works well

[jamesETsmith]

A slightly modified version of @catpatris's solution to keep the CMakeLists.txt as simple as possible. The github actions CI file gets a bit messier with this strategy, but I think it's worth keeping the CMakeLists.txt clean because more people will interact with that file (for my use case at least).

CMakeLists.txt

include(FetchContent)
FetchContent_Declare(
 <repo_name>
  GIT_REPOSITORY git@github.com:<my_org>/<repo_nam>.git
  GIT_TAG  <repo_hash>     
)

CI config file (note that we don't need any username here, just the PAT):

      - name: Configure CMake
        env:
          GH_PAT: ${{ YOUR_PAT }}
        run: |
          # Need a little regex here so we can download repo with PAT
          sed -i 's|git@github.com:|https://$ENV\{GH_PAT\}@github.com/|g' CMakeLists.txt
          cmake -S . -B build

[giallu]

I just stumbled upon this issue. The weird thing is that fetchcontent works if I use this:

git config --global url."https://${{ secrets.GH_ACCESS_TOKEN }}@github.com/".insteadOf "git@github.com:"
in a step before the cmake one. Please note I always use SSH to access repos, so the deps are declared like this:

FetchContent_Declare(
    mdtk
    GIT_REPOSITORY git@github.com:<org>/mdtk
    GIT_TAG fc8a658
)

However, passing the same secret in the token parameter of the checkout action do not work, apparently fetchcontent still wants to use ssh to get the sources, ignoring the previous global settings from the action:

[ 11%] Creating directories for 'mdtk-populate'
[ 22%] Performing download step (git clone) for 'mdtk-populate'
-- Configuring incomplete, errors occurred!
Cloning into 'mdtk-src'...
git@github.com: Permission denied (publickey).
fatal: Could not read from remote repository.

I am still not sure if this is a problem with the action or cmake/fetchcontent, can anyone help diagnose?

[phelter]

Is there a good example of how to do this properly yet?

I found this on the web as well - for supporting keys:
https://github.com/webfactory/ssh-agent?tab=readme-ov-file#ssh-agent-github-action

I would like to refrain from using a PAT as it is tied to a particular user I know I can use deploy keys but that is one per repo.

Any suggestions here? I know it's not a one-size fits all issue but an example would be nice.

[bvnp43]

Just use webfactory/ssh-agent github action and add your private ssh key to it via env var.

- uses: webfactory/ssh-agent@v0.9.0
  with:
    ssh-private-key: ${{ secrets.YOUR_SSH_PRIVATE_KEY }}

Then in cmake fetch repo like

FetchContent_Declare(your_private_repo
        GIT_REPOSITORY git@github.com:your_org/your_private_repo.git
        GIT_TAG ...
)