Switching revisions redownloads full repository

[PhilLab]

We use this awesome script to download our 3rdparty repository containing several gigabytes of pre-buillt thirdparty libraries.

However, when we add a new prebuild there and consequently update the referenced version in our main repository's cmake, the whole repository is checked out again (Instead of just downloading the diff). This takes hours for our +20 GB repository, especially via VPN.

Is there a way to achieve just a git pull using DownloadProject or FetchContent?

[craigscott-crascit]

Side note: If you can, please switch to FetchContent rather than using DownloadProject. The former is properly maintained, the latter is only really left here for those who can't yet move to CMake 3.11 or later.

I would have expected both DownloadProject and FetchContent to do an update rather than a full re-clone if all you've done is added commits to the existing repo. Potential things that come to mind:

• If you are doing a shallow clone, then there's no choice but to re-clone if your local copy doesn't have the requested tag/hash.
• Are you sure it is cloning the whole repo again, or is it just taking a while to download the added commits?

If neither of the above reveals the underlying cause for you, I suggest you confirm you still see the problem if using FetchContent and then post a complete description with output confirming the problem in the CMake forum. We can discuss it further there.

[PhilLab]

@craigscott-crascit Thanks for taking a look. We will take a look at FetchContent and if we can migrate to it.

Right now we have in our root CMakeLists.txt:

IF (THIRD_PARTY_DOWNLOAD)
    # Download the repository
    include(cmake/DownloadProject.cmake)
    MESSAGE(STATUS "Downloading thirdparty. If this repeatedly fails via VPN, please see the option THIRD_PARTY_DOWNLOAD...")
    download_project(PROJ                   thirdparty
                     GIT_REPOSITORY         ${THIRD_PARTY_REMOTE_URL}
                     GIT_TAG                ${THIRD_PARTY_GIT_TAG}
                     #To build without network
                     #UPDATE_DISCONNECTED    1
                     TLS_VERIFY             OFF
                     # Needed to disable TLS verification for LFS storage
                     GIT_CONFIG "http.sslVerify=false"
                     PREFIX                 ${THIRD_PARTY_PATH}/.tmp
                     SOURCE_DIR             ${THIRD_PARTY_PATH}/checkout
                     QUIET
    )
ELSE()
    MESSAGE(STATUS "Thirdparty is not downloaded, THIRD_PARTY_DOWNLOAD=OFF")
    SET(thirdparty_SOURCE_DIR "${THIRD_PARTY_PATH}/checkout")
    EXECUTE_PROCESS(
        COMMAND "${GIT_EXECUTABLE}" checkout ${THIRD_PARTY_GIT_TAG}
        WORKING_DIRECTORY "${thirdparty_SOURCE_DIR}"
        RESULT_VARIABLE result
        OUTPUT_QUIET
        )
    if(result)
        message(FATAL_ERROR "THIRD_PARTY_DOWNLOAD=OFF but local checkout does not contain requested thirdparty revision)
    endif()
ENDIF()

And we see that the SOURCE_DIR is always completely cleared and repopulated again, when THIRD_PARTY_DOWNLOAD=ON

[PhilLab]

@craigscott-crascit The complete deletion of the checkout folder happens if the local checkout does not yet contain the requested revision. If it does contain it, the revision is checked out without a full redownload.

We also migrated to FetchContent but the behavior is the same

[craigscott-crascit]

Ok. As mentioned previously, it is very unlikely that this DownloadProject repo will get any attention or further fixes. Please report this issue in CMake's issue tracker with a minimal example using FetchContent that demonstrates the problem. I'll try to take a look once it is reported there.

[craigscott-crascit]

Closing since DownloadProject is not being developed further and FetchContent is the appropriate replacement.