vcpkg's git strategy appears to neglect submodules

[heydojo]

Some git repositories use submodules.
Submodules are external repositories which need to exist in the build tree at build time but which are not checked out along with the repo by git without a follow up command.

Although eventually, it makes sense to split out submodule dependencies into their own individual ports (for each individual piece of software,) initially: doing so might not be ideal.

To update sub modules within a repository : the following command should be run once a source tree has been downloaded:-

git submodule update --init

An "update submodule" switch for the vcpkg_from_git, vcpkg_from_github and vcpkg_from_gitlab functions would really help ports which need to do this. Especially for software which incorporates obscure libraries as sub modules (which serve no use as individual ports.)

https://github.com/microsoft/vcpkg/blob/master/scripts/cmake/vcpkg_from_github.cmake#L114

A shallow clone, followed up by the sub module update command instead of an archive pull is my initial idea regarding a solution.

To try to solve this currently, for example, if you:-

	vcpkg_find_acquire_program(GIT)
	get_filename_component(GIT_EXE_PATH ${GIT} DIRECTORY)
	set(ENV{PATH} "${GIT_EXE_PATH};$ENV{PATH}")

	vcpkg_execute_required_process(
		COMMAND git submodule update --init
		WORKING_DIRECTORY ${SOURCE_PATH}
	)

Within a portfile; the result appears to be:

fatal: 'submodule' appears to be a git command, but we were not
able to execute it. Maybe git-submodule is broken?

I'm fairly new to git submodules but I think this:
https://github.blog/2016-02-01-working-with-submodules/
is a pretty good place to begin to learn about them (for anyone else interested.)

Also of note, the file .gitmodules in the root of the repository holds the sub module information. Should you want to add them manually to a port file or to fork them out into their own respective individual ports.

So is git submodule support within vcpkg some thing that could happen or have I missed an existing way vcpkg already deals with this?

[MVoz]

https://github.com/microsoft/vcpkg/blob/4921636f6bc92e041a410870ce564615c85a6cfb/ports/xlnt/portfile.cmake

find_program(GIT git)

set(GIT_URL "https://github.com/tfussell/xlnt.git")
set(GIT_REV "d7cd24c9f2092f691e266e872a3f297e10f60315")

if(NOT EXISTS "${DOWNLOADS}/xlnt.git")
    message(STATUS "Cloning")
    vcpkg_execute_required_process(
        COMMAND ${GIT} clone --bare ${GIT_URL} ${DOWNLOADS}/xlnt.git
        WORKING_DIRECTORY ${DOWNLOADS}
        LOGNAME clone
    )
endif()

COMMAND ${GIT} clone submodule update --init ${GIT_URL} ${DOWNLOADS}/xlnt.git

find_program(GIT git)

set(GIT_URL "https://github.com/LabSound/LabSound.git")
set(GIT_REV "1567ec802d1525569ad3a46c97ef054679714fa7")

set(SOURCE_PATH ${CURRENT_BUILDTREES_DIR}/src/${PORT})


if(NOT EXISTS "${SOURCE_PATH}/.git")
	message(STATUS "Cloning and fetching submodules")
	vcpkg_execute_required_process(
	  COMMAND ${GIT} clone --recurse-submodules ${GIT_URL} ${SOURCE_PATH}
	  WORKING_DIRECTORY ${SOURCE_PATH}
	  LOGNAME clone
	)

	message(STATUS "Checkout revision ${GIT_REV}")
	vcpkg_execute_required_process(
	  COMMAND ${GIT} checkout ${GIT_REV}
	  WORKING_DIRECTORY ${SOURCE_PATH}
	  LOGNAME checkout
	)
endif()

or

https://github.com/Voskrese/vcpkg/blob/4989f54d450070fd914fc6c9f77ec73aa763a7c8/ports/graphviz/portfile.cmake#L10

[heydojo]

Nice.
I have been adding them separately here. Makes more sense to move them into ports but the submodule command would still be nice in the fetch_from_git(hub,lab) commands IMO.

[MVoz]

use as you want
all in your hands ))

[heydojo]

Thanks for the advice guys. I'm still on the fence about whether I think --recurse-submodules should be the default. Because without it being default:

• On the one hand, it means you have to know that your port could use a recursive submodules checkout if you so wish. Which of course involves it failing to build first.
• But on the other it means that you are gently encouraged to reuse existing vcpkg ports or create new separate ones as required to replace sub modules. Thus reducing the possibility of other ports reproducing the same files and folders.

From what I've seen so far, developers who use sub modules tend to want to freeze the code they are pulling from upstream (at a certain commit) to facilitate consistent reproducible builds. So a vcpkg port which compiles against code which is further ahead or behind in development than that of the position of the sub module checkout could maybe create undesirable results in the resulting compiled outputs.

The solution here to use COMMAND ${GIT} clone --recurse-submodules ${GIT_URL} ${SOURCE_PATH} is great. And I suppose using it is really down to how appropriate it is to the individual situation at hand. The trade-offs for or against appear to be quite interesting.