Creating a GitHub compatible requirements file for CPM

[alandefreitas]

This discussion is a continuation of an idea started in issue #152 with @saxbophone, @iboB, @alandefreitas, and @TheLartians.

The idea is to create a requirements.txt (or equivalent) file in the project root. Any dependency included with CPM with be appended to this file in a format such as

-e git+https://github.com/catchorg/Catch2@v2.5.0#egg=Catch2

This would allow C++ projects and CPM itself to be part of the GitHub dependency graphs. Being part of the graph has lots of advantages, like automatically documenting dependencies, inferring how far your project has reached and being recommended for sponsorships.

Relevant questions for discussion (and some partial answers) are:

• Is there a better file format among the ones supported by GitHub?
  • While pip is just a list of dependencies in a text file, which is simple to replicate and append dependencies
  • It's worth having a look at each file format in the list to make sure the pip format is the best one
  • Formats based on json files are probably hard to manage from CMake scripts We can parse JSON files with the string() command
• Is it possible to make GitHub support another file format or filename?
  • That seems unlikely
• Can the code for package-lock.cmake be reused to make this happen?
• Is there a way to allow more than one requirements.txt file or python and C++ requirements in the same project?
  • It's probably OK to not have this feature from the beginning.
  • We could replicate whatever python projects with two dependency sets are doing.
  • GitHub might support another filename for dependencies.
  • GitHub can probably read the requirements.txt file from different directories.
  • Moving that to lock files instead of requirements.txt might solve the problem.
  • Annotate the dependencies with comments so CPM can ignore Python dependencies (small workaround that works only one way).
• Should there be a bidirectional relationship between this file and CPM?
  • The CMakeLists.txt file already serves the function of requirements.txt in a much better way because CMake allows us to create more complex logic. CPM doesn't really need to use this file, so the common direction would be from the CMake to the file.
  • There could be a command to make it happen in the opposite direction though. So CPM would read the file and add its dependencies. Any CPM command to add a dependency would do nothing. However, this might take some time to implement and it's not that useful if you are already using CMake.

Let the discussion begin...

[JohelEGP]

Formats based on json files are probably hard to manage from CMake scripts

From CMake 3.19 Release Notes,

The string() command gained a set of new JSON sub commands that provide JSON parsing capabilities.

[alandefreitas]

Cool! 😀

We can look into all file formats then!

[TheLartians]

I really love the idea of having a requirements.txt or equivalent file for dependencies to allow GitHub to build a C++ dependency graph! It might be a good idea to update the current package-lock functionality to use a GitHub compatible file format. Also imo we need to find a way to allow auto-updating of the package lock, e.g. when changing a dependency in the CMakeLists.txt, to make it more attractive to use. Currently it overwrites all dependencies defined elsewhere, meaning that we can only update dependencies by changing the entry in the package-lock itself.

Another thing I wonder is how GitHub resolved a dependencies original repository. E.g. for npm's package-lock.json I believe it actually checks the repository field inside the dependency's package.json file. I'm not sure how this would be resolved for C++ projects, as they usually don't contain a package.json or similar manifest. For research purposes it would be great if we had an example repo containing a C++ dependency manifest that GitHub is able to resolve.

[alandefreitas]

To be honest, I did read the documentation but I don't fully understand how the CPM package-lock.cmake works yet. I don't know the syntax, but the documentation makes it seem like it's going in the other direction from the file to CMake. If that's correct, it's more like a package.txt than a package-lock.txt because I think package locks are not supposed to be updated manually. I might be completely wrong about that.

In any case, even if we can't reuse what we have for package-lock.cmake, the initial implementation is just a matter of extending the CPMDeclarePackage function to write a new line to the file, with something that respects the file format:

if (CPM_GENERATE_REQUIREMENTS)
    file(APPEND ${REQUIREMENTS_FILE} "# Any comment from CPM\n")
    if (CPM_ARGS_GIT_REPOSITORY)
        file(APPEND ${REQUIREMENTS_FILE} "${CPM_ARGS_NAME}@git+${CPM_ARGS_URL}@${GIT_TAG}\n")
    elseif (CPM_ARGS_URL)
        list(GET CPM_ARGS_URL 0 URL)
        file(APPEND ${REQUIREMENTS_FILE} "${CPM_ARGS_NAME}@${URL}\n")
    endif()
endif()

where

set(REQUIREMENTS_FILE ${PROJECT_SOURCE_DIR}/requirements.txt)

That's all there is to it. The dependencies should be part of the graph. The obvious details, like conflicting requirements.txt and adding CPM itself are small and might come later if there is a demand for that.

The syntax is always one of these:

<requirement specifier> [; markers] [[--option]...]
<archive url/path>
[-e] <local project path>
[-e] <vcs project url>

but we just need <archive url/path> or [-e] <vcs project url>.

I'm not sure the target package also requires a local manifest for GitHub to be included in the graph. I've seen some dependencies without a manifest file in any common format (example) that are part of the graph.

At least the GitHub documentation makes it seem like all you need is to have the dependencies in your own repository. That makes sense since all they need to know is the GitHub address of the repository. For C++ projects, this would be straight in the file, since the repository is mentioned directly in the requirements file. They have all the extra information they might need on their side.

I created a repository to test that. The documentation says it might take a few minutes though. If nothing happens, we might need to change something or just ask in the support community. They usually don't take long to answer.

[TheLartians]

To be honest, I did read the documentation but I don't fully understand how the CPM package-lock.cmake works yet. I don't know the syntax, but the documentation makes it seem like it's going in the other direction from the file to CMake. If that's correct, it's more like a package.txt than a package-lock.txt because I think package locks are not supposed to be updated manually. I might be completely wrong about that.

I think you're quite right about that and, in retrospect, the name might be misleading. My main intention at the time was to be able to easily identify and update nested dependencies, without needing to touch the original declarations in CMake. This is in a way similar to what npm's package-lock.json allows, however you're right in that it should rarely be changed manually, but by npm itself (e.g. by calling npm update). My hope is that CPM.cmake eventually evolves enough to support automatically updating the package-lock file.

That's all there is to it. The dependencies should be part of the graph. The obvious details, like conflicting requirements.txt and adding CPM itself are small and might come later if there is a demand for that.

That does sound easy enough, imo it could be even better if we find a way to directly update CPM.cmake's package-lock format to one understandable by the GitHub graph instead. We could then start adding some basic functionally to allow CPM.cmake to automatically manage the lock-file. Personally, I really like the package-lock.json format as it's easy to query by any json reader, but I'm open to other formats as well.

I created a repository to test that. The documentation says it might take a few minutes though. If nothing happens, we might need to change something or just ask in the support community. They usually don't take long to answer.

That's awesome, once we get it to work it shouldn't be too difficult to create a first implementation for CPM.cmake!

[alandefreitas]

That does sound easy enough, imo it could be even better if we find a way to directly update CPM.cmake's package-lock format to one understandable by the GitHub graph instead.

Is there an informal syntax for the current CPM.cmake's package-lock format somewhere? Maybe we could look for something that looks similar.

We could then start adding some basic functionally to allow CPM.cmake to automatically manage the lock-file

I'm still a bit confused about that because I'm not sure the idea of a requirements-lock file makes sense without a requirements file. But again, I don't fully understand how the CPM.cmake's package-lock works.

For instance, if the requirements file says the project needs version >=3, the requirements lock might be used to state that the project uses version 3.2 specifically, but this doesn't seem to make much sense the way CPM works.

You could also use it to lock the specific version when the same project adds the same dependency twice. But this is kind of tricky because a project might legitimately use two different versions of a dependency, as long as there are no ODR violations, depending on how the targets are linked.

Personally, I really like the package-lock.json format as it's easy to query by any json reader, but I'm open to other formats as well.

Any format GitHub understand is fine. They are all very simple. I just thought dealing with json files from CMake must be a little tedious even with the new JSON sub commands.

And the pip format is stupidly simple. It's really just a list of links/dependencies. In principle, you could parse it to json but it would be always a json array.

On the other hand, now that I saw your package-lock.json, I guess it might be easier for GitHub to parse VCS dependencies from json files than from pip files.

[TheLartians]

Any progress or ideas on how to get GitHub to link dependencies to GH repos? I've asked in the GitHub.community, but so far no response.

[alandefreitas]

I also asked in the GitHub community but nothing. 😔

This Pacman ghost is haunting me