External repos #295

mdavezac · 2016-01-07T17:53:17Z

The objective is to allow for external repositories of recipes. In so far as I understand @alamaison's proposal in #12, this pull request goes some way towards separating two concerns, i.e. (i) the package manager, (ii) the managed recipes. It makes the following possible:

repos for sub-communities more specialised than what the main hunter repo might cater to
private repos for private people/organisations

Repos are specified by adding a directory to HUNTER_RECIPE_DIRS before any call to hunter_add_package:

HunterGate(...)
list(APPEND HUNTER_RECIPE_DIRS "/path/to/recipes/")

project(...)
hunter_add_package(...)

Each directory has the following format, following the scheme in hunter itself:

/root/
   |--- configs/
      |--- defaults.cmake  # defaults of packages in this repo
   |--- projects/
      |--- projectA/
         |--- hunter.cmake
         |--- component0/
            |--- hunter.cmake
         |--- projectB/
            |--- hunter.cmake

For what it's worth, there is a test of this approach at UCL/GreatCMakeCookOff#40.
Yep, it should really be here, but I couldn't quite figure out how testing works in hunter...
Any pointers appreciated!

Repos are specified by adding a directory to `HUNTER_RECIPE_DIRS` before any call to `hunter_add_package`: ~~~CMake HunterGate(...) list(APPEND HUNTER_RECIPE_DIRS "/path/to/recipes/") project(...) hunter_add_package(...) ~~~ Each directory has the following format: /root/ |--- defaults.cmake # defaults of packages in this repo |--- projectA/ |--- hunter.cmake |--- component0/ |--- hunter.cmake |--- projectB/ |--- hunter.cmake It's a simplification of the file structure in hunter, keeping only stuff for projects.

headupinclouds · 2016-01-07T18:23:23Z

Very useful feature.

Yep, it should really be here, but I couldn't quite figure out how testing works in hunter...
Any pointers appreciated!

If you are referring to how to run the hunter/example unit tests, I struggled with that one previously. From a recent post by @ruslo in the #76 GMock thread, this wiki page discusses how to run the local hunter testing .

# check you have python3
> python3 --version
Python 3.4.3

# download build.py script
> git clone https://github.com/ruslo/polly
> export PATH="`pwd`/polly/bin:$PATH"
> which build.py
/.../polly/bin/build.py

# clone Hunter repository
> git clone https://github.com/ruslo/hunter
> cd hunter

# run build for some package, like GTest
# environment variable TOOLCHAIN will be used as an argument for `build.py --toolchain`
[hunter]> TOOLCHAIN=default PROJECT_DIR=examples/GTest ./jenkins.py --verbose

# script will pack current archive in `_testing` directory
-- [hunter] Initializing Hunter workspace (f0550216489047a310a27c2b5ac95c70e7e878bf)
-- [hunter]   /.../hunter/_testing/hunter.tar.gz
-- [hunter]   -> /.../_testing/Hunter/_Base/Download/Hunter/unknown/f055021

ruslo · 2016-01-08T07:27:26Z

@mdavezac We will have next type of problems with such approach:

Extra recipes from root-A have package Foo with version 1.0 and SHA1 = ABC.
Extra recipes from root-B have package Foo with version 1.0 but with patch so SHA1 = CBA

The root directory will be choosed by:

SHA1 of Hunter archive (same for both root-A and root-B)
SHA1 of unified config.cmake (since version is the same for both A and B we will have same SHA1 too!)
SHA1 of toolchain (same too).

Hence after root-A is built there will be no instructions triggered for root-B and non-patched version will be used. Let me know if I'm missing something.

By the way here is feature request with implementation notes that can do quite the same job:

https://github.com/ruslo/hunter/issues/56

Beautifier commit cherry picked to develop, thanks:

049d7a2

mdavezac · 2016-01-08T13:22:06Z

If I understand @ruslo's comment, it is that with separate repos of recipes, we may have recipes over-shadowing each other if they declare the same package and the same version number. There is no one-to-one mapping between code and package name + version number.

Frankly, in some ways this is a feature, not a bug. If the same package declares the same version number, then the user most likely expects only one package, not two. It could mean hell to try and link a program to two ever-so-slightly different versions of the same library. Which versions overshadows which is then set by ordering the directories in HUNTER_RECIPE_DIRS.

That said, I think in the short term the simplest approach is to exit with an error message if overshadowing happens, possibly to the point where a package can only be in a single repo at a time (whether or not they declare the same version). This would ensure the uniqueness of the mapping from package+version to actual code.

Later on, we could add correct versioning to the external repos and add provenance to the information kept with a package. Then a fuse would blow only if a package+version is requested with different provenances. This would ensure that although over-shadowing is allowed, it should remain consistent for a given HUNTER_ROOT.

ruslo · 2016-01-09T11:17:52Z

Frankly, in some ways this is a feature, not a bug

From my point of view it's a design bug, not feature. For now there are no such "features" in Hunter and never will be. Currently everything will work correctly, both versions will live in separate hunter-id - hence in separate directories. In cache the SHA1 of archive will be saved (along with many other parameters) so cache will be in correct state too. Again, I think the easiest way to implement it is to follow https://github.com/ruslo/hunter/issues/56.

It could mean hell to try and link a program to two ever-so-slightly different versions of the same library

This conflict about two independent projects with different Foo packages which have same version number 1.0, it's not one project with two versions (this conflict is not possible by design).

If the same package declares the same version number, then the user most likely expects only one package, not two

User may not even know about this conflict. For instance there is project A which use patched Boost and version is 1.59-p0. And there is project B which use different patch of Boost but same version 1.59-p0. If you start with the empty Hunter directory and build A - everything will work fine, if you start with empty Hunter directory and build B - everything will work fine. But if you start with A, then build B - B will use patched version of Boost which is specified by A.

I think in the short term the simplest approach is to exit with an error message if overshadowing happens

The problem is that you can't detect it?

Project must be in single repo. Projects are identified by name only. If project PROJECT (with name PROJECT) lives in repo REPO, it cannot exist in repo X. More accurately, REPO and X cannot be included at the same time if package PROJECT is invoked.

Schemes can only exist in a single repo, e.g. repos are orthogonal. So two repos with schemes with the same name will fail. This commit also allow for schemes found within a schemes directory of the project. These schemes take precedence over all other schemes, and their name can overshadow another scheme.

mdavezac · 2016-01-11T10:29:34Z

The latest commits forces the repos to be orthogonal. By this I mean that a project, as identified exclusively by its name (and not version), can only live in a single repo at a time.

I've also thrown schemes into the mix, with the ability for repos to add their own schemes. Once again, they a scheme is enforced to live in a single repo.

However, schemes that live in projects/PROJECTNAME/schemes/schemename.cmake.in take priority. I thought I saw that feature request in an issue but can't find it now. If it's not wanted, I can remove it from this pull request.

Would this be a sensible solution for now, until I can grok #56 better? (assuming the tests pass, of course).

ruslo · 2016-01-11T13:13:46Z

The latest commits forces the repos to be orthogonal

Doesn't solve the issue. I've created repos for testing:

Experimental Hunter version with your changes: https://github.com/ruslo/hunter/releases/tag/v0.0.0-experimental
Package Foo (which is not in Hunter) with versions 1.0-patch-A and 1.0-patch-B: https://github.com/forexample/temp-foo/releases
Package Boo use Foo release 1.0-patch-A, names it locally 1.0-p0: https://github.com/forexample/temp-boo
Package Bar use Foo release 1.0-patch-B, names it locally 1.0-p0: https://github.com/forexample/temp-bar
Testing script: https://gist.github.com/ruslo/f5c6880462230c8d4dd6

> mkdir hunter-test
> cd hunter-test
[hunter-test]>  wget https://gist.githubusercontent.com/ruslo/f5c6880462230c8d4dd6/raw/8a870041499f37169df13384bdedc37e8d64e3b5/run-test.sh
[hunter-test]> chmod +x run-test.sh
[hunter-test> ./run-test.sh

Result:

========== Generate Boo
========== Build Boo
========== Test clean Boo
...
test 1
    Start 1: BooTest

1: Test command: /.../_builds/boo
1: Test timeout computed to be: 9.99988e+06
1: Foo say: A

If Hunter directory is clean, then Boo start build and correct version of Foo used, prints A

Same for Bar (Hunter directory cleaned before build):

========== Generate Bar
========== Build Bar
========== Test clean Bar
...
test 1
    Start 1: BooTest

1: Test command: /.../_builds/boo
1: Test timeout computed to be: 9.99988e+06
1: Foo say: B

But if we will not clean-up after building Bar and start Boo again, wrong version of Foo will be used:

========== Generate Boo
========== Build Boo
========== Test dirty Boo
...
test 1
    Start 1: BooTest

1: Test command: /home/ruslan/work/test/hunter-test/_builds/boo
1: Test timeout computed to be: 9.99988e+06
1: Foo say: B

As you can see it picks version of Foo which installation is triggered by Bar.

I've also thrown schemes into the mix, with the ability for repos to add their own schemes.

This will add additional moves to implementation, just for your information #56 limited only for CMake standard schemes.

mdavezac · 2016-01-11T13:49:45Z

Okay, thanks for laying out for me. It's very helpful. Now, just have to figure it out...

ruslo · 2016-02-24T03:16:19Z

Closing. For better approach and implementation details see https://github.com/ruslo/hunter/issues/56

@ruslo

1. Package with the same name but comming from different repos will induce a rebuild, since the sha of the unified defaults.config will change. From what I understood, this was @ruslo's main gripe with ruslo#295. 1. Nominally, users can specify which repo to get their recipe from by specifying the repo's directory in their config.cmake file. Since they need a directory at this point, it's not so easy. 1. The argument is not necessary per se. If it is absent, the first repo with the appropriate recipe name will be used, as per _hunter_find_project_dir in hunter_add_package.cmake. This should only happen when users set their default config in config.cmake. 1. The variable HUNTER_${packagname}_RECIPE_DIR references the repo for a given package.

Pkg.ceres solver

Mayeul d'Avezac added 2 commits January 7, 2016 17:20

💄 Just de-depthing an if statement

5a15654

mdavezac mentioned this pull request Jan 7, 2016

Hunter UCL/GreatCMakeCookOff#40

Merged

Mayeul d'Avezac added 2 commits January 11, 2016 10:14

Force repos to orthogonal

2c0bc13

Project must be in single repo. Projects are identified by name only. If project PROJECT (with name PROJECT) lives in repo REPO, it cannot exist in repo X. More accurately, REPO and X cannot be included at the same time if package PROJECT is invoked.

ruslo closed this Feb 24, 2016

mdavezac mentioned this pull request Apr 11, 2016

External repos -- trial 2 #384

Closed

Knitschi mentioned this pull request Apr 18, 2018

Implement a mechanism that allows switching between precompiled and inlined packages. Knitschi/CPFCMake#15

Open

NeroBurner pushed a commit to NeroBurner/hunter that referenced this pull request Dec 10, 2018

Merge pull request ruslo#295 from NeroBurner/pkg.ceres-solver

f8986fe

Pkg.ceres solver

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

External repos #295

External repos #295

mdavezac commented Jan 7, 2016

headupinclouds commented Jan 7, 2016

ruslo commented Jan 8, 2016

mdavezac commented Jan 8, 2016

ruslo commented Jan 9, 2016

mdavezac commented Jan 11, 2016

ruslo commented Jan 11, 2016

mdavezac commented Jan 11, 2016

ruslo commented Feb 24, 2016

External repos #295

External repos #295

Conversation

mdavezac commented Jan 7, 2016

headupinclouds commented Jan 7, 2016

ruslo commented Jan 8, 2016

mdavezac commented Jan 8, 2016

ruslo commented Jan 9, 2016

mdavezac commented Jan 11, 2016

ruslo commented Jan 11, 2016

mdavezac commented Jan 11, 2016

ruslo commented Feb 24, 2016