Recipes for packages with submodules

[djsutherland]

Some projects, for example osqp and shogun, don't have release tarballs that contain all of the source needed to build: they use github as the release mechanism and use submodules, but github release tarballs just leave submodules as empty folders.

One option to handle this is to use a git source, but that's frowned upon.

You can instead deal with this in conda-build 3 with multiple sources without too much difficulty with multiple sources in cb3; for example. But:

• The submodule version needs to be updated appropriately with the new release version. The bot of course doesn't know how to do this, so you need to (a) remember to do it, (b) go look up what version is actually needed, and (c) get the checksum yourself. If you forget to do this, depending on how the submodule is used the build may or may not fail.

• If submodules are added in a new release, you need to go figure that out (osqp v0.4.0 osqp-feedstock#2). Same for if they're removed or changed.

Some things that might make this nicer:

• If github would just give us a tarball that included submodules. Not much we can do about that. :/
• If the bot figured out how to update submodules. This would probably only be feasible for github sources, and it would probably need some kind of annotation in the recipe.
• If we decided it was okay to use git sources in cases like this. We could maybe use specific commit numbers instead of version tags to be more like using a checksum. How much of a problem is the Windows issue mentioned here? (Also, I think conda-build now does shallow clones by default, so the first point is now less relevant?)

Thoughts?

[scopatz]

I think it is probably OK to use git sourses in cases like these.

[ChrisBarker-NOAA]

I think it is probably OK to use git sources in cases like these. I agree - of the authors are using git to release, then that's what we

should follow. One reason git is frowned upon is that we want t use the source as released by the authors -- which ideally is a tarball of some sort (if even a tarball of a git tag)

…

-CHB

-- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

[scopatz]

I am renegging on my previous opinion. I think this is probably still not OK.

[carterbox]

Dependencies should be built separately and linked dynamically instead of revended and linked statically.

If the project is incompatible with a stable release of the dependency project (the one revended in a submodule), then you can list multiple URL sources and unpack them into the correct folder. You will have to manually update the submodules with each new release.

source:
  - url: https://github.com/foo/foo/releases/download/{{ version }}/foo-{{ version }}.zip
    sha256: 3485471defaec4f458a48db45523c87eff13c2a7eb6dec140f606faf13f8fe40
# List the submodules as separate sources only if foo isn't compatible with stable release of bar
  - url: https://github.com/bar/bar/archive/7cc409eeef9de1e7ac3ec01c8b76f640deec36bd.tar.gz
    sha256: 6dc30c9db1fb55e1009f76cafde7a5e0b9de0f14933336437748dfa25d8b3978
    folder: third-party/bar
  - url: https://github.com/spec/spec/archive/7cc409eeef9de1e7ac3ec01c8b76f640deec36bd.tar.gz
    sha256: 6dc30c9db1fb55e1009f76cafde7a5e0b9de0f14933336437748dfa25d8b3978
    folder: third-party/spec