boost consolidation/harmonization #137

h-vetinari · 2023-04-07T03:12:36Z

There's a bunch of discussions that touch upon changing our distribution of boost in some way. All of these come with quite some effort, but I believe we could solve them with essentially one PR plus a special migrator.

Open discussions I'm aware of:

Consolidate this feedstock with https://github.com/conda-forge/boost-feedstock (1, 2)
Consolidate naming of outputs with Anaconda (also in the context of this¹)
Solve the run-export situation; boost has a special role because boost can be used both as header-only (i.e. only in host), or compiled against (needs run-export)

Rough idea:

Unify all in this feedstock
Rename current boost-cpp --> libboost, and add a run-export
Add header-only output to this feedstock (without the export) -- This is now called libboost-headers discussion & a (basically unanimous) vote
Add boost from https://github.com/conda-forge/boost-feedstock (which already depends on boost-cpp)
- ~~Rename it to py-boost to match with Anaconda?~~ Rename it to libboost-python, at least in conda-forge - I also think this name would be much clearer to communicate that it's for boost's python bindings, rather than boost itself.
Add piggy-back to the next boost migration (finally found out how this should be done)
- If boost-cpp only in host, turn into libboost-headers. If also in run, remove it there but use libboost in host.
- Shepherd migration
Profit

In short:

	conda-forge: currently	Anaconda: currently	conda-forge (& Anaconda?): proposal
lib	`boost-cpp`	`libboost`	`libboost`
header-only lib	-	-	`libboost-headers`
python bindings	`boost`	`py-boost`	`libboost-python`

Am I overlooking something? Any thoughts/comments?
@conda-forge/boost-cpp @conda-forge/boost @conda-forge/core

where the following was said: "-cpp was a trend that I started with boost-cpp. That was a mistake. I'm in favour of changing it [...]" ↩

The text was updated successfully, but these errors were encountered:

beckermr · 2023-04-07T03:24:35Z

Doesn't the migrator need to do something with py-boost too? Maybe be replace boost?

h-vetinari · 2023-04-07T03:39:35Z

Doesn't the migrator need to do something with py-boost too? Maybe be replace boost?

I presume it could, but we could also solve the py-boost renaming separately.

sdebionne · 2023-04-19T09:06:32Z

I would prefer libboost-python over py-boost since it would follow the naming convention used for boost libraries built separately such as libboost-mpi. And Boost.Python is not a binding of boost but a library to write python bindings, I believe py-boost would be confusing.

h-vetinari · 2023-04-19T18:20:36Z

Recording what was discussed in the core-call (please correct me if anyone feels differently):

in general this is a good idea that we want to do
let's also rename py-boost to libboost-python

beckermr · 2023-04-19T18:22:01Z

I think libboost-dev for the headers would be most linux like?

h-vetinari · 2023-04-19T18:32:37Z

I think libboost-dev for the headers would be most linux like?

Hm. Usually -dev gets you the headers plus the binaries and everything else, no¹? And we're IMO doing something different, which is "just the headers"...

e.g. ubuntu's boost, openblas ↩

beckermr · 2023-04-19T18:34:23Z

Ahhhhhhh. You are right. Ignore me!

h-vetinari · 2023-04-19T18:47:37Z

No worries. It's good to explore the space a bit. Personally I like to have some variation of "-headers" in the name, so that it becomes very clear in the meta.yaml that the intent is to only use the headers.

Run-exports are a topic that's really hard to penetrate for newcomers, but at least I think it's more likely they'll understand that something that's only headers is only needed at compilation time.

If we don't mind longer names, we could even do libboost-header-only (perhaps people will have heard of "header only" libraries -- boost uses this formulation in their docs as well)

h-vetinari · 2023-04-19T21:42:29Z

I updated the summary in the OP to reflect libboost-python (as discussed) & libboost-header-only (which is now my preferred suggestion).

loriab · 2023-04-20T01:25:32Z

fwiw, Boost::headers is the CMake target for their header-only libraries (https://github.com/boostorg/boost_install/blob/develop/BoostConfig.cmake#L17-L18).

chenghlee · 2023-04-20T02:04:34Z

Hm. Usually -dev gets you the headers plus the binaries and everything else, no1? And we're IMO doing something different, which is "just the headers"...

FWIW, Debian and Fedora both seem to stuff the "headers only" components in their corresponding -dev/-devel packages.

However, I don't think that pattern works for conda packaging because that would make the run_exports a bit wonky. Maybe we should have a libboost-dev package containing the development binaries, which depends on libboost-headers and libboost (the runtime components and also the run export)?

I'm not 100% sold on the headers-only naming convention, mostly because some of those headers will actually need binary DSOs to be useful at build and runtime.

h-vetinari · 2023-04-20T03:11:11Z

I actually thought about reusing -dev in the way you described, though why I didn't end up proposing it is that it would be completely counter to how we're currently using library patterns everywhere (i.e. libfoo always comes with headers). I'm not sure we want to go that way TBH, even though we'd maybe design it like that on a green field.

I'm not 100% sold on the headers-only naming convention, mostly because some of those headers will actually need binary DSOs to be useful at build and runtime.

So to my mind, libboost would host- & run-depend on libboost-header-only, meaning that we always get the required headers when someone uses libboost. In turn, if you need a binary at runtime, libboost-header-only is the wrong host-dependence for that package (by design, because then you need the run-export)!

beckermr · 2023-04-20T13:05:57Z

I'd prefer to use the boost feedstock for consolidation and archive this one. That will enhance discoverability in the future.

sdebionne · 2023-04-21T07:25:32Z

The boost feedstock would then be headers only while all the libs that need to be built will have their own packages? IOW, the libboost-xxx would package only the libraries, not the headers and depends on boost that would provide all the headers (only?). I see advantages in having separate package for the libraries, for instance they can have their own variants (libboost-mpi is build for several mpi variants, libboost-iostream could be built with several compression variants). The build matrix would be unmanageable in a single package.

I wonder if a project that needs the latest libboost-xxx but also requires a python package that depends on a Boost.Python version that is pinned globally will be possible? Ideally, the Boost.Python lib would be statically linked in python extension packages to remove the dependency and solve this issue...

h-vetinari · 2023-04-21T07:44:46Z

All the 3 packages mentioned in the OP can be built in a single feedstock. technically speaking it doesn't matter if it's this one or the other, but "boost" is a better feedstock name than "boost-cpp" (especially if there's no more package of that name; though we have this situation elsewhere as well...)

I wonder if a project that needs the latest libboost-xxx but also requires a python package that depends on a Boost.Python version that is pinned globally will be possible?

In general everything that depends on the binaries would need a corresponding migration to move to a new (resp. the newest) version, but any feedstock can use conda_build_config.yaml to opt into newer versions.

Ideally, the Boost.Python lib would be statically linked in python extension packages to remove the dependency and solve this issue...

In conda-forge, we generally prefer shared libs (and migrating where necessary), so I doubt this is ideal here.

h-vetinari · 2023-04-28T05:04:08Z

Over in the land of opentelemetry, the header-only install has an -api suffix by upstream convention. I don't think this is too bad, but I do prefer the explicitness of -header-only. What I'd like to avoid is yet again having divergent names for the same thing (header only versions of a library) within conda-forge.

I haven't heard much more responses here. Should we do an informal poll (multi-vote possible)?

🎉 libboost-header-only / libopentelemetry-cpp-header-only / ...
❤️ libboost-headers / libopentelemetry-cpp-headers / ...
🚀 libboost-api / libopentelemetry-cpp-api / ...

Perhaps -headers is the best trade-off?

h-vetinari · 2023-05-03T11:30:31Z

Implementation PR is ready for a first round of reviews: conda-forge/boost-feedstock#164

Once we're converging towards agreement there, I'll try writing the piggyback migrator.

sdebionne · 2023-05-09T07:01:50Z

Something is still not clear to me regarding runtime vs devel dependencies. For instance, with the current implementation,

$mamba create -n test pytango boost-cpp=1.82
Looking for: ['pytango', 'boost-cpp=1.82']

Could not solve for environment specs
The following packages are incompatible
├─ boost-cpp 1.82**  is requested and can be installed;
└─ pytango   is uninstallable because there are no viable options
   ├─ pytango [9.3.3|9.3.4|9.3.5|9.3.6] would require
   │  └─ boost >=1.74.0,<1.74.1.0a0 , which requires
   │     └─ boost-cpp 1.74.0.* , which conflicts with any installable versions previously reported;
   └─ pytango [9.3.5|9.3.6|9.4.0|9.4.1] would require
      └─ boost >=1.78.0,<1.78.1.0a0 , which requires
         └─ boost-cpp 1.78.0.* , which conflicts with any installable versions previously reported.

why boost depends on boost-cpp since it does not depends on any other Boost (built) libraries?

At runtime, a python extension only requires the libboost_python* libraries that match the version it was built with -and the python version. In the other hand, building a python extension requires libboost_python* and the matching headers. Is there any chance to capture this?

With the new implementation, I wish installing a python extension that depends on libboost-python global pinning (1.78) and a more recent version of the libboost-headers would be possible.

h-vetinari · 2023-05-09T07:27:43Z

why boost depends on boost-cpp since it does not depends on any other Boost (built) libraries?

Because previously there was no way to only depend on the headers (or anything but the full shebang).

[...] Is there any chance to capture this?

conda-forge/boost-feedstock#164 does this - thanks for confirming that the link check did its job correctly (causing me to switch libboost-python to only depend on libboost-headers rather than the full libboost).

With the new implementation, I wish installing a python extension that depends on libboost-python global pinning (1.78) and a more recent version of the libboost-headers would be possible.

This would be easily possible as follows, but isn't yet implemented in conda-forge/boost-feedstock#164, mainly because a) I didn't think about it and b) I'm not sure (yet) that there's definitely no scenario where this could break.

--- a/recipe/meta.yaml
+++ b/recipe/meta.yaml
@@ -186,11 +186,11 @@ outputs:
       host:
         - python
         - numpy
-        - {{ pin_subpackage("libboost-headers", exact=True) }}
+        - libboost-headers >={{ version }}
       run:
         - python
         - {{ pin_compatible('numpy') }}
-        - {{ pin_subpackage("libboost-headers", exact=True) }}
+        - libboost-headers >={{ version }}
       run_constrained:
         # make sure we don't co-install with old version of old package name
         - boost ={{ version }}

beckermr · 2023-05-09T11:48:39Z

I'd prefer exact pins for now with the idea that we can relax them later if it is working.

h-vetinari · 2023-09-27T00:16:06Z

Closed by conda-forge/boost-feedstock#164

h-vetinari mentioned this issue Apr 7, 2023

Add run_exports to boost-cpp #82

Closed

5 tasks

h-vetinari mentioned this issue Apr 28, 2023

Build shared libs, add run-export, register in global pinning conda-forge/cpp-opentelemetry-sdk-feedstock#38

Closed

h-vetinari mentioned this issue May 3, 2023

Boost unification conda-forge/boost-feedstock#164

Merged

h-vetinari closed this as completed Sep 27, 2023

h-vetinari mentioned this issue Oct 18, 2023

Is there a way to install a boost header only package? #152

Closed

h-vetinari mentioned this issue Nov 20, 2023

Rebuild for libboost 1.82 conda-forge/mgis-feedstock#4

Merged

thomashampson mentioned this issue Feb 29, 2024

Update boost to 1.82 mantidproject/mantid#36687

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

boost consolidation/harmonization #137

boost consolidation/harmonization #137

h-vetinari commented Apr 7, 2023 •

edited

Loading

beckermr commented Apr 7, 2023

h-vetinari commented Apr 7, 2023

sdebionne commented Apr 19, 2023

h-vetinari commented Apr 19, 2023 •

edited

Loading

beckermr commented Apr 19, 2023

h-vetinari commented Apr 19, 2023

beckermr commented Apr 19, 2023

h-vetinari commented Apr 19, 2023 •

edited

Loading

h-vetinari commented Apr 19, 2023

loriab commented Apr 20, 2023

chenghlee commented Apr 20, 2023

h-vetinari commented Apr 20, 2023

beckermr commented Apr 20, 2023

sdebionne commented Apr 21, 2023

h-vetinari commented Apr 21, 2023

h-vetinari commented Apr 28, 2023

h-vetinari commented May 3, 2023

sdebionne commented May 9, 2023 •

edited

Loading

h-vetinari commented May 9, 2023

beckermr commented May 9, 2023

h-vetinari commented Sep 27, 2023

boost consolidation/harmonization #137

boost consolidation/harmonization #137

Comments

h-vetinari commented Apr 7, 2023 • edited Loading

Footnotes

beckermr commented Apr 7, 2023

h-vetinari commented Apr 7, 2023

sdebionne commented Apr 19, 2023

h-vetinari commented Apr 19, 2023 • edited Loading

beckermr commented Apr 19, 2023

h-vetinari commented Apr 19, 2023

Footnotes

beckermr commented Apr 19, 2023

h-vetinari commented Apr 19, 2023 • edited Loading

h-vetinari commented Apr 19, 2023

loriab commented Apr 20, 2023

chenghlee commented Apr 20, 2023

h-vetinari commented Apr 20, 2023

beckermr commented Apr 20, 2023

sdebionne commented Apr 21, 2023

h-vetinari commented Apr 21, 2023

h-vetinari commented Apr 28, 2023

h-vetinari commented May 3, 2023

sdebionne commented May 9, 2023 • edited Loading

h-vetinari commented May 9, 2023

beckermr commented May 9, 2023

h-vetinari commented Sep 27, 2023

h-vetinari commented Apr 7, 2023 •

edited

Loading

h-vetinari commented Apr 19, 2023 •

edited

Loading

h-vetinari commented Apr 19, 2023 •

edited

Loading

sdebionne commented May 9, 2023 •

edited

Loading