Dependency graph will always be computed completely, including build-requires

[memsharded]

Whenever the Conan dependency graph is evaluated and computed, all the transitive dependencies will be computed, including build_requires and their transitive dependencies. This includes many commands such as conan create, conan install, conan info, and conan lock among others.

One of the main advantages is that the build_requires will be able to affect the consumers package_ids in a controlled way (with different modes, selectable by users, as with current requires package_id_modes), and thus be part of Continuous Integration flows, forcing re-build of consumers when desired. On the other hand it will require the build_require recipes to be available to construct the dependency graph, and might produce some (likely very small, like 1-2% on average) more time to build the graph, and even much less if we factor the binaries, as by default the binaries will not be necessary to be downloaded.

Please note, as explained in the proposal, that it is not possible to have both. If we want the build_requires to be taken into account for the package_id computation, they must be always in the graph. And if we want build_requires not to be in the graph if not necessary, then, they cannot be part of the package_id.

---

• Upvote 👍 or downvote 👎 to show acceptance or not to the proposal (other reactions will be ignored)
  • Please, use 👀 to acknowledge you've read it, but it doesn't affect your workflow
• Comment and reviews to suggest changes to all (or part) of the proposal.

[DoDoENT]

I'm OK with that as long as there is still a possibility for some packages that are build-only dependency to not affect the package_id of the consumer package.

For example, package android-ndk/r22 should affect the package ID, as it is a toolchain (it will produce different binaries that android-ndk/r21e). However, package cmake/3.19 should not affect the package ID, as it will produce the same binaries as cmake/3.18 (actually this is true only for most cmake/package combinations, but it's possible that in some cases cmake/3.18 and cmake/3.19 produce different binaries (e.g. when using xcode generator)).

Also, I would argue that packages like the mentioned android-ndk/r22 should also be able to affect settings on the graph, as I proposed in this issue, however, this is also not important here.

[memsharded]

I'm OK with that as long as there is still a possibility for some packages that are build-only dependency to not affect the package_id of the consumer package.

Yes, what we had in mind is a similar model to the existing one for requires and python_requires, lets say something like:

[build_requires]
mytool/0.1

[conf]
core.package_id:build_requires_package_id_mode=minor_mode

So changes in the minor version of mytool would produce a new package_id of its consumers.

I hadn't thought about this case, but I guess defining it based on the package name, could make sense. In that case we could support something like:

[build_requires]
cmake/3.14
android-ndk/1.2.3

[conf]
core.package_id:build_requires_package_id_mode={"cmake": "major_mode", "android-ndk": "minor_mode"}

Just a rough UI idea.

Note: for in-recipe definition in the package_id(), that would be direct definitions, like self.info.build_requires["cmake"].package_id_mode = "major_mode" or self.info.build_requires["android-ndk"].package_id_mode = "recipe_revision_mode", so I have only exposed above the not so direct [conf] definition

[datalogics-kam]

Being able to specify the modes on a per-build-requirement basis resolved all of my concerns. Being able to have that level of control helps serve multiple needs.

[Daniel-Roberts-Bose]

Overall I think this proposal makes sense and is a good idea. However I have one or two questions.

1. If build_requires are basically going to be treated the same as requires, why even make them two different things? Perhaps I missed something in the proposal but what is the intended use case now for one over the other?

2. Perhaps we are currently using build_requires incorrectly but the exact reason we use them right now is to not do what this proposal is suggesting (ie. We use build_requires when we don't want it to affect the package_id). Would the correct way of having a package not affect package_ids of other packages be to use the package_id function in the conanfile instead?

[DoDoENT]

@Daniel-Roberts-Bose , I think the only difference between requires and build_requires now is that you will still be able to add build_requires to your profiles and thus inject it as a dependency of every package in the graph... Also, build_requires is affected by the build/host context and requires isn't.

Or did I also miss something?

[a4z]

everything that provides a solution for conan-io/conan#7876 is welcome !

[puetzk]

If you only mean that it must pick a recipe (and not that a package_id), that could maybe be OK. Or maybe that there's a build_requires(private=True) or similar to designate those that cannot influence package_id and are not part of the consumer's tree.

But we have a number of libraries that have complicated cross-compile setups (e.g. build_requires are almost wholly disjoint from their own package_id). Most are COM libraries, that may not even be written in the same language as their consumer, and potentially aren't even built on the same OS. The profile:build in effect for to build the consumer of this library may not be even remotely related to the profile:build needed to build the library, and such recipes are going to fail if you actually try to resolve them the consumer won't have have any of the needed settings.

I feel like I don't understand the motivation here, though, so maybe I'm missing something. Given the android-ndk API level be a part of the package_id via settings.compiler or settings.os? We don't capture the exact compiler binary for anything else...

Or looking at it more philosphically, what is a build_requires, except something that you only need to think about to build, not to consume? We already have self.requires(...,private=True) if you need something to be in the consumer's tree (and thus can affect package_id), but not be automatically linked into the consumer. And we already have self.build_requires(...,force_host_context=True) for build_requires that nonetheless use host profile. Do we just need something like self.requires(...,private=True,force_build_context=True) for something that the consumer needs to resolve, but not use?

---

Of course, given all these other knobs exist anyway, maybe it would make sense to just have a single kind of self.requires(...,context=["build","host"], visibility=["build","private","public"]). Where "build" really means truly private (what [build_requires] means today, don't even resolve it when consuming, can't impact package_id), "private" means what private=True means today (recipe has to be downloaded, and settings resolved, but that binary identified by that package_id won't actually be downloaded or seen by generators), and "public" means what private=False means today (consumer transitively gets this requirement in their deps_cpp_info). And I suppose one could introduce "interface" as well, ala CMake, for something that is transitive to the consumer but not used as part of this package, though I can't immediately think of a use for that (maybe if the profile-as-package thing came along, it would make sense the same way profiles adding [build_requires] did.

[memsharded]

@Daniel-Roberts-Bose

If build_requires are basically going to be treated the same as requires, why even make them two different things? Perhaps I missed something in the proposal but what is the intended use case now for one over the other?

They are not going to be identical:

• By default, they will not be automatically added to the package_id. It will be possible, and users wanting to do that, will be able to do it, but as a starting point we are proposing to be the default "unrelated_mode" which means, no effect in the package_id of the consumer
• They can still optimize the download and unzip of the binary, if not strictly necessary which is the heavy cost that could have some impact. Unless they are really necessary to build, of course.
• They also have some differences in what information they propagate to their consumers. For example, they will not propagate the cpp_info information, because they live in the build context, which can be different to the host one in cross-building. build_requires will be for reusing tools like CMake, as a conan package, not for libraries. For libraries, the current build-require in host context, that is necessary for test frameworks, will propagate the cpp_info.

Perhaps we are currently using build_requires incorrectly but the exact reason we use them right now is to not do what this proposal is suggesting (ie. We use build_requires when we don't want it to affect the package_id). Would the correct way of having a package not affect package_ids of other packages be to use the package_id function in the conanfile instead?

When you don't want mydep to affect the package_id of a consumer, you can do in the consumer conanfile.py:

def package_id(self):
    self.info.requires["mydep"].unrelated_mode()

Is this what you mean?

[memsharded]

what @puetzk is commenting here like self.requires(...,context=["build","host"], visibility=["build","private","public"]) is probably what the full new dependency model will look like. In this case we are assimilating the current build_requires to something in the "build" context, with "private" visibility, and also non overridable by downstream dependency.

We probably don't need to get into this level of detail now, we think it is possible to agree on the fact that having incomplete graphs based on the type of requirements is confusing, and having a full dependency graph containing all different types of requirements could help to reduce complexity and be more deterministic and understandable by users, irrespective of the different details of different types of requirements.

[kenfred]

@memsharded I have a question about this one. I can think of three ways to provide a custom toolchain to build:

1. A build_requires from within the conanfile (or conditionally from the build_requirements method)
2. A build_requires from a profile
3. Setting environment variables within a profile

I think you're saying that this proposal makes case 1 affect the package_id. But doing it from a profile (cases 2 and 3) would not affect the package_id?

This concerns me because the choice between these 3 cases is often stylistic/subjective or situation-dependent. There are been times within our organization that we've chosen one path or another. I would expect that whether I specify a build_requires in a profile or from the recipe, the result would be the same, including the package_id.

[puetzk]

We demand rigidly defined areas of doubt and uncertainty!.

I guess I just still reject this premise - there are a lot of package with carefully limited API/ABI that I want to build, in a very specific way, and then consume, in a very-much-less-specific way. The implementation conceals a great deal of complexity that I do not want consumer recipes to even have to think about, because it doesn't affect them one bit if it changes, and they wouldn't reasonably be expected to even know vocabulary to describe it. They want the x86-windows runtime, they don't know or care that its built with msvc16.8 (or that it's built with mingw-gcc, C#, or that it's using liblzma internally, etc). There may be some transitive dependencies that come along with that, but not nearly as many as things that were statically linked inside, and for the latter there are no exported symbols that are affected and the client is free to not care, or even to use a different version of the same library for their own (unrelated) purposes.

Of course, this is marginally supported even today (only [build_requires] can be disregarded by the consumer, though private=True is sometimes enough if it's simple enough to let the consumer resolve it, even though they may pick something other than what was actually used, and it it will just get thrown out). So, since conan-io/conan#4753 broke our previous workaround, more than a few of our recipes just have two entirely separate conanfiles: one to obtain dependencies for building the package in the local flow, and one used to export the resulting binaries, that fails to mention a lot of what was in the first one. That works, but I don't like it much, and I'd hate to see it become the near-universal pattern as soon as you have an irrelevant build_requires.

Now, I would love to capture all of this in some full_package_id at conan create or export-pkg, and then address the consumer's less-specific package_id separately. The current compatible_packages is kind of an attempt at this, but it's the wrong way around and mostly not very usable. One issue is simply that anything using compatible_packages will not work offline (the "preferred" binary likely doesn't exist, and negative results are not cached, so it will always want to go look on artifactory just in case before accepting the same fallback it accpted the previous N-1 times). The other, harder one, is that it still expects the consumer's dependency graph to be able to compute a specific package_id they would have preferred but not might not be uploaded, before it will consider falling back on some other binary that's actually available. That's just fundemantally underconstrained for some of our use-cases. A non-C++ consumer of (C-only, or VB, or matlab, or whatever) client has no idea what version of msvc it wants, or which liblzma was used, or any of a bunch of other questions. They want whatever binary we actually built, tested, and published to artifactory for the settings (generally just os/arch) that affect the consumer's ABI.

I have some thoughts on how that might work better but this probably isn't the place for detailed speculation. I'm thinking some kind of a binary alias to complement the existing support for recipe aliases, that was an actual artifact to upload, download and cache, under its package_id, which could have a conaninfo.txt that provided the missing profile information needed to resolve the newly-revealed transitive dependencies.

[memsharded]

@kenfred

I think you're saying that this proposal makes case 1 affect the package_id. But doing it from a profile (cases 2 and 3) would not affect the package_id?

No, it will affect both cases, defining build_requires in recipes and in the profiles. The only thing that is not covered is environment variables, but that is not connected to this proposal. Environment variables are not connected to the build_requires, and they do not currently affect the package_id. If we want env-vars to affect the package-id is a different story, and probably this would find a much better solution in the new [conf] configuration, which can inject custom behavior into recipes, and is specifically designed to be able to affect the package_id if desired, in a more controllable way than enviroment variables. But in any case I think these belong to a different discussion.

[memsharded]

@puetzk

I guess I just still reject this premise - there are a lot of package with carefully limited API/ABI that I want to build, in a very specific way, and then consume, in a very-much-less-specific way. The implementation conceals a great deal of complexity that I do not want consumer recipes to even have to think about, because it doesn't affect them one bit if it changes, and they wouldn't reasonably be expected to even know vocabulary to describe it. They want the x86-windows runtime, they don't know or care that its built with msvc16.8 (or that it's built with mingw-gcc, C#, or that it's using liblzma internally, etc). There may be some transitive dependencies that come along with that, but not nearly as many as things that were statically linked inside, and for the latter there are no exported symbols that are affected and the client is free to not care, or even to use a different version of the same library for their own (unrelated) purposes.

I don't think the current proposal is opposed at all to having a better package_id model, actually it is intended as a step towards having better and more powerful ways to compute and manage the package_id. The fact that you are fetching the recipes of the build-requires is only enabling more functionality, making available some possibilities to Conan users that were previously impossible. It doesn't impose a new package_id model that is worse in any aspect to the current 1.X proposal. The package_id model is something that will certainly require work in Conan 2.0, and all those other traits (visibility, private, propagation, etc) will be taken into account for better ways to model the package_id, and try to represent better the use cases you are exposing above.

I totally understand there could be in the past some problems like conan-io/conan#4753, that deserves some attention into the new dependencies model for Conan 2.0. Maybe if we can come up with a good modeling for that situation, this proposal wouldn't necessarily be a problem for your use case, I'll follow up on that issue, try to understand it better and gather feedback.

[puetzk]

If you only intend to fetch the recipes listed in build_requires, but not apply settings/options to them, or use their configure()/package_id() as requirements do, then I guess I can can retract my downvote. The recipes are readily identifiable without getting into the messy question that scared me (that transitive build_requires may well belong to neither the current package's host profile, nor the consumer's build profile). So if they were to affect package_id, they could only do so as far as full_recipe_mode, not full_package_mode (or more).

[ytimenkov]

As we identified earlier the recipe resolution is a bottleneck :( (when there is no cache, as in some CI configurations).

[memsharded]

Recipe resolution of a large dependency graph can take some time. But in the context of the build_requires, it is not like projects are having hundreds of different build requirements, but just a very few of them (cmake and similar tools), while we are aware of many users having dependency graphs with hundreds of dependencies. So fetching those build-requires should be relatively small compared with the resolution of the whole graph. Do you have tons of different build_requires in the same graph?

On the other hand, it is possible that a simplified dependency resolution could help in some tasks like parallelizing some tasks in it, and maybe speeding it up too.

[ytimenkov]

When you don't want mydep to affect the package_id of a consumer, you can do in the consumer conanfile.py:

def package_id(self):
self.info.requires["mydep"].unrelated_mode()

The problem is transitive dependencies. Even if I remove (say) CMake from the hash it will leave its requirements (like OpenSSL and bzip).

The second issue is that build requirements are affected by profiles which makes it hard to uncontrol. If on some particular machine you need some extra package to build (for example sysroot for other linux distro or so) which is in some local profile then all packages will need to be rebuilt (not nice when they are expected to be built by CI).

I think proposal is fine as long as it's off by default and needs to be opted in in some convenient way.

BTW, examples of Android NDK sound more like a wrong set of settings... This way all ABI settings like compiler etc can be removed by just depending on a compiler package...

[ytimenkov]

but the binary for the cross-compiler “mygcc-arm/1.2” will not exist for that platform,

It's not that binary doesn't exist. It's that the build requirement is completely missing (didn't come with profile). So I assume hash will be different. Or do I miss something?

[memsharded]

No, the build_requirement declaration is technically there. For most situations it is more than enough to know the version of it, in that example, lets say that you know that you want to rebuild binaries if your compiler bumps the major, to something like mygcc-arm/2.0. Then you declare a major_mode on that build-requirement.

If you are installing in Linux, you will not have a binary that can run in Linux, but you will still know the version, even down to the recipe revision of mygcc-arm/1.2 and you will know that you don't need new binaries, and don't need to fetch the whole mygcc-arm/1.2 package binaries because you don't need to build. But you know that the version is 1.2, and everything fits, with just the declaration of that package without needing its full binaries.

Please let me know if this clarifies it.

[ytimenkov]

Not really.
When I build package with 2 profiles I have a bunch of build requirements coming from it.
When I consume a package I don't specify any build profile therefore I have no build requirements => different graph => different package id.

[memsharded]

That is up to you. If you want that behavior, then you can have it, as by default build_requirements will not be automatically added to the package_id. Basically do nothing, and you will not have a different package_id when your build_requires are added by a profile or when they are not. If at some point you want to control it and decide that changing some of your build_requires and upgrading it to a newer version should automatically guarantee that packages using that build requires must be rebuilt, then you can add it to the package_id policy. From that time, yes, you should declare your build_requires in your profiles even if you don't intend to build, because that declaration is necessary to define which binaries to install.

[memsharded]

@ytimenkov

The problem is transitive dependencies. Even if I remove (say) CMake from the hash it will leave its requirements (like OpenSSL and bzip).

Yes, I also think it is necessary to have ways to model this package_id use case, and have possibility to make that "transitive" too. We will take that into account for the new package_id computation.

The second issue is that build requirements are affected by profiles which makes it hard to uncontrol. If on some particular machine you need some extra package to build (for example sysroot for other linux distro or so) which is in some local profile then all packages will need to be rebuilt (not nice when they are expected to be built by CI).

Not necessarily. Users will be able to define the logic they want, so if they don't want to rebuild packages, they will not have to do it. And they will be able to define under which conditions their packages need to be rebuilt when build-requires change.

I think proposal is fine as long as it's off by default and needs to be opted in in some convenient way.

Yes, the proposed default is off by default, and users will be able to opt-in if they want, with the package_id_mode that they want.

[kenfred]

No, it will affect both cases, defining build_requires in recipes and in the profiles.

Ok great. I didn't realize that seeing build_requires from a profile would factor into the package_id.

The only thing that is not covered is environment variables, but that is not connected to this proposal. Environment variables are not connected to the build_requires, and they do not currently affect the package_id.

Since a build_requires could set env, and a profile can do the same. It sounds like there is an inconsistency gap here if the former affects the package_id and the latter does not.

If we want env-vars to affect the package-id is a different story, and probably this would find a much better solution in the new [conf] configuration, which can inject custom behavior into recipes, and is specifically designed to be able to affect the package_id if desired, in a more controllable way than enviroment variables.

I'm not familiar with this. Sounds like a new kind of profile, whose change to env vars would affect the package_id. That sounds like it would address the inconsistency. What will become of profiles? In the end, I don't think any of these mechanisms should diverge from one another when it comes to how build requirements are handled. See below.

But in any case I think these belong to a different discussion.

Perhaps you're right, but It all seems connected in my mind. We need a model of how build requirements affects the "identity" of the resulting package (i.e. package_id). All the ways to specify build requirements (build_requires, build_requirements, profiles, confs, etc.] should simply be controllers of that model. I approve of making build info part of the dependency tree as a concept (which is the purpose of this proposal). But we need to be consistent in all of the different mechanisms used to inject build info.

[memsharded]

@kenfred

Since a build_requires could set env, and a profile can do the same. It sounds like there is an inconsistency gap here if the former affects the package_id and the latter does not.

It is a convention based on the version of the build_requires. A build_require could change the env-vars it defines, but if it doesn't bump its version, it will not create a new package_id in its users. Same as with normal requires, they can indeed change, but as long as their version follows the defined package_id_mode, that is what is used to compute package_id and decide if something needs to be built or not. The build_require could be even completely empty, and act as a pure proxy of some tool installed in the system, but the version that it declares can still be very useful to control when to rebuild things and manage custom ABI compatibility.

I'm not familiar with this. Sounds like a new kind of profile, whose change to env vars would affect the package_id. That sounds like it would address the inconsistency. What will become of profiles? In the end, I don't think any of these mechanisms should diverge from one another when it comes to how build requirements are handled. See below.

Yes, at the moment the current model is package_id = function(settings, options, dependencies-references). This current proposal is all about allowing the possibility of adding build_requires to the dependencies-references, that at the moment only contain regular requires, but the way the package_id will be affected is not novel, there is already the experience with normal requires and even the new python_requires.

It is kind of clear that adding env-vars to the package_id would be quite challenging for many reasons: env-vars are not structured, nor validated. It is challenging to distinguish Conan defined ones from system ones, and it is difficult to define models or semantic meaning for a predictable package_id. They are also not nice in the sense that they are a global, system wide variable. Any package can potentially change those values dramatically affecting other packages, unexpectedly.

It is possible that [conf] could fill this gap, being a structured, controlled, non-global and semantically meaningful way to pass information to Conan and Conan packages, and as such, it could be possibly added to package_id function. It is still a bit early, but so far is looking good.

[kenfred]

It is kind of clear that adding env-vars to the package_id would be quite challenging for many reasons: env-vars are not structured, nor validated. It is challenging to distinguish Conan defined ones from system ones, and it is difficult to define models or semantic meaning for a predictable package_id

Yeah you're right. I think what you'd have to do is consider a profile an injectable build_requires in and of itself. It would be like a package, that you dynamically add to build_requires from the cmd line. It would be represented in the dependency graph. Its existence is what impacts the package_id, not the env vars it defines nor its defined build requirements. If you could handle a profile like a package, then it would really be a stylistic choice if I insert it a run time or make it a build_requires of the consumer, and it's completely consistent in behavior in regard to package_id.

You're also right that the package_id can't factor in every thing about the whole system. We expect people to have their own installation of VS or CMake or whatever. We can and do conanize some of these tools, but at some point, we assume compatibility. I think what this tells us is, we shouldn't (and can't reasonably) track all build dependencies. Only the ones that impact the program behavior or ABI compatibility of a package should factor into the package_id. Unfortunately, this applies to some build requirements and not to others, so it's difficult to approve or disapprove of this proposal.

I'll read up on [conf] to understand.

[memsharded]

Only the ones that impact the program behavior or ABI compatibility of a package should factor into the package_id. Unfortunately, this applies to some build requirements and not to others, so it's difficult to approve or disapprove of this proposal.

Well, the idea is that this would be configurable, as it is already (in recipes for example) for normal requires. And users will be able to define which build_requires packages affect their package_ids and which ones do not.

[kenfred]

Sounds good. Upvoted!

[ytimenkov]

I think I got confused by package id mode: it was recommended by Conan developers (on trainings too) to use something like recipe_revision_mode by default which depends on dependency's package id, which in turn depends on OS setting (see table) which in turn makes it impossible to port in cross-compilation scenario.

However I subconsciously missed 2 things:

1. recipe_revision_mode used primarily with lockfiles. With lockfiles the graph is there already, so it doesn't really matter on which build OS the package was built.
2. It's not the only mode and in this thread more relaxed modes were proposed / assumed (like minor_mode), which indeed requires only a recipe (same table link).

Not sure if above realization helps anyone else, but it helped me 😊.

[ansutremel]

@memsharded how will this proposal impact the following problem with packages "A" (shared lib), "B" (static lib), and "C" (executable) and different compiler runtimes

• Shared library "A" consuming static library "B"
  • "A" built with a specific VS on Windows and a specific GNU gcc on Linux
  • "B" built with a specific Intel compiler on Windows and Linux
• "C" executable consuming "A" built similarly to "A"

Scenario 1: functions from static libraries not exported further

• "B" is only needed to build "A"
• "C" as consumer of "A" should not reference "B" at all but need to get the Intel compiler runtimes injected, otherwise tests won't run (missing redistributables)
  --> different flavors of "B" should not cause a problem, but different compiler runtimes should be raised as conflict at graph creation time

Scenario 2: functions from static libraries are exported

• same as above but all components used in "C" must use some flavor of "B"
  --> different flavors of "B" should raise a conflict at graph creation time of "C"

Will this proposal help here and how?

[memsharded]

@ansutremel

how will this proposal impact the following problem with packages "A" (shared lib), "B" (static lib), and "C" (executable) and different compiler runtimes

No, this proposal is not about library requirements to link other libraries or applications. This proposal is strictly about build_requires, please read the Important note in the docs: https://docs.conan.io/en/latest/devtools/build_requires.html#build-requires. Basically, it is for build tools, a package containing CMake or Gtest, but that doesn't belong to the final packaged artifacts. Despite the warnings and discouragement in the docs, in the Conan trainings, the build_requires have been historically abused for other purposes.

It is true that the relationship of static, shared and executables and static linking needs to be revisited, but that is intended to be represented by the requires relationship, which will likely get new features and improvements for Conan 2.0, but this is not covered directly by the current proposal. What it seems that will be true is that all different kind of relationships will always be modeled by a full dependency graph of recipes, and then the binaries, the linkage, the package_id, the propagation of link and run requirements would be more advance qualifiers, some of them might optimize and skip the download of the binaries under some circumstances, but in the same spirit as this proposal, the graph definition will always be complete.

Please let me know if this clarifies it.

[ansutremel]

@memsharded it's clear now, thanks.

[memsharded]

It seems the feedback and voting is compelling enough.
Lets move this forward, and implement it asap, to start learning and have a more solid base for further discussions about the graph model, and also to start having something, a Conan 2.0.0-dev.1 version with all the changes done so far that we could start testing as users and playing with it.

[KerstinKeller]

I realize, that you've merged this now, but I also wanted to add my two cents.
I have upvoted this, but I also start having second thoughts. It seems very reasonable to have the whole graph calculated always, if, but only if, all packages are available to you.

What we see now, that parts of our organization start collaborating with other companies, using conan. However, only binary releases will be distributed (no source code), and I see that this might make collaboration more complicated. It seems that organization will need to repackage their binaries with different conanfiles, if they like to hide implementation detail.

Which brings me of the point, that with conan, it's often not clear to me, which in formation is relevant for the build only, and which part is relevant for the consumer requirements, package_id, ...
In C++, we have .h and .cpp files to separate interface and build, but in other languages it's more subtle.
As for requirements, and the graph, it narrows down to the points: Are requirements private or public, and in which context are they required, build or host context.

So maybe keep this collaboration usecase in mind, and let's see where this goes 😄

[memsharded]

Hi @KerstinKeller

Yes, certainly this is one of the possible things that the new approach needs to handle somehow, and it is described in the proposal document as the "need to actually have at least the recipes" available when the graphs are resolved. It might be a possibility that for many cases it is good enough to promote just the recipes, but not the binaries, and consumers will be able to use them even if they cannot resolve those binaries, because they will be missing. Seems that it could make sense in many cases to have a complete definition of the things that are being pulled, even if not having access to binaries and couldn't reproduce the build. Of course, it this needs to be completely hidden, it could be that the re-packaging approach makes more sense, and the model could try to introduce something that makes the re-packaging a bit more explicit.

Thanks very much for the feedback, it is useful, we will keep this case in mind.