[workspaces] Workspaces should be Super Builds

[kenfred]

These comments were relevant as of version 1.18.

We were early adopters of Workspaces and we continue to rely it for our workflow. Unfortunately, the changes introduced in 1.13, have broken our workflow and we're stuck at a previous version of Conan. I have seen and commented on issues that are components of workspaces and deal with the issues I'm going to discuss (#4840, #4004, #4803, #4751, #5291, #4802, #4800) but I feel like we need an issue to discuss the design as a whole. My main focus is workspaces, but I'll share some opinions on editables as well, since they are closely related. Here is some feedback.

TLDR: Layouts, editables, and the current CMake workspace generator feel off the mark to me. I need more of a super build that retains the features we know and love from Conan.

Layouts

Layout files allow you to manually provide the paths Conan needs to generate conaninfo files. This is motivated by the fact that most packages in development have not had an install step and, therefore, the paths to libs, headers, and bins are somewhere in the build directory. Dependent packages need an alternate package_info to tell them where to find the goodies they need in the build directory.

Unfortunately, layout files are problematic for several reasons:

• Layout files can conflict with reality. Layout files allow you to specify externally what is happening internally, and the two can easily be in conflict. Layout files give the illusion that you can direct Conan to copy the output files to those locations, when actually you're simply telling Conan where you expect those files to end up in the build directory. In other words, layout files are presented as a remapping of output file locations, when actually they force Conan to look at particular location, whether the files are there or not.

• It defeats the convenience of package_info. The ability of a package to fill out its own info is a tremendous power of Conan. It allows dependent packages to automatically pick up paths and other features. This is analogous to modern CMake's use of target_link_libraries, which passes along include directories, library paths, and usage requirements. Using layout files, package_info is overridden with a manual file the consumer has to provide with some internal knowledge of the inner-workings of the package.

  Conan does provide a way to specify this layout within package_info. IMHO, this is much preferred, since it is the recipe that knows where the output files live. (See above - the user shouldn't have to assume these locations externally). The issue I have with this is the reliance on the self.in_local_cache switch. The condition should not be whether the recipe is being consumed from the local cache (in fact a recipe shouldn't know about the cache at all). The condition should be if the consumer is consuming from the build directory or the package directory. There is a direct CMake analog for this as well: package layout config files. Look at the following CMakeLists.txt:

  add_library(ClimbingStats climbingstats.cpp)
  target_include_directories(ClimbingStats INTERFACE
                             $<BUILD_INTERFACE:${CMAKE_CURRENT_BINARY_DIR}/generated>
                             $<INSTALL_INTERFACE:relative/path>
                             $<INSTALL_INTERFACE:$<INSTALL_PREFIX>/$<CONFIG>/generated>
  )

  You can see that the include directory is specified for both scenarios: if you're consuming from the build folder or if you're consuming from the install folder. CMake will generate a configuration file for each that contains all of the necessary include paths and usage requirements. (The actual config file generation is not shown in that snippet, but it involves CMake export and install functions).

  CMake is right to rely on the package's CMakeLists.txt to define how to consume the package. Similarly, Conan should do the same within the recipe. The ability to do this in package_info is sufficient, but we need a better condition than in_local_cache and a way for workspaces to trigger the in-build condition.

• New syntax for the same mechanism. Given that package_info and the full power of python is already available. Why introduce Jinja templating? Obviously this is necessary if you're going to do layout files in the ini format, but this only reinforces the notion that this logic should be in the recipe itself, not provided externally with new syntax.

• Out-of-source builds are broken. Layout files hardcode paths. There are some Jinja template expressions, but these are quite limited. Right now, the only way to do an out-of-source build is to blindly go up directories in relative path (ie. ../build). Not only is this dangerous (could overwrite an existing directory), it doesn't allow the user control over where the build directory is. This is especially critical when the user wishes to build more than one configuration at the same time with multiple build folders. CMake does things relative to CMAKE_CURRENT_BINARY_DIR or INSTALL_PREFIX and Conan should follow suite.

Editables

Editables allow you to short-circuit Conan to point to some local copy of a recipe instead of fetching it from the local cache. It does this globally for any other recipe that refers to it. As a developer who works on many things at once, this is problematic. It is too big of a hammer to globally override Conan to point to a alternate recipe for any and all dependent packages that reference it. I will happily continue to use workspaces, which explicitly states the intent for co-developed recipes and leaves the others alone. Please consider workspaces as the non-global editable and keep that functionality.

CMake Generator

The CMake Generator creates a cmake script that will add_subdirectory on the source folders (defined in the layout file). The actual generated script isn't very useful.

When you're developing recipes, you rely on conan build to perform non-trivial build steps. It is very useful to be able to reuse the settings and options in the recipe during the build. Even for end consumers, we use conan build as recommended in Mastering Conan. By using the generated cmake script, you lose this ability.

User/Channel

Why has user and channel been added as a requirement to the workspace file? When you are specifying the local path to the package in the workspace file, user and channel (and even version number) are meaningless.

Some Suggestions

I can see workspaces being used in two modes:

Super Build

This is the primary use-case. I have unique packages that I own and want to develop, sometimes together, sometimes individually. I temporarily create a workspace file that indicates my intent to develop them together. When I issue a conan workspace command, instead of generating a script with a bunch of add_subdirectory calls, it would generate a script with custom commands or similar that would invoke conan install and conan build. By doing this, you would gain:

• All of the conan build arguments, settings, options still work
• The packages don't need to be CMake anymore - you've enabled heterogeneous package co-development through the shared language of Conan
• The generated super build (can be a CMake script) can ensure that things rebuild when upstream package source changes
• You could potentially also call conan package as part of the custom command and then all of the special build folder mapping goes away
• Layout files are completely unnecessary. conan build already knows where and how to build the source, package_info already knows how to communicate paths and artifact info.
• Editables are unnecessary. Since workspaces are easier to specify without layouts, it should be preferred to make the non-global choice.

Self-Managed Build

In some cases, the source code of multiple packages are always developed together and their relative source locations are deterministic. This would be the case in a monorepo or git submodule situation. (This happens to be how I am using workspaces). In this case, I have my own CMakeLists.txt that knows where to add_subdirectory and a root conanfile.py that builds the monorepo via conan build. I also have other conanfile.py files scattered throughout the monorepo because packages within the monorepo should declare their own dependencies and remain independent, even if they are source controlled together. Finally, I have checked in a workspace file, because these packages are necessarily developed together. The workspace is critical to building the monorepo.

In this scenario, what I need from Conan is to walk all the dependencies and install all of the third-party packages like it normally would in a conan install. This is something I could do prior to Conan 1.13, although there were a few warts.

I think the Self-Managed Build is really just a special case of the Super Build where I don't bother to use the generated super build script and instead use my own CMakeLists.txt.

To help us debug your issue please explain:

• [x ] I've read the CONTRIBUTING guide.
• [x ] I've specified the Conan version, operating system version and any tool that can be relevant.
• [x ] I've explained the steps to reproduce the error or the motivation/use case of the question/suggestion.

[jgsogo]

🙌 Thanks a lot for your detailed overview and valuable feedback, also for gathering together all the issues related to workspaces.

I agree with you on most of your comments. We know we need to [re]define workspaces again, there are many issues related to them and it is a very valuable feature. Many of the issues are related to layouts, and your intuition about layouts being defined inside the recipe are in the right direction, I would like @memsharded to summarize here the work related to them before entering into details.

[monsdar]

Being able to work on multiple components at once is one of the main requests I get from our developers. Having workspaces "just work" would be awesome.

[kenfred]

@jgsogo Thanks for being receptive to my comments. I'm interested to know what is in store for the redesign.

As it is being redesigned, is there a stopgap that would allow it to work like pre-1.13.2 workspaces? I would like to get current with my conan version, but I need to be able to do workspaces without layout files. Perhaps we could revert back to that? Or enable the old or the current with a configuration option?

[kenfred]

@jgsogo I'd support the cmake generator to do a super build using externalproject_add as suggested in #3034. I saw your response to this in #3302, but I think its worth another look. externalproject_add seems like a great fit for this!

Specifically, I don't think we should mind the fact that externalproject_add uses import targets instead of the real cmake targets. In fact, this is the thing that will allow you to do support other build systems (a heterogeneous build tree) with the cmake workspace generator. It is also more true to how conan will operate when the packages are not joined in a workspace. externalproject_add has its own stamp files and DEPENDS arguments so that it will know to rebuild when one of the dependencies changes.

[petermbauer]

Another problem with the add_subdirectory approach is that one has to use e.g. CMAKE_CURRENT_BINARY_DIR instead of CMAKE_BINARY_DIR everywhere which is IMO not that common and leads to some migration effort.

[mbodmer]

#3034 (comment)

[wpalfi]

Very interesting. There seem to be many quite different ideas and use cases for workspaces. E.g.

• use conan as much as possible, if your architecture is already based on conan (as proposed here)
• avoid using conan as much as possible (as I proposed in [workspaces] Decoupled Workspaces #6844)
• lightweight workspaces, proposed in Lightweight workspaces aka Temporary editables #5566
• ...

The conan team should aim for a generic solution that enables all these use cases by just providing the required information and interfaces without actually creating any targets and without assuming any standard workspace layout:

• list of "editable" packages (together with source folders, etc.) in conaninfo.cmake or conanworkspace.cmake for custom workspace creation
• interface for linking binary packages to editables (e.g. by specifying the cmake-targets in the conanfile)

[melMass]

What is the state of Conan workspace, Conan 2.0?
Not being able to configure the CMake configuration of each target is a bit tricky when you have lots of dependencies.

[kenfred]

What is the state of Conan workspace, Conan 2.0?

I believe @jgsogo is best equipped to answer this question. I've tried to lobby for the workspaces feature reform in the Conan 2.0 Tribe discussions, especially as I think it is applicable to the proposals we've seen on local package development, secondary caches, package_info changes, etc. However, I don't think there have been any proposals specifically addressing workspaces, at least none that have surfaced for the tribe to vote on them. I believe @jgsogo is mindful of the workspaces feature and has some plans or drafts in the works. I'm fairly sure workspaces will exist in one form or another in Conan 2.0.

With relation to this thread, where I am advocating for specific workspaces behavior that involves super build and co-development of heterogeneous packages, I don't see that gaining any traction with the group. It seems one major sticking point is the desire to have individual source file dependencies for incremental builds. Of course, the only way to accomplish this is if all packages share the same build system, which is a narrow and unfortunate constraint to put on workspaces.

If we could loosen that restriction, we could have package-level dependencies which inform incremental builds at the super build level. That is, any changes to source would cause the individual package to rebuild and changes to the package folder would force other packages to do the same.

[melMass]

Thanks for the update, too bad I really liked your proposal and thought 2.0 would follow something like that.
I guess I will fallback to pure CMake but I don't understand how this is not "mandatory" for a Package Manager/Build Tool

[memsharded]

We have been recently doing a lot of effort in improving the foundation of the workspaces, the "editables". We have developed the new layout() feature, which is integrated in the recipes, and is more flexible and powerful. Please check latest Conan 1.38 and 1.39 releases.

We didn't want to resume work on the workspaces until this necessary piece was improved. Now it is looking much better, and the plan is to start allowing its usage in ConanCenter in next releases, so the feature will further mature and stabilize. We will keep investing in improving the editables via the layout() in the short term.

Furthermore, Conan 2.0 will bring many other very related changes, like pivoting the way the environment is managed, the imports of files, the local commands (for example conan build was already approved and changed in "develop2" branch to be complete, that is to compute the whole dependency graph), or the lockfiles. All these changes will really affect the workspace feature, so it didn't make sense to put efforts in it yet before all of that was ready. We will come back to this feature as soon as possible, can't guarantee it will be before Conan 2.0, or it might be likely to be just after it.

[melMass]

Sounds great! I've used layouts already but I was not able to run configure or build properly on my local dependencies. I think that's expected. For now, I'm just using conan install on each separately and then use the local cache, it's a bit slow but it works and it's predictable.

Are there public reports of the Conan 2.0 Tribes? What is the best way to stay up to date with the project?

EDIT: Oh I did not realize you were talking about a new method, I will recheck the new doc

[kenfred]

Thanks @memsharded! I'm a bit concerned about the persistence of "layout" and "editables" for the reasons I outlined at the top of this thread. Unless the concept of an editable has changed, it is problematic to keep that pattern going. There was lots of talk in the tribe about secondary caches and other strategies for local development, so I'm not sure if "editables" as I understand them is what you're talking about.

I see that the layout method is different from the layout external files, which is a great step and resolves a number of issues! (I think the external layout mechanism should be removed entirely.) However, consuming things from the build or source dir instead of the package dir will not support co-development of packages who use different build systems. So perhaps you'd consider a workspaces file that can optionally designate a source dependency in the generated workspace build (via source/build files from the layoud method, under the single CMake build system) OR a package dependency in the generated workspace build (via packaged files from the package method, under a super build - which is probably still CMake-based).

[memsharded]

Are there public reports of the Conan 2.0 Tribes? What is the best way to stay up to date with the project?

Yes, the best to stay up to date is the repo: https://github.com/conan-io/tribe, check the closed and open PRs

EDIT: Oh I did not realize you were talking about a new method, I will recheck the new doc

Yes, check also the new cmake_layout().

There was lots of talk in the tribe about secondary caches and other strategies for local development,

The secondary caches are a different approach that can resolve some related issues, but there are still many users that would still request the direct editable experience, so we think we are not ready to completely give up on the editable experience. Actually, the plan is to make package recipes "editable" by default. You can check some preliminary POC of how recipes in ConanCenter would be modernized: conan-io/conan-center-index#6474. The layout() will be there by default.

I see that the layout method is different from the layout external files, which is a great step and resolves a number of issues! (I think the external layout mechanism should be removed entirely.)

Yes, the external layout mechanism will be completely removed in Conan 2.0.

However, consuming things from the build or source dir instead of the package dir will not support co-development of packages who use different build systems.

Yes, they do, because the editables still rely on the generators like CMakeDeps, and the layout() folder definition is independent from the build system, it is an agnostic location of where the libraries are, but they could have been built with a different build system.

[kenfred]

The secondary caches are a different approach that can resolve some related issues,

Although I agree with the motivation of layered/tiered cache and think something is sorely needed, I don't support representing this as a "cache." I propose a similar approach around the workspace. You can see lengthy discussion on this topic: conan-io/tribe#19 (comment)

but there are still many users that would still request the direct editable experience...

I would like to probe this a bit. Of course users are asking for a way to edit packages and have those packages act as dependencies to other packages while in development. Are users asking specifically to globally override their cache to reference this "dirty" recipe? It is a pretty glaring liability to rewire your whole system to point to a particular reference that is in development. Many things can go wrong here.

Indeed this is what the "secondary cache" (and its variants, including my workspace proposal) are trying to address. I don't see how editables can continue to exist as-is with the prospect of these solutions on the horizon: they are meant to fix (or replace) editables. Of course, to be successful, the new design will need to satisfy the features enjoyed by the current users of editables.

Yes, they do, because the editables still rely on the generators like CMakeDeps, and the layout() folder definition is independent from the build system, it is an agnostic location of where the libraries are, but they could have been built with a different build system.

I think I'm missing something here. I understand that layouts are build system agnostic. However I need more information on what will be generated by the workspace and how it will form dependencies between packages in the workspace. The current workspace generator uses add_subdirectory to form CMake target dependencies, except not import targets, thus enablined source-level dependencies across those targets. But this is obviously not build system agnostic. So is the intent to use CMakeDeps to create imported targets of the libraries in the build tree? If so, then why not do this from the install tree and remove the need for the layout method entirely? The only benefit I can see is to avoid the install/package step.

My understanding is that layout is to package_info as CMake's $<BUILD_INTERFACE> generator is to $<INSTALL_INTERFACE> generator. However, in CMake, the BUILD_INTERFACE is needed for the add_subdirectory case. Yes, it may also be used to create a valid package configuration file (xxxx-config.cmake), but since those only have imported targets, there is no real advantage to consuming from the build tree vs. the install tree. Therefore, I don't see a good rationale for the introduction of layout.

[hpe-ykoehler]

Sorry if this is not fully related, but when I started to read about conan editable and this post I was quite surprise to learn that once you have an editable package, any other packages start to use it.

I was always trying to figure out how my own package would "link" to this editable package I added, and now I understand, but I think Conan is getting it wrong.

Assuming the scenario "MyApp" is using "MyLib" and both are independent Conan packages, I would expect having to do something special when installing "MyLib" and also when building "MyApp".

Today, we only have to do something special when installing "MyLib" -> use "conan editable add".

What I was somehow expecting is something like this:

conan install --dev MyLib # Special install instruction to install locally, here --dev is kind of an alias to --output-folder ./MyLib
conan build MyApp --dev # Special instruction to build using the locally installed MyLib requirements instead of the cached one.

that would give me the following hierarchy:

MyApp/
  conanfile.py
  MyLib/
    conanfile.py

This could be implemented by using some default rules:
"Requirements" have a new property "editable_folder" # self.requires("pkg/version", editable_folder="pkg")
"editable_folder" by default have the value "current_folder"/"pkgname" # above that would be ./MyLib
--dev is an indication to use --editable when possible.

There would be no need for a "conan editable" command, it would ensure that CI script that uses "conan build" without --dev always does the proper thing, in case they get tested locally on the developer machine.

It would also ensure that the editable package is clearly identified, by default at the same level, or otherwise, you have to specify.

Maybe the "workspace" then is basically "currentdir", one could then set "workspace" to "parentdir" (relative or absolute path) and then all packages would appear at the same level:

<workspace>:
  MyApp/
    conanfile.py
  MyLib/ 
    conanfile.py

By default, the workspace would be the current folder you execute "conan install" in...

Anyway, total newbie here as far as Conan goes. Let me know if that makes no sense ;-)

[memsharded]

Hi @hpe-ykoehler

Thanks for your comment.
We didn't invent the editable concept out of the blue. It is copied from the Python pip -e editable mode.

There are some challenges that would make your proposal not work nicely:

• The need to to have something in the recipes that says "editable" is not a great approach. Packages should be prepared for both local independent development and for creation in the cache, but explicit "editable" in recipes will not work
• Relative nested layouts don't work nicely with transitive dependencies. If MyLib depends on another MyDep package, that is also editable, it should be nested one level inside MyLib, so MyLib finds it in the same way you specified for MyApp. This doesn't work at all when there are diamond structures, and several packages depending on MyDep.
• Nested layouts are problematic with source code, binaries and repos. If MyApp and MyLib live in different git repos, putting a git repo inside another git repo is not great and will produce issues. Every possible globbing approach, temporary build folders of the different packages, might unexpectedly pollute the folder structure.

So in the development of editables it is very important that editable packages are almost "transparent" to their consumers. It doesn't matter if the package is in "editable" mode or in the cache, for the consumer/consumers of that package, it should be exactly the same. Editables don't aim to be a workspace which is what it seems you are proposing.

[hpe-ykoehler]

For most case, the need to have something in the recipe would not exists, since the default package name is likely to be the value used by all, only the "dir" or "workspace" could change and that could be set in the global conan config.

Regarding the diamond approach, likely when a developer works on one app, it doesn't need to have all the requirements locally made available but only some. In a situation of a diamond approach, that is where you may want to set the requirement editable_folder to be something else. when you install a package with --dev its own dependency is not installed locally, you would have to install those explicitly.

conan install -dev MyLib would install only MyLib, not MyDep locally, then if you want to have MyDep installed locally, you would install it under MyLib if you want MyLib to take it or at a different place, in which case then you would have to edit the editable_folder for the requirement that you want to use that folder.

So you could decide between

   MyApp (if MyApp depends on MyDep you would specify an editable_folder to adjust the location.)
     MyLib
       MyDep (This would have to be installed explicitely)

or

   MyApp 
     MyLib (MyLib then has to be edited to have the MyDep editable_folder pointing to MyDep at the same level)
     MyDep

or

   MyApp
   MyLib
   MyDep

Since the workspace is "parent", it works without any change.

Using a git repo for another one is not that problematic, and you can control how you want the structure to be. I used git within git often and other than having to know where you are when you type a command, there is not many issues occurring. I cannot say for other VCS, but the idea here is you are in control of how those things will show up on your disk, so you pick the structure that fits you need.

I agree that a solution that would not require you to edit your requirement inside your own conanfile.py would be great, but somehow this to me feel less of an issue since it only becomes applicable when you use --dev and you probably will want the structure to be the same once you start working on a workspace and then wants to reconstruct it.

Seems more explicit and doesn't impact other packages that you do not use --dev or are outside of the workspace.

There is the "name" clash that can happen but when you install the package if there is something with the same name, conan can just stop there and tell you, you then have to pass --output folder to override the installation name, and adjust the editable_folder on the recipe using it.

[kenfred]

Hello @memsharded :

Editables don't aim to be a workspace which is what it seems you are proposing.

I thought that is exactly what editables were meant for. When Conan removed the previous concept of workspaces, it pointed to this editable scheme as the alternative. If this is not the way to do a "workspace", is there going to be a way?

I useful reference is the spack environment. Environments are similar to the old Conan workspaces. Is there anything like this in Conan?

So in the development of editables it is very important that editable packages are almost "transparent" to their consumers. It doesn't matter if the package is in "editable" mode or in the cache, for the consumer/consumers of that package, it should be exactly the same.

The issue is the editables become global to your system and you can't have multiple versions of a package in development (without jumping through big hoops, like creating a new cache). Packages under development should not be transparent to the consumer - they should be very obvious and limited in scope, only affecting the "workspace" / group of packages you are co-developing.

[hpe-ykoehler]

There may be 2 views on this.

1. you edit a package because something about it is wrong on the official cache and you want to adjust it and then use your version all over the place.
2. you edit a package for development purposes, in which case, you want to control how that package gets used and limits its scope.

[kenfred]

@hpe-ykoehler

Option 1 is bad because it points your whole system to a mutable package, which negates much of Conan's efforts to ensure reproducible builds. If the "official" version is bad and you never want to use it, then a new "official" version should be created and used. And, very importantly, that particular revision of that new official version should be immutable!

[petermbauer]

Using a separate dedicated Conan Local Cache seems acceptable and cleaner to me. We are anyway using project-specific ones to keep the profiles and so on separated.

[kenfred]

@petermbauer

Using a separate dedicated Conan Local Cache seems acceptable and cleaner to me.

The downside of that is you need to populate your new cache with all of your dependency tree, even if you're only developing a couple of them. For large packages, this can be a pretty heavy operation.

The whole concept of "cache" is to prevent those long fetches by having a local mirror. Once you decide to make a cache serve as a workspace, it is no longer useful as a cache.

IMO, Conan needs the cache and a workspace concept to make multi-package co-development reasonable.

[memsharded]

I thought that is exactly what editables were meant for. When Conan removed the previous concept of workspaces, it pointed to this editable scheme as the alternative. If this is not the way to do a "workspace", is there going to be a way?

It is not an alternative. The 1.X workspaces were already using editables as building blocks, but the previous editable management based on external .ini files was simply not working. The concept has not changed, editables are still the necessary building block to be able to restore workspaces sometime in 2.X. I think you are still conflating both concepts.

As I said, we are not building or inventing something out of the blue. The Python pip editable mode has been there for years, being a very useful tool for Python developers when they need to work simultaneously on more than one package, or develop the current package. We use it daily in our development and both the concept and execution is solid.

The fact that there is not yet in Conan 2.0 a "workspace" layer that manages the orchestration, local relative location, bootstraping, project-local definition of editables doesn't mean that that the current editables are incorrect or that they should implement those features. Those features still belong to the "workspace" layer, not to the editable layer, and when work on workspaces is resumed (it is part of 2.X roadmap), it will be focused on those things. I am not saying that this will not be considered, it is already in the 2.X roadmap.

[kenfred]

Hello @memsharded. Thanks for your reply!

The 1.X workspaces were already using editables as building blocks

I was talking about the original Conan workspaces. Editables were not introduced until around 1.12 or 1.13. Agreed that both the original workspaces and the post-1.13 workspaces had flaws that needed fixing.

As I said, we are not building or inventing something out of the blue. The Python pip editable mode has been there for years, being a very useful tool for Python developers when they need to work simultaneously on more than one package, or develop the current package. We use it daily in our development and both the concept and execution is solid

Another well-established Python tool is the virtual environment to sandbox your dependencies. I do not have a problem with using a python editable within a virtual environment. That is effectively the behavior I think we need for Conan workspaces. It's the global-ness of the editables that I find offensive and I would like a workspace akin to a virtual environment so I can restrict which builds the mutable package is affecting.

[memsharded]

#15992 is centralizing feedback and effort for Conan 2 workspaces, lets close this one and track progress there.