Subproject limitations #422
Comments
|
Using If you use subproject foo that uses other subprojects, the master project must add those to its own toplevel subprojects directory. See test case 79 to see an example. The reason Meson forces you to do this is disambiguation. Suppose you have top project A and subprojects B and C, both of which use subproject D. It is mandatory that there is only one instance of D and both subprojects use the same instance. Otherwise they would each embed a copy, their versions would be different, symbols would clash and things would crash and burn in a way that is very hard to debug if you don't know beforehand that this can be an issue. |
|
Just to clarify if anyone else is wondering: However, you will still run into problems, if you have 2 (or more) subprojects that both use the same library, but in a differently named folder inside their The issue here has nothing to do with subprojects in general, because this is an issue of the I am in no way an expert on the C linkage process, so it would be great to get some more details why this is really necessary or does this make it just simpler for the main project maintainer, because he must decide upfront (by setting up the top level subprojects) how the compatibilty between subprojects and their dependencies is fulfilled? |
|
I randomly opened this discussion, but already have an opinion :) I think we can agree that there are cases when different versions of same library in one project are useful (when done cautiously, as it is indeed quite fragile). I also think that this is probably rare case, but is there a reason to explicitly forbid it? Then, even if forbidding it, could meson allow for recursive detection of subprojects, and then issuing an error, if different versions of same sub project are used? Or even when just same sub project is used twice. And only then user would be forced to do something for disambiguation. This will avoid breaking builds just because some sub-sub-project decided to use some other subproject. Thank you for your time! |
|
I think the case where two subprojects use the same library (possibly with different versions) is kinda rare compared to the more common case where a subproject just uses some other subproject that has nothing to do with any other subproject in the build. This should definitely be supported. Otherwise, as the maintainer of the main project, whenever I update any of my subprojects, I'd have to recursively check every subproject and if it was updated then copy those changes to my own subprojects folder. This quickly gets out of hand e.g. when using git submodules for subprojects. Since it is not possible (at least not feasable) for meson to detect the same-library-used-twice situation reliable (at the moment it just relies on the same subproject name), it's a bid odd to forbid the common case of nested subprojects because of a potential issue that is not really solved with a single subprojects folder. |
|
Having two different versions of the same library in the same project is very, very bad. Suppose you have lib1 and lib2 and functions When you call into Having multiple different versions of any library used in the same executable will not work and, more importantly, can not be made to work. Any system that permits this kind of intermixing is inherently broken.
That is true. Such is the responsibility of being a maintainer. The unavoidable fact is that someone, somewhere has to linearize the list of dependencies to avoid duplicate deps. There is no way around it. Wrapdb aims to make this easier by providing a single source for most of deps. Not using subprojects of subprojects automatically has other benefits, too. For example if you have a common dependency (zlib, sqlite, etc) that is used by 10 different subprojects then that would mean compiling the dependency 10 times. This quickly becomes infeasible. If you use distro packages for your dependencies, all of this is taken care of for you. If you don't have the distro's support behind your back, you have to do it yourself. |
|
I am onboard with the issues you described when using the same library - or even two libraries that both export the same symbol. My situation is:
There are no conflicts with any symbols in the hierarchy, so I would expect meson to just apply the subprojects recursively, because managing the correct version of I therefore propose to allow recursive This proposal:
I can't think of any downside to this approach. If meson really wants to provide a way to prevent the issues of same symbols in different libraries (which it currently does NOT), it should not conflict with easy to use subprojects, imho. |
|
There are two different failure cases here. The first on is that two completely different expose the same symbol name. If this happens you are screwed in any case. There's nothing any build or packaging system can do to help you. Fortunately this is rare. The second case is that you have subprojects that depend on the same subsubproject but they each bundle their own version with a different name. This is the most common failure case. Forbidding nested subprojects makes this case impossible because you need to hoist all deps to the top level and there can only be one subdirectory with a given subproject name. This is the Highlander way of dependency management: there can be only one. Another advantage of the flat hierarchy is that it makes it very simple to determine what subprojects any given project uses. All you need to do is One of the main design points of subprojects is that any project can be used as a master project or a subproject without any changes to the code. Having an That being said we might very well develop some tools that make hoisting dependencies to the top level as easy as possible. (Further: it might not be obvious but the suggested file layout is one git repo per project. You should not have a repository with stuff in its subprojects dir unless you have a monolith app whose deps never change. Subprojects should be gathered by e.g. using wrap files to do git checkouts.) |
|
How does a flat subprojects hierarchy prevent any issues in this case:
with What should I do now? I need a way to decare in my main This would be easily solved with either I don't see why an Example from the beginning, but taken a step further:
meson build files:
or:
I get that its much simpler to flatten the hierarchy (but even then we do need a way to resolve same subprojects with different names), but it is also much harder for a user to work with nested subprojects. I'll want to clarify why I find the current subproject concept limiting: Basically there are 2 possible solutions to my issue:
I hope I got my point across at least a bit :) BTW: We could also simplify the whole |
There's the problem right there.
There may be a misunderstanding here. Wrap files can be used just fine without wrapdb. You can even run your own instance of wrapdb if you wish. A wrap file just specifies how to obtain the dependency in question. For your case you would create a wrap file for your project called This tracks git head. You can specify a hash id instead if you wish to do version tracking manually. Meson will do the checkout and all that good stuff for you. For all the details please read the manual.
CMake has this approach. The problem with this is that subprojects are not isolated but instead can change their parents' state. I have had the displeasure of debugging projects that do this (usually not by intention but by accident). It is misery on end that I don't wish upon my worst enemy. Running subprojects in isolated sandboxes is the only way to make it work. In fact let me put my project lead hat on and state the following for the record: Permitting the use of subprojects with |
This requires all project maintainers to do the right thing without any chance for me to correct issues in those projects. Also, there are many reasons for different names for the same (in the sense of a common base, must not be identical) library, e.g.:
So, something like an
This is not helpful during debugging sessions, since it only can checkout code that I already pushed to the repo. When I'm currently trying to fix a bug I don't want to push every change I make, but rather change the code, run / test it, and then do a push when I know that it works.
You mean like it does with subprojects? This would make meson.build files easier to read, because a variable in a subdir can only be defined in the current file or in one of the parent's. Currently, the variable can be defined in any subdir that was executed prior to the current one, which makes it kinda hard to hunt down variables. Overall, How would you tackle the issue of including a library that is created in an outside project (which is owned by me) in a way that allows me to easily modify the libraries code until I fixed a bug or implemented a new feature?
The |
If you work on your own machine you can define the git url int the wrap file as file:///path/to/local/checkout. Alternatively you can put a symlink in As far as the package naming go, could you please give an example of multiple providing incompatible versions of a dependency and a project that links to them in the same process via current tools (preferably pkg-config). |
|
Pointing wrap to my local git repo would still require me to commit the code first. The symlink works and should be fine during debugging sessions. I am not sure what example you'd like to see. I am not talking about incompatible versions, but about compatible ones with different names (in the subproject folder, they very well might both produce a library called I have the feeling that I need to recap what issues I think should be solved with subprojects.
I would gladly provide pull requests for these features, but won't start implementing these until I get an official 'go' :) |
|
I'm going to make this a long one because this very issue has driven me away from the likes of Bazel, Buck, etc. In my opinion, as much as I hate to say this, CMake is the only build-system to get subprojects even remotely correct. Unfortunately, Meson's subproject design is going to force me to drop it after just a few short hours.
Please stop with these strange assumptions and absolutions about libraries. You're missing the point about the possibility of two unrelated executables being used in a later build step with the same library. For example, I have two tools that I need to use in order to convert some file formats. They are called They do not link to one another in any way. By your logic, I have to choose one of the xopts, wrap it, and then define it at the top level. This makes no sense and solves nothing.
This design and philosophy will drive away users from Meson. I'm now regretting the several hours I've spent evaluating Meson for my projects. It's clear nested dependencies are entirely broken. Here's a simple example: $ mkdir -p subprojects/foo/subprojects/bar
$ cat > meson.build <<EOF
project('top')
message('hello from top')
foo = subproject('foo')
EOF
$ cat > subprojects/foo/meson.build <<EOF
project('foo')
message('hello from foo')
bar = subproject('bar')
EOF
$ cat > subprojects/foo/subprojects/bar/meson.build <<EOF
project('bar')
message('hello from bar')
EOF
$ meson build
The Meson build system
Version: 0.49.2
Source dir: /private/tmp/test-meson-come-on-bro
Build dir: /private/tmp/test-meson-come-on-bro/build
Build type: native build
Project name: top
Project version: undefined
Build machine cpu family: x86_64
Build machine cpu: x86_64
Message: hello from top
|
|Executing subproject foo
|
|Project name: foo
|Project version: undefined
|Message: hello from foo
|WARNING: Dependency bar not found but it is available in a sub-subproject.
|To use it in the current project, promote it by going in the project source
|root and issuing the following command:
|meson wrap promote subprojects/foo/subprojects/bar
subprojects/foo/meson.build:3:0: ERROR: Failed to initialize 'subprojects/bar':
Subproject directory not found and bar.wrap file not found
A full log can be found at /private/tmp/test-meson-come-on-bro/build/meson-logs/meson-log.txtThere's absolutely no reason why this hierarchy shouldn't work, and the fact that it doesn't on the latest version means I simply cannot use Meson for my projects. I rely on the nested nature of things to build static executables and do WPO.
No. This is not at all the goal here. A build system is supposed to relieve these pain-points, not exasperate them.
I'm sorry but this isn't feasible in my opinion. The best way to make this work is to give users the ability to vendor Meson scripts in next to either vendored or submoduled code. This is what repositories are already doing, and it has the added benefit with Github that they can do automatic vulnerability alerts based on the submodule links you have set up in your repo - including across forks. With wrapfiles and URLs being specified, that's all lost. Further, wrapdb entries don't have their patches in any sort of source control, but instead zip them up as a release. Aside from the fact that's a non-starter from a security point of view, it's very obtuse from a maintenance standpoint.
As a project maintainer of many heavily-nested projects with CMake, I simply do not care about this as long as the builds are incremental. This is very close to the bottom of my concerns - initial build times are almost a guarantee that they will be long, and this is something most developers I know have come to terms with. What matters more is that the build system gets out of my way and works intuitively, without needing me to think or reference documentation. This is why CMake is so
So rare, in fact, that anecdotally this has only happened to me maybe once or twice in my entire career as a C/C++ developer. This is not a case I'm worried about at all because it's incredibly easy to debug.
I don't see how this is an issue unless you're using URL-based acquisition, which - personally - I see as a huge negative feature that shouldn't even be included in build systems because it is A) not their job (they are not a package manager), and B) causes even more unreliability as your build now becomes dependent upon the availability of the remote server. This is horrifying from a sys-ops and dev-ops perspective and is one of the biggest failures of Go's dependency management system. Further, the use of remote HTTP URLs, as you said, reduces your ability for introspection. You're trying to solve it by forbidding nested submodules in order to protect your HTTP acquisition - however, you should be doing it the other way around.
How can they be hidden if they all follow the directory convention of
Please don't. "Hoisting" dependencies sounds like a maintenance nightmare and even in the simple cases feels wrong, looks wrong, and behaves wrong. Please fix the real issue here.
Not if they're within two disparate subprojects. You can assert that two libraries are the same, albeit maybe different versions, if their names are the same. That's all fine and well. However, you can also assert the relationships between these dependencies since we very clearly define them using So what you're describing isn't a problem with the project layout. It's a problem with your narrow array of ways you envision people using Meson. This isn't meant to be a 'dig' at you, I'm simply pointing out the shortcomings of many new-age build systems and I'm sad to see Meson have so much potential but ultimately break down at the same place most other build systems break down.
I 100% agree this is a poor design decision in CMake. However, this doesn't negate the philosophy of nested submodules in any way.
Literally nobody on this thread is arguing for this.
☝️ Seconded.
This doesn't scale. This runs the risk of pushing bad configuration to a remote, which in the case of CI/CD will break production systems if they're naïvely configured.
Please don't make me spell out why this is a terrible idea.
DLL proxying, dependency patching, debugging tools that do function call detouring. While I agree that the build system shouldn't allow this by default, your assertions that there is never a valid reason for linking two libraries with identical symbols is objectively wrong. Also, you're still missing the point entirely about how libraries are used in nested subprojects. At this rate, I don't see how Meson is a viable alternative to any build system. I hope you make the right decision to change your philosophical views on project hierarchies because as of right now they are very, very broken. |
|
If you want to have multiple different versions of a library in one project, you can do so fairly easily. Suppose you want to have versions 1 and 2 of project And then you could get them via This has the additional benefit that if you want to know how many different versions of |
|
@jpakkane you're missing all of the points yet again. |
I don't understand why you're saying that. I was going to reply exactly the same as what @jpakkane just said, that answer your issue regarding having different versions of the same library. One thing I have been working on lately is adding the ability to override dependencies. For example: dep = declare_dependency(...)
meson.override_dependency('foo', dep)
# dep == dep2 here
dep2 = dependency('foo')What else do you need regarding that issue? Your other main concern is about nested subprojects, and I agree with you it's not ideal (however, I don't see how that can be blocker in any way). Copying the .wrap from a subproject to the parent project doesn't feels right, we should have a way to rather "link" to it. I already thought we should have some kind of config file in |
|
I have the same issue now. Exactly the same problem as @Qix- described above.
Example of my project layout: :( I really love Meson, but for this project I need to find an alternative. |
|
@TheAifam5 I just implemented a feature for meson to support nested subprojects: https://github.com/semasquare/meson/tree/nested-subprojects I tested it with nested subprojects that use https://github.com/meisenzahl/meson-dependency-test Try it out to see if it solves your issue. |
|
@jpakkane I was thinking I think the name "subproject" might make people continue to ask for recursive subprojects. If instead we had just projects and one "metaproject", I think people might get the point better. |
|
@Ericson2314 it's not about terminology, it's about functionality. I've used an army of build systems throughout my career and know exactly what I want. Meson gets in my way with its philosophy and dogma, so therefore I cannot use it. |
|
@Qix- I have worked on an army of build systems in mine, and I agree with @jpakkane that hierarchical subprojects are indeed wrong. Depedencies form a directed acyclic graph, not a tree. Your example with the two executable is what we call a private dependency. Private dependencies are the exception, not the norm, and trees/hierarchies only work if every dependency is private. I suggest you read about https://www.well-typed.com/blog/2015/03/qualified-goals/ . Cabal, Cargo, NPM, etc are solving build systems in that you must decoratively state your dependencies so that the dependencies can be simultaneously choosen. [Meson has one imperatively state dependencies, which is arguably a mistake as it makes version solving a lot harder.] The "qualified goals" on executables and is basically equivalent to saying that executables only have private dependencies, since nobody links with them. Cabal also has one "project" which contains "packages" which must have all their dependencies reconciled. This matches my project vs metaproject proposal. |
And I'm the pope. What's your point? Pissing matches don't belong on Github, take them elsewhere. Cabal, Cargo and NPM are not build systems, they are package managers. I wouldn't expect a build system to solve versioning constraints. I also don't feel build systems and package managers are one and the same, nor should they be. Your link describes version conflicts, which are NOT analogous to linkage conflicts. A build system is not solving a system of constraints, but instead solve a system of dependencies. Those are very much different domains and problem spaces. Let me repeat: your build system should not be solving systems of versions. This is a large reason why I moved away from Meson because it's scope creep. There's a reason why Aptitude, brew, npm, etc. don't dictate underlying build systems (I could use automake, cmake, ant, maven, etc. for any or all of those) - so your point doesn't really make sense to me. This is also why projects like Glide exist - because Go literally has package management built into the language, which people have had to re-learn is bad, it seems. This is also why people tend not to like Cargo to manage their Rust builds and turn to other build systems entirely. Instead of telling me what is/is not allowed based on your own philosophies, I just need a tool that lets me do my work and doesn't force me to shoot myself in the foot later on down the road. Meson is not that. I have my own philosophies (clearly) so let me operate within them. |
|
My comments on working on multiple build systems and on imperative dependency specification perhaps being a bad idea weren't supposed to kick off a pissing contest and change the subject, respectively, but rather try to offer some evidence that I'm not some fanatic with blinders that doesn't know anything but Meson and also thinks that Meson is already perfect. If you want a strict separation between package managers and build systems as those are currently defined, the meson shouldn't have sub-projects at all. If you want to keep talking a bit off talking on solving and the division in labor, OK consider me now nerd sniped :).
I actually agree. I like that Ninja does all the building and nothing else. I use NixOS and Nix (http://nixos.org/) and someday hope to write a Nix backend; Nix also does no solving. The issue is configuration vs solving. I think all build-time inspect-the-environment configuration is a workaround the solver not knowing enough statically. If you top-down decide everything, packages should build directly from those assumptions, and what was configure steps instead become tests ensuring the system as assumed matches what was built. Sub-projects allow one to short-cut environment detection in that a depency required by one project is required by the other, and the configure step is OK with that even though the dependency isn't yet built and cannot be detected the usual way. They also allow for cross-project incremental rebuilds, which is very useful for interesting cross-project refactors. In short 2 things: incremental builds accross boundaries and transposition all configuration before all building. Take that to its logic limit in the context of something like NixOS and you are building entire machines images with compiler-invocation granularity. Since there can be no environment detection when everything is built from scratch, everything has to be top-down specified. If you pin versions for everything then like NixOS you don't need to solve. If you don't wish to pin versions (or more generally the declarative requirements don't fully constrain the build plan), then some solving naturally comes into play because there are potentially multiple unique solutions and requirements need to be combined. If you want a separate solver, "configurer" (Meson/Cmake), and builder (Ninja/CMake/Bazel/..), what is left for the configurer to do in this stage? Just be a pure function from what the solver decides to some Ninja? Part of Meson is environment detection and the other part is domain-specific knowledge about how to invoke compilers. So just the other part is needed. So that's a vision of a far future. Nowhere between now and then do I see a good use for hierarchical subprojects. Subproject combining is always package management in that you are deciding how to resolve dependencies. And it always precludes that dependency from being used by anything else. How many private dependencies do you have exactly?
Perhaps you have your own opinions which is good, but it seems that many people have no regard for their build system and just hack things up until it works. (Perhaps it's not their fault, but something they learned from decades of terrible build systems). That usually leads to hard to understand projects that don't compose very well, so I rather the build tool be more opinionated. |
My mistake, apologies.
I don't see how that follows - using git submodules, for example, leaves the "package management" element to Git and would allow Meson to work seemlessly with recursive submodules.
I still don't see how this has to do with file hierarchy, seeing as how those subprojects would be present at the initial configuration stage.
I still cannot fathom how this is limited by what you call "recursive submodules", nor how it's fixed by enforcing a single level of subprojects.
Has nothing to do with nested subprojects, I'm not even sure what this is talking about.
You're assuming a lot about projects, and making generalizations that simply are not true. When building an OS it is entirely common to use packages that are intended to be used by non-standard platforms and thus provide a number of configuration variables to specify how they should be built. This is no different than how CMake handles it - you can absolutely specify the cross-compilation toolchain, and even override sub-sub-subproject target flags/settings/etc, even at the file level. What you're describing is already possible with existing build systems that also allow nested subprojects. Meson is creating an artificial limitation here out of stubbornness.
Again, I don't see how "versions" have anything to do with anything. Build systems should look at the filesystem and be configured by the file system and build whatever you tell it to build. I don't see how versions have anything to do with Meson if you don't use Meson for managing your dependencies.
If you are entertaining the idea that build systems should care about building multiple versions (in build system terminology they're called "variants") then this is also already done by many build systems just fine, even with nested subprojects. I'm not sure where you get the idea that this is somehow impossible. If you're referring to dependency management, something your build system shouldn't care about, then it should at the very most warn/error about conflicting errors depending on how the packages are going to be used. If I have This sort of configuration happens all the time and the fact Meson
Tup seems to do this just fine.
Generate the ninja build files. CMake doesn't build anythinng, and I'm not sure where Bazel comes in because it does both (and subsequently sucks at doing both FWIW). Ninja is the only thing creating a pseudoterminal, forking, and execing. A build system comprises both the configuration and the build, hence why "CMake" is widely considered a build system even though it doesn't actually run the builds. I don't really understand the question.
That's all fine and well, but I seriously do not see how this is at-all related to the discussion.
You haven't even addressed subprojects..... I simply do not understand your point. With all due respect, your response is a word soup of nonsense. Please clarify your points.
Yes, and as the developer of the project, I'm pretty certain I can understand the dependencies that I am consuming just fine. We've been doing it for decades without issue.
That's a bizarre conclusion, one that is objectively false...
I do not understand this question, as it makes no sense.
It's not that difficult to understand. Meson does a lot of things right but falls short of being a real build system. Here's a great resource that is packed full of falsehoods about compilation and build systems. I don't need a bible, I need a tool. https://pozorvlak.livejournal.com/174763.html Here are a few highlights:
And from the comments, another gem:
I do not understand why build systems need to force their religion down my throat. |
|
@Qix- I still don't know what problems nested subproject solve for you. I only see them luring users in general---if not yourself---into doing terrible things. If you really think there's something we're missing that would make me change our minds, go find me on IRC or something. |
|
Please go read my earlier comments, I explain everything in detail. Meson's job isn't to tell me how to build my projects. Meson's job is to build my projects. |
|
This is going nowhere fast. I think everyone's clearly stated their positions on the matter, and no one seems interested in changing them. I'm going to close and lock this thread. |
Currently,
subprojectsare somewhat limited.Example:
B and C are completely unrelated, so I it is not possible to have all projects in a single source tree and just use
subdir.Also, nesting subprojects is impossible, so I can't do:
If I don't have control over
XandY, I can't create a simpleZproject that usesY. As the creator ofZ, I should not have to worry if any of my subprojects use other subprojects, which would require me to list all needed subprojects in my own subproject folder.Did I miss something? Note: I did not test these, I just want to clarify if this is how it is supposed to be and discuss possible future improvements.
The text was updated successfully, but these errors were encountered: