[RFC] Make dub self hosted #1972
Conversation
|
It would likely break every packages out there which is why I'm in no hurry to pull this. |
|
Thanks a lot for working on this. I actually needs this to due to the undead movement of std.xml. Also it would ease the |
There was a problem hiding this comment.
This will get a strong -1 from me in its current form.
Can you describe the bootstrapping process? As far as I can tell, there is none. You either download a dub binary and use it to build dub, which is not an option for me, or you build an older version which still can be built standalone. The latter, which is the only available option for me, will likely break in the future, leaving us with no way to bootstrap dub. Am I missing something?
My suggestion would be to just use a proper build system. vibe.d installs pkgconfig files which can be used for dependency resolution. This would require you to ask the maintainers of other libraries you want to use to do the same.
I think making dub build itself is a bad idea, given the other available options. But if you really want to do it, you need to add a way for us to build a minimal dub, which can then be used to build the "full" dub.
Checkout dub v1.21.0, build it with
I will add a bootstrapping check in the CI, ensuring that master can be built with v1.21.0 for a relatively long time. How do you guys handle DMD at the moment ? Alternatively, I could add an option to |
|
Having a build system would also allow you to dynamically link to vibe.d and all other dependencies. This is very desirable for distributions. The current approach goes against arch guidelines, but I guess there's no option so it will have to do. |
This does not give me enough confidence. "relatively long time" is not enough for me. At some point things will change enough that this will be very difficult to maintain. Might be in 1 year, might be in 5 years, but it will likely happen at some point.
I want to manage sources myself, so there is no need. I fetch the sources, validate the checksums, and proceed with the build. |
|
I have a fear of dependencies down the dependency chain introducing breaking issues like using toolchainRequirements or new potential future dub features (like git submodules, new repository types or preBuildCommands with new variables) which wouldn't necessarily break the build but get rejected then anyway, causing headaches. I think personally I would prefer the most if dependencies were normally specified and built using dub, however for the bootstrapping process you are expected to install all the packages (like |
I agree and I'm well aware of how at odds the D ecosystem is with traditional Linux packaging practices. However, this change does not make the situation worse, does it ?
The wording was intentionally vague, as I'm not the only stakeholder, and I can't tell you how long it will be without gathering opinion. This is also why I asked how you are handling DMD, as it requires a D compiler to build. |
Copying or symlinking from the system to
It wouldn't. But you have a situation here that would be much better resolved by adhering the standard practices, and I am still getting pushback. What are the cons of having a proper build system?
This is exactly the problem. And opinions change with time. I have had to deal with with similar issues in the past, and I am advising that from my experience, I think going down this route is not a great option.
We use LDC to build DMD. Anyway, it's not my call. I am just giving my opinion as it was asked from me. |
Then the bootstrap issue is with
You mean switching to
I understand the frustration (and experienced it myself). I'll make sure we reach an agreement before this gets merged (if it does). But I'd like an answer on the
And it is very welcome! |
Yes, they went for the same approach as you propose. I do not think it is very maintainable, but they will have to deal with it. While I think the approach the LDC developers choose isn't great, it is
Dub is a decent package manager but an awful build system. I could have a lengthy discussion about this, but when it comes to our issue I think it all boils down to dub not being able to resolve dependencies on the system. Nobody uses pkgconfig for D dependencies because dub, the de-facto "build system", does not support them. Dub also doesn't support installing to the system which does not help in any way. There is a real need for this, The main motivation for this change is to be able to have external dependencies, instead of using dub and making everything more complicated just provide a simple Makefile or meson definition that finds the dependencies via pkgconfig and builds dub. If you also want to maintain a dub file for dub to be able to build itself, go ahead, but do not make it so that dub is un-bootstrapable by itself, especially since the Makefile/meson definition would be super simple. |
By making DUB self-hosted, we will allow a bunch of improvements, most importantly using libraries and getting rid of the duplication with the vibe.d code copied over. However, in doing so, we want to make the process as easy for package manager as possible, hence build.d will be left in place to provide user-friendly error messages. The ones that wish to switch to a direct 'dub build' can do so, and the ones that need backward compatibility can specify both HOST_DUB and DMD.
For a long time we were using copies of the module, with some adaptations. However this was error prone, required extra care, and was not overly scalable. Now that we are self-hosted, we can simply use a dependency.
|
I wonder whether this alternative idea for bootstrapping would work:
If CMake is not available on the system to be bootstrapped, Makefiles can be used that are generated by CMake on the fully functional system. Does this solve any bootstrapping concerns? |
|
It would be much better if the cmake files were committed, but overall, it would be a reasonable approach. What I am afraid of is in the future the cmake generation be changed in a way that it relies on dub, like calling dub for something. This is something I feel would be reasonable to do otherwise, but would absolutely break the bootstrapping. |
There are a bunch of places I need to go through and polish, mostly CI scripts, to make sure the
build.dscript is correctly called, but I wanted to get an early-ish feedback on what people think about this before I put the 80% of work required to get those last 20% of feature.This PR will make
dubdepend ondubfor building. This mean that we will need to have a bootstrap step in the CI to ensure we do not depend on new versions "too soon". As a concession to user-friendliness, thebuild.dscript was left in place as a shell to calldub. It also allows backward compatibility if one needs that.The main motivation to make
dubself-hosted is to open it up to the usage of libraries. Currentlydubhas a few modules that are copy-pasted from Vibe.d because no dependency management way could be decided on for the dependency-management tool. I personally wanted to experiment with a few things (including #1832) but was blocked by the addition of a dependency. I considered usinggit submodulesbut the usage felt like reinventing the wheel because of the logic needed inbuild.d. @PetarKirov suggested self-hosting, and while originally hostile to the idea, I had grown convinced that it might be the time to do so.I hope the diff is convincing enough: Literally no addition to dub, only a sed on import, and -5k LoCs.
CC @s-ludwig @WebFreak001 @MartinNowak @wilzbach @andre2007 : Destroy (or 👍 )!