Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Unify build directories. #6668
This unifies the
For backwards compatibility, tests and benchmarks are still stored in
This may be a disruptive change, so I'd like to make sure this is a good layout for the future, or if a different strategy would be better. There is some flexibility to change things around (for example, we could make it
As I mentioned earlier, this slightly breaks rustbuild, but the fix is simple.
I feel like the backwards compatibility for tests is a little inelegant. Not sure if there's a better approach.
Continuing discussion from #6577:
Directory junctions are supported back to Windows 2000. However, they are absolute-paths only, and only work on NTFS. std does not have a way to create them (it has one for internal tests, though).
I feel like this is a good next step on eventually moving to a separate global cache, if that is the direction the team decides to go.
I see two concerns:
I wonder if this is a good moment to define what is "stable" about the file structure and what is "implementation details"?
Difficult questions to answer. I think it will definitely break a few projects. I just did some searching, and I'm unfortunately finding a large number of projects digging in the deps dir (over 40), which is making me reconsider this strategy.
I'm not sure how to proceed. It would be depressing to have undocumented behavior be perma-stable because people are using it, and it would also be frustrating to break a bunch of projects.
I could change it to hard-link everything into the old deps dir. I worry about filesystems that don't support hard-links, in which case you'd have two copies of everything. I could go with the symlink/directory-junction approach, but that means Windows wouldn't support non-NTFS filesystems.
Another approach would be to segregate build-dependencies, and hard-link the shared ones. That will be more difficult to implement, but might be a good outcome.
Could also just give up on trying to share, and just say build scripts need to be compiled separately. That will make some projects very unhappy, though.
Since we're wondering about possible breakage here, I think it'd be most actionable to go from concrete breakage to develop a plan to roll this out. @ehuss you mentioned that you saw some projects digging around in
I think with that we might be able to develop a strategy to roll this out (for example do we roll it out with warnings on internals? do we have an unstable flag and opt-in for awhile before switching defaults? do we switch immediately, use hard links where possible, and warn on internals? etc/etc).
I did a survey of projects that appear to be accessing inside deps: https://gist.github.com/ehuss/7e3de6b732d276bb9639de11bf911af3
This is not a complete survey, but maybe gives an idea of what is being used. It might be interesting to consider these will also likely break with a global, shared cache. How disruptive does that survey look?
I don't see flags as being a good way to transition. An unstable flag would only be usable on nightly, and I feel it would be unlikely that projects would support both modes simultaneously.
I'm considering a different approach of keeping build dependencies separate, but with a fallback mechanism so that it would use the deps in
Thanks for collecting that info @ehuss!
I continue to pretty strongly feel that we should continue to find a way forward with this change or something pretty similar. I don't want us to get into a situation where we feel shackled to make changes because you're definitely right that doing something like a global cache is going to cause problems regardless, but I feel like those problems are worth the tradeoff of the benefits we'd otherwise get.
Along those lines I see this as a "game" of figuring out how to make the impact as reasonable as possible. A good number of those crates, for example, are largely just interested in the "final artifact", but something that isn't handled well. For example the "final artifact" may include something via
Another category seems to be about executing rustc to do things like more doctests or compiletest. Thankfully though there's a small handful of foundational crates we can fix in one way or another to get them working again.
I wonder if we could take a plan of attack like so?
From there we'd have some clear messaging saying that a change is coming as well as a way to force experimentation with the new scheme (but a way to locally fix builds if necessary). Depending on the fallout and schemes we have for fixing things we could then continue to evaluate. If it breaks everything and everyone's unhappy then we revert the boolean switch and figure out a different strategy. If it breaks lots and there's enthusiasm to fix, then it'll fix itself in time. Or maybe we have very little breakage in practice!
How's that sound?
I think we should do this. The main problem with the current situation is that we don't actually say which things inside