How to manage large build dependencies? #5703
Replies: 1 comment
-
How big is "very large"? In our environment we have extremely large build artifacts - many gigabytes. We use rsync to do incremental archiving from the build machine to an archive machine, where we store the artifacts by revision number, incrementally (like time machine does). Downstream test machines then are triggered with the appropriate archive key (revision number) which they use with rsync to incrementally sync the artifacts. Given that we have a broad project with widely changing artifacts this works for us. Some revisions cause very small changes, while others have broad changes. I wouldn't say this is instant, but we can get 20s incremental syncs on our many gigabyte project. Our downstream tests do no building at all, instead relying on the build artifacts. It operates outside of buildbot as we didn't think it was really a built-in responsibility. Would something like this help? |
Beta Was this translation helpful? Give feedback.
-
We use buildbot to test a library that depends on LLVM for code generation. We support building against three LLVM versions, so we need to build each LLVM version regularly while testing (trunk changes the most often, but we need to be on top of backported patches, too). This must happen on every worker for every supported configuration.
However, to build LLVM in-line (ie. as additional steps) with our own builds is untenable because it takes an extremely long time (slowing our workflow) and the artifacts are very large (making caching too expensive).
So what we do instead is create—for each LLVM branch, build configuration, and buildbot worker—a builder and nightly scheduler pair that runs the build and puts the artifacts at a predictable location on the worker.
But this is messy for several reasons. It clutters the UI with combinatorially many force schedulers and builders. Buildbot doesn't know about the relationship between our own builds and the LLVM dependency builds; even though we achieve synchronization through manual locks, spinning up a new worker requires manually starting LLVM builds to get the dependencies ready. Work is duplicated across replicated workers that could share those artifacts, etc.
Is there a better way to do this?
Beta Was this translation helpful? Give feedback.
All reactions