Consolidate C++ conda recipes and add libraft-tests package #641
Conversation
jjacobelli
added
improvement
jjacobelli
force-pushed
the
consolidate
branch
5 times, most recently
from
5f6f2f2
to
e622f73
Compare
|
rerun tests |
jjacobelli
force-pushed
the
consolidate
branch
2 times, most recently
from
92e8ca2
to
8ad57f5
Compare
|
@Ethyling This would be a good PR to activate the staging conda label. Any chance we could do that here? This would allow us to physically compare the deployed artifacts before consuming projects use them. |
Yes, let me update the PR with this |
jjacobelli
force-pushed
the
consolidate
branch
2 times, most recently
from
0bc5365
to
26f4925
Compare
|
@Ethyling I had to back out a couple recently merged PRs at the beginning of the week because they had caused some builds to break downstream (cuml specifically). IIRC, the changes in those PRs were intended to fix the cmake exports from consolidating and the build steps in this PR, right? |
@cjnolet Yes. This PR requires to have a CMake component installation working. Maybe we can try to work on fixing the issue you were facing with cuml? |
jjacobelli
force-pushed
the
consolidate
branch
2 times, most recently
from
fa214ce
to
27f29e5
Compare
jjacobelli
force-pushed
the
consolidate
branch
2 times, most recently
from
8224d06
to
ef6f83a
Compare
|
@vyasr CI in this PR seems to be indicating that the cmake for pylibraft cannot find the distance library even though it's being built and installed earlier in the build script. Have you seen this behavior or know how to handle it? |
jjacobelli
force-pushed
the
consolidate
branch
2 times, most recently
from
425b4bb
to
b63c3cb
Compare
|
No, I haven't seen this issue before. Right now what I see in CI is that it's not finding cuco, and you're saying that implicates the distance lib because cuco should be exported as a global target from there, right? |
|
@vyasr talking with @robertmaynard and @Ethyling offline, we came to the realization that the changes in this PR to consolidate the 3 c++ components into a single build is not going to work so we are going to keep that part the way it is for now. To follow up about my earlier question- the problem is that when we run cmake install on an individual component, it's only going to install that component, so dependencies will not be installed. |
Signed-off-by: Jordan Jacobelli <jjacobelli@nvidia.com>
Signed-off-by: Jordan Jacobelli <jjacobelli@nvidia.com>
Signed-off-by: Jordan Jacobelli <jjacobelli@nvidia.com>
Signed-off-by: Jordan Jacobelli <jjacobelli@nvidia.com>
Signed-off-by: Jordan Jacobelli <jjacobelli@nvidia.com>
|
@gpucibot merge |
…#641) This PR includes the following changes: - Adds a `libraft-tests` package to the `libraft-split` recipe - This is a prerequisite for removing "Project Flash" from our build/CI scripts - The `libraft-tests` package was added as an additional output to the `libraft-split` recipe - Consolidates remaining C++ recipes into `libraft-split` recipe - This gets rid of a lot of duplicate code between the recipes and reduces the number of times we have to call conda build in our CI scripts - Migrate the "from sources" builds done in GPU tests by building packages using conda. This is done for the following reasons: - This is required step to improve the Ops CI/CD setup to a more convenient pipeline - This is required to start using conda compilers and `mamba` to build RAPIDS packages - This prevent us from manually managing and installing the dependencies in GPU job - This ensure the packages can be installed (no conda conflict while installing) - This ensure the tests are running and working against the package content and not the build results. Currently the Python packages are not tested Dependency version specs are now specified in `conda/recipes/libraft/conda_build_config.yaml`. The exception here is the version spec used for cudatoolkit since that comes from an environment variable in the CI process. Authors: - Jordan Jacobelli (https://github.com/Ethyling) Approvers: - Corey J. Nolet (https://github.com/cjnolet) - AJ Schmidt (https://github.com/ajschmidt8) URL: rapidsai#641
Our `devel` Docker containers need to be switched to using `conda` compilers to resolve a linking error. `raft` is in those containers, but hasn't yet been built with `conda` compilers. This PR addresses that. These changes won't cleanly merge into `branch-22.08` unfortunately due to the changes in #641, but we can address that another time. Authors: - AJ Schmidt (https://github.com/ajschmidt8) - Corey J. Nolet (https://github.com/cjnolet) - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Corey J. Nolet (https://github.com/cjnolet)
This PR includes the following changes:
libraft-testspackage to thelibraft-splitrecipelibraft-testspackage was added as an additional output to thelibraft-splitrecipelibraft-splitrecipemambato build RAPIDS packagesDependency version specs are now specified in
conda/recipes/libraft/conda_build_config.yaml. The exception here is the version spec used for cudatoolkit since that comes from an environment variable in the CI process.