-
Notifications
You must be signed in to change notification settings - Fork 879
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add logging to libcudf #12637
Add logging to libcudf #12637
Conversation
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## branch-23.04 #12637 +/- ##
===============================================
Coverage ? 43.53%
===============================================
Files ? 158
Lines ? 24868
Branches ? 0
===============================================
Hits ? 10827
Misses ? 14041
Partials ? 0 Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report in Codecov by Sentry. |
FYI, we need to fix some issues with the way rmm is packaging/using spdlog. I have a PR here that moves in that direction, but more changes may be needed. Please don’t merge this PR until I can confirm that rmm is fixed up. See: rapidsai/rmm#1177 |
You may want to make the design a bit more flexible such that it doesn't assume one is always logging to a file. @jlowe or @revans2 can correct me if I'm wrong, but iirc Spark wants to be able to log to something like cout/cerr. In the RMM logging adaptor we enabled this by adding a constructor that takes an arbitrary |
We'd like the ability to glue the cudf logging into the existing Spark/Java logging mechanisms, e.g.: log4j. Being able to specify a custom spdlog sink, ideally at runtime, would give us the flexibility to do that and more. |
The singleton spdlog object is accessible via |
One caveat: the API that changes the sink is not thread-safe. @jlowe is this an issue?
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This PR appears to rely on rmm's spdlog/fmt transitively. I think we want to update our conda recipes and/or CPM logic to explicitly depend on spdlog/fmt? What should our approach be for downstream packages like cudf @robertmaynard ? I'd think the conda packages should be explicit dependencies but I'm not sure how to handle the CMake/CPM side.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The fmt/spdlog conda packaging looks good. The fmt/spdlog CMake packaging looks good to me also, but I would like a second eye from @robertmaynard before merging.
Looks good to me |
Has anything happened with this? I'm curious how urgent it is to change RMM's spdlog dependency to use the pre-compiled library. |
I tried using the build metrics to evaluate the impact of spdlog dependency. There's a measurable impact on .so size, but compilation time seems hard to compare because of caching. The time difference does not look alarming given the difference between cache hit rates. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Couple of minor things but looks basically ready to go.
cpp/CMakeLists.txt
Outdated
@@ -676,8 +678,11 @@ if(NOT USE_NVTX) | |||
target_compile_definitions(cudf PUBLIC NVTX_DISABLE) | |||
endif() | |||
|
|||
# Define RMM logging level | |||
target_compile_definitions(cudf PUBLIC "RMM_LOGGING_LEVEL=LIBCUDF_LOGGING_LEVEL") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think you would run into any issues. RMM only sets this for compiled components i.e. its tests and benchmarks. The main rmm code is header-only so there are no compiled components where SPDLOG_ACTIVE_LEVEL
would be set to conflict with whatever libcudf sets.
Co-authored-by: Vyas Ramasubramani <vyas.ramasubramani@gmail.com>
Co-authored-by: Vyas Ramasubramani <vyas.ramasubramani@gmail.com>
/merge |
Description
Adds a global
spdlog
logger.Logging should be done through
CUDF_LOG_*
macros declared in detailnamespace
.Logging level and output file path can be set via environment variables.
Logging can also be configured directly through the singleton object using spdlog APIs (should only be used outside of libcudf).
Added sample (useful) logging to
getenv_or
.Fixed a bug discovered through the log :) Environment variable was being read in each
hostdevice_vector
constructor call.Checklist