[SPARK-50427][CONNECT][PYTHON] Expose configure_logging as a public API #48802
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nija-at
changed the title
nija-at
force-pushed
the
expose-log-method
branch
from
e37431c to
de3ec77
Compare
HyukjinKwon
reviewed
Nov 26, 2024
HyukjinKwon
approved these changes
Nov 26, 2024
Member
|
Merged to master. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Expose
configure_loggingas a public API that can be usedto configure the log level for Pyspark connect component.
Why are the changes needed?
We currently offer the mechanism to configure the connect-specific logger
based on the environment variable
SPARK_CONNECT_LOG_LEVEL.The logger is configured once at the the time of "module load". In some cases,
Python frameworks (eg. IPythonKernel) can modify the Python log level after the
fact leading to unintended log output.
There is no good way to restore the logger to restore its previous functionality
to honor the environment variable configured.
Does this PR introduce any user-facing change?
Yes.
Provide a new API
configure_loggingin the modulepyspark.sql.connect.logging.How was this patch tested?
Local testing by calling
configure_loggingwith different log levels.Further tested with IPythonKernel instance which changes the log level
and confirmed that calling this API during app startup fixes it back to the
correct log level.
Was this patch authored or co-authored using generative AI tooling?
No.