[SPARK-33863][PYTHON] Respect session timezone in udf workers #53161
Conversation
|
@cloud-fan , @ueshin , @zhengruifeng we've discussed this but did not reach to a conclusion. I had a draft here and a few questions. We probably need to further discuss about the implementation and implication. |
gaogaotiantian
changed the title
zhengruifeng
left a comment
zhengruifeng
left a comment
There was a problem hiding this comment.
It is a behavior change, I think we need a flag for it.
Also we need new tests in test_udf.py
|
|
||
| # Use the local timezone to convert the timestamp | ||
| tz = datetime.datetime.now().astimezone().tzinfo | ||
| tzname = os.environ.get("SPARK_SESSION_LOCAL_TIMEZONE", None) |
There was a problem hiding this comment.
To confirm, we will hit this branch for every udf execution, not just once per python worker initialization, right?
There was a problem hiding this comment.
That's correct, but it doesn't seem like spark.session.conf.set("spark.sql.session.timeZone") impacts the result. This only works when I create the session with the conf. Any ideas? I can investigate if that's an issue or we want to understand it. I just thought you might understand immediately.
There was a problem hiding this comment.
@gaogaotiantian We can use the same way as the other configs to get the runtime config, like hideTraceback or simplifiedTraceback above. Please take a look at PythonRunner and its subclasses.
There was a problem hiding this comment.
Ah, so basically overwrite this for every subclassed worker?
There was a problem hiding this comment.
Yes. Also if we have a flag, the subclasses should decide whether it returns the session local timezone or None.
There was a problem hiding this comment.
So the flag should be a conf in the same level as session local timezone? Or just Python udf level? Will it be default to the original behavior or the new behavior?
There was a problem hiding this comment.
Yes, the flag should be the same level as the session local timezone, a runtime conf in SQLConf.
It can be enabled by default, but when disabled, the behavior should be the original behavior.
WDYT? cc @zhengruifeng @HyukjinKwon
There was a problem hiding this comment.
The Arrow-based UDFs already handles the session local timezone, so it may be ok to just update BasePythonUDFRunner.envVars to have the env var there instead of PythonRunner?
| envVars.put("SPARK_SIMPLIFIED_TRACEBACK", "1") | ||
| } | ||
| if (sessionLocalTimeZone.isDefined) { | ||
| envVars.put("SPARK_SESSION_LOCAL_TIMEZONE", sessionLocalTimeZone.get) |
There was a problem hiding this comment.
for arrow-based UDFs, sessionLocalTimeZone is actually already passed to the python side
However this workerConf is not available in vanilla Python UDF, probably we can consider supporting it in vanilla Python UDF in the future. also cc @HeartSaVioR
There was a problem hiding this comment.
yea it's better to pass the configs via a proper protocol, instead of system variables. But it's already the case for vanilla python runner and I think it's fine to follow it.
|
can we add a test which sets |
4 participants
What changes were proposed in this pull request?
Respect
spark.sql.session.timeZonein UDF workers.This is discussed in #52980 but we decided to move it to a separate PR. There are still open questions left
spark.conf.set. I believe this is trivial to people who are familiar with the configs so I did not investigate too much.Why are the changes needed?
Relying on the timezone of local machine does not make any sense.
Does this PR introduce any user-facing change?
Yes. The UDF behavior regarding to timestamps and timezones will be changed.
How was this patch tested?
Manually
Was this patch authored or co-authored using generative AI tooling?
No