[SPARK-51278][PYTHON] Use appropriate structure of JSON format for `PySparkLogger` #50038

itholic · 2025-02-21T07:00:14Z

What changes were proposed in this pull request?

This PR proposes to use appropriate structure of JSON format for PySparkLogger by updating JSONFormatter.

Why are the changes needed?

To align the log structure with JVM side. Currently the "stacktrace" does not align.

Does this PR introduce any user-facing change?

The output JSON structure will include "stacktrace" in correct format:

Before

{
  "ts": "2025-02-21 15:40:52,043",
  "level": "ERROR",
  "logger": "TestLogger",
  "msg": "Exception occurred",
  "context": {"user": "test_user_stacktrace"},
  "exception": {
    "class": "ValueError",
    "msg": "Test Exception",
    "stacktrace": [
      "Traceback (most recent call last):", "  File \"/spark/python/pyspark/logger/tests/test_logger.py\", line 115, in test_log_exception_with_stacktrace", "    raise ValueError(\"Test Exception\")", "ValueError: Test Exception"
    ]
  }
}

After

{
  "ts": "2025-02-21 15:40:52,043",
  "level": "ERROR",
  "logger": "TestLogger",
  "msg": "Exception occurred",
  "context": {"user": "test_user_stacktrace"},
  "exception": {
    "class": "ValueError",
    "msg": "Test Exception",
    "stacktrace": [
      {
        "class": null,
        "method": "test_log_exception_with_stacktrace",
        "file": "/spark/python/pyspark/logger/tests/test_logger.py",
        "line": "115"
      }
    ]
  }
}

How was this patch tested?

Added related UT

Was this patch authored or co-authored using generative AI tooling?

No

…SparkLogger

ueshin

Otherwise, LGTM, pending tests.

python/pyspark/logger/logger.py

Co-authored-by: Takuya UESHIN <ueshin@happy-camper.st>

…to fix_json_format

…ySparkLogger` ### What changes were proposed in this pull request? This PR proposes to use appropriate structure of JSON format for `PySparkLogger` by updating `JSONFormatter`. ### Why are the changes needed? To align the log structure with JVM side. Currently the "stacktrace" does not align. ### Does this PR introduce _any_ user-facing change? The output JSON structure will include "stacktrace" in correct format: **Before** ```json { "ts": "2025-02-21 15:40:52,043", "level": "ERROR", "logger": "TestLogger", "msg": "Exception occurred", "context": {"user": "test_user_stacktrace"}, "exception": { "class": "ValueError", "msg": "Test Exception", "stacktrace": [ "Traceback (most recent call last):", " File \"/spark/python/pyspark/logger/tests/test_logger.py\", line 115, in test_log_exception_with_stacktrace", " raise ValueError(\"Test Exception\")", "ValueError: Test Exception" ] } } ``` **After** ```json { "ts": "2025-02-21 15:40:52,043", "level": "ERROR", "logger": "TestLogger", "msg": "Exception occurred", "context": {"user": "test_user_stacktrace"}, "exception": { "class": "ValueError", "msg": "Test Exception", "stacktrace": [ { "class": null, "method": "test_log_exception_with_stacktrace", "file": "/spark/python/pyspark/logger/tests/test_logger.py", "line": "115" } ] } } ``` ### How was this patch tested? Added related UT ### Was this patch authored or co-authored using generative AI tooling? No Closes #50038 from itholic/fix_json_format. Lead-authored-by: Haejoon Lee <haejoon.lee@databricks.com> Co-authored-by: Hyukjin Kwon <gurwls223@gmail.com> Signed-off-by: Haejoon Lee <haejoon.lee@databricks.com> (cherry picked from commit 67a337e) Signed-off-by: Haejoon Lee <haejoon.lee@databricks.com>

itholic · 2025-02-24T09:59:02Z

Merged to master and branch-4.0.
Thanks @ueshin @HyukjinKwon for the review!

### What changes were proposed in this pull request? This PR followups #50038 to update documentation following the fixed JSON format. ### Why are the changes needed? To match the docs with real logging format ### Does this PR introduce _any_ user-facing change? No API changes, but the user-facing documentation updated. ### How was this patch tested? CI ### Was this patch authored or co-authored using generative AI tooling? No. Closes #50100 from itholic/json_format_followup. Authored-by: Haejoon Lee <haejoon.lee@databricks.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>

### What changes were proposed in this pull request? This PR followups #50038 to update documentation following the fixed JSON format. ### Why are the changes needed? To match the docs with real logging format ### Does this PR introduce _any_ user-facing change? No API changes, but the user-facing documentation updated. ### How was this patch tested? CI ### Was this patch authored or co-authored using generative AI tooling? No. Closes #50100 from itholic/json_format_followup. Authored-by: Haejoon Lee <haejoon.lee@databricks.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org> (cherry picked from commit b05f18b) Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>

…ySparkLogger` ### What changes were proposed in this pull request? This PR proposes to use appropriate structure of JSON format for `PySparkLogger` by updating `JSONFormatter`. ### Why are the changes needed? To align the log structure with JVM side. Currently the "stacktrace" does not align. ### Does this PR introduce _any_ user-facing change? The output JSON structure will include "stacktrace" in correct format: **Before** ```json { "ts": "2025-02-21 15:40:52,043", "level": "ERROR", "logger": "TestLogger", "msg": "Exception occurred", "context": {"user": "test_user_stacktrace"}, "exception": { "class": "ValueError", "msg": "Test Exception", "stacktrace": [ "Traceback (most recent call last):", " File \"/spark/python/pyspark/logger/tests/test_logger.py\", line 115, in test_log_exception_with_stacktrace", " raise ValueError(\"Test Exception\")", "ValueError: Test Exception" ] } } ``` **After** ```json { "ts": "2025-02-21 15:40:52,043", "level": "ERROR", "logger": "TestLogger", "msg": "Exception occurred", "context": {"user": "test_user_stacktrace"}, "exception": { "class": "ValueError", "msg": "Test Exception", "stacktrace": [ { "class": null, "method": "test_log_exception_with_stacktrace", "file": "/spark/python/pyspark/logger/tests/test_logger.py", "line": "115" } ] } } ``` ### How was this patch tested? Added related UT ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#50038 from itholic/fix_json_format. Lead-authored-by: Haejoon Lee <haejoon.lee@databricks.com> Co-authored-by: Hyukjin Kwon <gurwls223@gmail.com> Signed-off-by: Haejoon Lee <haejoon.lee@databricks.com>

### What changes were proposed in this pull request? This PR followups apache#50038 to update documentation following the fixed JSON format. ### Why are the changes needed? To match the docs with real logging format ### Does this PR introduce _any_ user-facing change? No API changes, but the user-facing documentation updated. ### How was this patch tested? CI ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#50100 from itholic/json_format_followup. Authored-by: Haejoon Lee <haejoon.lee@databricks.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>

[SPARK-51278][PYTHON] Use appropriate structure of JSON format for Py…

13dd2b5

…SparkLogger

itholic requested a review from ueshin February 21, 2025 07:00

github-actions bot added the PYTHON label Feb 21, 2025

remove testing-purpose code

01aaa2b

ueshin approved these changes Feb 21, 2025

View reviewed changes

python/pyspark/logger/logger.py Outdated Show resolved Hide resolved

python/pyspark/logger/logger.py Outdated Show resolved Hide resolved

ueshin requested review from HyukjinKwon and allisonwang-db February 21, 2025 19:55

HyukjinKwon approved these changes Feb 23, 2025

View reviewed changes

HyukjinKwon and others added 5 commits February 23, 2025 10:26

Apply suggestions from code review

45436bf

Co-authored-by: Takuya UESHIN <ueshin@happy-camper.st>

Applied the suggestions

cce1f5f

Merge branch 'fix_json_format' of https://github.com/itholic/spark in…

023e43e

…to fix_json_format

Resolved conflicts

32da6f2

Updated related tests

3a89b7f

itholic closed this in 67a337e Feb 24, 2025

itholic mentioned this pull request Feb 27, 2025

[SPARK-51278][FOLLOWUP][DOCS] Update JSON format from documentation #50100

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SPARK-51278][PYTHON] Use appropriate structure of JSON format for `PySparkLogger` #50038

[SPARK-51278][PYTHON] Use appropriate structure of JSON format for `PySparkLogger` #50038

Uh oh!

itholic commented Feb 21, 2025 •

edited

Loading

Uh oh!

ueshin left a comment

Uh oh!

Uh oh!

Uh oh!

itholic commented Feb 24, 2025

Uh oh!

Uh oh!

[SPARK-51278][PYTHON] Use appropriate structure of JSON format for PySparkLogger #50038

[SPARK-51278][PYTHON] Use appropriate structure of JSON format for PySparkLogger #50038

Uh oh!

Conversation

itholic commented Feb 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

ueshin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

itholic commented Feb 24, 2025

Uh oh!

Uh oh!

[SPARK-51278][PYTHON] Use appropriate structure of JSON format for `PySparkLogger` #50038

[SPARK-51278][PYTHON] Use appropriate structure of JSON format for `PySparkLogger` #50038

itholic commented Feb 21, 2025 •

edited

Loading