Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-47112][INFRA] Write logs into a file in SparkR Windows build #45192

Closed
wants to merge 4 commits into from
Closed

[SPARK-47112][INFRA] Write logs into a file in SparkR Windows build #45192

wants to merge 4 commits into from
+1 −1

Conversation

HyukjinKwon
Copy link
Member

@HyukjinKwon HyukjinKwon commented Feb 21, 2024 •
edited

What changes were proposed in this pull request?

We used to write Log4J logs into target/unit-tests.log instead of console. This seems to be broken in SparkR Windows job. This PR fixes it.

Why are the changes needed?

https://github.com/apache/spark/actions/runs/7977185456/job/21779508822#step:10:89
This write too many logs, and difficult to see the real test cases.

Does this PR introduce any user-facing change?

No, dev-only.

How was this patch tested?

In my fork

Was this patch authored or co-authored using generative AI tooling?

No.

@HyukjinKwon HyukjinKwon marked this pull request as draft February 21, 2024 03:31
@github-actions github-actions bot added the INFRA label Feb 21, 2024
@HyukjinKwon
Copy link
Member Author

It works good: https://github.com/HyukjinKwon/spark/actions/runs/7985005685/job/21802732830

@HyukjinKwon HyukjinKwon marked this pull request as ready for review February 21, 2024 07:14
@HyukjinKwon
Copy link
Member Author

HyukjinKwon commented Feb 21, 2024 •
edited

BTW, I know why it shows an error like:

Traceback (most recent call last):
  File "C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\runpy.py", line 183, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\runpy.py", line 109, in _get_module_details
    __import__(pkg_name)
  File "D:\a\spark\spark\python\lib\pyspark.zip\pyspark\__init__.py", line [53](https://github.com/HyukjinKwon/spark/actions/runs/7985005685/job/21802732830#step:10:54), in <module>
  File "D:\a\spark\spark\python\lib\pyspark.zip\pyspark\rdd.py", line [54](https://github.com/HyukjinKwon/spark/actions/runs/7985005685/job/21802732830#step:10:55), in <module>
  File "D:\a\spark\spark\python\lib\pyspark.zip\pyspark\java_gateway.py", line 33, in <module>
  File "D:\a\spark\spark\python\lib\pyspark.zip\pyspark\serializers.py", line 69, in <module>
  File "D:\a\spark\spark\python\lib\pyspark.zip\pyspark\cloudpickle\__init__.py", line 1, in <module>
  File "D:\a\spark\spark\python\lib\pyspark.zip\pyspark\cloudpickle\cloudpickle.py", line 80, in <module>
ImportError: cannot import name 'CellType' from 'types' (C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\types.py)

which is actually a warning. I will make a separate fix: #45196

@HyukjinKwon
Copy link
Member Author

This should be good to go.

@HyukjinKwon
Copy link
Member Author

HyukjinKwon commented Feb 21, 2024 •
edited

cc @dongjoon-hyun @LuciferYang 🙏

LuciferYang
LuciferYang approved these changes Feb 21, 2024
View reviewed changes
Copy link
Contributor

@LuciferYang LuciferYang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@HyukjinKwon
Copy link
Member Author

Merged to master.

@dongjoon-hyun
Copy link
Member

+1, LGTM.

ericm-db pushed a commit to ericm-db/spark that referenced this pull request Mar 5, 2024
@HyukjinKwon @ericm-db
[SPARK-47112][INFRA] Write logs into a file in SparkR Windows build
17b77e7 
### What changes were proposed in this pull request?

We used to write Log4J logs into `target/unit-tests.log` instead of console. This seems to be broken in SparkR Windows job. This PR fixes it.

### Why are the changes needed?

https://github.com/apache/spark/actions/runs/7977185456/job/21779508822#step:10:89
This write too many logs, and difficult to see the real test cases.

### Does this PR introduce _any_ user-facing change?

No, dev-only.

### How was this patch tested?

In my fork

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#45192 from HyukjinKwon/reduce-logs.

Authored-by: Hyukjin Kwon <gurwls223@apache.org>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@LuciferYang LuciferYang LuciferYang approved these changes

Assignees
No one assigned
Labels
INFRA
Projects
None yet
Milestone
No milestone
3 participants
@HyukjinKwon @dongjoon-hyun @LuciferYang