Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-40851][INFRA ][SQL][TESTS][3.2] Make GA run successfully with the latest Java 8/11/17 #38365

Closed
wants to merge 1 commit into from
Closed

[SPARK-40851][INFRA ][SQL][TESTS][3.2] Make GA run successfully with the latest Java 8/11/17 #38365

wants to merge 1 commit into from

Conversation

LuciferYang
Copy link
Contributor

What changes were proposed in this pull request?

The main change of this pr as follows:

  • Replace Antarctica/Vostok to Asia/Urumqi in Spark code
  • Replace Europe/Amsterdam to Europe/Brussels in Spark code
  • Regenerate gregorian-julian-rebase-micros.json using generate 'gregorian-julian-rebase-micros.json' in RebaseDateTimeSuite with Java 8u352
  • Regenerate julian-gregorian-rebase-micros.json using generate 'julian-gregorian-rebase-micros.json' in RebaseDateTimeSuite with Java 8u352

Why are the changes needed?

Make GA run successfully with the latest Java 8/11/17:

  • Java 8u352
  • Java 11.0.17
  • Java 17.0.5

Does this PR introduce any user-facing change?

No

How was this patch tested?

  • Pass GitHub Actions
  • Manual test: the following commands can test pass with Java 8u345, 8u352, 11.0.16, 11.0.17, 17.0.4 and 17.0.5
    • build/sbt "catalyst/test"
    • build/sbt "sql/testOnly *SQLQueryTestSuite -- -t \"timestamp-ltz.sql\""
    • build/sbt "sql/testOnly *SQLQueryTestSuite -- -t \"timestamp-ntz.sql\""

@LuciferYang
[SPARK-40851][INFRA ][SQL][TESTS] Make GA run successfully with the l…
2cd6574 
…atest Java 8/11/17

The main change of this pr as follows:

- Replace `Antarctica/Vostok` to `Asia/Urumqi` in Spark code
- Replace `Europe/Amsterdam` to `Europe/Brussels` in Spark code
- Regenerate `gregorian-julian-rebase-micros.json` using generate 'gregorian-julian-rebase-micros.json' in `RebaseDateTimeSuite` with Java 8u352
- Regenerate `julian-gregorian-rebase-micros.json` using generate 'julian-gregorian-rebase-micros.json' in RebaseDateTimeSuite with Java 8u352
- Revert change of SPARK-40846

Make GA run successfully with the latest Java 8/11/17:

- Java 8u352
- Java 11.0.17
- Java 17.0.5

No

- Pass GitHub Actions
- Manual test: the following commands can test pass with Java 8u345, 8u352, 11.0.16, 11.0.17, 17.0.4 and 17.0.5
   - `build/sbt "catalyst/test"`
   - `build/sbt "sql/testOnly *SQLQueryTestSuite -- -t \"timestamp-ltz.sql\""`
   - `build/sbt "sql/testOnly *SQLQueryTestSuite -- -t \"timestamp-ntz.sql\""`

Closes apache#38317 from LuciferYang/SPARK-40851.

Lead-authored-by: yangjie01 <yangjie01@baidu.com>
Co-authored-by: YangJie <yangjie01@baidu.com>
Signed-off-by: Max Gekk <max.gekk@gmail.com>
@LuciferYang LuciferYang mentioned this pull request Oct 24, 2022
Closed
@github-actions github-actions bot added the SQL label Oct 24, 2022
@LuciferYang
Copy link
Contributor Author

waiting ci

@LuciferYang
Copy link
Contributor Author

https://github.com/LuciferYang/spark/actions/runs/3309640149/jobs/5464365656

ImportError: Missing optional dependency 'Jinja2'. DataFrame.style requires jinja2. Use pip or conda to install Jinja2.

----------------------------------------------------------------------
Ran 153 tests in 332.174s

FAILED (errors=1, skipped=3)

Generating XML reports...

Had test failures in pyspark.pandas.tests.test_dataframe with python3.6; see logs.
Error:  running /__w/spark/spark/python/run-tests --modules=pyspark-pandas-slow --parallelism=1 --python-executable=python3.6,python3.9,pypy3 ; received return code 255
Error: Process completed with exit code 19.

https://github.com/LuciferYang/spark/actions/runs/3309640149/jobs/5464366180

starting pycodestyle test...
pycodestyle checks failed:
./python/pyspark/ml/tests/test_tuning.py:203:23: E275 missing whitespace after keyword
./python/pyspark/broadcast.py:102:23: E275 missing whitespace after keyword
./python/pyspark/sql/pandas/conversion.py:174:20: E275 missing whitespace after keyword
./python/pyspark/sql/readwriter.py:662:60: E275 missing whitespace after keyword
./python/pyspark/sql/readwriter.py:694:60: E275 missing whitespace after keyword
./python/pyspark/sql/tests/test_pandas_udf_grouped_agg.py:512:[15](https://github.com/LuciferYang/spark/actions/runs/3309640149/jobs/5464366180#step:14:16): E275 missing whitespace after keyword
./python/pyspark/sql/tests/test_context.py:[16](https://github.com/LuciferYang/spark/actions/runs/3309640149/jobs/5464366180#step:14:17)6:[19](https://github.com/LuciferYang/spark/actions/runs/3309640149/jobs/5464366180#step:14:20): E275 missing whitespace after keyword
./python/pyspark/worker.py:583:27: E275 missing whitespace after keyword

@HyukjinKwon Known issue?

@HyukjinKwon
Copy link
Member

Yeah, let's just get this in first.

@itholic mind taking a look when you find some time? The tests fail in branch-3.2.

@HyukjinKwon
Copy link
Member

Merged to branch-3.2.

HyukjinKwon pushed a commit that referenced this pull request Oct 24, 2022
@LuciferYang @HyukjinKwon
[SPARK-40851][INFRA ][SQL][TESTS][3.2] Make GA run successfully with …
8448038 
…the latest Java 8/11/17

### What changes were proposed in this pull request?
The main change of this pr as follows:

- Replace `Antarctica/Vostok` to `Asia/Urumqi` in Spark code
- Replace `Europe/Amsterdam` to `Europe/Brussels` in Spark code
- Regenerate `gregorian-julian-rebase-micros.json` using generate 'gregorian-julian-rebase-micros.json' in `RebaseDateTimeSuite` with Java 8u352
- Regenerate `julian-gregorian-rebase-micros.json` using generate 'julian-gregorian-rebase-micros.json' in RebaseDateTimeSuite with Java 8u352

### Why are the changes needed?
Make GA run successfully with the latest Java 8/11/17:

- Java 8u352
- Java 11.0.17
- Java 17.0.5

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?

- Pass GitHub Actions
- Manual test: the following commands can test pass with Java 8u345, 8u352, 11.0.16, 11.0.17, 17.0.4 and 17.0.5
   - `build/sbt "catalyst/test"`
   - `build/sbt "sql/testOnly *SQLQueryTestSuite -- -t \"timestamp-ltz.sql\""`
   - `build/sbt "sql/testOnly *SQLQueryTestSuite -- -t \"timestamp-ntz.sql\""`

Closes #38365 from LuciferYang/SPARK-40851-32.

Authored-by: yangjie01 <yangjie01@baidu.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
@LuciferYang
Copy link
Contributor Author

Thanks @HyukjinKwon @MaxGekk

sunchao pushed a commit to sunchao/spark that referenced this pull request Jun 2, 2023
@LuciferYang @huaxingao
[SPARK-40851][INFRA ][SQL][TESTS][3.2] Make GA run successfully with …
bcae416 
…the latest Java 8/11/17

### What changes were proposed in this pull request?
The main change of this pr as follows:

- Replace `Antarctica/Vostok` to `Asia/Urumqi` in Spark code
- Replace `Europe/Amsterdam` to `Europe/Brussels` in Spark code
- Regenerate `gregorian-julian-rebase-micros.json` using generate 'gregorian-julian-rebase-micros.json' in `RebaseDateTimeSuite` with Java 8u352
- Regenerate `julian-gregorian-rebase-micros.json` using generate 'julian-gregorian-rebase-micros.json' in RebaseDateTimeSuite with Java 8u352

### Why are the changes needed?
Make GA run successfully with the latest Java 8/11/17:

- Java 8u352
- Java 11.0.17
- Java 17.0.5

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?

- Pass GitHub Actions
- Manual test: the following commands can test pass with Java 8u345, 8u352, 11.0.16, 11.0.17, 17.0.4 and 17.0.5
   - `build/sbt "catalyst/test"`
   - `build/sbt "sql/testOnly *SQLQueryTestSuite -- -t \"timestamp-ltz.sql\""`
   - `build/sbt "sql/testOnly *SQLQueryTestSuite -- -t \"timestamp-ntz.sql\""`

Closes apache#38365 from LuciferYang/SPARK-40851-32.

Authored-by: yangjie01 <yangjie01@baidu.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
(cherry picked from commit 8448038)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@MaxGekk MaxGekk MaxGekk approved these changes

@HyukjinKwon HyukjinKwon HyukjinKwon approved these changes

Assignees
No one assigned
Labels
SQL
Projects
None yet
Milestone
No milestone
3 participants
@LuciferYang @HyukjinKwon @MaxGekk