[BEAM-8830] Fix flatten in the new spark runner #10293

echauchot · 2019-12-05T10:53:13Z

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang	SDK	Apex	Dataflow	Gearpump	Samza	Spark
Go		---	---	---	---
Java
Python		---		---	---
XLang	---	---	---	---	---	---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website
Non-portable
Portable	---		---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

aromanenko-dev

Thanks, LGTM, just a minor question (see inline).
Also, please squash all commits before merging.

aromanenko-dev · 2019-12-05T13:56:07Z

runners/spark/src/test/resources/log4j-test.properties

Do we really need to have DEBUG log level in tests by default?

in test I think so because it prints Beam DAG and spark DAG

I'm afraid to have too much logs for all Spark tests that could consume too much disk space on Jenkins servers (since Spark is very verbose).

well it is just for the runner not for the spark engine. And the runner puts so few logs (I think only the DAGs).
But anyway, it appears that for some reason, to have the debug logs even in UTests, it requires to change the production conf log file! Spark has always been quite difficult to set up with log4j. Maybe it is due to the test file named log4j-test.properties. Anyway I'll put it back to error until we figure out how to configure it properly.

aromanenko-dev · 2019-12-05T13:56:34Z

runners/spark/src/test/resources/log4j-test.properties

The same question as above

Same answer than above :)

echauchot · 2019-12-05T14:10:12Z

@aromanenko-dev thanks for the prompt review !
I'll squash the commits into the last one but commit 1 has nothing to do with the rest so I'll keep it separate.

echauchot · 2019-12-05T14:14:39Z

done

aromanenko-dev · 2019-12-05T15:23:16Z

Run Spark StructuredStreaming ValidatesRunner

aromanenko-dev · 2019-12-05T15:52:03Z

Some ValidatesRunner tests started to fail even if they passed before

16:24:54 > Task :runners:spark:validatesStructuredStreamingRunnerBatch
16:39:08 
16:39:08 org.apache.beam.sdk.transforms.SplittableDoFnTest > testPairWithIndexWindowedTimestampedBounded FAILED
16:39:08     java.lang.RuntimeException at SplittableDoFnTest.java:220
16:39:08         Caused by: org.apache.spark.SparkException
16:39:08 
16:39:08 org.apache.beam.sdk.transforms.SplittableDoFnTest > testPairWithIndexBasicBounded FAILED
16:39:08     java.lang.IllegalStateException at SplittableDoFnTest.java:153
16:39:08 
16:39:08 org.apache.beam.sdk.transforms.SplittableDoFnTest > testOutputAfterCheckpointBounded FAILED
16:39:08     java.lang.IllegalStateException at SplittableDoFnTest.java:311
16:39:08 
16:39:08 org.apache.beam.sdk.transforms.SplittableDoFnTest > testAdditionalOutputBounded FAILED
16:39:08     java.lang.IllegalStateException at SplittableDoFnTest.java:637
16:39:36 
16:39:36 179 tests completed, 4 failed, 1 skipped

…at sets the schema to binary. Remove test exclusions for now passing tests Set test log level to debug

echauchot · 2019-12-05T16:00:43Z

Some ValidatesRunner tests started to fail even if they passed before

16:24:54 > Task :runners:spark:validatesStructuredStreamingRunnerBatch
16:39:08 
16:39:08 org.apache.beam.sdk.transforms.SplittableDoFnTest > testPairWithIndexWindowedTimestampedBounded FAILED
16:39:08     java.lang.RuntimeException at SplittableDoFnTest.java:220
16:39:08         Caused by: org.apache.spark.SparkException
16:39:08 
16:39:08 org.apache.beam.sdk.transforms.SplittableDoFnTest > testPairWithIndexBasicBounded FAILED
16:39:08     java.lang.IllegalStateException at SplittableDoFnTest.java:153
16:39:08 
16:39:08 org.apache.beam.sdk.transforms.SplittableDoFnTest > testOutputAfterCheckpointBounded FAILED
16:39:08     java.lang.IllegalStateException at SplittableDoFnTest.java:311
16:39:08 
16:39:08 org.apache.beam.sdk.transforms.SplittableDoFnTest > testAdditionalOutputBounded FAILED
16:39:08     java.lang.IllegalStateException at SplittableDoFnTest.java:637
16:39:36 
16:39:36 179 tests completed, 4 failed, 1 skipped

Strange because it does not use emptyDataset (the only thing that has changed). It might be a side effect, I'll take a look

echauchot · 2019-12-05T16:18:55Z

Some ValidatesRunner tests started to fail even if they passed before

16:24:54 > Task :runners:spark:validatesStructuredStreamingRunnerBatch
16:39:08 
16:39:08 org.apache.beam.sdk.transforms.SplittableDoFnTest > testPairWithIndexWindowedTimestampedBounded FAILED
16:39:08     java.lang.RuntimeException at SplittableDoFnTest.java:220
16:39:08         Caused by: org.apache.spark.SparkException
16:39:08 
16:39:08 org.apache.beam.sdk.transforms.SplittableDoFnTest > testPairWithIndexBasicBounded FAILED
16:39:08     java.lang.IllegalStateException at SplittableDoFnTest.java:153
16:39:08 
16:39:08 org.apache.beam.sdk.transforms.SplittableDoFnTest > testOutputAfterCheckpointBounded FAILED
16:39:08     java.lang.IllegalStateException at SplittableDoFnTest.java:311
16:39:08 
16:39:08 org.apache.beam.sdk.transforms.SplittableDoFnTest > testAdditionalOutputBounded FAILED
16:39:08     java.lang.IllegalStateException at SplittableDoFnTest.java:637
16:39:36 
16:39:36 179 tests completed, 4 failed, 1 skipped

Strange because it does not use emptyDataset (the only thing that has changed). It might be a side effect, I'll take a look

The test passes on my machine. Maybe an unrelated flaky test

echauchot · 2019-12-05T16:19:11Z

Run Spark StructuredStreaming ValidatesRunner

echauchot · 2019-12-06T10:21:05Z

@aromanenko-dev : OOM on last run of VR tests. one splittableDoFn test is taking 5 min I guess it is not normal. But anyway, SDF is not implemented on the runner. I think we should exclude all SDF tests. WDYT ?

echauchot · 2019-12-09T09:31:52Z

@aromanenko-dev with @iemejia last comment I just reduced the SDF tests exclusion scope to only unbounded tests and testPairWithIndexWindowedTimestampedBounded that took 5 min in last run and that might be the case of the last OOM. Can we merge on green light ?

aromanenko-dev · 2019-12-09T09:42:01Z

Run Spark StructuredStreaming ValidatesRunner

aromanenko-dev · 2019-12-09T14:49:14Z

Run Java PreCommit

echauchot · 2019-12-10T09:13:52Z

@aromanenko-dev timeout in un unrelated portability test.

echauchot · 2019-12-10T09:14:00Z

Run Java PreCommit

echauchot · 2019-12-11T10:14:34Z

@aromanenko-dev all the tests pass, as you gave LGTM, I'll auto-merge. Thanks for the review !

echauchot · 2019-12-11T10:16:04Z

Run Spark StructuredStreaming ValidatesRunner

echauchot · 2019-12-11T10:16:30Z

just reworded the last commit message

echauchot · 2019-12-11T14:10:52Z

Run Java PreCommit

[BEAM-8830] flatten: better detect empty collections

4dae3ae

echauchot requested a review from aromanenko-dev December 5, 2019 10:53

aromanenko-dev approved these changes Dec 5, 2019

View reviewed changes

echauchot force-pushed the BEAM-8830-flattens-spark branch from 33ae949 to 41f3aa6 Compare December 5, 2019 14:13

[BEAM-8830] Fix empty dataset creation: use the beam coder wrapper th…

62bd3ff

…at sets the schema to binary. Remove test exclusions for now passing tests Set test log level to debug

echauchot force-pushed the BEAM-8830-flattens-spark branch from 41f3aa6 to 62bd3ff Compare December 5, 2019 15:56

echauchot force-pushed the BEAM-8830-flattens-spark branch 2 times, most recently from 7b6f604 to 7dd8354 Compare December 9, 2019 09:30

echauchot force-pushed the BEAM-8830-flattens-spark branch from 7dd8354 to f67bc41 Compare December 9, 2019 09:36

[BEAM-8830] Exclude failing SDF tests

a9a68a8

echauchot force-pushed the BEAM-8830-flattens-spark branch from f67bc41 to a9a68a8 Compare December 11, 2019 10:15

echauchot merged commit 2080520 into apache:master Dec 11, 2019

echauchot deleted the BEAM-8830-flattens-spark branch December 11, 2019 15:52

[BEAM-8830] Fix flatten in the new spark runner #10293

[BEAM-8830] Fix flatten in the new spark runner #10293

Uh oh!

Conversation

echauchot commented Dec 5, 2019

Post-Commit Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

Uh oh!

aromanenko-dev left a comment

Choose a reason for hiding this comment

Uh oh!

aromanenko-dev Dec 5, 2019

Choose a reason for hiding this comment

Uh oh!

echauchot Dec 5, 2019

Choose a reason for hiding this comment

Uh oh!

aromanenko-dev Dec 5, 2019

Choose a reason for hiding this comment

Uh oh!

echauchot Dec 5, 2019

Choose a reason for hiding this comment

Uh oh!

aromanenko-dev Dec 5, 2019

Choose a reason for hiding this comment

Uh oh!

echauchot Dec 5, 2019

Choose a reason for hiding this comment

Uh oh!

echauchot commented Dec 5, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

echauchot commented Dec 5, 2019

Uh oh!

aromanenko-dev commented Dec 5, 2019

Uh oh!

aromanenko-dev commented Dec 5, 2019

Uh oh!

echauchot commented Dec 5, 2019

Uh oh!

echauchot commented Dec 5, 2019

Uh oh!

echauchot commented Dec 5, 2019

Uh oh!

echauchot commented Dec 6, 2019

Uh oh!

echauchot commented Dec 9, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aromanenko-dev commented Dec 9, 2019

Uh oh!

aromanenko-dev commented Dec 9, 2019

Uh oh!

echauchot commented Dec 10, 2019

Uh oh!

echauchot commented Dec 10, 2019

Uh oh!

echauchot commented Dec 11, 2019

Uh oh!

echauchot commented Dec 11, 2019

Uh oh!

echauchot commented Dec 11, 2019

Uh oh!

echauchot commented Dec 11, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

echauchot commented Dec 5, 2019 •

edited

Loading

echauchot commented Dec 9, 2019 •

edited

Loading