Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BEAM-7448] Remove redundant windowing information and unused accumulators on Spark runner #8711

Merged
merged 2 commits into from May 29, 2019
Merged

[BEAM-7448] Remove redundant windowing information and unused accumulators on Spark runner #8711

merged 2 commits into from May 29, 2019

Conversation

echauchot
Copy link
Contributor

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Choose reviewer(s) and mention them in a comment (R: @username).
  • Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

Post-Commit Tests Status (on master branch)

Lang SDK Apex Dataflow Flink Gearpump Samza Spark
Go Build Status --- --- --- --- --- ---
Java Build Status Build Status Build Status Build Status
Build Status
Build Status		 Build Status Build Status Build Status
Python Build Status
Build Status		 --- Build Status
Build Status		 Build Status --- --- ---

Pre-Commit Tests Status (on master branch)

--- Java Python Go Website
Non-portable Build Status Build Status Build Status Build Status
Portable --- Build Status --- ---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

@echauchot echauchot requested a review from iemejia May 29, 2019 08:25
@echauchot
Copy link
Contributor Author

Run Spark ValidatesRunner

@echauchot
[BEAM-7448] Remove unneeded external windowedValue in collections for…
fa96678 
… window merging as the windowedValue is already with the elements. Clean related functions and avoid calling so mappers.
@echauchot echauchot force-pushed the spark-clean-windowing-accum branch from 1f81191 to fa96678 Compare May 29, 2019 09:05
@echauchot
Copy link
Contributor Author

Run Spark ValidatesRunner

@iemejia iemejia changed the title Spark: clean unneeded windowing and accumulators [BEAM-7448] Spark: clean unneeded windowing and accumulators May 29, 2019
@iemejia iemejia changed the title [BEAM-7448] Spark: clean unneeded windowing and accumulators [BEAM-7448] Remove redundant windowing information and unused accumulators on Spark runner May 29, 2019
@echauchot
Copy link
Contributor Author

Better PR title, thanks !

iemejia
iemejia approved these changes May 29, 2019
View reviewed changes
Copy link
Member

@iemejia iemejia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Nice fix and nice to see improvements of the Structured Streaming runner getting backported for the benefit of the current users.

@iemejia iemejia merged commit cd8c6ec into apache:master May 29, 2019
@echauchot
Copy link
Contributor Author

Thanks for the review @iemejia !

@echauchot echauchot deleted the spark-clean-windowing-accum branch August 19, 2019 08:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@iemejia iemejia iemejia approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@echauchot @iemejia