Rw beam 9557 #11253
Closed
Rw beam 9557 #11253
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Add WithKeys lesson in Python Katas * Add Windowing Adding Timestamp section using ParDo lesson * Fix window import statement * Add Windowing Fixed Time Window lesson * Add WithKeys lesson in Python Katas * Add Windowing Adding Timestamp section using ParDo lesson * Fix window import statement * Add Windowing Fixed Time Window lesson * Update course on Stepik * Add missing license header
[BEAM-5898] Upgrading gRPC to 1.27
…EAD by default in SQL
…tegration with DirectRunner [BEAM-8335] TeststreamService integration with DirectRunner
Updating BigQuery client APIs
…dencies (apache#11093) * Exclude signature files from test dependencies * Apply suggestions from code review Thanks Luke! Co-Authored-By: Lukasz Cwik <lcwik@google.com> Co-authored-by: Lukasz Cwik <lcwik@google.com>
…le with Flink 1.10
…get and Make FlinkRunner compatible with Flink 1.10
[BEAM-8681] Upgrading grpc-google-cloud-pubsub-v1 to 1.85.1
The cleanup code in DefaultJobBundleFactory and its RemoteEnvironments may leak resources. This is especially a concern when the execution engines reuses the same JVM or underlying machines for multiple runs of a pipeline. Exceptions encountered during cleanup should not lead to aborting the cleanup procedure. Not all code handles this correctly. We should also ensure that the cleanup succeeds even if the runner does not properly close the bundle, e.g. when a exception occurs during closing the bundle.
[BEAM-9074] Update Python roadmap for 2.7 eol
…d_gbk_side_input [BEAM-9402] Remove options overwrite
Change-Id: Ib61aa3fac53d9109178744e11eeebe5c5da0929c
…ndleFactory and ProcessEnvironment
…Cache to subclass the CacheManager [BEAM-8335] Modify the StreamingCache to subclass the CacheManager
…ache#11106) * Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head (apache#11106) * Adding flink 1.10 to pipelineoptions
[BEAM-9731] Add SideInputLoadTest implementation
… holds Benchmarks have shown that the minimum output timestamp computation over all timers is very expensive because we have to iterate over all keys in the state backend. This is especially costly for RocksDB. This change introduces a cache which stores and updates the timer output timestamps such that the minimum can be retrieved efficiently. Only at startup the cache is populated from the state backend. This should give us roughly the same performance as before introducing output timestamps for timers. The following have been run with RocksDB: Nexmark Query 12 before: ``` Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline) 0000 105.3 950.0 9039 0001 2.2 4466.3 196 ``` Nexmark query 12 with this change: ``` Performance: Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline) 0000 6.0 16730.8 1919 0001 0.8 12391.6 196 ``` For more details please see BEAM-9566.
Reenable test_streaming_wordcount_it Rename the test in streaming_wordcount_debugging_it_test.py
[BEAM-9340] Validate pipeline requirements in PipelineValidator.
…ue for output timestamp watermark holds
…nvironment capabilities.
…ordcount_debugging_it
pyarrow was released on Feb 2020.
[BEAM-9562] Remove TimerSpec from Proto
…Option to setOption on Option builder [BEAM-9605] BIP-1: Rename setRowOption to setOption on Option builder
Added missing rename.
…Option to setOption on Option builder [BEAM-9605] BIP-1: Rename setRowOption to setOption on Option builder
Update the range for pyarrow
|
O actually this is not right way to create cherry-pick PR. closing now. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
R @aaltay
cherry-pick #11252
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
R: @username).[BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replaceBEAM-XXXwith the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
Post-Commit Tests Status (on master branch)
Pre-Commit Tests Status (on master branch)
See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.