rebasing DSL_SQL branch from master#3246
Closed
xumingming wants to merge 367 commits intoapache:DSL_SQLfrom
Closed
rebasing DSL_SQL branch from master#3246xumingming wants to merge 367 commits intoapache:DSL_SQLfrom
xumingming wants to merge 367 commits intoapache:DSL_SQLfrom
Conversation
quickstart and apex runner pages. apache/beam-site#232 moves small bits of missing content.
If too many tables are generated in a bundle, spill and group the results before writing files. Generating hundreds or thousands of file write buffers in a single bundle was causing workers to crash with out of memory.
Now that users can implement and register custom FileSystems, we can no longer really effectively validate filesystems they can read or write files from. They can even register file:// to point to some HDFS path, e.g.,
find . -type f -name '*.java' | xargs sed -i '' 's/\([.]..code[(].*\), *context.nested..[)]/\1)/'
find . -type f -name '*.java' | xargs sed -i '' 's/\([.]..code[(].*\), *nestedContext[)]/\1)/'
find . -type f -name '*.java' | xargs sed -i '' 's/\([.]..code[(].*\), *Context.NESTED[)]/\1)/'
find . -type f -name '*.java' | xargs sed -i '' 's/\([.]..code[(].*\), *[^ ]*.Context.NESTED[)]/\1)/'
Added back explicit context in CoGbkResult.java due to compile error.
Now only the context-free encode() and decode() methods are abstract.
It will be logged later if not caught. This reduces duplication for failing tests, and noise for tests expecting failure.
For now, this does not need to be a Beam concept
When a worker dies because of an error, propagate that error and fail the Pipeline.
Add an API Surface Test
Removes withKeyCoder() and withValueCoder() methods. Their meaning changed when KafkaIO added support for Deserializers. The coders can be explicitly specified using withKeyDeserializerAndCoder(), likewise for value. This makes it explicit to the user that Deserializer is still required and JavaDoc explains why/when both are required.
Always ensure that the producer of a value is visited before that value is visited for the first time. Visit a compoosite before visiting any of its child nodes.
Only override CreatePCollectionView transforms
Add the value of PTransform.getAdditionalInputs in the inputs of a TransformHierarchy node. Fork the Node constructor to reduce nullability This slightly simplifies the constructor implementation(s).
This is different than a StateBinder: for a binder, the id is needed and the StateSpec controls the return type. For case dispatch, the dispatcher controls the type and it should just be reading the spec, which does not require the id. Eventually, StateBinder could be removed in favor of StateSpec.Cases<Function<String, StateT>>.
…m Nodes" This reverts commit 2e2ae9c.
This reverts commit bd1dfdf.
* Add unit and integration tests * Add MutationSizeEstimator * Improve display data * Upgrade libraries
Tests that pickle the main session but are run from setup.py attempt to pickle everything visible in setup.py.
Since it checks the entire project anyway.
|
Changes Unknown when pulling 2fed60f on xumingming:DSL_SQL into ** on apache:DSL_SQL**. |
|
Changes Unknown when pulling 2fed60f on xumingming:DSL_SQL into ** on apache:DSL_SQL**. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Be sure to do all of the following to help us incorporate your contribution
quickly and easily:
[BEAM-<Jira issue #>] Description of pull requestmvn clean verify.<Jira issue #>in the title with the actual Jira issuenumber, if there is one.
Individual Contributor License Agreement.