[Beam-6858] Validate that side-input parameters match the type of the PCollectionView #9372
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
salmanVD
reviewed
Aug 20, 2019
| @@ -107,7 +108,8 @@ public void cleanup() throws Exception { | |||
| List<PCollectionView<?>> sideInputs, | |||
| TupleTag<OutputT> mainOutputTag, | |||
| List<TupleTag<?>> additionalOutputTags, | |||
| DoFnSchemaInformation doFnSchemaInformation) | |||
| DoFnSchemaInformation doFnSchemaInformation, | |||
| Map<String, String> sideInputMapping) | |||
There was a problem hiding this comment.
@reuvenlax This should be String, String? or PCollectionView
|
Yes, we should change to Map<String, PCollectionView<?>>
…On Mon, Aug 19, 2019 at 10:50 PM Salman Raza ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In
runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluatorFactory.java
<#9372 (comment)>:
> @@ -107,7 +108,8 @@ public void cleanup() throws Exception {
List<PCollectionView<?>> sideInputs,
TupleTag<OutputT> mainOutputTag,
List<TupleTag<?>> additionalOutputTags,
- DoFnSchemaInformation doFnSchemaInformation)
+ DoFnSchemaInformation doFnSchemaInformation,
+ Map<String, String> sideInputMapping)
@reuvenlax <https://github.com/reuvenlax> This should be String, String?
or PCollectionView
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#9372?email_source=notifications&email_token=AFAYJVOIDLSSQKSCWIFEWL3QFOA37A5CNFSM4IMWQU52YY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCCA45TQ#pullrequestreview-276942542>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AFAYJVIGE5T2FH655BWGMJLQFOA37ANCNFSM4IMWQU5Q>
.
|
reuvenlax
force-pushed
the
side_input_pardo
branch
from
9c0002b
to
83dc627
Compare
reuvenlax
changed the title
|
run Flink ValidatesRunner |
|
Run Dataflow ValidatesRunner |
|
Run Portable_Python PreCommit |
TheNeuralBit
approved these changes
Sep 5, 2019
| sideInput.sideInputId()); | ||
| TypeDescriptor<?> viewType = view.getViewFn().getTypeDescriptor(); | ||
|
|
||
| // Currently check that the types exactly match, even if the types are convertible. |
There was a problem hiding this comment.
Are there plans to handle convertible types? Should there be a JIRA thats linked here?
There was a problem hiding this comment.
We have the same issue with Element parameters. I'll check to see if we have a JIRA for this (I think a user actually filed one some time back).
| .apply(ParDo.of(fn).withSideInput(sideInputTag1, sideInput1)); | ||
| pipeline.run(); | ||
| } | ||
|
|
There was a problem hiding this comment.
Would you mind adding tests for the remaining view types - Map, Multimap, Iterable?
|
all comments addressed |
soyrice
pushed a commit
to soyrice/beam
that referenced
this pull request
Sep 19, 2019
…parameters match the type of the PCollectionView
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Validate that the type of a processElement side-input parameter matches that of the PCollectionView.
Note: Suppliers are used because often the TypeDescriptor is not Serializable, and so cannot be captured in the PCollectionView.