Skip to content

Extract the Main Input PCollection in ParDos #3190

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
tgroh wants to merge 4 commits into from
Closed

Extract the Main Input PCollection in ParDos #3190

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
3fd85c2
Remove Unused DoFnInfo methods
tgroh May 19, 2017
6698929
Extract the Main Input PCollection in ParDos
tgroh May 19, 2017
1930e3c
fixup! Extract the Main Input PCollection in ParDos
tgroh May 19, 2017
cf12477
fixup! Extract the Main Input PCollection in ParDos
tgroh May 22, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions ...ore-construction-java/src/main/java/org/apache/beam/runners/core/construction/ParDos.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@

import com.google.auto.value.AutoValue;
import com.google.common.base.Optional;
import com.google.common.collect.Iterables;
import com.google.common.collect.Sets;
import com.google.protobuf.Any;
import com.google.protobuf.ByteString;
import com.google.protobuf.BytesValue;
Expand Down Expand Up @@ -146,6 +148,20 @@ public static TupleTag<?> getMainOutputTag(ParDoPayload payload)
return doFnAndMainOutputTagFromProto(payload.getDoFn()).getMainOutputTag();
}

public static RunnerApi.PCollection getMainInput(
Copy link
Member

@kennknowles kennknowles May 20, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems unused except in test. Have I missed something?

Should javadoc the requirement that it is a ParDo, or just decompose the input outside the method, and perhaps also make it private.

Copy link
Member Author

@tgroh tgroh May 22, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is going to be used within the Dataflow Worker

RunnerApi.PTransform ptransform, Components components) throws IOException {
checkArgument(
ptransform.getSpec().getUrn().equals(PAR_DO_PAYLOAD_URN),
"Unexpected payload type %s",
ptransform.getSpec().getUrn());
ParDoPayload payload = ptransform.getSpec().getParameter().unpack(ParDoPayload.class);
String mainInputId =
Iterables.getOnlyElement(
Sets.difference(
ptransform.getInputsMap().keySet(), payload.getSideInputsMap().keySet()));
return components.getPcollectionsOrThrow(ptransform.getInputsOrThrow(mainInputId));
}

// TODO: Implement
private static StateSpec toProto(StateDeclaration state) {
throw new UnsupportedOperationException("Not yet supported");
Expand Down
4 changes: 4 additions & 0 deletions ...construction-java/src/test/java/org/apache/beam/runners/core/construction/ParDosTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,10 @@ public void toAndFromTransformProto() throws Exception {
view.getWindowingStrategyInternal().fixDefaults()));
assertThat(restoredView.getCoderInternal(), equalTo(view.getCoderInternal()));
}
String mainInputId = components.registerPCollection(mainInput);
assertThat(
ParDos.getMainInput(protoTransform, protoComponents),
equalTo(protoComponents.getPcollectionsOrThrow(mainInputId)));
}

private static class DropElementsFn extends DoFn<KV<Long, String>, Void> {
Expand Down
25 changes: 0 additions & 25 deletions ...gle-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/DoFnInfo.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,25 +53,6 @@ public static <InputT, OutputT> DoFnInfo<InputT, OutputT> forFn(
doFn, windowingStrategy, sideInputViews, inputCoder, mainOutput, outputMap);
}

/** TODO: remove this when Dataflow worker uses the DoFn overload. */
@Deprecated
@SuppressWarnings("unchecked")
public static <InputT, OutputT> DoFnInfo<InputT, OutputT> forFn(
Serializable doFn,
WindowingStrategy<?, ?> windowingStrategy,
Iterable<PCollectionView<?>> sideInputViews,
Coder<InputT> inputCoder,
long mainOutput,
Map<Long, TupleTag<?>> outputMap) {
return forFn(
(DoFn<InputT, OutputT>) doFn,
windowingStrategy,
sideInputViews,
inputCoder,
mainOutput,
outputMap);
}

public DoFnInfo<InputT, OutputT> withFn(DoFn<InputT, OutputT> newFn) {
return DoFnInfo.forFn(newFn,
windowingStrategy,
Expand All @@ -96,12 +77,6 @@ private DoFnInfo(
this.outputMap = outputMap;
}

/** TODO: remove this when Dataflow worker uses {@link #getDoFn}. */
@Deprecated
public Serializable getFn() {
return doFn;
}

/** Returns the embedded function. */
public DoFn<InputT, OutputT> getDoFn() {
return doFn;
Expand Down
2 changes: 1 addition & 1 deletion ...va/harness/src/test/java/org/apache/beam/fn/harness/control/ProcessBundleHandlerTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -340,7 +340,7 @@ public void testCreatingAndProcessingDoFn() throws Exception {
new TestDoFn(),
WindowingStrategy.globalDefault(),
ImmutableList.of(),
STRING_CODER,
StringUtf8Coder.of(),
mainOutputId,
ImmutableMap.of(
mainOutputId, TestDoFn.mainOutput,
Expand Down