Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@
import java.util.List;

/**
* Base operator for the combineGroup transformation. It receives the UDF GroupCombineOperator as an input.
* Base operator for the combineGroup transformation. It receives the UDF GroupCombineFunction as an input.
* This class is later processed by the compiler to generate the plan.
* @see org.apache.flink.api.common.functions.CombineFunction
*/
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -465,20 +465,20 @@ public <R> GroupReduceOperator<T, R> reduceGroup(GroupReduceFunction<T, R> reduc
}

/**
* Applies a CombineFunction on a non-grouped {@link DataSet}.
* Applies a GroupCombineFunction on a non-grouped {@link DataSet}.
* A CombineFunction is similar to a GroupReduceFunction but does not perform a full data exchange. Instead, the
* CombineFunction calls the combine method once per partition for combining a group of results. This
* operator is suitable for combining values into an intermediate format before doing a proper groupReduce where
* the data is shuffled across the node for further reduction. The GroupReduce operator can also be supplied with
* a combiner by implementing the RichGroupReduce function. The combine method of the RichGroupReduce function
* demands input and output type to be the same. The CombineFunction, on the other side, can have an arbitrary
* output type.
* @param combiner The CombineFunction that is applied on the DataSet.
* @return A GroupCombineOperator which represents the combined DataSet.
* @param combiner The GroupCombineFunction that is applied on the DataSet.
* @return A {@link GroupCombineOperator} which represents the combined DataSet.
*/
public <R> GroupCombineOperator<T, R> combineGroup(GroupCombineFunction<T, R> combiner) {
if (combiner == null) {
throw new NullPointerException("GroupReduce function must not be null.");
throw new NullPointerException("GroupCombine function must not be null.");
}

String callLocation = Utils.getCallLocationName();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -167,20 +167,20 @@ public <R> GroupReduceOperator<T, R> reduceGroup(GroupReduceFunction<T, R> reduc
}

/**
* Applies a CombineFunction on a grouped {@link DataSet}.
* Applies a GroupCombineFunction on a grouped {@link DataSet}.
* A CombineFunction is similar to a GroupReduceFunction but does not perform a full data exchange. Instead, the
* CombineFunction calls the combine method once per partition for combining a group of results. This
* operator is suitable for combining values into an intermediate format before doing a proper groupReduce where
* the data is shuffled across the node for further reduction. The GroupReduce operator can also be supplied with
* a combiner by implementing the RichGroupReduce function. The combine method of the RichGroupReduce function
* demands input and output type to be the same. The CombineFunction, on the other side, can have an arbitrary
* output type.
* @param combiner The CombineFunction that is applied on the DataSet.
* @return A GroupCombineOperator which represents the combined DataSet.
* @param combiner The GroupCombineFunction that is applied on the DataSet.
* @return A {@link GroupCombineOperator} which represents the combined DataSet.
*/
public <R> GroupCombineOperator<T, R> combineGroup(GroupCombineFunction<T, R> combiner) {
if (combiner == null) {
throw new NullPointerException("GroupReduce function must not be null.");
throw new NullPointerException("GroupCombine function must not be null.");
}
TypeInformation<R> resultType = TypeExtractor.getGroupCombineReturnTypes(combiner, this.getDataSet().getType());

Expand All @@ -191,7 +191,7 @@ public <R> GroupCombineOperator<T, R> combineGroup(GroupCombineFunction<T, R> co
/**
* Returns a new set containing the first n elements in this grouped and sorted {@link DataSet}.<br/>
* @param n The desired number of elements for each group.
* @return A ReduceGroupOperator that represents the DataSet containing the elements.
* @return A GroupReduceOperator that represents the DataSet containing the elements.
*/
public GroupReduceOperator<T, T> first(int n) {
if(n < 1) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -162,20 +162,20 @@ public <R> GroupReduceOperator<T, R> reduceGroup(GroupReduceFunction<T, R> reduc
}

/**
* Applies a CombineFunction on a grouped {@link DataSet}.
* A CombineFunction is similar to a GroupReduceFunction but does not perform a full data exchange. Instead, the
* Applies a GroupCombineFunction on a grouped {@link DataSet}.
* A GroupCombineFunction is similar to a GroupReduceFunction but does not perform a full data exchange. Instead, the
* CombineFunction calls the combine method once per partition for combining a group of results. This
* operator is suitable for combining values into an intermediate format before doing a proper groupReduce where
* the data is shuffled across the node for further reduction. The GroupReduce operator can also be supplied with
* a combiner by implementing the RichGroupReduce function. The combine method of the RichGroupReduce function
* demands input and output type to be the same. The CombineFunction, on the other side, can have an arbitrary
* output type.
* @param combiner The CombineFunction that is applied on the DataSet.
* @return A GroupCombineOperator which represents the combined DataSet.
* @param combiner The GroupCombineFunction that is applied on the DataSet.
* @return A {@link GroupCombineOperator} which represents the combined DataSet.
*/
public <R> GroupCombineOperator<T, R> combineGroup(GroupCombineFunction<T, R> combiner) {
if (combiner == null) {
throw new NullPointerException("GroupReduce function must not be null.");
throw new NullPointerException("GroupCombine function must not be null.");
}
TypeInformation<R> resultType = TypeExtractor.getGroupCombineReturnTypes(combiner, this.getDataSet().getType());

Expand All @@ -185,7 +185,7 @@ public <R> GroupCombineOperator<T, R> combineGroup(GroupCombineFunction<T, R> co
/**
* Returns a new set containing the first n elements in this grouped {@link DataSet}.<br/>
* @param n The desired number of elements for each group.
* @return A ReduceGroupOperator that represents the DataSet containing the elements.
* @return A GroupReduceOperator that represents the DataSet containing the elements.
*/
public GroupReduceOperator<T, T> first(int n) {
if(n < 1) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -34,14 +34,14 @@
/**
* GroupReduceDriver task which is executed by a Task Manager. The task has a
* single input and one or multiple outputs. It is provided with a GroupReduceFunction
* implementation or a RichGroupFunction. This Driver performs
* implementation or a RichGroupReduceFunction. This Driver performs
* multiple tasks depending on the DriverStrategy. In case of a ALL_GROUP_REDUCE_COMBINE
* it uses the combine function of the supplied user function. In case
* of the ALL_GROUP_REDUCE, it uses the reduce function of the supplied user function to
* process all elements. In either case, the function is executed on all elements.
* <p>
* The GroupReduceTask creates a iterator over all records from its input. The iterator returns all records grouped by their
* key. The iterator is handed to the <code>reduce()</code> method of the GroupReduceFunction.
* The AllGroupReduceDriver creates an iterator over all records from its input.
* The iterator is handed to the <code>reduce()</code> method of the GroupReduceFunction.
*
* @see org.apache.flink.api.common.functions.GroupReduceFunction
*/
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,8 @@
* single input and one or multiple outputs. It is provided with a ReduceFunction
* implementation.
* <p>
* The ReduceTask creates a iterator over all records from its input. The iterator returns all records grouped by their
* key. The iterator is handed to the <code>reduce()</code> method of the ReduceFunction.
* The AllReduceDriver creates an iterator over all records from its input.
* The elements are handed pairwise to the <code>reduce()</code> method of the ReduceFunction.
*
* @see org.apache.flink.api.common.functions.ReduceFunction
*/
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@
* to have the same input and output type to be able to reduce the elements after the combine from
* {@code IN} to {@code OUT}.
*
* <p>The CombineTask uses a combining iterator over its input. The output of the iterator is emitted.</p>
* <p>The GroupReduceCombineDriver uses a combining iterator over its input. The output of the iterator is emitted.</p>
*
* @param <IN> The data type consumed by the combiner.
* @param <OUT> The data type produced by the combiner.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@
* single input and one or multiple outputs. It is provided with a GroupReduceFunction
* implementation.
* <p>
* The GroupReduceTask creates a iterator over all records from its input. The iterator returns all records grouped by their
* The GroupReduceDriver creates a iterator over all records from its input. The iterator returns all records grouped by their
* key. The iterator is handed to the <code>reduce()</code> method of the GroupReduceFunction.
*
* @see org.apache.flink.api.common.functions.GroupReduceFunction
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,12 +30,12 @@
import org.apache.flink.util.MutableObjectIterator;

/**
* Reduce task which is executed by a Task Manager. The task has a
* Reduce driver which is executed by a Task Manager. The task has a
* single input and one or multiple outputs. It is provided with a ReduceFunction
* implementation.
* <p>
* The ReduceTask creates a iterator over all records from its input. The iterator returns all records grouped by their
* key. The iterator is handed to the <code>reduce()</code> method of the ReduceFunction.
* The ReduceDriver creates an iterator over all records from its input. The iterator returns all records grouped by their
* key. The elements are handed pairwise to the <code>reduce()</code> method of the ReduceFunction.
*
* @see org.apache.flink.api.common.functions.ReduceFunction
*/
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@
import java.util.List;

/**
* Chained variant of the GroupCombineDriver
* Chained variant of the GroupReduceCombineDriver
*
* Acts like a combiner with a custom output type OUT.
*
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -641,17 +641,17 @@ class DataSet[T: ClassTag](set: JavaDataSet[T]) {
}

/**
* Applies a CombineFunction on a grouped [[DataSet]]. A
* CombineFunction is similar to a GroupReduceFunction but does not
* perform a full data exchange. Instead, the CombineFunction calls
* Applies a GroupCombineFunction on a grouped [[DataSet]]. A
* GroupCombineFunction is similar to a GroupReduceFunction but does not
* perform a full data exchange. Instead, the GroupCombineFunction calls
* the combine method once per partition for combining a group of
* results. This operator is suitable for combining values into an
* intermediate format before doing a proper groupReduce where the
* data is shuffled across the node for further reduction. The
* GroupReduce operator can also be supplied with a combiner by
* implementing the RichGroupReduce function. The combine method of
* the RichGroupReduce function demands input and output type to be
* the same. The CombineFunction, on the other side, can have an
* the same. The GroupCombineFunction, on the other side, can have an
* arbitrary output type.
*/
def combineGroup[R: TypeInformation: ClassTag](
Expand All @@ -666,17 +666,17 @@ class DataSet[T: ClassTag](set: JavaDataSet[T]) {
}

/**
* Applies a CombineFunction on a grouped [[DataSet]]. A
* CombineFunction is similar to a GroupReduceFunction but does not
* perform a full data exchange. Instead, the CombineFunction calls
* Applies a GroupCombineFunction on a grouped [[DataSet]]. A
* GroupCombineFunction is similar to a GroupReduceFunction but does not
* perform a full data exchange. Instead, the GroupCombineFunction calls
* the combine method once per partition for combining a group of
* results. This operator is suitable for combining values into an
* intermediate format before doing a proper groupReduce where the
* data is shuffled across the node for further reduction. The
* GroupReduce operator can also be supplied with a combiner by
* implementing the RichGroupReduce function. The combine method of
* the RichGroupReduce function demands input and output type to be
* the same. The CombineFunction, on the other side, can have an
* the same. The GroupCombineFunction, on the other side, can have an
* arbitrary output type.
*/
def combineGroup[R: TypeInformation: ClassTag](
Expand Down