ConcatenateListMergeFunction should return lazy iterables #2829
Labels
enhancement
Improvement to existing functionality/feature
federated-store
Specific to/touches the federated-store module
Milestone
Describe the new feature you'd like
The federated store's
ConcatenateListMergeFunction
should return results in a lazy iterable.Why do you want this feature?
Pre v2.0 the results were in a lazy iterable so this functionality should be retained.
Additional context
Currently, the koryphe function
ToList
is used to turn all results into iterables. However, this stops the results from being returned lazily as they are read into a list:Gaffer/store-implementation/federated-store/src/main/java/uk/gov/gchq/gaffer/federatedstore/util/ConcatenateListMergeFunction.java
Line 36 in 2a33be8
As well as this,
Lists.newArrayList
has the same issue:Gaffer/store-implementation/federated-store/src/main/java/uk/gov/gchq/gaffer/federatedstore/util/ConcatenateListMergeFunction.java
Line 39 in 2a33be8
Koryphe functions do not really need to be used at all here in fact, and
IterableConcat
can be replaced by directly using aChainedIterable
, which is what used to happen and what is happening indirectly anyway.If the proposed changes were implemented, then gchq/koryphe#286 can be closed.
As well as these functionality changes, there should be proper javadoc and tests added to this class to ensure it works as expected and the expected behaviour is documented.
Additionally, I would argue the class should be renamed to
ConcatenateMergeFunction
andConcatenateSetMergeFunction
should be deleted.This is because results are concatenated whether they are in a list or not, and returned in an Iterable. The results are not necessarily returned in a list. If a user specifically wanted a list or set return type, they could use ToList or ToSet in their operation chain.
The text was updated successfully, but these errors were encountered: