Investigate parallelizing TransactionImpl.readUnread()

[keith-turner]

When a transaction only writes to a row+col and then has a collision Fluo will read the row+col after the collision to look for orphaned locks. If there are multiple row+cols then the code reads them one at time. The could possibly be parallelized by calling some of the existing methods in Fluo that read multiple row+cols at once.

The code in question is at TransactionImpl line 519

[keith-turner]

Actually it does an entire row and all of it cols at once. Could still do all row cols at once.

[jkosh44]

Is this still valid? I was looking at Map<RowColumn, Bytes> get(Collection<RowColumn> rowColumns) and get(Collection<Bytes> rows, Set<Column> columns) as options but they don't have the Consumer<Entry<Key, Value>> locksSeen parameter.

Another option would be something like the following, which just runs all the getImpl call concurrently. It's a little messy but I'd think of someway to clean it up.

List<CompletableFuture> reads = columnsToRead.entrySet()
            .stream()
            .map(entry -> CompletableFuture.runAsync(()->getImpl(entry.getKey(), entry.getValue(), locksSeen)))
            .collect(Collectors.toList());
    return CompletableFuture.allOf(reads.toArray(new CompletableFuture[]{})).get();

[jkosh44]

I was also thinking, though I haven't fully thought it through yet, that we could change readUnread(CommitData cd, Consumer<Entry<Key, Value>> locksSeen) and some of the other helper methods to return a CompletableFutures and chain them together with thenComposeAsync. That way if we go with the completable future method above we wouldn't have to block with .get().

[keith-turner]

Is this still valid?

Yes.

I was looking at Map<RowColumn, Bytes> get(Collection<RowColumn> rowColumns) and get(Collection<Bytes> rows, Set<Column> columns) as options but they don't have the Consumer<Entry<Key, Value>> locksSeen parameter.

I feel like this is the right path. Looking inside get(Collection<RowColumn> rowColumns) there is a ParallelSnapshotScanner, I wonder if this could be used. Create a parallel snapshot scanner and then read all the data from it. Can pass locksSeen to the constructor of ParallelSnapshotScanner.

Another option would be something like the following, which just runs all the getImpl call concurrently.

I am wary of using CompletableFuture.runAsync() for operations that do I/O because it runs in a JVM wide shared thread pool. If the Fluo client library does this it could impact user code that is also using runAsync()

[jkosh44]

Gotcha, we could go a similar route of Map<Column, Bytes> getImpl(Bytes row, Set<Column> columns, Consumer<Entry<Key, Value>> locksSeen) and create a public Map<RowColumn, Bytes> getImpl(Collection<RowColumn> rowColumns, Map<Bytes, Set<Column>> locksSeen) method which allows you to provide the locksSeen for the ParallelSnapshotScanner constructor.