ISPN-9812 Streaming Cluster Publisher #7523

wburns · 2019-10-31T02:35:35Z

https://issues.jboss.org/browse/ISPN-9812

wburns · 2019-10-31T02:36:17Z

I still have to organize some of the classes and add more docs for the internal classes. This is still in Preview as only core tests should be pretty much passing.

wburns · 2019-10-31T02:37:11Z

There is a random failure that I can't seem to isolate or get a trace from that I have been still trying to track down. It seems to only happen when running without trace and always seems to fail two different tests. So I am not sure if it is related to running those in parallel somehow.

wburns · 2019-10-31T02:37:55Z

I have not deleted the old distributed stream classes, but that will be the next PR and it should remove needless to say a lot of lines of code (more than was added for this I believe).

wburns · 2019-10-31T03:14:44Z

I was able to squash the remaining intermittent failure I could find. Was caused by the async closure of a publisher was too slow and a failover could request again with the same id. Now it properly overwrites an old state if it is present and complete (not completed still throws exception).

wburns · 2019-10-31T04:42:22Z

Looks like I need to tweak some query stuff to no longer return a null value :)

wburns · 2019-10-31T13:55:17Z

Query should hopefully be fixed now too :)

wburns · 2019-10-31T15:58:28Z

CI is looking pretty good - it appears my segment completion isn't quite on par yet though

https://ci.infinispan.org/job/Infinispan/job/PR-7523/4/testReport/junit/org.infinispan.client.hotrod.impl.iteration/DistFailOverRemoteIteratorTest/testFailOver/

wburns · 2019-11-01T13:01:45Z

CI appears to be passing just fine now. Just have to add Javadoc and maybe reorg classes slightly.

wburns · 2019-11-01T13:38:31Z

Testing with https://github.com/infinispan/infinispan-benchmarks/tree/master/iteration I have anywhere from 13-33% increase in throughput when rehash is disabled. When rehash is not disabled the increase is anywhere from 38-141% in throughput.

wburns · 2019-11-01T13:48:21Z

Averaging the various runs non rehash had a performance increase of 20.7% and rehash had an increase of 76.4%.

https://docs.google.com/spreadsheets/d/1p3rArTx-lIZ66vALMcuXjxeCW7ivxfCqeYCfluwzQyw/edit?usp=sharing

wburns · 2019-11-05T13:14:18Z

Some recent changes have introduced a failure, probably a typo. Fixing.

core/src/main/java/org/infinispan/reactive/publisher/impl/PublisherHandler.java

commons/src/main/java/org/infinispan/commons/marshall/Ids.java

core/src/main/java/org/infinispan/cache/impl/EncodingFunction.java

core/src/main/java/org/infinispan/filter/CacheFilters.java

core/src/main/java/org/infinispan/reactive/publisher/impl/ClusterPublisherManagerImpl.java

ryanemerson · 2019-11-06T11:01:21Z

core/src/main/java/org/infinispan/reactive/publisher/impl/ClusterPublisherManagerImpl.java

+                     }
+                     segments.forEach(completedSegmentConsumer);
+                  })
+            );


Should we also handle doOnError?

What were you thinking would go in there? We don't want to complete the segments. The error would be propagated to the caller, so the entire Subscription would be cancelled anyways.

core/src/main/java/org/infinispan/reactive/publisher/impl/MaybeValueRetainedFunction.java

core/src/main/java/org/infinispan/reactive/publisher/impl/LocalPublisherManagerImpl.java

core/src/main/java/org/infinispan/reactive/publisher/impl/ClusterPublisherManagerImpl.java

core/src/main/java/org/infinispan/reactive/publisher/impl/PublisherHandler.java

core/src/main/java/org/infinispan/reactive/publisher/impl/commands/batch/PublisherResponse.java

...ava/org/infinispan/reactive/publisher/impl/commands/batch/PublisherResponseExternalizer.java

wburns · 2019-11-06T18:00:06Z

Okay flatMap should now be fixed. It can probably be optimized to not have to store the entries in a list by counting the entries and using onComplete but my first pass at that didn't work. So we can leave that for a future endeavor maybe.

wburns · 2019-11-06T19:24:13Z

Key tracking map/flatMap is better optimized now as it doesn't have to eagerly retrieve the values into a list, which is especially costly for map as it is always 1:1 or 1:0.

wburns · 2019-11-06T22:32:06Z

Addressed all the current comments as well as fixed another bug with flatMap if there was a large number of entries generated from it.

ryanemerson · 2019-11-13T17:32:40Z

core/src/main/java/org/infinispan/reactive/publisher/impl/ModifiedValueFunction.java

+    * can be useful for some optimizations that may need to track produced values from the original.
+    * @return if the values in the publisher are unchanged
+    */
+   boolean isModified();


This needs updating to say that is should return true when the function does change the value.

Sure, let me make sure this correct again.

Should be fixed now.

ryanemerson · 2019-11-13T18:01:27Z

core/src/main/java/org/infinispan/stream/impl/DistributedCacheStream.java

+         log.tracef("Completing last segments of: %s", completedSegments);
+         listener.accept(completedSegments::iterator);
+         completedSegments.clear();
+      }
   }

   private class RehashIterator<S> extends AbstractIterator<S> implements CloseableIterator<S> {


It seems we can now remove this and CompletionListenerRehashIterator

Yup that will be handled in https://issues.jboss.org/browse/ISPN-10895. I didn't want to bloat this PR too much.

wburns · 2019-11-13T18:42:23Z

Rebased and squashed to 1 commit.

ryanemerson · 2019-11-14T09:49:43Z

@wburns #7591 was merged before I came online, so this needs a rebase. Then I promise I'll merge!

wburns · 2019-11-15T01:50:40Z

CI found a small gap in partition handling between when a publisher is created and when it is subscribed to. Recheck the partition status after registering with listener to remove gap. I am not sure what those PooledConnectionOperation failures are, but I can't seem to reproduce and it seems like a setup issue more likely as everyone failed consistently.

wburns · 2019-11-15T03:02:32Z

It looks like I fixed flatMap when rehash is enabled, but non rehash flatMap is still broken :(

I need to actually fix this at the LocalPublisherManager level, which will in turn make the PublisherHandler code simpler.

wburns · 2019-11-15T03:39:23Z

Non rehash flatMap should work properly now. I just piggy back off the rehash version, but don't return the keys to keep the response size smaller. This can be optimized later though.

wburns · 2019-11-15T04:22:44Z

Got a chance to think about it and the flatMap with non rehash is now properly optimized. In fact the LocalPublisherManager now should always properly return the segment completed or lost after all values as it was originally envisioned. Unfortunately, KeyPublisherState can't utilize this as it has to track by the original key to a segment, not just the values - so that will have to stay the same.

ryanemerson · 2019-11-15T11:42:35Z

@wburns The transformer changes have broken SimpleClusterPublisherManagerTest.

wburns · 2019-11-15T12:21:37Z

I messed up something with the predicate. I ran a bunch of of the tests, but if course missed that one last night as I went to bed ☹️

…

On Fri, Nov 15, 2019, 6:42 AM Ryan Emerson ***@***.***> wrote: @wburns <https://github.com/wburns> The transformer changes have broken SimpleClusterPublisherManagerTest. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7523?email_source=notifications&email_token=AAE6H33WUSCSLDX6DD5EKLTQT2DK3A5CNFSM4JHDHD3KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEFGFDY#issuecomment-554328719>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAE6H35GEDO3SJJ52LCGETTQT2DK3ANCNFSM4JHDHD3A> .

wburns · 2019-11-15T13:28:19Z

Context should be only applied once now.

ryanemerson · 2019-11-15T15:01:49Z

Great stuff @wburns, thanks!

wburns added the Preview label Oct 31, 2019

wburns force-pushed the ISPN-9812_cluster_publisher branch from d8c281c to 9cd4444 Compare October 31, 2019 03:14

wburns force-pushed the ISPN-9812_cluster_publisher branch from a8dd157 to 5bdccf9 Compare October 31, 2019 13:55

wburns force-pushed the ISPN-9812_cluster_publisher branch from 011b1ac to 47750bc Compare November 1, 2019 13:02

wburns changed the title ~~[PREVIEW] ISPN-9812 Streaming Cluster Publisher~~ ISPN-9812 Streaming Cluster Publisher Nov 4, 2019

wburns removed the Preview label Nov 4, 2019

wburns requested review from ryanemerson and danberindei November 4, 2019 15:31

wburns commented Nov 6, 2019

View reviewed changes

core/src/main/java/org/infinispan/reactive/publisher/impl/PublisherHandler.java Outdated Show resolved Hide resolved

ryanemerson reviewed Nov 6, 2019

View reviewed changes

wburns requested a review from ryanemerson November 6, 2019 22:32

ryanemerson reviewed Nov 13, 2019

View reviewed changes

wburns force-pushed the ISPN-9812_cluster_publisher branch from 1f7833e to 3325a83 Compare November 13, 2019 18:42

ryanemerson added the Needs Rebase label Nov 14, 2019

wburns force-pushed the ISPN-9812_cluster_publisher branch from 3325a83 to 17a8913 Compare November 14, 2019 14:12

wburns removed the Needs Rebase label Nov 14, 2019

wburns force-pushed the ISPN-9812_cluster_publisher branch from 17a8913 to d36e2ac Compare November 14, 2019 15:53

wburns added the Changes Required label Nov 15, 2019

wburns removed the Changes Required label Nov 15, 2019

wburns force-pushed the ISPN-9812_cluster_publisher branch from 31bddb8 to 9c96504 Compare November 15, 2019 04:18

ryanemerson added this to the 10.1.0.Beta1 milestone Nov 15, 2019

ryanemerson added the Changes Required label Nov 15, 2019

wburns removed the Changes Required label Nov 15, 2019

wburns force-pushed the ISPN-9812_cluster_publisher branch from 9d269fa to 059f174 Compare November 15, 2019 13:40

ISPN-9812 Implement streaming response publisher method

791c64f

wburns force-pushed the ISPN-9812_cluster_publisher branch from 059f174 to 791c64f Compare November 15, 2019 13:41

ryanemerson merged commit 8684d79 into infinispan:master Nov 15, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ISPN-9812 Streaming Cluster Publisher #7523

ISPN-9812 Streaming Cluster Publisher #7523

wburns commented Oct 31, 2019 •

edited

wburns commented Oct 31, 2019

wburns commented Oct 31, 2019

wburns commented Oct 31, 2019

wburns commented Oct 31, 2019 •

edited

wburns commented Oct 31, 2019

wburns commented Oct 31, 2019

wburns commented Oct 31, 2019

wburns commented Nov 1, 2019

wburns commented Nov 1, 2019

wburns commented Nov 1, 2019

wburns commented Nov 5, 2019

ryanemerson Nov 6, 2019

wburns Nov 6, 2019

wburns commented Nov 6, 2019 •

edited

wburns commented Nov 6, 2019

wburns commented Nov 6, 2019

ryanemerson Nov 13, 2019

wburns Nov 13, 2019

wburns Nov 13, 2019

ryanemerson Nov 13, 2019 •

edited

wburns Nov 13, 2019

wburns commented Nov 13, 2019

ryanemerson commented Nov 14, 2019

wburns commented Nov 15, 2019

wburns commented Nov 15, 2019

wburns commented Nov 15, 2019

wburns commented Nov 15, 2019

ryanemerson commented Nov 15, 2019

wburns commented Nov 15, 2019 via email

wburns commented Nov 15, 2019

ryanemerson commented Nov 15, 2019

ISPN-9812 Streaming Cluster Publisher #7523

ISPN-9812 Streaming Cluster Publisher #7523

Conversation

wburns commented Oct 31, 2019 • edited

wburns commented Oct 31, 2019

wburns commented Oct 31, 2019

wburns commented Oct 31, 2019

wburns commented Oct 31, 2019 • edited

wburns commented Oct 31, 2019

wburns commented Oct 31, 2019

wburns commented Oct 31, 2019

wburns commented Nov 1, 2019

wburns commented Nov 1, 2019

wburns commented Nov 1, 2019

wburns commented Nov 5, 2019

ryanemerson Nov 6, 2019

Choose a reason for hiding this comment

wburns Nov 6, 2019

Choose a reason for hiding this comment

wburns commented Nov 6, 2019 • edited

wburns commented Nov 6, 2019

wburns commented Nov 6, 2019

ryanemerson Nov 13, 2019

Choose a reason for hiding this comment

wburns Nov 13, 2019

Choose a reason for hiding this comment

wburns Nov 13, 2019

Choose a reason for hiding this comment

ryanemerson Nov 13, 2019 • edited

Choose a reason for hiding this comment

wburns Nov 13, 2019

Choose a reason for hiding this comment

wburns commented Nov 13, 2019

ryanemerson commented Nov 14, 2019

wburns commented Nov 15, 2019

wburns commented Nov 15, 2019

wburns commented Nov 15, 2019

wburns commented Nov 15, 2019

ryanemerson commented Nov 15, 2019

wburns commented Nov 15, 2019 via email

wburns commented Nov 15, 2019

ryanemerson commented Nov 15, 2019

wburns commented Oct 31, 2019 •

edited

wburns commented Oct 31, 2019 •

edited

wburns commented Nov 6, 2019 •

edited

ryanemerson Nov 13, 2019 •

edited