Pinning Consumers - integration

[rpalcolea]

Hi @toolbear,

I was wondering if there are plans to introduce the "Pinning Consumers" mechanism as a default in Hollow.

I think it would be nice if we could know when a consumer is pinned and expose this via the metrics or the API. something like consumer.isPinned() and/or consumer.getPinnedVersion(). This could help to triage issues when you have tens/hundreds of consumers with the same dataset but because of the nature of eventual consistency, a few of them could be stuck.

I don't have a clear idea on how this could be accomplish, my guess is that the AnnouncementWatcher could be modified to support the pinning mechanism and HollowConsumer expose the information, similar to:

public long getCurrentVersionId() {
        return updater.getCurrentVersionId();
}

Something like

public long getPinnedVersion() {
        return announcementWatcher.getPinnedVersion();
} 

It would be up to the users to store the pinned version in the announcementWatcher or read it from their database/blob storage every time they need this. getPinnedVersion should be another abstract method in AnnouncementWatcher.

From the HollowProducer perspective, I think Hollow could provide a versionPinner in the builder that takes an VersionPinner (new interface). e.g.

HollowProducer producer = HollowProducer.withPublisher(new FakeBlobPublisher())
                                                .withAnnouncer(new MyAnnouncer(tmpFolder))
                                                .withVersionPinner(new MyVersionPinner());

Then, probably the HollowProducer could have a pinVersion method that takes the version number as an argument and invokes versionPinner.pin(long pinnedVersion). This way users could programmatically pin version from the producer either by background jobs based on their own business rules or as simple as expose the pinVersion as a HTTP endpoint.

Also it could have the unpin.

thoughts?