Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for callback when records in hybrid log become read-only #133

Merged
merged 3 commits into from
Jun 12, 2019

Conversation

badrishc
Copy link
Contributor

@badrishc badrishc commented Jun 11, 2019

This PR allows users to register a user-defined callback, which is invoked when records in the hybrid log become read-only. It can be used for various reasons:

(1) to perform data write-through to an external store
(2) to perform stream processing on the change log created by the key-value store

How it Works

The FASTER log accessor now supports the IObservable<IFasterScanIterator<Key, Value>> interface. This means you can Subscribe to FASTER log, in order to receive (as push calls) updates whenever records enter the read-only region of the hybrid log:

        var s = fht.Log.Subscribe(new LogObserver());
        ...
        s.Dispose();

The parameter LogObserver is an instance of type IObserver<IFasterScanIterator<Key, Value>>. User gets push-based callbacks via the OnNext method of the observer. Each callback provides an iterator for a batch of records that have just become read-only, in bulk. The user can then perform operations such as stream processing or write-through to the remote database.

Here is an example of the callback:

        class LogObserver : IObserver<IFasterScanIterator<Key, Value>>
        {
            public void OnCompleted()
            {
            }

            public void OnError(Exception error)
            {
            }

            public void OnNext(IFasterScanIterator<Key, Value> iter)
            {
                while (iter.GetNext(out RecordInfo info, out Key key, out Value value))
                {
                    // Process key and value
                }
                iter.Dispose();
            }
        }

… be used to perform data write-through to external store.
@badrishc
Copy link
Contributor Author

badrishc commented Jun 11, 2019

We have also added an extension method to convert an IObservable<IFasterScanIterator<Key, Value>> to an easier-to-use IObservable<Record<Key, Value>>. This converter copies the key, value, and header info from the log to the Record<Key, Value> struct, so it will not directly work with special inline variable-length structs (see #120).

Usage:

        var s = fht.Log.ToRecordObservable.Subscribe(...);
        ...
        s.Dispose();

…e) to convert to more easily consumable record-oriented IObservable.
@badrishc
Copy link
Contributor Author

Currently, we support only one subscriber to the log (easy to extend in future). Also, note that the subscriber only receives new log updates from the time of subscription onwards. In other words, no I/O is associated with this operation. If you need to scan the historical part of the log, use the Scan(...) method instead (see #90).

Copy link

@peterfreiling-zz peterfreiling-zz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@badrishc badrishc merged commit e05f967 into master Jun 12, 2019
@badrishc badrishc deleted the evict-callback branch September 7, 2020 02:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants