New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Ro replica object store integration #444
Conversation
d4e70c2
to
71d6eb5
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good! Just need a clarification about wait()
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Thanks!
34c3159
to
e49d754
Compare
bd83b5c
to
c6edf7c
Compare
8f93fbb
to
3aee377
Compare
9d36230
to
6095e0d
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the offline discussion and the cleanup @toly-kournik ! I think it looks good and we can merge it as is. I have a few minor comments that I will leave and you can decide whether you want to address them in a separate PR.
Thanks @toly-kournik and @salieri11 for the effort, good work! It is great that we can now save data to S3! And the code is clearer and simpler. |
The commit includes new code that support S3 as possible implementation of IDBClient. We plan to use it with ReadOnly replica for the replication of the data to external storage. Introduce has(Key) method in IDBClient interface and implement it in all its derived classes ROReplica: handoff state transfer messages to a dedicated thread to avoid process stuck in case of long blocking call to an external object store. Re-factoring: replace various BCStateTran config params by their Config struct equivalent. IDBclient: - add has(Sliver key) - ReplicaForStateTransfer owns the IStateTransfer* instance instead of kvbc::ReplicaImp - remove RorAppState - IAppState implemented by kvbc::ReplicaImp - v1::DBAdapter - setLatestBlock() - setLastReachableBlock() - add an ability to add free keys - ecs::S3Client -> s3::client DBAdapter to use additional IDBClient for metadata
6095e0d
to
baba245
Compare
Introducing S3-compatible external object store support. Key points: S3: - concord::storage::s3::client with S3KeyGenerator - handing off state transfer messages in BCStateTransfer if used with remote object store to avoid process stuck in case of long blocking call to an external object store. General: - kvbc::ReplicaImp implements IAppState - removed ReplicaImp::StorageWrapperForIdleMode Co-authored-by: Igor Golikov <salieri11@gmail.com> Co-authored-by: @toly-kournik
Introducing S3-compatible external object store support.
Co-authored-by: @salieri11
Co-authored-by: @toly-kournik
Key points:
S3:
General: