FIFO the Datalog! #35548
Merged
FIFO the Datalog! #35548
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
adamemerson
force-pushed
the
wip-datalog-fifo
branch
from
5f266c5
to
d5472a7
Compare
adamemerson
force-pushed
the
wip-datalog-fifo
branch
2 times, most recently
from
ea17b14
to
900e01e
Compare
adamemerson
force-pushed
the
wip-datalog-fifo
branch
6 times, most recently
from
4e52f7d
to
ef3102c
Compare
adamemerson
force-pushed
the
wip-datalog-fifo
branch
from
ef3102c
to
4aaa9f6
Compare
adamemerson
force-pushed
the
wip-datalog-fifo
branch
6 times, most recently
from
3c61232
to
3e13103
Compare
adamemerson
force-pushed
the
wip-datalog-fifo
branch
5 times, most recently
from
5dd94e6
to
5cd78e0
Compare
|
@adamemerson I understand there to be successful teuthology runs? |
|
@mattbenjamin Almost, and I have no idea what's up with the CLI test failure on make check. I think that's unrelated to anything I'm doing. But I've got reliable local reproducers for the issues and think I've almost hit a fix, so I think I can get it green tomorrow. |
|
Once I DO get this in, I think I'll do another pull request to get a local 'nose' frontend on multisite tests. It shouldn't be much work now that I've pulled them apart/etc. to get local versions and should make this kind of thing faster. |
adamemerson
force-pushed
the
wip-datalog-fifo
branch
2 times, most recently
from
e01f425
to
5898b44
Compare
|
Jenkins, retest this please. |
From https://github.com/phalpern/uses-allocator To handle uses_allocator construction Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Bind an allocator to a completion, analogously to boost::async::bind_executor. Based on bind_executor.hpp which is copyright 2003-2019 Christopher M. Kohlhoff <chris@kohlhoff.com> and under the Boost Software License, Version 1.0 Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
A template to ensure that one completion has the same executor and allocator as another. Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
adds an allocator-aware version of std::make_unique(). this is similar to std::allocate_shared(), though it's slightly less convenient because, unlike std::shared_ptr<T>, the Deleter has to be specified as a template parameter in std::unique_ptr<T, Deleter> Signed-off-by: Casey Bodley <cbodley@redhat.com>
Support using a librados::AioCompletion as a completion handler in Boost.Asio when the arguments are void or only an error code. (To support more arguments we'd have to wrap up the AioCompletion with pointers to hold them.) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Specifically boost::container::flat_map and boost::container::flat_set Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Having Thread::create store a pointer to a string that is passed to ceph_pthread_setname in Thread::entry_wrapper can lead to using a pointer in the calling thread's stack that gets freed before use. Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Also these don't have the stringstream overhead. Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
This is an implementation of fifo queue over rados. Data is appended to rados object until it's full. At that point data will be written to a new object. Data is read from the tail sequentially (can be iterated using marker). Data can be trimmed (up to and including marker). Queue has a header object (meta), and zero or more data objects (parts). The software has two layers: the higher level client operations side that deals with the application apis, and manages the meta and parts, and there’s the objclass level that deals with the rados layout of meta and part objects. There are different objclass methods that deal with reading and modifying each of these entities. A single part has max possible size, however, it may become full once a certain smaller size is reached (full_size_threshold). It is imperative that once a part has reached its capacity, it will not allow any more writes into it. For this reason, it is important that data being written to the queue does not exceed max_entry_size . This is enforced, by the higher level client api. Written entries go to the current head object, and when it’s full, a new head part is created. When listing entries, data is iterated from tail to the current head. Trim can either change the pointer within the current tail object, and if needed it removes tail objects. A unitest has been created to test functionality. Signed-off-by: Yehuda Sadeh <yehuda@redhat.com> Squashed-by: Adam C. Emerson <aemerson@redhat.com>
Rewrite around the asynchronous ASIO-based RADOS interface, and include support for calls from multiple threads. `allocate_unique` pattern contributed by Casey Bodley <cbodley@redhat.com> Co-authored-by: Casey Bodley <cbodley@redhat.com> Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Use optional_yield for most operations and provide an AioCompletion* overload for trim. Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Remove date-oriented trim/list functionality from MDLOG, DataLog, and Sync-Error-Log. Don't provide start/end markers. Support only a start marker for list and an end marker for trim. Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Also only accept one marker. Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
As suggested by Casey, so the compiler can catch failures to return errors. Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Pull out the stuff we aren't using so FIFO and Timelog Datalog can use the same interface. Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
By default, use whatever is present (and use FIFO if nothing is present.) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
To support RGW's MetadataLog, add a flag to FIFO::trim that when true trims up to but not including the given entry. Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
adamemerson
force-pushed
the
wip-datalog-fifo
branch
from
5898b44
to
dbd2ee5
Compare
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Run Datalog atop FIFO (and Omap, depending on configuration.)