rgw: notifications on object replication #43371

liavt · 2021-09-30T17:53:40Z

Optionally publishes notifications upon object replication between zones in the same zone group.

New notification types were added to facilitate this change.

Uses commits and API from #39192

Fixes: no ticket

Signed-off-by: Liav Turkia liav.turkia@gmail.com

Checklist

References tracker ticket
Updates documentation if necessary
Includes tests for new functionality or reproducer for bug

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test dashboard cephadm
jenkins test api
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox

github-actions · 2021-09-30T18:17:54Z

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

cbodley · 2021-09-30T18:57:14Z

src/rgw/rgw_data_sync.cc

+             * we need to fetch info about source object, so that we can determine
+             * the correct policy configuration. This can happen if there are multiple
+             * policy rules, and some depend on the object tagging */
+            yield call(new RGWStatRemoteObjCR(sync_env->async_rados,


so you're making this Stat request unconditional? my vague understanding is that need_more_info is very rarely needed at the moment, so this could be an expensive change. this Stat is also inherently racy, because the remote could overwrite the object between our calls to Stat and FetchRemoteObj

whatever info you need from this Stat request, you should get from FetchRemoteObj instead

cbodley · 2021-09-30T18:58:52Z

src/rgw/rgw_data_sync.cc

+        int ret = rgw::notify::publish_reserve(dpp, rgw::notify::ObjectSyncedCreate, notify_res, &obj_tags);
+        if (ret < 0) {
+          ldpp_dout(dpp, 1) << "ERROR: reserving notification failed, with error: " << ret << dendl;
+          // no need to return, the sync already happened
+        } else {
+          ret = rgw::notify::publish_commit(&obj, src_size, src_mtime, src_etag, 0/* version id */, rgw::notify::ObjectSyncedCreate, notify_res, dpp);


these are blocking calls to librados, which we shouldn't use in coroutines. blocking here means that sync can't make progress with other buckets/objects in the meantime

it would probably be easier to trigger this stuff from RGWRados::fetch_remote_obj(), which is already being called synchronously via the RGWAsyncRadosProcessor thread pool

cbodley · 2021-09-30T19:12:20Z

src/rgw/rgw_data_sync.cc

@@ -2441,6 +2442,33 @@ class RGWObjFetchCR : public RGWCoroutine {
          }
          return set_cr_error(retcode);
        }
+
+        // notify that object has synced to this zone


note that a call to FetchRemoteObj doesn't actually mean that we synced the object. it sends a GET request with the If-Modified-Since header so that it only transfers the object if it's newer than our local copy. if not, we just get ERR_NOT_MODIFIED and don't have to do anything

you can find some extra logic in RGWAsyncFetchRemoteObj::_send_request() for perf counters that tries to detect whether or not it actually transferred anything

thanks @cbodley !

@liavt, moving the notification code to RGWAsyncFetchRemoteObj::_send_request() would simplify the code, on top of resolving the issues in the PR.
it already has: RGWObjectCtx, rgw::sal::Attrs attr, rgw::sal::RadosBucket and rgw::sal::RadosObject available.

in addition, you can use the value of bytes_transferred to verify that the object was indeed synced

I moved the notification code to RGWAsyncFetchRemoteObj::_send_request() and reverted changes to the coroutine, including the one for need_more_info
Commit: 6211ee4

yuvalif · 2021-10-03T15:35:06Z

@liavt please don't forget to sign your commits (use: git commit -s to automatically do that).
note that you don't have to do that right now, you can sign the one commit you will have after you squash them.

stale · 2022-01-09T00:27:33Z

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

mattbenjamin · 2022-01-09T02:45:54Z

@liavt hi; the original lc notifications pr has merged, please try to rebase this on master

yuvalif · 2022-02-09T13:42:44Z

@liavt please remove Matt's commit: 9f6ffe1 (since it is already merged to master) and rebase

liavt · 2022-02-09T14:31:58Z

I removed Matt's commit and rebased the pull request, it still requires testing however (and will be squashed and signed after everything is done)

mattbenjamin · 2022-02-09T14:33:36Z

thanks guys, yay. Just fyi, this should have a tracker ticket, as well :)

yuvalif · 2022-02-09T14:59:46Z

I removed Matt's commit and rebased the pull request, it still requires testing however (and will be squashed and signed after everything is done)

this is great!
please follow these instructions for testing manual testing (let me know if you run into issues): https://gist.github.com/yuvalif/d50bcc3d27121732ffbbafe7b7146112

when i tried that (several months ago), it was eventually crashing, with something that seemed unrelated to the code changed here :-(
note that you cannot start the RGW with gdb, instead, you attach it to gdb after it already started (but before you do something that may crash it...)

run:

pgrep -a radosgw

to get the pid of the RGW you want to attach to.
and then:

gdb bin/radosgw -p <pid>

note that, for some unknown reason, after a while, gdb will stop on a "read()" function (even if no breakpoint is set).
just hit ENTER and the "c" (=continue)

yuvalif · 2022-02-09T15:16:39Z

regarding the rebase, i don't think we should see this commit here: 08e7383

on your local branch, keep only the commits with your code changes
probably better to squash your commits at that point (git rebase -i <hash>^), so that you have less work later on in fixing conflicts
fetch the latest upstream master
rebase your code on top of the latest master (and fix conflicts): git rebase origin/master (assuming "origin" is the upstream and not your fork)

liavt · 2022-02-10T14:16:33Z

regarding the rebase, i don't think we should see this commit here: 08e7383

on your local branch, keep only the commits with your code changes

probably better to squash your commits at that point (git rebase -i <hash>^), so that you have less work later on in fixing conflicts

fetch the latest upstream master

rebase your code on top of the latest master (and fix conflicts): git rebase origin/master (assuming "origin" is the upstream and not your fork)

The rebase should be fixed, the extra commit is gone

yuvalif · 2022-02-14T13:23:10Z

src/rgw/rgw_cr_rados.cc

@@ -713,20 +712,22 @@ int RGWAsyncFetchRemoteObj::_send_request(const DoutPrefixProvider *dpp)
          }
        }

+        // NOTE: we create a mutable copy of bucket.get_tenant as the get_notification function expects a std::string&, not const


i think you can just change the function to use const std::string&
nothing should change the tenant inside the function

Changing the function to use const std::string& has a lot of other side effects and changes to other code, namely the class reservation_t from rgw_notify.h will need to be updated to have a const tenant, as well as any class that deals with notifications (which is a couple)
Ideally I think it should be a separate PR if anything as it touches a lot of unrelated files, and it isn't necessarily required for this one

agree. making all of them const is a good thing.
but lets keep it out of scope of this PR

yuvalif · 2022-02-15T16:09:44Z

src/rgw/rgw_cr_rados.cc

-	   ceph::make_timespan(g_conf()->rgw_op_thread_timeout),
-	   ceph::make_timespan(g_conf()->rgw_op_thread_suicide_timeout),
-	   &m_tp) {
+       ceph::make_timespan(g_conf()->rgw_op_thread_timeout),


nit: please avoid indentation changes in existing code in the file. here as well as lines 77, 78, 112, 130, 667

yuvalif · 2022-02-15T16:10:36Z

src/rgw/rgw_notify_event_type.cc

@@ -42,6 +42,14 @@ namespace rgw::notify {
      return "s3:ObjectLifecycle:Transition:Current";
    case ObjectTransitionNoncurrent:
      return "s3:ObjectLifecycle:Transition:Noncurrent";
+	case ObjectSynced:


nit: seems like indentation is off
(we use 2 spaces and expand tabs)

yuvalif · 2022-02-16T18:21:23Z

src/rgw/rgw_cr_rados.cc

+        std::string tenant(bucket.get_tenant());
+
+        std::unique_ptr<rgw::sal::Notification> notify 
+                 = store->get_notification(dpp, &dest_obj, &src_obj, &obj_ctx, rgw::notify::ObjectSyncedCreate,


please use nullptr instead of src_obj
names are confusing, but for this function "src" and "dest" mean that an object is copied to another object (which is not the case here).

yuvalif · 2022-02-16T18:30:59Z

src/rgw/rgw_cr_rados.cc

+		ldpp_dout(dpp, 0) << "sending sync notification" << dendl;
+
+        // send notification that object was succesfully synced
+        std::string user_id = "0";


probably better to have "req_id" = "0" and "user_id" = "rgw sync"

yuvalif · 2022-02-16T18:31:21Z

src/rgw/rgw_cr_rados.cc

+                 = store->get_notification(dpp, &dest_obj, &src_obj, &obj_ctx, rgw::notify::ObjectSyncedCreate,
+                  &bucket, user_id,
+                  tenant,
+                  user_id, null_yield);


this should be "req_id" and not "user_id"

cbodley · 2022-02-16T21:28:18Z

src/rgw/rgw_cr_rados.cc

+          ldpp_dout(dpp, 1) << "ERROR: reserving notification failed, with error: " << ret << dendl;
+          // no need to return, the sync already happened
+        } else {
+          ret = rgw::notify::publish_commit(&src_obj, src_obj.get_obj_size(), src_mtime, "" /* etag */, "0"/* version id */, rgw::notify::ObjectSyncedCreate, notify_res, dpp);


can you not get the realm etag and version id here?

it looks like you could get the etag from the string *petag argument of fetch_remote_obj()

dest_obj.get_instance() should give you the version id

cbodley · 2022-02-16T21:29:59Z

src/rgw/rgw_cr_rados.cc

+  } else {
+      // r >= 0
+      if (bytes_transferred) {
+		ldpp_dout(dpp, 0) << "sending sync notification" << dendl;


can we drop this log message? in most cases, notifications for sync won't be enabled on the bucket, so this message would be misleading

👍
It was here for debugging right now but it can be removed once the commit is finalized for the PR

sounds good

yuvalif · 2022-02-23T08:50:02Z

@liavt I think the code is ready! next steps for this PR:

move out of draft state
squash commits
sign commit
add documentation
- new types compatibility doc
- add a note there with short description of the new notifications and also explaining they have to be set explicitly on each zone
teuthology regression testing (I will own that part)

next steps for other PRs:

add integration test
finish the "example" PR
fix comment

yuvalif · 2022-02-28T08:50:55Z

jenkins test make check

yuvalif · 2022-03-07T08:45:35Z

jenkins test docs

yuvalif · 2022-03-07T17:16:15Z

teuthology run has 6 failures:
http://pulpito.front.sepia.ceph.com/yuvalif-2022-03-07_12:52:41-rgw-wip-yuval-sync-notifications-distro-basic-smithi/
mostly overlap with baseline: http://pulpito.front.sepia.ceph.com/yuriw-2022-03-04_22:06:30-rgw-master-distro-default-smithi/.

however, bucket notification test is failing due to RGW crash:

1: /lib64/libpthread.so.0(+0x12c20) [0x7ff38ba43c20]
2: /lib64/librados.so.2(+0xfe053) [0x7ff38dd7c053]
3: /lib64/librados.so.2(+0xff01b) [0x7ff38dd7d01b]
4: /lib64/librados.so.2(+0x1041d6) [0x7ff38dd821d6]
5: (DispatchQueue::fast_dispatch(boost::intrusive_ptr<Message> const&)+0x1a0) [0x7ff38c642ba0]
6: (ProtocolV2::handle_message()+0x12c6) [0x7ff38c72b346]
7: (ProtocolV2::handle_read_frame_dispatch()+0x238) [0x7ff38c73dc28]
8: (ProtocolV2::_handle_read_frame_epilogue_main()+0x85) [0x7ff38c73dd15]
9: (ProtocolV2::handle_read_frame_epilogue_main(std::unique_ptr<ceph::buffer::v15_2_0::ptr_node, ceph::buffer::v15_2_0::ptr_node::disposer>&&, int)+0x204) [0x7ff38c73f2c4]
10: (ProtocolV2::run_continuation(Ct<ProtocolV2>&)+0x3c) [0x7ff38c726a3c]
11: (AsyncConnection::process()+0x789) [0x7ff38c6ed0f9]
12: (EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0x1547) [0x7ff38c74cf87]
13: /usr/lib64/ceph/libceph-common.so.2(+0x61aa4e) [0x7ff38c754a4e]
14: /lib64/libstdc++.so.6(+0xc2ba3) [0x7ff38aa69ba3]
15: /lib64/libpthread.so.0(+0x817f) [0x7ff38ba3917f]
16: clone()

the backtrace does not seem related to the code changes, however, it is probably due to something in this PR.
log around the crash is here:
http://qa-proxy.ceph.com/teuthology/yuvalif-2022-03-07_12:52:41-rgw-wip-yuval-sync-notifications-distro-basic-smithi/6724876/remote/smithi161/crash/posted/2022-03-07T13%3A17%3A35.979405Z_1c4627dc-6ad8-4206-b24e-d327760efea1/log

liavt · 2022-03-08T14:39:41Z

The pulpito links do not seem to work for me, the requests time out

Regarding the backtrace, it doesn't seem like that passes through anything changed by the PR. Do you know if there is a way to reproduce this locally or have an area of the code that could be affected by the changes?

yuvalif · 2022-03-13T12:53:08Z

The pulpito links do not seem to work for me, the requests time out

Regarding the backtrace, it doesn't seem like that passes through anything changed by the PR. Do you know if there is a way to reproduce this locally or have an area of the code that could be affected by the changes?

sorry, can you try this link (the above use the sepia VPN)?
https://pulpito.ceph.com/yuvalif-2022-03-07_12:52:41-rgw-wip-yuval-sync-notifications-distro-basic-smithi/

you can run the bucket notification test locally to see why it crashes. the directory holding the test code and instructions is here:
https://github.com/ceph/ceph/tree/master/src/test/rgw/bucket_notification
you can try first without the kafka./amqp setup (their tests will ber skipped).
if the crash is not reproduced then you can try these tests as well.

yuvalif · 2022-04-07T09:57:48Z

src/rgw/rgw_cr_rados.cc

+                 = store->get_notification(dpp, &dest_obj, nullptr, &obj_ctx, rgw::notify::ObjectSyncedCreate,
+                  &dest_bucket, user_id,
+                  tenant,
+                  req_id, null_yield);


using null_yield here may have a negative impact on sync performance.
in such a case, the notification sending code would just block using a mutex until the ack for the notification is received.
this needs more investigation since we are already inside a coroutine, there should be a way to yield and do async waits.

this needs more investigation since we are already inside a coroutine, there should be a way to yield and do async waits.

unfortunately these are two different kinds of coroutines. they could potentially work together like you describe (i.e. a RGWCoroutine could spawn::spawn() a yield context and wait on it), but that would be a big project. that could be a stepping stone on the way to a better multisite, but we might also be able to avoid it - that's worth discussion

but classes like RGWAsyncFetchRemoteObj that inherit from RGWAsyncRadosRequest are actually being handed off to a thread pool in RGWAsyncRadosProcessor to run synchronously. fetch_remote_obj() itself is blocking, so this blocking null_yield is harmless

yuvalif · 2022-04-07T15:37:47Z

can still see crashes here: http://qa-proxy.ceph.com/teuthology/yuvalif-2022-04-07_08:28:12-rgw:notifications-wip-yuval-sync-notifications-distro-basic-smithi/6780814/remote/smithi135/crash/posted/

crash here makes more sense (even though it is not in the code introduced in this PR). from the log:

-174> 2022-04-07T08:48:14.734+0000 7f6fc0119700 20 AMQP run: multiple n/acks received with tag=131 and result=0
-173> 2022-04-07T08:48:14.734+0000 7f6fc0119700 20 AMQP run: invoking callback with tag=125
-154> 2022-04-07T08:48:14.739+0000 7f6fc0119700 -1 *** Caught signal (Aborted) **
 in thread 7f6fc0119700 thread_name:amqp_manager

 ceph version 17.0.0-11458-gd46c8509 (d46c85091b2d82ebff5510920a39fc8c7071290c) quincy (dev)
 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f6ff50a3ce0]
 2: gsignal()
 3: abort()
 4: /lib64/libc.so.6(+0x21c89) [0x7f6ff36c9c89]
 5: /lib64/libc.so.6(+0x473a6) [0x7f6ff36ef3a6]
 6: /lib64/libpthread.so.0(+0x13ace) [0x7f6ff50a4ace]
 7: (RGWPubSubAMQPEndpoint::Waiter::finish(int)+0xc3) [0x7f6ff80fa0d3]
 8: (rgw::amqp::Manager::run()+0x234e) [0x7f6ff840fe3e]
 9: /lib64/libstdc++.so.6(+0xc2ba3) [0x7f6ff40c9ba3]
 10: /lib64/libpthread.so.0(+0x81cf) [0x7f6ff50991cf]
 11: clone()
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

looks like an AMQP ack was received, triggering the call to RGWPubSubAMQPEndpoint::Waiter::finish() that would release the async wait.

yuvalif · 2022-04-12T05:04:29Z

after another rebase, notification tests are not crashing anymore:
http://pulpito.front.sepia.ceph.com/yuvalif-2022-04-12_04:19:06-rgw:notifications-wip-yuval-sync-notifications-distro-basic-smithi/
the failure is expected, unrelated to this change, and handled by another PR

@liavt please rebase and force push the rebased code

yuvalif · 2022-04-13T09:47:23Z

multisite tests are also passing:
http://pulpito.front.sepia.ceph.com/yuvalif-2022-04-13_08:44:21-rgw:multisite-wip-yuval-sync-notifications-distro-basic-smithi/
failures are expected:

version_suspended_incremental_sync is also failing on master
9 pubsub tests are also failing on master

yuvalif · 2022-04-25T12:51:59Z

src/rgw/rgw_cr_rados.cc

+        std::string tenant(dest_bucket.get_tenant());
+
+        std::unique_ptr<rgw::sal::Notification> notify 
+                 = store->get_notification(dpp, &dest_obj, nullptr, &obj_ctx, rgw::notify::ObjectSyncedCreate,


this does not compile now:

error: no matching function for call to ‘rgw::sal::RadosStore::get_notification(...

the function changes to:

rgw_sal_rados.h:97:43: note: candidate: ‘virtual std::unique_ptr<rgw::sal::Notification> rgw::sal::RadosStore::get_notification(const DoutPrefixProvider*, rgw::sal::Object*, rgw::sal::Object*, rgw::notify::EventType, rgw::sal::Bucket*, std::string&, std::string&, std::string&, optional_yield)’

I think you should remove: obj_ctx

Signed-off-by: liavt <liav.turkia@gmail.com>

liavt · 2022-04-26T19:02:28Z

@yuvalif I removed obj_ctx and it compiles for me now

yuvalif · 2022-04-27T14:27:40Z

notification tests are passing after rebase (error exist in baseline as well):
http://pulpito.front.sepia.ceph.com/yuvalif-2022-04-27_12:08:42-rgw:multisite-wip-yuval-syn-notifications-distro-basic-smithi/

yuvalif · 2022-04-28T10:03:58Z

full test suite is passing: http://pulpito.front.sepia.ceph.com/yuvalif-2022-04-27_14:28:25-rgw-wip-yuval-syn-notifications-distro-basic-smithi/
with failures similar to baseline:
http://pulpito.front.sepia.ceph.com/cbodley-2022-04-26_18:09:12-rgw-master-distro-default-smithi/

ceph-jenkins · 2022-04-28T10:04:00Z

Can one of the admins verify this patch?

mattbenjamin

lgtm

mattbenjamin · 2021-09-30T18:01:51Z

src/rgw/rgw_notify_event_type.cc

@@ -30,6 +30,14 @@ namespace rgw::notify {
      return "s3:ObjectLifecycle:NoncurrentExpiration";
    case ObjectDeleteMarkerExpiration:
      return "s3:ObjectLifecycle:DeleteMarkerExpiration";
+	case ObjectSynced:
+	  return "s3:ObjectSynced:*";


I did use s3:ObjectLifecycle in my lifecycle notifications change, but should we possibly use rgw: here (and maybe there, I don't know--it's a little less clear given that the lifecycle features are AWS compatible)

yuvalif · 2022-04-28T16:11:45Z

remaining work on the feature: https://gist.github.com/yuvalif/0db188fc63db40af3229f7bd63407bfb
should be done in a followup PR

tchaikov · 2022-04-30T01:58:34Z

@yuvalif hi Yuval, in future, could you please add "Reviewed-by" lines in the merge commit when merging pull requests?

mattbenjamin · 2022-04-30T02:35:34Z

@yuvalif hi Yuval, in future, could you please add "Reviewed-by" lines in the merge commit when merging pull requests?

Isn't that something we can rely on github to do, since we have approving github reviewers?

github-actions bot added the rgw label Sep 30, 2021

mattbenjamin requested review from yuvalif and cbodley September 30, 2021 17:58

github-actions bot added the needs-rebase label Sep 30, 2021

cbodley reviewed Sep 30, 2021

View reviewed changes

stale bot added the stale label Jan 9, 2022

stale bot removed the stale label Jan 9, 2022

liavt force-pushed the master branch from 6211ee4 to 08e7383 Compare February 9, 2022 14:30

github-actions bot removed the needs-rebase label Feb 9, 2022

liavt force-pushed the master branch from 08e7383 to 716cc7c Compare February 10, 2022 14:15

yuvalif reviewed Feb 14, 2022

View reviewed changes

yuvalif reviewed Feb 15, 2022

View reviewed changes

yuvalif reviewed Feb 16, 2022

View reviewed changes

cbodley reviewed Feb 16, 2022

View reviewed changes

yuvalif added needs-doc needs-qa labels Feb 23, 2022

yuvalif self-requested a review February 28, 2022 08:51

yuvalif approved these changes Feb 28, 2022

View reviewed changes

liavt force-pushed the master branch from 7db9592 to f9b29c4 Compare March 5, 2022 17:50

yuvalif removed the needs-doc label Mar 7, 2022

yuvalif added the wip-yuval-testing label Mar 7, 2022

yuvalif reviewed Apr 7, 2022

View reviewed changes

yuvalif added the needs-rebase label Apr 12, 2022

github-actions bot removed the needs-rebase label Apr 12, 2022

liavt force-pushed the master branch from f9b29c4 to 3eab59a Compare April 13, 2022 19:31

yuvalif reviewed Apr 25, 2022

View reviewed changes

rgw: notifications on object replication

b303ae1

Signed-off-by: liavt <liav.turkia@gmail.com>

liavt force-pushed the master branch from 3eab59a to b303ae1 Compare April 26, 2022 19:01

yuvalif removed the needs-qa label Apr 27, 2022

mattbenjamin approved these changes Apr 28, 2022

View reviewed changes

yuvalif merged commit cc5354e into ceph:master Apr 28, 2022
11 checks passed

rgw: notifications on object replication #43371

rgw: notifications on object replication #43371

Conversation

liavt commented Sep 30, 2021 • edited by yuvalif

Checklist

github-actions bot commented Sep 30, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuvalif Oct 3, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuvalif commented Oct 3, 2021

stale bot commented Jan 9, 2022

mattbenjamin commented Jan 9, 2022

yuvalif commented Feb 9, 2022

liavt commented Feb 9, 2022 • edited

mattbenjamin commented Feb 9, 2022

yuvalif commented Feb 9, 2022

yuvalif commented Feb 9, 2022

liavt commented Feb 10, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuvalif Feb 15, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuvalif commented Feb 23, 2022 • edited

yuvalif commented Feb 28, 2022

yuvalif commented Mar 7, 2022

yuvalif commented Mar 7, 2022

liavt commented Mar 8, 2022 • edited

yuvalif commented Mar 13, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuvalif commented Apr 7, 2022

yuvalif commented Apr 12, 2022 • edited

yuvalif commented Apr 13, 2022

Choose a reason for hiding this comment

liavt commented Apr 26, 2022

yuvalif commented Apr 27, 2022

yuvalif commented Apr 28, 2022

ceph-jenkins commented Apr 28, 2022

mattbenjamin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuvalif commented Apr 28, 2022

tchaikov commented Apr 30, 2022

mattbenjamin commented Apr 30, 2022

liavt commented Sep 30, 2021 •

edited by yuvalif

yuvalif Oct 3, 2021 •

edited

liavt commented Feb 9, 2022 •

edited

yuvalif Feb 15, 2022 •

edited

yuvalif commented Feb 23, 2022 •

edited

liavt commented Mar 8, 2022 •

edited

yuvalif commented Apr 12, 2022 •

edited