Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mds/quiesce: fix timeouts, a crash, and overdrive a tree export when possible #57579

Merged
merged 11 commits into from
May 28, 2024

Conversation

leonid-s-usov
Copy link
Contributor

@leonid-s-usov leonid-s-usov commented May 20, 2024

Fixes: https://tracker.ceph.com/issues/66152 - mds/quiesce: holding remote authpins for the duration of the quiesce op may cause deadlocks
Fixes: https://tracker.ceph.com/issues/66123 - Quiesce timeout due to exporting
Fixes: https://tracker.ceph.com/issues/66208 - Segfault when quiesce_overdrive_fragmenting synchronously calls dispatch_fragment_dir for an abort
Fixes: https://tracker.ceph.com/issues/66219 - [quiesce timeout] QuiesceAgent may send an async QUIESCED ack before the QuiesceManager does the sync QUIESCING ack, which causes the QUIESCED ack to be lost
Fixes: https://tracker.ceph.com/issues/66225 - [quiesce] disable debug parameters on quiesce roots

Show available Jenkins commands
  • jenkins retest this please
  • jenkins test classic perf
  • jenkins test crimson perf
  • jenkins test signed
  • jenkins test make check
  • jenkins test make check arm64
  • jenkins test submodules
  • jenkins test dashboard
  • jenkins test dashboard cephadm
  • jenkins test api
  • jenkins test docs
  • jenkins render docs
  • jenkins test ceph-volume all
  • jenkins test ceph-volume tox
  • jenkins test windows
  • jenkins test rook e2e

@github-actions github-actions bot added cephfs Ceph File System common tests labels May 20, 2024
@leonid-s-usov leonid-s-usov changed the title Quiesce should actively cancel an ongoing quiesce if it hasn't frozen the tree yet mds/quiesce: overdrive exporting that is still freezing May 20, 2024
@leonid-s-usov leonid-s-usov changed the title mds/quiesce: overdrive exporting that is still freezing mds/quiesce: overdrive export that is still freezing May 20, 2024
@leonid-s-usov leonid-s-usov force-pushed the wip-lusov-quiesce-overdrive-export branch from 3b748ea to 7956b5e Compare May 21, 2024 07:37
@leonid-s-usov leonid-s-usov marked this pull request as ready for review May 21, 2024 07:39
@leonid-s-usov
Copy link
Contributor Author

jenkins test make check arm64

1 similar comment
@batrick
Copy link
Member

batrick commented May 21, 2024

jenkins test make check arm64

Copy link
Member

@batrick batrick left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mds/quiesce: don't force a remote authpin for the quiesce lock was not part of the two runs I did for

https://pulpito.ceph.com/?branch=wip-lusov-quiesce-overdrive-export

I would prefer to keep that in another PR/ticket. Let's get the export fix merged quickly.

@leonid-s-usov
Copy link
Contributor Author

mds/quiesce: don't force a remote authpin for the quiesce lock was not part of the two runs I did for

https://pulpito.ceph.com/?branch=wip-lusov-quiesce-overdrive-export

I would prefer to keep that in another PR/ticket. Let's get the export fix merged quickly.

I'm not sure. I think they work together, where the absence of the remote authpin in most of the cases eliminates the possibility for a deadlock. It also reduces the inter-rank messaging. And it's true to the current design of the quiesce. And it's required to fix that test that fails otherwise.

Your run shows one quiesce timeout due to exporting. I have looked at that one and it appears to be the renaming issue or rather a different kind of it.

I'm re-running just the exporter with full replication x8 to stress-test this PR: https://pulpito.ceph.com/leonidus-2024-05-21_18:10:18-fs-wip-lusov-quiesce-overdrive-export-distro-default-smithi/

@leonid-s-usov
Copy link
Contributor Author

I'm re-running just the exporter with full replication x8 to stress-test this PR: https://pulpito.ceph.com/leonidus-2024-05-21_18:10:18-fs-wip-lusov-quiesce-overdrive-export-distro-default-smithi/

[leonidus@vossi04 leonidus-2024-05-21_18:10:18-fs-wip-lusov-quiesce-overdrive-export-distro-default-smithi]$ grep "ERROR:tasks.quiescer" */teuthology.log
[leonidus@vossi04 leonidus-2024-05-21_18:10:18-fs-wip-lusov-quiesce-overdrive-export-distro-default-smithi]$

No quiesce errors. I do think that the absence of the authpin plays the role. To validate it I'll rerun the same suite as above but with the previous version of the branch, without the authpin change

@leonid-s-usov
Copy link
Contributor Author

To validate it I'll rerun the same suite as above but with the previous version of the branch, without the authpin change

https://pulpito.ceph.com/leonidus-2024-05-21_21:13:35-fs-wip-lusov-quiesce-overdrive-export-distro-default-smithi/

@leonid-s-usov
Copy link
Contributor Author

leonid-s-usov commented May 22, 2024

This test puts some strain on the system. However, I couldn't find any correlation between the presence of the AP commit and the results.

Here are the quiesce times from both runs above, sorted by duration. One test had 4 less quiesces, but both did ~200

image

It is still to be validated whether the rename issue would be affected by the AP change

@leonid-s-usov
Copy link
Contributor Author

@leonid-s-usov
Copy link
Contributor Author

leonid-s-usov commented May 22, 2024

Yes, I found that issue with the rename, and it's indeed due to the remote authpin from the quiesce request.

quiesce request on mds.1:

 "description": "internal op quiesce_inode:mds.1:50612 fp=#0x1000000383a"

took authpin on mds 0:

2024-05-21T03:24:08.269+0000 7f6319567700  7 mds.0.server dispatch_peer_request request(mds.1:50612 nref=2 peer_to mds.1 sr=0x56059607a700) peer_request(mds.1:50612.0 authpin)
2024-05-21T03:24:08.269+0000 7f6319567700 10 mds.0.server handle_peer_auth_pin request(mds.1:50612 nref=2 peer_to mds.1 sr=0x56059607a700)
2024-05-21T03:24:08.269+0000 7f6319567700 10 mds.0.server auth_pinning [inode 0x1000000383a [c,head] /volumes/qa/sv_1/d8fbe2d3-1fdc-4c3a-b7dc-a7b20043535f/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-52/comment-9.xml.tmp auth{1=1} v1151 DIRTY
PARENT s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={24833=0-4194304@b} caps={24833=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=24833 | request=0 caps=1 dirtyparent=1 replicated=1 dirty=1 0x56059c154000]
2024-05-21T03:24:08.269+0000 7f6319567700 10 mds.0.cache.ino(0x1000000383a) auth_pin by 0x56059e100d80 on [inode 0x1000000383a [c,head] /volumes/qa/sv_1/d8fbe2d3-1fdc-4c3a-b7dc-a7b20043535f/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-52/comm
ent-9.xml.tmp auth{1=1} v1151 ap=1 DIRTYPARENT s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={24833=0-4194304@b} caps={24833=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=24833 | request=0 caps=1 dirtyparent=1 replicated=1 dirty=1 authpin=1 0x5
6059c154000] now 1

then the rename request tried to authpin-freeze the same inode:

2024-05-21T03:24:08.271+0000 7f6319567700  4 mds.0.server handle_peer_request client.24833:456994 from mds.1
2024-05-21T03:24:08.271+0000 7f6319567700  7 mds.0.cache request_start_peer request(client.24833:456994 nref=2 peer_to mds.1) by mds.1
2024-05-21T03:24:08.271+0000 7f6319567700  7 mds.0.server dispatch_peer_request request(client.24833:456994 nref=2 peer_to mds.1 sr=0x56059ff5f500) peer_request(client.24833:456994.0 authpin)
2024-05-21T03:24:08.271+0000 7f6319567700 10 mds.0.server handle_peer_auth_pin request(client.24833:456994 nref=2 peer_to mds.1 sr=0x56059ff5f500)
2024-05-21T03:24:08.271+0000 7f6319567700 20 mds.0.cache.dir(0x20000001f84.100*) lookup (comment-9.xml.tmp, 'head')
2024-05-21T03:24:08.271+0000 7f6319567700 20 mds.0.cache.dir(0x20000001f84.100*)   hit -> (comment-9.xml.tmp,head)
2024-05-21T03:24:08.271+0000 7f6319567700 10 mds.0.server  freezing auth pin on [inode 0x1000000383a [c,head] /volumes/qa/sv_1/d8fbe2d3-1fdc-4c3a-b7dc-a7b20043535f/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-52/comment-9.xml.tmp auth{1=1} v1
151 ap=1 DIRTYPARENT s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={24833=0-4194304@b} caps={24833=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=24833 | request=0 caps=1 dirtyparent=1 replicated=1 dirty=1 authpin=1 0x56059c154000]
2024-05-21T03:24:08.271+0000 7f6319567700 10 mds.0.cache.ino(0x1000000383a) auth_pin by 0x56059e103180 on [inode 0x1000000383a [c,head] /volumes/qa/sv_1/d8fbe2d3-1fdc-4c3a-b7dc-a7b20043535f/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-52/comm
ent-9.xml.tmp auth{1=1} v1151 ap=2 DIRTYPARENT s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={24833=0-4194304@b} caps={24833=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=24833 | request=0 caps=1 dirtyparent=1 replicated=1 dirty=1 authpin=1 0x5
6059c154000] now 2 <--- 
2024-05-21T03:24:08.271+0000 7f6319567700 15 mds.0.cache.dir(0x20000001f84.100*) adjust_nested_auth_pins 1 on [dir 0x20000001f84.100* /volumes/qa/sv_1/d8fbe2d3-1fdc-4c3a-b7dc-a7b20043535f/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-52/ [2,head] auth{1=1,2=1} v=1152 cv=763/763 REP ap=0+9 state=1610874881|complete f(v5 m2024-05-21T03:24:04.636446+0000 26=26+0) n(v7 rc2024-05-21T03:24:04.636446+0000 b566673 26=26+0) hs=26+12,ss=31+0 dirty=69 | ptrwaiter=0 child=1 frozen=0 subtree=0 importing=0 importbound=0 sticky=0 replicated=1 dirty=1 authpin=0 0x560592cacd00] by 0x56059c154000 count now 0/9
2024-05-21T03:24:08.271+0000 7f6319567700 10 mds.0.cache.ino(0x1000000383a) freeze_inode - waiting for auth_pins to drop to 1
2024-05-21T03:24:08.271+0000 7f6319567700 10 mds.0.cache.ino(0x1000000383a) add_waiter tag 2 0x56059bda5d40 !ambig 1 !frozen 1 !freezing 0

this inode is now freezing and is blocking the quiesce but it can't freeze until the quiesce is over and that first authpin is lifted --> deadlock

@leonid-s-usov leonid-s-usov force-pushed the wip-lusov-quiesce-overdrive-export branch from 7956b5e to 0cfaf3a Compare May 22, 2024 12:08
@leonid-s-usov
Copy link
Contributor Author

scheduling 20 jobs of the workload that has seen the rename issue.

without the AP change: https://pulpito.ceph.com/leonidus-2024-05-22_06:28:15-fs-wip-lusov-quiesce-overdrive-export-distro-default-smithi/

[leonidus@vossi04 leonidus-2024-05-22_06:28:15-fs-wip-lusov-quiesce-overdrive-export-distro-default-smithi]$ grep "ERROR:tasks.quiescer" */teuthology.log
7720264/teuthology.log:2024-05-22T07:08:02.140 ERROR:tasks.quiescer.fs.[cephfs]:Couldn't quiesce root with rc: 110 (ETIMEDOUT), stdout:
7720266/teuthology.log:2024-05-22T07:06:03.878 ERROR:tasks.quiescer.fs.[cephfs]:Couldn't quiesce root with rc: 110 (ETIMEDOUT), stdout:
7720277/teuthology.log:2024-05-22T07:23:18.124 ERROR:tasks.quiescer.fs.[cephfs]:Couldn't quiesce root with rc: 110 (ETIMEDOUT), stdout:
7720278/teuthology.log:2024-05-22T07:23:15.774 ERROR:tasks.quiescer.fs.[cephfs]:Couldn't quiesce root with rc: 110 (ETIMEDOUT), stdout:
7720279/teuthology.log:2024-05-22T07:28:07.997 ERROR:tasks.quiescer.fs.[cephfs]:Couldn't quiesce root with rc: 1 (EPERM), stdout:

with the AP change: https://pulpito.ceph.com/leonidus-2024-05-22_06:28:39-fs-wip-lusov-quiesce-overdrive-export-distro-default-smithi/

[leonidus@vossi04 leonidus-2024-05-22_06:28:39-fs-wip-lusov-quiesce-overdrive-export-distro-default-smithi]$ grep "ERROR:tasks.quiescer" */teuthology.log
[leonidus@vossi04 leonidus-2024-05-22_06:28:39-fs-wip-lusov-quiesce-overdrive-export-distro-default-smithi]$

@leonid-s-usov
Copy link
Contributor Author

@batrick can we please have a single PR for those two tickets? They will have to be backported everywhere in a batch... 🙏🏻 🙏🏻

@leonid-s-usov leonid-s-usov changed the title mds/quiesce: overdrive export that is still freezing mds/quiesce: fix deadlocks due to export and remote auth pinning May 22, 2024
@leonid-s-usov leonid-s-usov force-pushed the wip-lusov-quiesce-overdrive-export branch from 0cfaf3a to 9dc8c01 Compare May 22, 2024 13:11
@leonid-s-usov
Copy link
Contributor Author

jenkins test make check arm64

Copy link
Member

@batrick batrick left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't forget to QA -s fs:functional --filter quiesce.

mds/qiuesce: allow quiescing a single file in dispatch_quiesce_path

will break test_quiesce_path_regfile

mds: quiesce_path: don't block the asock thread and return an adequate rc -> mds: do not block the asok thread via quiesce_path and return an adequate rc

otherwise the change is 👍

mds: add lifetimecmd param tolock path`` 👍

qa/cephfs: quiesce: test that a quiesce op doesn't hold remote ap 👍

Other fixes are good. You've convinced me to combine the changes.

@leonid-s-usov leonid-s-usov force-pushed the wip-lusov-quiesce-overdrive-export branch 2 times, most recently from 240841e to 6e074e1 Compare May 22, 2024 21:42
@leonid-s-usov leonid-s-usov force-pushed the wip-lusov-quiesce-overdrive-export branch 4 times, most recently from 8e0427a to 11ffcf2 Compare May 23, 2024 21:03
@leonid-s-usov
Copy link
Contributor Author

leonid-s-usov commented May 25, 2024

Latest:
Functional - pass

Stress exporting/renaming/fragmenting - pass (*half of the jobs had failed because of the backtrace scrub, but no quiesce failures)

[leonidus@vossi04 leonidus-2024-05-24_04:47:38-fs-wip-lusov-quiesce-distro-default-smithi]$ grep "ERROR:tasks.quiescer" */teuthology.log
[leonidus@vossi04 leonidus-2024-05-24_04:47:38-fs-wip-lusov-quiesce-distro-default-smithi]$ grep "Successfully quiesced" -c */teuthology.log
7724261/teuthology.log:25
7724262/teuthology.log:25
7724263/teuthology.log:25
7724264/teuthology.log:25
7724265/teuthology.log:25
7724266/teuthology.log:25
7724268/teuthology.log:25
7724269/teuthology.log:78
7724270/teuthology.log:78
7724271/teuthology.log:74
7724272/teuthology.log:75
7724273/teuthology.log:75
7724274/teuthology.log:78
7724275/teuthology.log:78
7724276/teuthology.log:78

General with-quiesce: 2 timeouts

[leonidus@vossi04 leonidus-2024-05-24_04:50:13-fs-wip-lusov-quiesce-distro-default-smithi]$ grep "ERROR:tasks.quiescer" */teuthology.log
7724309/teuthology.log:2024-05-24T06:39:43.050 ERROR:tasks.quiescer.fs.[cephfs]:Couldn't quiesce root with rc: 110 (ETIMEDOUT), stdout:
7724309/teuthology.log:2024-05-24T06:39:43.051 ERROR:tasks.quiescer.fs.[cephfs]:exception:
7724310/teuthology.log:2024-05-24T06:38:56.474 ERROR:tasks.quiescer.fs.[cephfs]:Couldn't quiesce root with rc: 110 (ETIMEDOUT), stdout:
7724310/teuthology.log:2024-05-24T06:38:56.474 ERROR:tasks.quiescer.fs.[cephfs]:exception:

7724309 and 772410 is a problem with the QuiesceAgent ack reordering, reported this in a new defect: https://tracker.ceph.com/issues/66219

… adequate rc

Signed-off-by: Leonid Usov <leonid.usov@ibm.com>
Signed-off-by: Leonid Usov <leonid.usov@ibm.com>
Signed-off-by: Leonid Usov <leonid.usov@ibm.com>
Signed-off-by: Leonid Usov <leonid.usov@ibm.com>
Signed-off-by: Leonid Usov <leonid.usov@ibm.com>
1. avoid taking a remote authpin for the quiesce lock
2. drop remote authpins that were taken because of other locks

We should not be forcing a mustpin when taking quiesce lock.
This creates unnecessary overhead due to the distributed nature
of the quiesce: all ranks will execute quiesce_inode, including
the auth rank, which will authpin the inode.

Auth pinning on the auth rank is important to synchronize quiesce
with operations that are managed by the auth, like fragmenting
and exporting.

If we let a remote quiesce process take a foreign authpin then
it may block freezing on the auth, which will stall quiesce locally.
This wouldn't be a problem if the quiesce that is blocked on the auth
and the quiesce that's holding a remote authpin from the replica side
were unrelated, but in our case it may be the same logical quiesce
that effectively steps on its own toes. This creates an opportunity
for a deadlock.

Fixes: https://tracker.ceph.com/issues/66152
Signed-off-by: Leonid Usov <leonid.usov@ibm.com>
Just like with the fragmenting, we should abort an ongoing export
if a quiesce is attempted for the directory.

To minimize the stress for the system, we only allow the abort
if the export hasn't yet managed to freeze the tree. If that is the case,
then quiesce will have to wait for the export to finish.

Fixes: https://tracker.ceph.com/issues/66123
Signed-off-by: Leonid Usov <leonid.usov@ibm.com>
In this scenario, the agent thread is able to run and generate an ack
before the db_update call returns to the caller.

Fixes: https://tracker.ceph.com/issues/66219
Signed-off-by: Leonid Usov <leonid.usov@ibm.com>
Defer to the agent thread to perform all acking.
This avoids race conditions between the updating thread
and the acking thread.

Fixes: https://tracker.ceph.com/issues/66219
Signed-off-by: Leonid Usov <leonid.usov@ibm.com>
@leonid-s-usov leonid-s-usov force-pushed the wip-lusov-quiesce-overdrive-export branch from 11ffcf2 to 9a4c585 Compare May 26, 2024 08:34
@leonid-s-usov leonid-s-usov changed the title mds/quiesce: fix deadlocks due to export and remote auth pinning mds/quiesce: fix deadlocks due to export, remote auth pinning, and reordered agent acks May 26, 2024
@leonid-s-usov leonid-s-usov changed the title mds/quiesce: fix deadlocks due to export, remote auth pinning, and reordered agent acks mds/quiesce: fix three quiesce timeouts and a crash May 26, 2024
@leonid-s-usov leonid-s-usov changed the title mds/quiesce: fix three quiesce timeouts and a crash mds/quiesce: fix two quiesce timeouts and a crash May 26, 2024
@leonid-s-usov leonid-s-usov changed the title mds/quiesce: fix two quiesce timeouts and a crash mds/quiesce: fix timeouts, a crash, and overdrive a tree export when possible May 26, 2024
@leonid-s-usov
Copy link
Contributor Author

jenkins test make check

@leonid-s-usov
Copy link
Contributor Author

jenkins test windows

@leonid-s-usov
Copy link
Contributor Author

A rerun with the fix of https://tracker.ceph.com/issues/66219 found by the tests in #57579 (comment):

Functional - pass

Export/replication kernel_untar_build stress 8/20 pass, no quiesce errors

[leonidus@vossi04 leonidus-2024-05-26_10:47:39-fs-wip-lusov-quiesce-distro-default-smithi]$ grep "ERROR:tasks.quiescer" */teuthology.log
[leonidus@vossi04 leonidus-2024-05-26_10:47:39-fs-wip-lusov-quiesce-distro-default-smithi]$

A generic with-quiesce suite - 19/28 pass, no quiesce errors

[leonidus@vossi04 leonidus-2024-05-26_10:46:27-fs-wip-lusov-quiesce-distro-default-smithi]$ grep "ERROR:tasks.quiescer" */teuthology.log
[leonidus@vossi04 leonidus-2024-05-26_10:46:27-fs-wip-lusov-quiesce-distro-default-smithi]$

A second generic with-quiesce suite - 23/30 passed, 1 quiesce timeout:

[leonidus@vossi04 leonidus-2024-05-26_10:48:16-fs-wip-lusov-quiesce-distro-default-smithi]$ grep "ERROR:tasks.quiescer" */teuthology.log
7726344/teuthology.log:2024-05-26T12:51:16.035 ERROR:tasks.quiescer.fs.[cephfs]:Couldn't quiesce root with rc: 110 (ETIMEDOUT), stdout:

The analysis suggests that this is a client issue, see https://tracker.ceph.com/issues/66229.

In total, there were 980 successful quiesces over the three runs of this batch. The quiesce times are measured by the script, and as such are not reliable, but still, here's the distribution:

image

@leonid-s-usov
Copy link
Contributor Author

A re-run with the change for [quiesce] disable debug parameters on quiesce roots

Functional - pass

A generic with-quiesce suite - 28/36 passed, 2 quiesce errors:

[leonidus@vossi04 leonidus-2024-05-27_07:25:29-fs-wip-lusov-quiesce-distro-default-smithi]$ grep "ERROR:tasks.quiescer" */teuthology.log
7727755/teuthology.log:2024-05-27T08:58:00.992 ERROR:tasks.quiescer.fs.[cephfs]:Couldn't quiesce root with rc: 110 (ETIMEDOUT), stdout:

7727756/teuthology.log:2024-05-27T08:36:38.524 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.4299 quiesce db --set-id 494d7fba --release --await'
7727756/teuthology.log:2024-05-27T08:38:38.557 ERROR:tasks.quiescer.fs.[cephfs]:exception: (124)

Both issues are due to messaging errors preventing the delivery of an ack:

7727755 - during the quiesce

2024-05-27T08:56:30.985+0000 7f8886fbe640 20 quiesce.agt <agent_thread_main> asyncrhonous ack for a new version: q-map[v:(14:201) roots:1/0]
2024-05-27T08:56:30.985+0000 7f8886fbe640 10 quiesce.mds.1 <operator()> sending ack q-map[v:(14:201) roots:1/0] to the leader 4288
2024-05-27T08:56:30.985+0000 7f8886fbe640  1 -- [v2:172.21.15.81:6834/957495742,v1:172.21.15.81:6837/957495742] send_to--> mds [v2:172.21.15.81:6835/1830870643,v1:172.21.15.81:6838/1830870643] --  -- ?+0 0x56329891fe00
2024-05-27T08:56:30.985+0000 7f8886fbe640  1 -- [v2:172.21.15.81:6834/957495742,v1:172.21.15.81:6837/957495742] --> [v2:172.21.15.81:6835/1830870643,v1:172.21.15.81:6838/1830870643] --  -- 0x56329891fe00 con 0x563297f4d400
2024-05-27T08:56:30.985+0000 7f88927d5640  1 -- [v2:172.21.15.81:6834/957495742,v1:172.21.15.81:6837/957495742] >> [v2:172.21.15.81:6835/1830870643,v1:172.21.15.81:6838/1830870643] conn(0x563297f4d400 msgr2=0x563298204680 crc :-1 s=STATE_CONNECT
ION_ESTABLISHED l=0).read_bulk peer close file descriptor 38

7727756 - during the release

22151270:2024-05-27T08:36:38.779+0000 7f59bed33640 20 quiesce.mgr.4299 <check_peer_reports> [494d7fba@127,file:/] up_to_date_peers: 2 min_reported_state: QS_QUIESCED max_reported_state: QS_QUIESCED peer_acks: {{QS_QUIESCED=4282,(13:127),QS_QUIES
CED=4296,(13:126),QS_QUIESCED=4299,(13:127)}}
22151274:2024-05-27T08:36:38.779+0000 7f59c7d45640 -1 quiesce.mds.0 <quiesce_dispatch> failed to decode message of type 1286 v1: End of buffer [buffer:2]

Stress test with replication and exporting
- 6/20 passed. Very hight failure rate due to different reasons, but 0 quiesce errors with ~30 successful quiesces per job

[leonidus@vossi04 leonidus-2024-05-27_07:27:31-fs-wip-lusov-quiesce-distro-default-smithi]$ grep "ERROR:tasks.quiescer" */teuthology.log
[leonidus@vossi04 leonidus-2024-05-27_07:27:31-fs-wip-lusov-quiesce-distro-default-smithi]$ grep "Successfully quiesced" -c */teuthology.log
7727791/teuthology.log:28
7727792/teuthology.log:26
7727793/teuthology.log:26
7727795/teuthology.log:29
7727797/teuthology.log:25
7727798/teuthology.log:26
7727799/teuthology.log:27
7727800/teuthology.log:29
7727801/teuthology.log:30
7727802/teuthology.log:27
7727803/teuthology.log:30
7727804/teuthology.log:32
7727805/teuthology.log:29
7727806/teuthology.log:30
7727807/teuthology.log:12
7727808/teuthology.log:30
7727809/teuthology.log:25
7727810/teuthology.log:29

Copy link
Member

@batrick batrick left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fantastic!

@batrick batrick merged commit 25e4ee2 into main May 28, 2024
14 of 15 checks passed
@batrick batrick deleted the wip-lusov-quiesce-overdrive-export branch May 28, 2024 16:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
2 participants