Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rgw: dump remaining coroutines when cr deadlock is detected #11580

Merged
merged 1 commit into from Oct 20, 2016

Conversation

cbodley
Copy link
Contributor

@cbodley cbodley commented Oct 20, 2016

when we detect a coroutine deadlock, dump remaining coroutines to lderr before asserting. this should help us figure out what's causing the deadlocks in teuthology runs. example output:

2016-10-20 12:17:28.236470 7f52959dea80 -1 run(): ERROR: deadlock detected, dumping remaining coroutines:
[
    {
        "stack": "0x55f9f6bdc3d0",
        "run_count": 0,
        "ops": [
            {
                "type": "17RGWFetchAllMetaCR",
                "spawned": [
                    "0x55f9f6bde3c0"
                ],
                "status": {
                    "status": "acquiring lock",
                    "timestamp": "2016-10-20 16:17:28.235549Z"
                }
            }
        ]
    },
    {
        "stack": "0x55f9f6bde3c0",
        "run_count": 0,
        "ops": [
            {
                "type": "11MockLeaseCR"
            }
        ]
    }
]

Signed-off-by: Casey Bodley <cbodley@redhat.com>
}
formatter.close_section();
lderr(cct) << __func__ << "(): ERROR: deadlock detected, dumping remaining coroutines:\n";
formatter.flush(*_dout);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cbodley shouldn't we use _derr?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is referring to the std::ostream* _dout variable from the dout_impl macro in common/dout.h, which the other dout/derr macros are using

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i.e. it's the stream associated with the lderr() on the line above, and closed with dendl on the line below

@yehudasa yehudasa merged commit 4e4feb0 into ceph:master Oct 20, 2016
@cbodley cbodley deleted the wip-rgw-deadlock-dump-crs branch October 20, 2016 18:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants