Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

msg/async: reduce write_lock contention #15092

Merged
merged 6 commits into from May 23, 2017

Conversation

Projects
None yet
4 participants
@yuyuyu101
Copy link
Member

yuyuyu101 commented May 15, 2017

No description provided.

msg/async: remove send message inline option
Signed-off-by: Haomai Wang <haomai@xsky.com>
@yuyuyu101

This comment has been minimized.

Copy link
Member Author

yuyuyu101 commented May 15, 2017

we saw write_lock acquire cosume nearly 60us for each message(30 us each lock, one send_message, one handle_write). it will cause osd connection limited in iops.

@yuyuyu101

This comment has been minimized.

Copy link
Member Author

yuyuyu101 commented May 16, 2017

retest this please

@yuyuyu101 yuyuyu101 force-pushed the yuyuyu101:wip-msgr-writelock branch 3 times, most recently from afab44b to 9e10392 May 16, 2017

yuyuyu101 added some commits May 15, 2017

msg/async: remove write_lock on outcoming_bl and write_message
since we only write_message within own thread, we could free outcoming_bl
without lock protection. then it could hugely reduce write_lock contention
with other threads want to queue message.

the only exception is that when replacing another thread may modify non-own
outcoming_bl, we delay this into safe time.

Signed-off-by: Haomai Wang <haomai@xsky.com>
msg/async: make sure write_message work in own thread
Signed-off-by: Haomai Wang <haomai@xsky.com>

@yuyuyu101 yuyuyu101 force-pushed the yuyuyu101:wip-msgr-writelock branch from 9e10392 to c70b356 May 16, 2017

msg/async: avoid was_session_reset involving outcoming_bl operation
Signed-off-by: Haomai Wang <haomai@xsky.com>

@yuyuyu101 yuyuyu101 requested review from tchaikov and liewegas May 16, 2017

msg/async: remove unused perf counter
Signed-off-by: Haomai Wang <haomai@xsky.com>

@yuyuyu101 yuyuyu101 force-pushed the yuyuyu101:wip-msgr-writelock branch 3 times, most recently from a7d4acf to ca1a7cf May 16, 2017

msg/async: add perf counter for msgr thread to know lat distribute
Signed-off-by: Haomai Wang <haomai@xsky.com>

@yuyuyu101 yuyuyu101 force-pushed the yuyuyu101:wip-msgr-writelock branch from ca1a7cf to cad4cff May 16, 2017

@yuyuyu101

This comment has been minimized.

Copy link
Member Author

yuyuyu101 commented May 16, 2017

"AsyncMessenger::Worker-0": {
    "msgr_recv_messages": 304,
    "msgr_send_messages": 316,
    "msgr_recv_bytes": 423759723,
    "msgr_send_bytes": 323189354,
    "msgr_created_connections": 10,
    "msgr_active_connections": 7,
    "msgr_running_total_time": 0.453655656,
    "msgr_running_send_time": 0.109688463,
    "msgr_running_recv_time": 0.323284624,
    "msgr_running_fast_dispatch_time": 0.019052846
},

@liewegas liewegas added the needs-qa label May 19, 2017

@markhpc

This comment has been minimized.

Copy link
Member

markhpc commented May 19, 2017

Some quick performance tests with this PR look good so far. Lower lock contention as a result of the async messenger during 4K random writes with bluestore on NVMe.

@yuriw yuriw merged commit 27f7146 into ceph:master May 23, 2017

3 checks passed

Signed-off-by all commits in this PR are signed
Details
Unmodifed Submodules submodules for project are unmodified
Details
default Build finished.
Details

@yuyuyu101 yuyuyu101 deleted the yuyuyu101:wip-msgr-writelock branch May 24, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.