Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

osd: avoid the config's get_val() overhead on the read path. #20217

Merged
merged 1 commit into from Feb 5, 2018
Merged

osd: avoid the config's get_val() overhead on the read path. #20217

merged 1 commit into from Feb 5, 2018

Conversation

rzarzynski
Copy link
Contributor

@rzarzynski rzarzynski commented Feb 1, 2018 •
edited

Profiling shows the overhead of the md_config_t::get_val<T> can be significant. Unfortunately, it is being used in two critical places across the read path.

The patch resorts to the legacy config infrastructure to mitigate the performance penalty. It is expected it will be superseded with a solution that allows to use the new, intended routines without hurting performance.

Before:

Samples: 924K of event 'cycles:p', Event count (approx.): 225233537585
  Children      Self  Command          Shared Object        Symbol                                                                                                                                           ◆
-   56,06%     0,13%  tp_osd_tp        ceph-osd             [.] ShardedThreadPool::shardedthreadpool_worker                                                                                                  ▒
   - 55,93% ShardedThreadPool::shardedthreadpool_worker                                                                                                                                                      ▒
      - 55,70% OSD::ShardedOpWQ::_process                                                                                                                                                                    ▒
         - 24,53% PGOpItem::run                                                                                                                                                                              ▒
            - 23,53% OSD::dequeue_op                                                                                                                                                                         ▒
               - 21,74% PrimaryLogPG::do_request                                                                                                                                                             ▒
                  - 20,37% PrimaryLogPG::do_op                                                                                                                                                               ▒
                     - 9,59% PrimaryLogPG::execute_ctx                                                                                                                                                       ▒
                        - 5,30% PrimaryLogPG::prepare_transaction                                                                                                                                            ▒
                           - 5,21% PrimaryLogPG::do_osd_ops                                                                                                                                                  ▒
                              + 3,56% PrimaryLogPG::do_read                                                                                                                                                  ▒
                              - 1,25% md_config_t::get_val<bool>                                                                                                                                             ▒
                                 - 1,22% md_config_t::get_val_generic                                                                                                                                        ▒
                                      1,04% md_config_t::_get_val_generic                                                                                                                                    ▒
                        + 2,04% PrimaryLogPG::OpContext::start_async_reads                                                                                                                                   ▒
                     + 1,76% PrimaryLogPG::maybe_await_blocked_head                                                                                                                                          ▒
                       1,28% MOSDOp::finish_decode                                                                                                                                                           ▒
                     + 1,05% std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >::basic_ostringstream                                                                     ▒
                     + 0,85% PrimaryLogPG::find_object_context                                                                                                                                               ▒
                       0,84% PG::op_has_sufficient_caps                                                                                                                                                      ▒
                     + 0,76% OpRequest::mark_flag_point                                                                                                                                                      ▒
                       0,68% Connection::get_priv                                                                                                                                                            ▒
                       0,59% PrimaryLogPG::OpContext::OpContext                                                                                                                                              ▒
                    0,56% Connection::get_priv                                                                                                                                                               ▒
                 0,69% Connection::get_priv                                                                                                                                                                  ▒
               + 0,67% OpRequest::mark_flag_point                                                                                                                                                            ▒
            + 0,92% Mutex::Unlock                                                                                                                                                                            ▒
         - 18,66% PGRecoveryContext::run                                                                                                                                                                     ▒
            - 18,29% PrimaryLogPG::execute_ctx                                                                                                                                                               ▒
               + 5,84% OpTracker::unregister_inflight_op                                                                                                                                                     ▒
               - 4,90% PrimaryLogPG::complete_read_ctx                                                                                                                                                       ▒
                  - 2,58% PG::publish_stats_to_osd                                                                                                                                                           ▒
                     - 1,17% md_config_t::get_val<unsigned long>                                                                                                                                             ▒
                        - 1,14% md_config_t::get_val_generic                                                                                                                                                 ▒
                             0,98% md_config_t::_get_val_generic                                                                                                                                             ▒
                  + 1,65% AsyncConnection::send_message                                                                                                                                                      ▒
               - 1,72% PrimaryLogPG::prepare_transaction                                                                                                                                                     ▒
                  - 1,57% PrimaryLogPG::do_osd_ops                                                                                                                                                           ▒
                     - 1,17% md_config_t::get_val<bool>                                                                                                                                                      ▒
                        - 1,14% md_config_t::get_val_generic                                                                                                                                                 ▒
0,97% md_config_t::_get_val_generic

After:

Samples: 686K of event 'cycles:p', Event count (approx.): 153042142576
  Children      Self  Command          Shared Object        Symbol                                                                                                                                           ◆
-   50,64%     0,13%  tp_osd_tp        ceph-osd             [.] ShardedThreadPool::shardedthreadpool_worker                                                                                                  ▒
   - 50,51% ShardedThreadPool::shardedthreadpool_worker                                                                                                                                                      ▒
      - 50,22% OSD::ShardedOpWQ::_process                                                                                                                                                                    ▒
         - 22,16% PGOpItem::run                                                                                                                                                                              ▒
            - 21,38% OSD::dequeue_op                                                                                                                                                                         ▒
               - 19,92% PrimaryLogPG::do_request                                                                                                                                                             ▒
                  - 18,58% PrimaryLogPG::do_op                                                                                                                                                               ▒
                     - 8,15% PrimaryLogPG::execute_ctx                                                                                                                                                       ▒
                        - 4,02% PrimaryLogPG::prepare_transaction                                                                                                                                            ▒
                           - 3,93% PrimaryLogPG::do_osd_ops                                                                                                                                                  ▒
                              - 3,68% PrimaryLogPG::do_read                                                                                                                                                  ▒
                                 + 3,37% PrimaryLogPG::OpContext::read_maybe_async                                                                                                                           ▒
                        + 2,24% PrimaryLogPG::OpContext::start_async_reads                                                                                                                                   ▒
                     + 1,63% PrimaryLogPG::maybe_await_blocked_head                                                                                                                                          ▒
                       1,39% MOSDOp::finish_decode                                                                                                                                                           ▒
                     + 1,13% std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >::basic_ostringstream                                                                     ▒
                       0,74% PG::op_has_sufficient_caps                                                                                                                                                      ▒
                     + 0,71% PrimaryLogPG::find_object_context                                                                                                                                               ▒
                       0,63% PrimaryLogPG::OpContext::OpContext                                                                                                                                              ▒
                     + 0,59% OpRequest::mark_flag_point                                                                                                                                                      ▒
                       0,51% Connection::get_priv                                                                                                                                                            ▒
                 0,56% OpRequest::mark_flag_point                                                                                                                                                            ▒
              0,69% Mutex::Unlock                                                                                                                                                                            ▒
         - 15,62% PGRecoveryContext::run                                                                                                                                                                     ▒
            - 15,26% PrimaryLogPG::execute_ctx                                                                                                                                                               ▒
               + 5,39% OpTracker::unregister_inflight_op                                                                                                                                                     ▒
               - 3,89% PrimaryLogPG::complete_read_ctx                                                                                                                                                       ▒
                  + 2,00% AsyncConnection::send_message                                                                                                                                                      ▒
                    1,17% PG::publish_stats_to_osd                                                                                                                                                           ▒
               + 1,44% PrimaryLogPG::close_op_ctx                                                                                                                                                            ▒
                 0,68% PrimaryLogPG::do_osd_op_effects                                                                                                                                                       ▒
               + 0,65% TrackedOp::put

Tested on the top of PR #19380 which means old master. However, the benefit could be higher now because of the recently introduced third call: cct->_conf->get_val<uint64_t>("osd_max_object_size").

See also:

Signed-off-by: Radoslaw Zarzynski rzarzyns@redhat.com

xiexingguo
xiexingguo reviewed Feb 1, 2018
View reviewed changes
src/osd/PrimaryLogPG.cc Outdated
auto osd_max_object_size = cct->_conf->get_val<uint64_t>(
"osd_max_object_size");
cct->_conf->osd_skip_data_digest;
const uint64_t osd_max_object_size = \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why newline?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've though we would cross the 80th column boundary otherwise. Fortunately, just mislooked. Thanks!

liewegas
liewegas approved these changes Feb 1, 2018
View reviewed changes
Copy link
Member

@liewegas liewegas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good except for that stray newline :)

@rzarzynski
osd: avoid the config's get_val() overhead on the read path.
64c7f31 
Profiling shows the overhead of the md_config_t::get_val<T>
can be significant. Unfortunately, it is being used in two
critical places across the read path.

The patch resorts to the legacy config infrastructure to
mitigate the performance penalty. It is expected it will
be superseded with a solution that allows to use the new,
intended routines without hurting performance.
`
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
@tchaikov
Copy link
Contributor

tchaikov commented Feb 2, 2018

tchaikov
tchaikov approved these changes Feb 2, 2018
View reviewed changes
Copy link
Contributor

@tchaikov tchaikov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will need to cache these settings, add add them to get_tracked_conf_keys()

@tchaikov
Copy link
Contributor

tchaikov commented Feb 5, 2018

@tchaikov tchaikov merged commit 764814c into ceph:master Feb 5, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@xiexingguo xiexingguo xiexingguo left review comments

@tchaikov tchaikov tchaikov approved these changes

@liewegas liewegas liewegas approved these changes

Assignees
No one assigned
Labels
needs-qa performance wip-kefu-testing
Projects
None yet
Milestone
No milestone
4 participants
@rzarzynski @tchaikov @liewegas @xiexingguo