Debug for sharded index server by iilyak · Pull Request #3933 · apache/couchdb

iilyak · 2022-02-18T12:32:06Z

Overview

Recent refactor of couch_index_server made it a sharded implementation. This means multiple processes are in play now. This made it somewhat harder to bounce in production in case of problems. Also the couch_debug:print_linked_processes(couch_index_server) had an assumption that couch_index_server is a single process.

This PR does two very related things

fixes couch_debug:print_linked_processes(couch_index_server)
adds couch_index_debug module with auxiliary functions to restart couch_index_server processes safely.

Testing recommendations

compile, start, remsh
couch_debug:print_linked_processes(couch_index_server).

|name                                              | reductions | message_queue_len |    memory    |id
|--------------------------------------------------|------------|-------------------|--------------|--
|index_server_1[<0.320.0>]                         |    1127    |         0         |    17000     |
|  couch_secondary_services[<0.312.0>]             |   93596    |         0         |    68632     |
|  couch_event_listener:do_init/3[<0.323.0>]       |    203     |         0         |     2856     |
|    index_server_1[<0.320.0>]                     |    1127    |         0         |    17000     |
|                                                  |            |                   |              |
|index_server_2[<0.324.0>]                         |    290     |         0         |     6088     |
|  couch_secondary_services[<0.312.0>]             |   93598    |         0         |    68632     |
|  couch_event_listener:do_init/3[<0.326.0>]       |    169     |         0         |     2856     |
|    index_server_2[<0.324.0>]                     |    290     |         0         |     6088     |
....

couch_index_debug:help().

[names,print_linked_processes,busy,restart_busy,restart]

couch_index_debug:help(names)
couch_index_debug:help(print_linked_processes)
couch_index_debug:help(busy)
couch_index_debug:help(restart_busy)
Note pid of whereis(index_server_8). and do couch_index_debug:restart(index_server_8)
Use output of couch_debug:print_linked_processes(couch_index_server). to select a value for a reduction property somewhere in between. Then call couch_index_debug:restart_busy(SelectedValue, 50, reductions).. Finally use couch_debug:print_linked_processes(couch_index_server). and verify pids of some index_server_x processes change.

Related Issues or Pull Requests

Sharded couch index server #3860

Checklist

Code is written and works correctly
Changes are covered by tests
Any new configurable parameters are documented in rel/overlay/etc/default.ini
A PR for documentation changes has been made in https://github.com/apache/couchdb-documentation

rnewson · 2022-02-18T12:35:23Z

not a refactor :P

iilyak · 2022-02-18T14:17:26Z

src/couch_index/src/couch_index_debug.erl

+print_linked_processes() ->
+    couch_debug:print_linked_processes(couch_index_server).
+
+-spec busy(Thershold :: pos_integer()) ->


This is not couch_index_server specific. Therefore I am considering to move it into couch_debug.erl for reuse. With a slight change in funuction signature. It would become busy(Threshold, ProcessList) -> [proccess_name()].

done in 6c7a1ac

iilyak · 2022-02-18T14:20:25Z

src/couch_index/src/couch_index_debug.erl

+        names()
+    ).
+
+-spec restart_busy(Threshold :: pos_integer()) ->


I think I should move it into couch_debug.erl for re-use.

done in 8da3715

iilyak · 2022-02-18T18:15:34Z

src/couch/src/couch_debug.erl

+    Pid :: pid() | timeout.
+
+restart(Name) ->
+    Res = test_util:with_process_restart(Name, fun() ->


Note: I am calling a function from test_util module. Is it worth to move the with_process_restart (and friends) into couch_debug and create a dispatch in test_util?

The main downside IMO to moving them is that with_process_restart uses the same sneaky trick with respect to time units as test_util:wait and friends, so keeping them together might be slightly less bewildering to someone unfamiliar with that code. I'm pretty neutral, though, and just wish all implementations were a bit less sneaky.

jaydoane

Followed testing instructions, and seems to work as expected.

jaydoane · 2022-02-20T04:15:19Z

src/couch/src/couch_debug.erl

+
+    If Property is not specified we use message box size
+
+    Properties which can be used are listed bellow


s/bellow/below

jaydoane · 2022-02-20T04:15:51Z

src/couch/src/couch_debug.erl

+
+    If Property is not specified we use message box size
+
+    Properties which can be used are listed bellow


s/bellow/below

jaydoane · 2022-02-20T04:16:31Z

src/couch/src/couch_debug.erl

    io:format("    ---~n", []),
    ok.

+-spec busy(ProcessList :: [process_name()], Thershold :: pos_integer()) ->


s/Thershold/Threshold

jaydoane · 2022-02-20T04:16:56Z

src/couch/src/couch_debug.erl

+    busy(ProcessList, Threshold, message_queue_len).
+
+-spec busy(
+    ProcessList :: [process_name()], Thershold :: pos_integer(), Property :: busy_properties()


s/Thershold/Threshold

jaydoane · 2022-02-20T04:23:50Z

src/couch/src/couch_debug.erl

+    Pid :: pid() | timeout.
+
+restart(Name) ->
+    Res = test_util:with_process_restart(Name, fun() ->


The main downside IMO to moving them is that with_process_restart uses the same sneaky trick with respect to time units as test_util:wait and friends, so keeping them together might be slightly less bewildering to someone unfamiliar with that code. I'm pretty neutral, though, and just wish all implementations were a bit less sneaky.

jaydoane · 2022-02-20T04:26:07Z

src/couch/src/couch_debug.erl

+    end.
+
+-spec restart_busy(ProcessList :: [process_name()], Threshold :: pos_integer()) ->
+    throw({timeout, Name :: process_name()}).


does this also return ok?

(node1@127.0.0.1)28> couch_index_debug:restart_busy(4000, 50, reductions). ok

jaydoane · 2022-02-20T04:27:48Z

src/couch_index/src/couch_index_debug.erl

+
+    If Property is not specified we use message box size
+
+    Properties which can be used are listed bellow


s/bellow/below

jaydoane · 2022-02-20T04:28:03Z

src/couch_index/src/couch_index_debug.erl

+
+    If Property is not specified we use message box size
+
+    Properties which can be used are listed bellow


s/bellow/below

…ndex_server

iilyak force-pushed the debug-for-sharded-index-server branch from 4e72316 to 4743389 Compare February 18, 2022 12:55

iilyak commented Feb 18, 2022

View reviewed changes

jaydoane approved these changes Feb 20, 2022

View reviewed changes

iilyak added 2 commits February 22, 2022 11:13

Fix couch_debug:print_linked_processes/1 to work with sharded couch_i…

d5a964a

…ndex_server

Add couch_index_debug.erl and restart functions

3412a96

iilyak force-pushed the debug-for-sharded-index-server branch from 3879f10 to 3412a96 Compare February 22, 2022 19:20

iilyak merged commit e4b8a46 into apache:3.x Feb 23, 2022

iilyak deleted the debug-for-sharded-index-server branch February 23, 2022 12:19


		If Property is not specified we use message box size

		Properties which can be used are listed bellow

Conversation

iilyak commented Feb 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Testing recommendations

Related Issues or Pull Requests

Checklist

Uh oh!

rnewson commented Feb 18, 2022

Uh oh!

iilyak Feb 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jaydoane left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

iilyak commented Feb 18, 2022 •

edited

Loading

iilyak Feb 18, 2022 •

edited

Loading