Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

messaging_service: Clear list of clients on shutdown #14632

Closed
wants to merge 1 commit into from

Conversation

xemul
Copy link
Contributor

@xemul xemul commented Jul 11, 2023

When messaging_service shuts down it first sets _shutting_down to true and proceeds with stopping clients and servers. Stopping clients, in turn, is calling client.stop() on each.

Setting _shutting_down is used in two places.

First, when a client is stopped it may happen that it's in the middle of some operation, which may result in call to remove_error_rpc_client() and not to call .stop() for the second time it just does nothing if the shutdown flag is set (see 357c91a).

Second, get_rpc_client() asserts that this flag is not set, so once shutdown started it can make sure that it will call .stop() on all clients and no new ones would appear in parallel.

However, after shutdown() is complete the _clients vector of maps remains intact even though all clients from it are stopped. This is not very debugging-friendly, the clients are better be removed on shutdown.

fixes: #14624

@xemul xemul requested a review from kbr-scylla July 11, 2023 10:54
@scylladb-promoter
Copy link
Contributor

@xemul
Copy link
Contributor Author

xemul commented Jul 12, 2023

CI state FAILURE - https://jenkins.scylladb.com/job/scylla-master/job/scylla-ci/2391/

#14612 , re-kicking

@scylladb-promoter
Copy link
Contributor

CI state SUCCESS - https://jenkins.scylladb.com/job/scylla-master/job/scylla-ci/2427/

@xemul
Copy link
Contributor Author

xemul commented Jul 24, 2023

@scylladb/scylla-maint , review/merge ping

When messaging_service shuts down it first sets _shutting_down to true
and proceeds with stopping clients and servers. Stopping clients, in
turn, is calling client.stop() on each.

Setting _shutting_down is used in two places.

First, when a client is stopped it may happen that it's in the middle of
some operation, which may result in call to remove_error_rpc_client()
and not to call .stop() for the second time it just does nothing if the
shutdown flag is set (see 357c91a).

Second, get_rpc_client() asserts that this flag is not set, so once
shutdown started it can make sure that it will call .stop() on _all_
clients and no new ones would appear in parallel.

However, after shutdown() is complete the _clients vector of maps
remains intact even though all clients from it are stopped. This is not
very debugging-friendly, the clients are better be removed on shutdown.

fixes: scylladb#14624

Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
@xemul xemul force-pushed the br-ms-unlist-clients-always branch from 1fcaf34 to 5a538ba Compare July 25, 2023 08:32
@xemul
Copy link
Contributor Author

xemul commented Jul 25, 2023

upd:

  • clear list of clients in .finally() block and drop the comment explaining why .then() was unavoidable

@scylladb-promoter
Copy link
Contributor

patjed41 pushed a commit to patjed41/scylladb that referenced this pull request Jul 27, 2023
When messaging_service shuts down it first sets _shutting_down to true
and proceeds with stopping clients and servers. Stopping clients, in
turn, is calling client.stop() on each.

Setting _shutting_down is used in two places.

First, when a client is stopped it may happen that it's in the middle of
some operation, which may result in call to remove_error_rpc_client()
and not to call .stop() for the second time it just does nothing if the
shutdown flag is set (see 357c91a).

Second, get_rpc_client() asserts that this flag is not set, so once
shutdown started it can make sure that it will call .stop() on _all_
clients and no new ones would appear in parallel.

However, after shutdown() is complete the _clients vector of maps
remains intact even though all clients from it are stopped. This is not
very debugging-friendly, the clients are better be removed on shutdown.

fixes: scylladb#14624

Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>

Closes scylladb#14632
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

The messaging_service::_clients list is not cleared on shutdown
3 participants