rpc graceful shutdown #291

beef9999 · 2023-12-05T10:53:22Z

rpc支持graceful shutdown，即等待客户端主动关闭链接。
配合socket server现有的terminate（即只关listen fd，阻止建立新链接；不关active connection），可以用如下代码实现优雅关闭：

socket_server->terminate();
delete socket_server;
rpc_skeletion->shutdown(true);
delete rpc_skeletion;

beef9999 · 2023-12-05T10:54:35Z

rpc/rpc.cpp

@@ -318,7 +317,6 @@ namespace rpc {
        };
        intrusive_list<ThreadLink> m_list;
        uint64_t m_serving_count = 0;
-        bool m_concurrent;


删除同步模式，降低代码复杂度

beef9999 · 2023-12-05T10:55:43Z

rpc/rpc.cpp

@@ -381,48 +373,55 @@ namespace rpc {
            auto ctx = (Context*)args_;
            Context context(std::move(*ctx));
            ctx->got_it = true;
-            thread_yield_to(nullptr);
+            thread_yield();


thread_yield_to 第一步会有一个unlikely的判断，thread_yield 是直接调用

beef9999 · 2023-12-05T11:08:26Z

rpc/rpc.cpp

@@ -222,7 +222,7 @@ namespace rpc {
            bool got_it;
            int* stream_serv_count;
            photon::condition_variable *stream_cv;
-            std::shared_ptr<photon::mutex> w_lock;
+            photon::mutex* w_lock;



现在的代码中已经有cv，能够保证mutex在所有request完成后再析构，所以这里不需要用共享指针了。

beef9999 · 2023-12-09T10:28:21Z

rpc/rpc.cpp

-            DEFER(if (ownership) delete stream;);
+            DEFER(if (ownership) {
+                delete stream;
+                stream = nullptr;


防止被socket server double delete

As stream is a local variable, assigning it with nullptr won't affect the caller side.

beef9999 · 2023-12-09T10:29:19Z

rpc/rpc.cpp

-                // means shutdown called by rpc serve, should return to give chance to shutdown
-                if ((m_serving_count == 1) && (m_list.front()->thread == nullptr))
-                    return 0;
-                m_cond_served.wait_no_lock();


现有代码已经可以保证在stream析构前等待所有requests结束，因此这段可以删了

lihuiba · 2023-12-11T02:21:44Z

rpc/rpc.cpp

-            DEFER(if (ownership) delete stream;);
+            DEFER(if (ownership) {
+                delete stream;
+                stream = nullptr;


As stream is a local variable, assigning it with nullptr won't affect the caller side.

lihuiba · 2023-12-11T02:38:30Z

rpc/rpc.cpp

+            if (!passive) {
+                m_running = false;
+            }
+            std::vector<thread*> stream_threads;


We can avoid copying the threads out by:

while(m_list) { // wait for the 1st thread in m_list to complete }

if we need to wait for them.

lihuiba · 2023-12-11T02:42:50Z

rpc/rpc.h

+         * @warning DO NOT invoke this function within the RPC request.
+         *          You should create a thread to invoke it, or just use shutdown_inplace.
+         */
+        virtual int shutdown(bool passive = false) = 0;


bool no_more_requests = true, bool wait = true

in order to behave as before, and allow for more control

lihuiba · 2023-12-11T02:45:47Z

rpc/rpc.h


-        virtual int shutdown() = 0;
+        /** @brief Shutdown the server within a RPC request */
+        virtual void shutdown_inplace(bool passive = false) = 0;


effectively not waiting for the completion of shutndown(), i.e. shutdown_no_wait()?

lihuiba · 2023-12-11T02:46:55Z

rpc/rpc.h

@@ -168,9 +168,16 @@ namespace rpc
        // the default allocator is defined in iovector.h/cpp
        virtual void set_allocator(IOAlloc allocation) = 0;

-        virtual int shutdown_no_wait() = 0;


int shutdown_no_wait(bool no_more_requests = true) { return shutdown(no_more_requests, false); }

lihuiba · 2023-12-11T09:42:14Z

rpc/rpc.h

-        virtual int serve(IStream* stream, bool ownership_stream = false) = 0;
+        virtual int serve(IStream* stream) = 0;
+
+        [[deprecated]]


use __attribute__((deprecated)) to fit for C++11 (and gcc 4.9)

similar for new_skeleton()

It's contradictory to support Clang/Windows and old gcc compiler at the same time

lihuiba · 2023-12-11T09:44:27Z

rpc/rpc.cpp

-                m_cond_served.wait_no_lock();
+        virtual int shutdown(bool no_more_requests) override {
+            if (no_more_requests)
+                m_running = false;


m_running = !no_more_requests;

lihuiba · 2023-12-11T09:46:14Z

rpc/rpc.cpp

            return 0;
        }
+        void shutdown_no_wait(bool no_more_requests) override {
+            thread_create11(&SkeletonImpl::shutdown, this, no_more_requests);


simply m_running = ! no_more_requests;

lihuiba · 2023-12-11T09:47:33Z

rpc/rpc.cpp

+            if (no_more_requests)
+                m_running = false;
+            while (m_list) {
+                thread_enable_join(m_list.front()->thread);


auto th = m_list.front()->thread;
then use th instead.

beef9999 changed the base branch from main to release/0.7 December 5, 2023 10:53

beef9999 commented Dec 5, 2023

View reviewed changes

beef9999 marked this pull request as ready for review December 5, 2023 12:58

beef9999 force-pushed the beef9999/rpc-graceful-shutdown branch 2 times, most recently from 9ae94ed to e4a5806 Compare December 9, 2023 10:26

beef9999 commented Dec 9, 2023

View reviewed changes

beef9999 force-pushed the beef9999/rpc-graceful-shutdown branch from e4a5806 to e16c15d Compare December 9, 2023 10:31

beef9999 requested review from lihuiba and Coldwings December 9, 2023 10:32

lihuiba reviewed Dec 11, 2023

View reviewed changes

beef9999 force-pushed the beef9999/rpc-graceful-shutdown branch 6 times, most recently from e1c039f to e1eaa1a Compare December 11, 2023 09:24

lihuiba reviewed Dec 11, 2023

View reviewed changes

beef9999 force-pushed the beef9999/rpc-graceful-shutdown branch 3 times, most recently from a8829b9 to c322fc1 Compare December 11, 2023 13:43

lihuiba approved these changes Dec 12, 2023

View reviewed changes

RPC support graceful shutdown

67172d2

beef9999 force-pushed the beef9999/rpc-graceful-shutdown branch from c322fc1 to 67172d2 Compare December 12, 2023 04:05

beef9999 merged commit a8a55aa into alibaba:release/0.7 Dec 12, 2023
7 checks passed

beef9999 deleted the beef9999/rpc-graceful-shutdown branch December 12, 2023 06:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rpc graceful shutdown #291

rpc graceful shutdown #291

beef9999 commented Dec 5, 2023 •

edited

beef9999 Dec 5, 2023

beef9999 Dec 5, 2023

beef9999 Dec 5, 2023

beef9999 Dec 9, 2023

lihuiba Dec 11, 2023

beef9999 Dec 9, 2023

lihuiba Dec 11, 2023

lihuiba Dec 11, 2023

lihuiba Dec 11, 2023

lihuiba Dec 11, 2023 •

edited

lihuiba Dec 11, 2023 •

edited

lihuiba Dec 11, 2023

beef9999 Dec 11, 2023

lihuiba Dec 11, 2023

lihuiba Dec 11, 2023

lihuiba Dec 11, 2023

rpc graceful shutdown #291

rpc graceful shutdown #291

Conversation

beef9999 commented Dec 5, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lihuiba Dec 11, 2023 • edited

Choose a reason for hiding this comment

lihuiba Dec 11, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

beef9999 commented Dec 5, 2023 •

edited

lihuiba Dec 11, 2023 •

edited

lihuiba Dec 11, 2023 •

edited