Support hot upgrade or smooth upgrade, Upgrade smoothly, Gracefully Upgrade, Source cleaning

[winlinvip]

Usage

SRS supports two signals:

• SIGTERM: Fast exit, quickly clean up actively disconnected connections, and then exit. K8s sends this signal during preStop, and then sends SIGKILL to forcefully kill the Pod after a timeout. We can configure force_grace_quit to consider SIGTERM as Gracefully QUIT as well.
• SIGQUIT: Graceful exit, close listening and wait for all clients to disconnect before exiting. If there are still connections, SRS will not exit, but the longest exit waiting time configuration in K8s is terminationGracePeriodSeconds, and it will force exit after waiting for this long. If there are no connections, it will wait for grace_final_wait before exiting.

Note: SRS does not implement a maximum waiting time. It will wait for clients to disconnect indefinitely without forcing an exit. In conjunction with the terminationGracePeriodSeconds configuration in K8s for managing Pods, K8s will send SIGKILL to forcefully shut down SRS after a timeout.

Other

In order to simplify the handling process, SRS does not clean up memory objects when stopping the stream, as the stream may be re-pushed. If cleaning is required, it would result in complex and careful handling of Source objects, which is not conducive to problem simplification.

Not cleaning up Source objects will cause continuous memory growth. This may not be a noticeable issue in scenarios where there is less streaming and more playback. However, in scenarios with a lot of streaming, such as monitoring and conference scenarios, cleaning up the streams becomes necessary. Reference:

1. PR for Source cleanup submitted by Nobody2 (fix: clean up source and add publisher status #1568) discussed various scenarios that require cleaning up. Of course, Nobody2 did a great job with the submitted PR, but the issue itself is too complex.
2. Online reports (Source Cleanup: When there is a large amount of streaming, Source leakage causes OOM (Out of Memory). #1509, #1271, After stopping the stress test, the CPU and memory have remained consistently high. #1507) indicate memory leaks and OOM caused by not cleaning up Source objects.

Currently, partial optimizations have been implemented to alleviate this issue.

• Resolve coroutine issue: link
• Reduce memory usage in source: link

At the same time, we are also considering the most stable and easiest solution. There is another idea to make SRS support smooth exit and smooth upgrade, roughly as follows:

1. Disable exclusive access to the PID file, allowing a new SRS to be started.
2. Use REUSEPORT to open a new SRS, allowing both the old and new SRS to provide services using the same PID file.
3. The old SRS will no longer accept new connections and the API port will be closed. After serving existing clients or after a certain period of time, such as 12 hours, the old SRS will exit.

This way, the old SRS can easily and safely release the created sources and potential other memory issues. Users can smoothly upgrade and exit SRS during off-peak periods according to their business needs, minimizing the impact on users.

The only issue is that when both the new and old SRS are providing services, the API is provided by the new SRS, which means that the system count is not accurate, and the number of users served by the old SRS may be missed.

Remark: If it is a source station cluster, the stream is on the old SRS, which may result in the inability to detect the stream. In this case, it is necessary to forcefully disconnect the stream. The client needs to support retries in order to smoothly support this. One solution is to place an Edge before the source station, so that retries can be supported through the Edge.

TRANS_BY_GPT3

[winlinvip]

Users can choose:

1. According to their business situation, users can choose to smoothly upgrade and exit SRS during the low peak period of their business.
2. If there is a significant increase in the memory usage of SRS, exceeding the warning level, users can choose to urgently initiate a smooth upgrade.
3. Users can choose to upgrade in a planned manner after a new stable version is released, in order to avoid impacting a small number of existing users.

TRANS_BY_GPT3

[winlinvip]

When SRS supports K8S deployment, services need to support upgrades, rollbacks, and canary releases. The basic requirement for these mechanisms is that SRS needs to support Gracefully Quit/Upgrade. Only when SRS can do its part well, can K8S or other release mechanisms meet the requirements for production-level releases.

The SRS cluster is divided into Origin and Edge clusters, and this issue can be viewed separately.

• The Origin cluster can be directly restarted because there is an Edge cluster that can retry. However, it can be improved when exiting, for example, not exiting abruptly at once, as this may cause the flow to be directed to another Origin server.
• The Edge cluster cannot be directly restarted because it directly serves the clients. It can only exit after the service is finished. Whether it is a long connection or a short connection, the requirement is the same, only the duration differs.

Therefore, we focus on the issue of Gracefully Quit in the Edge cluster, which can refer to the mechanism of Nginx.

1. Update the Nginx binary.
2. Send the SIGUSR2 signal to Nginx.
3. The Nginx master modifies the PID file to /var/run/nginx.pid.oldbin, allowing the new master to start. This file can also be used to send signals to the old master.
4. Start the new master using execve, with the PID set to /var/run/nginx.pid. Pass the listen file descriptor to the new master through the ENV, allowing both the old and new masters to listen on the same port.
5. Send the SIGWINCH signal to the old master to gracefully terminate the workers after serving the existing file descriptors. Meanwhile, the new master is already working, and the new workers are serving new connections.
6. Send the SIGQUIT signal to the old master to initiate a graceful shutdown.
7. After a certain period, the old master can also be sent the SIGTERM signal to exit directly.

Since Nginx chooses to start the master using execve, inheriting the listen file descriptor, this process can be more complex. SRS can choose to use REUSEPORT to directly start a new process listening on the same file descriptor, making this solution simpler.

Additionally, SRS3 has been released with the following plans:

• SRS3 supports some key features that require script coordination or K8S management.
• SRS4 will provide improved support for Gracefully Upgrade and offer more comprehensive features.

TRANS_BY_GPT3

[winlinvip]

To publish updates, rollbacks, and gray releases, there are two main requirements for SRS:

• Gracefully Quit: Smoothly exit by closing the listening port, no longer accepting new connections, and waiting for existing connections to end before quitting.
• Gracefully Upgrade: Smoothly upgrade by starting a new SRS instance while the old one continues to run. The old instance will begin a Gracefully Quit process to smoothly exit.

The key to Gracefully Quit is to no longer accept new connections and wait for the existing connections to exit. We can achieve this by closing the listening file descriptor (fd) in SRS. Another approach is to remove the backend Pod from the SLB (Server Load Balancer), which will naturally prevent new fds from being created.

TRANS_BY_GPT3