-
Notifications
You must be signed in to change notification settings - Fork 40
Redundancy
The application contains several tools designed to build redundant play-out system with minimal (->zero) channel downtime. These tools are shown below. Sometimes it's required to shut down the playout application or server. If you have two servers with CasparCG, it can be done without play-out interruption. CasparCG will continue play current clip until it ends. TVPlay will resume rundown after restart if the clip is still playing on application startup (if clip is finished, TVPlay will assume that something other - from different source - is on air).
In TVPlay it's possible to assign two playout outputs on separate servers as PRImary and SECondary channels. They should be defined in Play-out servers section. Media paths provided should be UNC-style (\\server\share format) - this will allow Application redundancy. Then in Rundown engine config assign these channels to rundown engine. The both (PRI and SEC) servers will receive the same commands to play, stop etc.
PRV preview channel (in rundown engine configuration) can be the same as SECondary (when previewing, program content will be covered with preview and muted) or set up as second CasparCG output on one of servers.
When a clip on PRImary server is successfully verified, it's automatically copied to SECondary server. If a clip is deleted on primary, the same occurs on secondary. When primary media metadata is changed, the same change is applied to secondary.
When TVPlay starts, it also checks that all files from PRImary server are available on SECondary. If not, they are scheduled to copy.
It is possible to have more clips on SEC server than on PRImary (e.g. when file cannot be deleted on secondary when it is in use by preview).
Having two copies of the database is first requirement when we setup redundant solution. Database mirroring is implemented internally in application core, not via MySQL replication, as it is easier to implement and manage. Setup is easy: in TVPlay.config.exe, in section Config file and database, first define connection parameters to redundant server, and then use button "Clone primary database to mirror". Beginning from this, TVPlay will read at first from primary database, then, only if failed - from secondary. It will also execute every database statement on both databases and check if autoincrement values are equal. If an error occurs, it will show problem with database on status bar.
Having two media sets and database on two servers is not enough, when machine running TVPlay appllication is failed. So it should be able to run from second PC as failover. It should be executed only when its primary instance is unavailable, because it can't follow primary instance state (e.g.) database changes. The failover installation should have different local configuration (TVPlay.exe.config file and Configuration\IngestFolders.xml), taking into account primary machine unavailability. Additionally, marking it as Backup instance in Config file and database will warn user against accidental execution (and, for example, break database consistency).
These features, of course, are not only means that should be provided to make play-out system reliable. We need to know what will happen when any particular element of the system will fail. It can be network switch, a mouse or a cable break. They all should be identified, estimated and secured if needed. Furthermore, the emergency play-out should be tested with different scenarios.