Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: decklink consumer "secondary" ports #1493

Merged
merged 25 commits into from Sep 12, 2023
Merged

Conversation

Julusian
Copy link
Member

@Julusian Julusian commented Aug 4, 2023

This work was sponsored by the BBC.

Testing build: https://builds.julusian.dev/casparcg/casparcg-server--pr-1493-80f8ee76b3bc43efa6a6ed78f5bdb95ec7e355c6-windows.zip

This expands upon #1433, to resolve some synchronisation issues that were observed on the output when splitting a channel up over multiple decklink outputs.
Since v11.0 of the Decklink SDK, some cards support a concept of 'playback groups'. This is currently supported by the Duo2, Quad2 and 8K Pro (I expect any newer multi-channel cards will also support it). By adding outputs to a playback group, the driver/card will ensure that they are run in sync with each other with the same playback timings used across each.
We can use this new functionality to allow for grouping multiple ports on a card in combination with subregions, to allow for slicing up a channel across the ports on card.

To utilise this, the decklink consumer configuration has been expanded, to allow for defining these secondary ports:

<decklink>
    <device>[1..]</device>
    <key-device>device + 1 [1..] (This is only used with the external_separate_device mode)</key-device>
    <embedded-audio>false [true|false]</embedded-audio>
    <latency>normal [normal|low|default]</latency>
    <keyer>external [external|external_separate_device|internal|default]</keyer>
    <key-only>false [true|false]</key-only>
    <buffer-depth>3 [1..]</buffer-depth>
    <video-mode>(Run the decklink at a different video-mode. Note: the framerate must match that of the channel)</video-mode>
    <subregion>
        <src-x>0 (x offset into the channel)</src-x>
        <src-y>0 (y offset into the channel)</src-y>
        <dest-x>0 (x offset of the video onto the output)</dest-x>
        <dest-y>0 (y offset of the video onto the output)</dest-y>
        <width>0 (width of the region to copy. 0 means no-limit)</width>
        <height>0 (height of the region to copy. 0 means no-limit)</height>
    </subregion>

    <wait-for-reference>auto [auto|enable|disable]</wait-for-reference>
    <wait-for-reference-duration>10 (seconds)</wait-for-reference-duration>

    <ports>
        (Add secondary ports to be run in sync with the primary. This allows for splitting a wide channel across multiple decklinks, with sync across the outputs guaranteed by the driver on supported cards)
        <port>
            <device>[1..]</device>
            <key-only>false [true|false]</key-only>
            <video-mode>(Run the decklink at a different video-mode. Note: the framerate must match that of the channel)</video-mode>
            <subregion>
                <src-x>0 (x offset into the channel)</src-x>
                <src-y>0 (y offset into the channel)</src-y>
                <dest-x>0 (x offset of the video onto the output)</dest-x>
                <dest-y>0 (y offset of the video onto the output)</dest-y>
                <width>0 (width of the region to copy. 0 means no-limit)</width>
                <height>0 (height of the region to copy. 0 means no-limit)</height>
            </subregion>
        </port>
    </ports>
</decklink>

The additional ports block defines the secondary ports to run. The SDK requires them to be the same video-mode for the driver level synchronisation, but it is possible to use different modes (with the same framerate) with less guarantees on the timing.

It is also possible to manually define a key-only output as a secondary port against a fill output, internally the external_separate_device has been rewritten to use this same flow. If one of the outputs used does not support the playback group functionality, it will still be run but will have less guarantees on the timing. It should run as well as the external_separate_device used to.

For this to guarantee the accuracy, most cards (except the 8K pro) require a reference to be provided and for lock to be aquired before CasparCG starts playback. When running with these secondary ports, the default configuration will wait for reference (in my setup this takes 3.5 seconds) with a max wait of 10s. If no secondary ports are defined, no wait will be done by default. Both the max wait, and whether to wait are configurable.

Example usages:
3x1080p wide videowall

<?xml version="1.0" encoding="utf-8"?>

<configuration>
    <paths>
        <media-path>media/</media-path>
        <log-path disable="false">log/</log-path>
        <data-path>data/</data-path>
        <template-path>template/</template-path>
    </paths>
    <lock-clear-phrase>secret</lock-clear-phrase>
    <video-modes>
        <video-mode>
            <id>wide</id>
            <width>5760</width>
            <height>1080</height>
            <time-scale>50000</time-scale>
            <duration>1000</duration>
            <cadence>960</cadence>
        </video-mode>
    </video-modes>
    <channels>
        <channel>
            <video-mode>wide</video-mode>
            <consumers>
                <decklink>
                    <device>5</device>
                    <video-mode>1080p5000</video-mode>

                    <ports>
                        <port>
                            <device>6</device>
                            <video-mode>1080p5000</video-mode>
                            <subregion>
                                <src-x>1920</src-x>
                            </subregion>
                        </port>
                        <port>
                            <device>9</device>
                            <video-mode>1080p5000</video-mode>
                            <subregion>
                                <src-x>3840</src-x>
                            </subregion>
                        </port>
                    </ports>
                </decklink>
            </consumers>
        </channel>
    </channels>
    <controllers>
        <tcp>
            <port>5250</port>
            <protocol>AMCP</protocol>
        </tcp>
    </controllers>
    <amcp>
        <media-server>
            <host>localhost</host>
            <port>8000</port>
        </media-server>
    </amcp>
</configuration>

Manual key output:

<?xml version="1.0" encoding="utf-8"?>

<configuration>
    <paths>
        <media-path>media/</media-path>
        <log-path disable="false">log/</log-path>
        <data-path>data/</data-path>
        <template-path>template/</template-path>
    </paths>
    <lock-clear-phrase>secret</lock-clear-phrase>
    <channels>
        <channel>
            <video-mode>1080p5000</video-mode>
            <consumers>
                <decklink>
                    <device>5</device>

                    <ports>
                        <port>
                            <device>6</device>
                            <key-only>true</key-only>
                        </port>
                    </ports>
                </decklink>
            </consumers>
        </channel>
    </channels>
    <controllers>
        <tcp>
            <port>5250</port>
            <protocol>AMCP</protocol>
        </tcp>
    </controllers>
    <amcp>
        <media-server>
            <host>localhost</host>
            <port>8000</port>
        </media-server>
    </amcp>
</configuration>

@ryanwmckenna
Copy link

Thanks again for this @Julusian

@saerich
Copy link

saerich commented Aug 9, 2023

Thanks from me too @Julusian -- Was great to see it.

@Sidonai-1
Copy link

Thanks a lot for this @Julusian, outstanding work!
If I understand your post correctly, the sync is only guaranteed within the same Decklink card? Or would it work as well with more than one card providing a reference signal to both?

For example: you need to output via HDMI so you have to use 2 Decklink Extreme (if you want to avoid converting from SDI to HDMI).
Or if you need more than 4 outputs but you don't have a Quad2 Card so you use two Duo 2.

I have some Mini Monitor 4k that have only 1 output and no ref input, so I'm guessing the sync is not guaranteed at all between them.

I'll try to test those scenarios soon, but just to know what to expect.

Thanks!

@Julusian
Copy link
Member Author

@Sidonai-1 yes, it is only guaranteed within the same decklink, as we are relying on the card internally to keep it in sync.
It should work reasonably well across cards, but it is very possible that they will drift. Commonly this will happen if there are performance issues and the decklinks are starved of frames briefly.
When used across cards, I expect it will perform very similarly to the external_separate_device keyer mode in previous versions. I dont remember seeing many complaints about that, so either it typically works reasonably well, or it was barely used (I never did find a good reason to use it myself).
Even in the multicard mode, this will be a lot more reliable than attaching multiple decklink consumers, as the way frames are fed to the decklinks will be more predictable and consistent.

@Julusian Julusian merged commit 206b384 into master Sep 12, 2023
8 checks passed
@Julusian Julusian deleted the feat/decklink-sync-group branch September 12, 2023 09:53
@Sidonai-1
Copy link

Hi @Julusian , one question: I saw that the diag window doesn't show the "ports" devices, only the main one. I'm guessing that's intended behaviour?
Thanks!

@Julusian
Copy link
Member Author

@Sidonai-1 correct, it is expected to only show the primary in the diag graph.
I am not sure what it should show, as many/all of the timings are shared with the primary

@Sidonai-1
Copy link

Sidonai-1 commented Sep 27, 2023

Great, good to know. Thanks.

One more thing I noticed. When using Extreme 4K BM cards with the new ports option, it needs a reference signal no matter what you put in the config: the "disable" option doesn't seem to work. This doesn't happen with 'MiniMonitor 4K' or '8K Pro cards', as one doesn't have reference input, and the other has its own internal one or something like that (don't quote me on this).

This is the config/log I had:

[2023-09-27 13:43:53.436] [info]    ############################################################################
[2023-09-27 13:43:53.437] [info]    CasparCG Server is distributed by the Swedish Broadcasting Corporation (SVT)
[2023-09-27 13:43:53.437] [info]    under the GNU General Public License GPLv3 or higher.
[2023-09-27 13:43:53.437] [info]    Please see LICENSE.TXT for details.
[2023-09-27 13:43:53.437] [info]    http://www.casparcg.com/
[2023-09-27 13:43:53.437] [info]    ############################################################################
[2023-09-27 13:43:53.437] [info]    Starting CasparCG Video and Graphics Playout Server 2.4.0 8062a41 Dev
[2023-09-27 13:43:53.438] [info]    "C:/Users\Avid\Desktop\casparcg-server-8062a41c09d132fe75dd8d549d4a2900048ec55e-windows\casparcg.config":
[2023-09-27 13:43:53.438] [info]    -----------------------------------------
[2023-09-27 13:43:53.438] [info]    <?xml version="1.0" encoding="utf-8"?>
[2023-09-27 13:43:53.438] [info]    <configuration>
[2023-09-27 13:43:53.438] [info]       <log-level>info</log-level>
[2023-09-27 13:43:53.438] [info]       <paths>
[2023-09-27 13:43:53.438] [info]          <media-path>C:\APP\MEDIA\</media-path>
[2023-09-27 13:43:53.438] [info]          <log-path>C:\APP\LOG\</log-path>
[2023-09-27 13:43:53.438] [info]          <data-path>C:\APP\data\</data-path>
[2023-09-27 13:43:53.438] [info]          <font-path>font/</font-path>
[2023-09-27 13:43:53.438] [info]          <template-path>C:\APP\TEMPLATES\</template-path>
[2023-09-27 13:43:53.438] [info]       </paths>
[2023-09-27 13:43:53.438] [info]       <lock-clear-phrase>secret</lock-clear-phrase>
[2023-09-27 13:43:53.438] [info]       <channels>
[2023-09-27 13:43:53.438] [info]          <channel>
[2023-09-27 13:43:53.438] [info]             <video-mode>7680x2160px30</video-mode>
[2023-09-27 13:43:53.438] [info]             <consumers>
[2023-09-27 13:43:53.438] [info]                <decklink>
[2023-09-27 13:43:53.438] [info]                   <device>1</device>
[2023-09-27 13:43:53.438] [info]                   <video-mode>2160p3000</video-mode>
[2023-09-27 13:43:53.438] [info]                   <subregion>
[2023-09-27 13:43:53.438] [info]                      <src-x>0</src-x>
[2023-09-27 13:43:53.438] [info]                      <src-y>0</src-y>
[2023-09-27 13:43:53.438] [info]                   </subregion>
[2023-09-27 13:43:53.438] [info]                   <buffer-depth>4</buffer-depth>
[2023-09-27 13:43:53.438] [info]                   <embedded-audio>false</embedded-audio>
[2023-09-27 13:43:53.438] [info]                   <wait-for-reference>disable</wait-for-reference>
[2023-09-27 13:43:53.438] [info]                   <wait-for-reference-duration>1</wait-for-reference-duration>
[2023-09-27 13:43:53.438] [info]                   <ports>
[2023-09-27 13:43:53.438] [info]                      <port>
[2023-09-27 13:43:53.438] [info]                         <device>2</device>
[2023-09-27 13:43:53.438] [info]                         <video-mode>2160p3000</video-mode>
[2023-09-27 13:43:53.438] [info]                         <subregion>
[2023-09-27 13:43:53.438] [info]                            <src-x>3841</src-x>
[2023-09-27 13:43:53.438] [info]                            <src-y>0</src-y>
[2023-09-27 13:43:53.438] [info]                         </subregion>
[2023-09-27 13:43:53.438] [info]                         <buffer-depth>4</buffer-depth>
[2023-09-27 13:43:53.438] [info]                         <embedded-audio>false</embedded-audio>
[2023-09-27 13:43:53.438] [info]                      </port>
[2023-09-27 13:43:53.438] [info]                   </ports>
[2023-09-27 13:43:53.438] [info]                </decklink>
[2023-09-27 13:43:53.438] [info]             </consumers>
[2023-09-27 13:43:53.438] [info]          </channel>
[2023-09-27 13:43:53.438] [info]       </channels>
[2023-09-27 13:43:53.438] [info]       <controllers>
[2023-09-27 13:43:53.438] [info]          <tcp>
[2023-09-27 13:43:53.438] [info]             <port>5250</port>
[2023-09-27 13:43:53.438] [info]             <protocol>AMCP</protocol>
[2023-09-27 13:43:53.438] [info]          </tcp>
[2023-09-27 13:43:53.438] [info]          <tcp>
[2023-09-27 13:43:53.438] [info]             <port>5251</port>
[2023-09-27 13:43:53.438] [info]             <protocol>AMCP</protocol>
[2023-09-27 13:43:53.438] [info]          </tcp>
[2023-09-27 13:43:53.438] [info]       </controllers>
[2023-09-27 13:43:53.438] [info]       <amcp>
[2023-09-27 13:43:53.438] [info]          <media-server>
[2023-09-27 13:43:53.438] [info]             <host>localhost</host>
[2023-09-27 13:43:53.438] [info]             <port>8000</port>
[2023-09-27 13:43:53.438] [info]          </media-server>
[2023-09-27 13:43:53.438] [info]       </amcp>
[2023-09-27 13:43:53.438] [info]       <html>
[2023-09-27 13:43:53.438] [info]          <remote-debugging-port>8091</remote-debugging-port>
[2023-09-27 13:43:53.438] [info]          <enable-gpu>false</enable-gpu>
[2023-09-27 13:43:53.438] [info]       </html>
[2023-09-27 13:43:53.438] [info]       <osc>
[2023-09-27 13:43:53.438] [info]          <default-port>6250</default-port>
[2023-09-27 13:43:53.438] [info]          <disable-send-to-amcp-clients>false</disable-send-to-amcp-clients>
[2023-09-27 13:43:53.438] [info]          <predefined-clients>
[2023-09-27 13:43:53.438] [info]             <predefined-client>
[2023-09-27 13:43:53.438] [info]                <address>127.0.0.1</address>
[2023-09-27 13:43:53.438] [info]                <port>6251</port>
[2023-09-27 13:43:53.438] [info]             </predefined-client>
[2023-09-27 13:43:53.438] [info]             <predefined-client>
[2023-09-27 13:43:53.438] [info]                <address>127.0.0.1</address>
[2023-09-27 13:43:53.438] [info]                <port>6252</port>
[2023-09-27 13:43:53.438] [info]             </predefined-client>
[2023-09-27 13:43:53.438] [info]             <predefined-client>
[2023-09-27 13:43:53.438] [info]                <address>127.0.0.1</address>
[2023-09-27 13:43:53.438] [info]                <port>6253</port>
[2023-09-27 13:43:53.438] [info]             </predefined-client>
[2023-09-27 13:43:53.438] [info]             <predefined-client>
[2023-09-27 13:43:53.438] [info]                <address>127.0.0.1</address>
[2023-09-27 13:43:53.438] [info]                <port>6254</port>
[2023-09-27 13:43:53.438] [info]             </predefined-client>
[2023-09-27 13:43:53.438] [info]             <predefined-client>
[2023-09-27 13:43:53.438] [info]                <address>127.0.0.1</address>
[2023-09-27 13:43:53.438] [info]                <port>6255</port>
[2023-09-27 13:43:53.438] [info]             </predefined-client>
[2023-09-27 13:43:53.438] [info]             <predefined-client>
[2023-09-27 13:43:53.438] [info]                <address>127.0.0.1</address>
[2023-09-27 13:43:53.438] [info]                <port>7071</port>
[2023-09-27 13:43:53.438] [info]             </predefined-client>
[2023-09-27 13:43:53.438] [info]          </predefined-clients>
[2023-09-27 13:43:53.438] [info]       </osc>
[2023-09-27 13:43:53.438] [info]       <video-modes>
[2023-09-27 13:43:53.438] [info]          <video-mode>
[2023-09-27 13:43:53.438] [info]             <id>7680x2160px30</id>
[2023-09-27 13:43:53.438] [info]             <width>7680</width>
[2023-09-27 13:43:53.438] [info]             <height>2160</height>
[2023-09-27 13:43:53.438] [info]             <time-scale>30000</time-scale>
[2023-09-27 13:43:53.438] [info]             <duration>1000</duration>
[2023-09-27 13:43:53.438] [info]             <cadence>1600</cadence>
[2023-09-27 13:43:53.438] [info]          </video-mode>
[2023-09-27 13:43:53.438] [info]       </video-modes>
[2023-09-27 13:43:53.438] [info]    </configuration>
[2023-09-27 13:43:53.438] [info]    -----------------------------------------
[2023-09-27 13:43:53.438] [info]    Initialized video modes.
[2023-09-27 13:43:53.720] [info]    Initializing OpenGL Device.
[2023-09-27 13:43:53.722] [info]    Initialized OpenGL 4.5.0 NVIDIA 516.94 NVIDIA Corporation
[2023-09-27 13:43:53.753] [info]    Initialized OpenGL Accelerated GPU Image Mixer for channel 1
[2023-09-27 13:43:53.753] [info]    video_channel[1|7680x2160px30] Successfully Initialized.
[2023-09-27 13:43:53.753] [info]    Initialized channels.
[2023-09-27 13:43:53.753] [info]    Initialized command repository.
[2023-09-27 13:43:53.754] [info]    Initialized image module.
[2023-09-27 13:43:53.754] [info]    Initialized ffmpeg module.
[2023-09-27 13:43:53.754] [info]    Initialized oal module.
[2023-09-27 13:43:53.754] [info]    Initialized decklink module.
[2023-09-27 13:43:53.754] [info]    Initialized screen module.
[2023-09-27 13:43:53.754] [info]    Initialized newtek module.
[2023-09-27 13:43:53.754] [info]    Initialized artnet module.
[2023-09-27 13:43:53.800] [info]    Initialized html module.
[2023-09-27 13:43:53.800] [info]    Flash support is disabled
[2023-09-27 13:43:53.800] [info]    Initialized flash module.
[2023-09-27 13:43:53.802] [info]    Initialized bluefish module.
[2023-09-27 13:43:53.802] [info]    Initialized modules.
[2023-09-27 13:43:53.810] [warning] Device supports video-format with conversion: 2160p30
[2023-09-27 13:43:53.817] [info]    Device may not support video-format: 2160p30
[2023-09-27 13:43:53.817] [info]    DeckLink 4K Extreme 12G [1-1|2160p3000] Disabled low-latency mode.
[2023-09-27 13:43:53.818] [info]    DeckLink 4K Extreme 12G [1-1|2160p3000] Enabled external keyer.
[2023-09-27 13:43:53.819] [info]    DeckLink 4K Extreme 12G [1-1|2160p3000] && DeckLink 4K Extreme [2|2160p3000] Disabled low-latency mode.
[2023-09-27 13:43:53.819] [info]    DeckLink 4K Extreme 12G [1-1|2160p3000] && DeckLink 4K Extreme [2|2160p3000] Enabled external keyer.
[2023-09-27 13:43:53.976] [info]    [decklink_consumer] Uninitialized.
[2023-09-27 13:43:53.976] [error]   Exception: D:\a\server\server\src\modules\decklink\consumer\decklink_consumer.cpp(636): Throw in function void __cdecl caspar::decklink::decklink_consumer::start_playback(void)
[2023-09-27 13:43:53.976] [error]   Dynamic exception type: class boost::exception_detail::clone_impl<struct caspar::caspar_exception>
[2023-09-27 13:43:53.976] [error]   [struct caspar::tag_msg_info * __ptr64] = DeckLink 4K Extreme 12G [1-1|2160p3000] && DeckLink 4K Extreme [2|2160p3000] Failed to schedule primary playback.
[2023-09-27 13:43:53.976] [error]   [struct caspar::tag_stacktrace_info * __ptr64] =  0# 0x00007FF77BD62787 in casparcg
[2023-09-27 13:43:53.976] [error]    1# 0x00007FF77BD8D0F6 in casparcg
[2023-09-27 13:43:53.976] [error]    2# 0x00007FF77BEF488E in casparcg
[2023-09-27 13:43:53.976] [error]    3# 0x00007FF77BEED3E9 in casparcg
[2023-09-27 13:43:53.976] [error]    4# 0x00007FF77BEEEBBB in casparcg
[2023-09-27 13:43:53.976] [error]    5# 0x00007FF77BD9B99F in casparcg
[2023-09-27 13:43:53.976] [error]    6# 0x00007FF77BDFD9DE in casparcg
[2023-09-27 13:43:53.976] [error]    7# 0x00007FF77BDF7D86 in casparcg
[2023-09-27 13:43:53.976] [error]    8# configthreadlocale in ucrtbase
[2023-09-27 13:43:53.976] [error]    9# BaseThreadInitThunk in KERNEL32
[2023-09-27 13:43:53.976] [error]   10# RtlUserThreadStart in ntdll
[2023-09-27 13:43:53.976] [error]   
[2023-09-27 13:43:53.976] [error]   

[2023-09-27 13:43:53.976] [error]    0# 0x00007FF77BD62787 in casparcg
[2023-09-27 13:43:53.976] [error]    1# 0x00007FF77BD62215 in casparcg
[2023-09-27 13:43:53.976] [error]    2# 0x00007FF77C028FE7 in casparcg
[2023-09-27 13:43:53.976] [error]    3# 0x00007FFB0C421080 in VCRUNTIME140_1
[2023-09-27 13:43:53.976] [error]    4# _NLG_Return2 in VCRUNTIME140_1
[2023-09-27 13:43:53.976] [error]    5# RtlCaptureContext2 in ntdll
[2023-09-27 13:43:53.976] [error]    6# 0x00007FF77BD88FE8 in casparcg
[2023-09-27 13:43:53.976] [error]    7# 0x00007FF77BD8D68E in casparcg
[2023-09-27 13:43:53.976] [error]    8# 0x00007FF77BD644CA in casparcg
[2023-09-27 13:43:53.976] [error]    9# 0x00007FF77BD67984 in casparcg
[2023-09-27 13:43:53.976] [error]   10# 0x00007FF77BFE4BD4 in casparcg
[2023-09-27 13:43:53.976] [error]   11# BaseThreadInitThunk in KERNEL32
[2023-09-27 13:43:53.976] [error]   12# RtlUserThreadStart in ntdll
[2023-09-27 13:43:53.976] [error]   
[2023-09-27 13:43:54.050] [info]    Initialized startup producers.
[2023-09-27 13:43:54.051] [info]    Initialized controllers.
[2023-09-27 13:43:54.051] [info]    Initialized osc.

@Sidonai-1
Copy link

Sidonai-1 commented Oct 5, 2023

Hi again @Julusian, one more question. Only the main device outputs embedded audio, so none of the 'ports' have audio output even if you set it to 'true'. Is this intended behaviour?

Thanks a lot!

@Julusian
Copy link
Member Author

Julusian commented Oct 5, 2023

I can't explain that issue you are seeing, and I dont have access to a similar card to be able to test with.. The SDK does say that ref must be locked when doing this except for with the 8k card, so it is possible that some cards are pickier than others if ref is not locked.

It is expected that only the primary will have embedded audio currently. The client had no need for embedded audio, so I didnt spend any time on it. The structure should be in place for it to slot in easily, when someone wants to do it.

@Sidonai-1
Copy link

The lack of embedded audio in the ports is not a big issue, it just caught me by surprise and I struggled to find the cause for a bit.

I was able to test this feature in a big LED screen and the synchronization between two Decklink 8K Pro cards was perfect (providing ref. signal).

Using the 8 outputs at 4K30p the total was around 23.800 px wide; one channel and one single file containing the 23k-wide-video and 8 audios.

Everything was smooth, even if the CPU was around 83%.

In summary: amazing work! hats off

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants