feat: Implement Luma Player

[dazzatronus]

User story

As an end-user, I want the Shotstack Studio SDK's luma rendering to match the Shotstack Edit API, so I can accurately preview edits using luma assets.

Motivation & context

Motivation

Currently, numerous Shotstack templates can't be accurately previewed in the SDK due to missing or incorrect luma masking capabilities. The Studio lacks a compatible luma player for accurate previews of Luma assets.

Context

The Problem

The Shotstack Edit API and PixiJS handle luma masks in opposite ways:

• Shotstack Edit API: White areas in luma masks create transparency, revealing content behind; black areas remain opaque
• PixiJS: White areas in masks make content visible; black areas create transparency

This incompatibility creates incorrect masking when implemented with PixiJS masks.

Acceptance criteria

• Video Luma Edit in Studio (below) is identical to Edit API export
• Image Luma Edit in Studio (below) is identical to Edit API export
• Track layering is working correctly, with tracks with a lower index showing in front of tracks with a higher index
• No performance impact on frame rate
• No WebGL context loss
• Updated documentation

Video Luma Edit payload

{
    "timeline": {
        "soundtrack": {
            "src": "https://shotstack-assets.s3-ap-southeast-2.amazonaws.com/music/unminus/palmtrees.mp3",
            "effect": "fadeOut"
        },
        "background": "#000000",
        "tracks": [
            {
                "clips": [
                	{
                        "asset": {
                            "type": "luma",
                            "src": "https://shotstack-assets.s3-ap-southeast-2.amazonaws.com/examples/luma-mattes/radial.mp4"
                        },
                        "start": 2.24,
                        "length": 1.76
                    },
                    {
                        "asset": {
                            "type": "video",
                            "src": "https://shotstack-assets.s3-ap-southeast-2.amazonaws.com/footage/table-mountain.mp4"
                        },
                        "start": 0,
                        "length": 4,
                        "transition": {
                            "in": "fade"
                        }
                    }
                ]
            },
            {
                "clips": [
                	{
                        "asset": {
                            "type": "luma",
                            "src": "https://shotstack-assets.s3-ap-southeast-2.amazonaws.com/examples/luma-mattes/blocks-in.mp4"
                        },
                        "start": 4.92,
                        "length": 1.32
                    },
                    {
                        "asset": {
                            "type": "video",
                            "src": "https://shotstack-assets.s3-ap-southeast-2.amazonaws.com/footage/road.mp4"
                        },
                        "start": 2.24,
                        "length": 4
                    }
                ]
            },
            {
                "clips": [
                    {
                        "asset": {
                            "type": "luma",
                            "src": "https://shotstack-assets.s3-ap-southeast-2.amazonaws.com/examples/luma-mattes/lines.mp4"
                        },
                        "start": 8.08,
                        "length": 1.84
                    },
                    {
                        "asset": {
                            "type": "video",
                            "src": "https://shotstack-assets.s3-ap-southeast-2.amazonaws.com/footage/lake.mp4"
                        },
                        "start": 4.92,
                        "length": 5,
                        "transition": {
                            "out": "fade"
                        }
                    }
                ]
            },
            {
                "clips": [
                    {
                        "asset": {
                            "type": "video",
                            "src": "https://shotstack-assets.s3-ap-southeast-2.amazonaws.com/footage/beach-cliffs.mp4"
                        },
                        "start": 8.08,
                        "length": 5,
                        "transition": {
                            "out": "fade"
                        }
                    }
                ]
            }
        ]
    },
    "output": {
        "format": "mp4",
        "resolution": "hd"
    }
}

Image Luma Edit payload

{
    "timeline": {
        "soundtrack": {
            "src": "https://shotstack-assets.s3-ap-southeast-2.amazonaws.com/music/unminus/palmtrees.mp3",
            "effect": "fadeOut"
        },
        "background": "#000000",
        "tracks": [
            {
                "clips": [
                    {
                        "asset": {
                            "type": "luma",
                            "src": "https://shotstack-assets.s3-ap-southeast-2.amazonaws.com/luma-mattes/static/circle-sd.jpg"
                        },
                        "start": 0,
                        "length": 10
                    },
                    {
                        "asset": {
                            "type": "video",
                            "src": "https://s3-ap-southeast-2.amazonaws.com/shotstack-assets/footage/cat.hd.mp4"
                        },
                        "start": 0,
                        "length": 10
                    }
                ]
            },
            {
                "clips": [
                    {
                        "asset": {
                            "type": "video",
                            "src": "https://shotstack-assets.s3-ap-southeast-2.amazonaws.com/footage/night-sky.mp4"
                        },
                        "start": 0,
                        "length": 10
                    }
                ]
            }
        ]
    },
    "output": {
        "format": "mp4",
        "resolution": "hd"
    }
}

Implementation notes

Previous Implementation

The addPlayer method identifies luma clips and applies them as masks to their containing track:

public async addPlayer(trackIdx: number, clipToAdd: Player): Promise<void> {
    // Track container setup...
    
    // Identify luma clips
    const isClipMask = clipToAdd instanceof LumaPlayer;
    
    // Add clip to track...
    await clipToAdd.load();
    
    // Apply mask if luma clip
    if (isClipMask) {
        trackContainer.mask = clipToAdd.getMask();
    }
    
    // Update duration...
}

This results in inverted masking behaviour compared to Shotstack's expected output.

Solution Options

1. Custom Shader Approach

Implement a custom shader that inverts the masking logic.

2. Pre-inversion Approach

Process luma mask textures before applying them by inverting their color values. While conceptually simple, this approach has previously resulted in implementation challenges.

Recommendation

Initially try the pre-inversion approach, and if unsuccessful replace the direct mask application with a custom shader solution that properly handles the inverted masking behavior to match Shotstack's expected output.

[Sniper32191]

Hi, ddazzatronus
How are you?
I'd like to talk with you about this issue.

[dazzatronus]

@Sniper32191 this is not working for you? There remain issues in the old implementation. This has been resolved in the upcoming 2.0.0. You can already access the beta if you'd like to test this out.

[Sniper32191]

Thank you for your reply.
Are you familiar with the workdev1211 Git account? If so, that’s me. I’d like to chat with you more freely. Could you please share your contact information?

[Sniper32191]

In my memory, I worked on the mask issue for Luma Player using that Git account six month's ago.

[Sniper32191]

However, that Git account is suspended, and I don’t know the reason.
I use the email workdev1211@gmail.com and am waiting from you.
Thanks.