Proper clipping of SwingPanel interop

[MatkovIvan]

Proposed Changes

• Adopt an approach that we're using for iOS - using custom blending to respect clip/shape modifiers and overlapped drawings.
• Fix bounds in window calculation - it should work properly inside scaled content now.

Known limitations

• Works only with Metal (macOS), Direct3D (Windows) (requires Transparency support for D3D swap chain skiko#837) and offscreen rendering (might be enabled by another feature flag: compose.swing.render.on.graphics)
• On DirectX, it cannot overlay another DirectX component (due to OS blending limitation)
• On macOS, render and event dispatching order differ. It means that interop view might catch the mouse event even if visually it renders below Compose content

Testing

The new behavior is under a feature flag, so you can test it by setting system property:

System.setProperty("compose.interop.blending", "true")

Clipping

SwingPanel(
    modifier = Modifier.clip(RoundedCornerShape(6.dp))
    ...
)

Overlapping

Box(modifier = Modifier.fillMaxSize()) {
    SwingPanel(factory = {
        JPanel().also { panel ->
            panel.background = java.awt.Color.red
            panel.add(JButton().also { button ->
                button.text = "JButton"
            })
        }
    }, modifier = Modifier.fillMaxSize())
    Snackbar(
        action = { Button(onClick = {}) { Text("OK") } },
        modifier = Modifier.padding(8.dp).align(Alignment.BottomCenter),
    ) {
        Text("Snackbar")
    }
    Popup(alignment = Alignment.Center) {
        Box(
            modifier = Modifier.size(200.dp, 100.dp).background(Color.Gray),
            contentAlignment = Alignment.Center,
        ) {

            Text("Popup")
        }

    }
}

Issues Fixed

Fixes JetBrains/compose-multiplatform#1521
Fixes (partially) JetBrains/compose-multiplatform#3823
Fixes (partially) JetBrains/compose-multiplatform#3739
Fixes (partially) JetBrains/compose-multiplatform#3353
Fixes (partially) JetBrains/compose-multiplatform#3474

[igordmn]

I agree with Alexander, we use so many hacks for transparency that we can easily break the behavior in the future.

I tested main variations on Windows/macOs - it works, but to be safe, let's write java.awt.Robot screenshot tests for different OS'es, as we wrote them in Skiko (look at SkiaLayerTest).

Disable it for now, as we should run them only on CI. I will configure it later myself.

We can merge the PR now to receive feedback earlier, but If we agree that we can/should write the tests, we should write them before the 1.6.0 release (please, create a task).

Even if Robot tests can be flaky and it is hard to run them locally, they are very useful.

To write tests for transparency, we can place an opaque window behind a transparent window.

[igordmn]

https://youtrack.jetbrains.com/issue/COMPOSE-672/Test-Proper-clipping-of-SwingPanel-interop#focus=Comments-27-8631837.0-0

[JuBan1]

Hi @MatkovIvan,

I've tried the newest alpha on Linux Desktop but Swing panels still seem to occlude all composables.

Could you confirm that this PR affects MacOS and Windows, but not Linux? If so it would be worth opening another issue to keep track of that part of the problem.

[MatkovIvan]

I've tried the newest alpha

This PR is not included in the 1.6.0-alpha01.

Could you confirm that this PR affects MacOS and Windows, but not Linux?

Yes, it's explicitly mentioned in the description:

Currently it supports Metal (macOS), Direct3D (Windows)

Also, please note that it's under the feature flag and does NOT enabled by default even on Windows/macOS. Mainly because it's not supported consistently across all platforms.

it would be worth opening another issue to keep track of that part of the problem.

JetBrains/compose-multiplatform#2926 is not closed yet

[JuBan1]

This PR is not included in the 1.6.0-alpha01.

Sorry, I mean I tried the latest dev, 1.6.0-dev1340 (including using the feature flag).

I was asking because I don't know the whole context of this PR. Closing the related issues suggests that the problem is addressed on all platforms. So I wasn't sure where Linux is standing. But if that's tracked via #2926 then all is well. Thanks for your response.

[MatkovIvan]

Closing the related issues suggests that the problem is addressed on all platforms

Well, all of the mentioned issues were closed as duplicates, but not by this PR. I've updated the PR description to highlight that the solution only partially addresses it. Sorry for confusion

[ahamilt]

This is a big improvement for Swing / JavaFX interop, much better experience mixing components. However as far as I can see it is not possible to interact with the Compose components that are "on top" of the SwingPanel? I tried the simple example in this PR and I could not click on the OK button in the SnackBar. Is it possible to make this work?

[MatkovIvan]

@ahamilt thanks for feedback!

it is not possible to interact with the Compose components that are "on top" of the SwingPanel

Let's track this as a separate issue.
UPD: I reproduced this. It's only macOS-specific issue, on Windows it works fine. It's due to this hack:

compose-multiplatform-core/compose/ui/ui/src/desktopMain/kotlin/androidx/compose/ui/scene/ComposeSceneMediator.desktop.kt

Lines 384 to 388 in 3f9010f

	if (renderApi == GraphicsApi.METAL && contentComponent !is SkiaSwingLayer) {
	// Applying layer on macOS makes our bridge non-transparent
	// But it draws always on top, so we can just add it as-is
	// TODO: Figure out why it makes difference in transparency
	add(component, 0)

So, all input events go to interop view, so Compose just didn't aware of them. I'll try to find a workaround for this

[MatkovIvan]

@ahamilt I've fixed handling mouse events above SwingPanel on macOS in #1119.
Elements like SwingPanel/JPanel (the code from the description above) work as expected now. However, it still cannot receive events consumed by Swing elements. It's about the situation where you have a Compose button right about JButton inside SwingPanel.
Hope this fix will mitigate most of the use cases, including yours.

[ahamilt]

Hi @MatkovIvan,

Thanks for the quick turnaround! I was able to test the Snackbar example with #1119 and could see interacting with the button on the Snackbar is now working on macos.

Unfortunately it doesn't work for our use case. We are trying to show a JavaFX map component, and support showing a BottomSheet over the map. As far as I can tell, even a basic JavaFX Pane inside a SwingPanel is not allowing interaction with a Compose component on top, even with #1119.

I had difficulty testing #915 on Windows (which is the main platform we are interested in), I was getting this message when launching the app, and the Swing component was always on top:

• org.jetbrains.skiko.RenderException: Failed to create DirectX12 device.

I assume #915 needs working DirectX12. I'll try on another Windows machine when I get access to one.

[MatkovIvan]

@ahamilt Thanks for sharing your use case. Currently, I can only suggest manually redispatching AWT events to compose (see example below). In future, we'll try to find a way to catch such events out of the box.

⚠️ Please note that this code sample depends on implementation details that might be changed in the future without notice

fun main() {
    System.setProperty("compose.interop.blending", "true")
    singleWindowApplication {
        App()
    }
}

@Composable
fun App() {
    Box(modifier = Modifier.fillMaxSize()) {
        SwingPanel(
            factory = { InteropPanel().also { panel ->
                panel.background = java.awt.Color.gray
                panel.add(JButton().also { button ->
                    button.preferredSize = Dimension(400, 200)
                    button.text = "JButton"

                    panel.subscribeToMouseEvents(button)
                })
            } },
            modifier = Modifier.fillMaxSize()
        )
        Button(
            onClick = { },
            modifier = Modifier.padding(100.dp).alpha(0.5f).fillMaxSize(),
        ) {
            Text("Compose Button")
        }
    }
}

private class InteropPanel : JPanel(), MouseListener, MouseWheelListener, MouseMotionListener {
    override fun mouseClicked(e: MouseEvent) = dispatchToCompose(e)
    override fun mousePressed(e: MouseEvent) = dispatchToCompose(e)
    override fun mouseReleased(e: MouseEvent) = dispatchToCompose(e)
    override fun mouseEntered(e: MouseEvent) = dispatchToCompose(e)
    override fun mouseExited(e: MouseEvent) = dispatchToCompose(e)
    override fun mouseDragged(e: MouseEvent) = dispatchToCompose(e)
    override fun mouseMoved(e: MouseEvent) = dispatchToCompose(e)
    override fun mouseWheelMoved(e: MouseWheelEvent) = dispatchToCompose(e)

    fun subscribeToMouseEvents(component: Component) {
        component.addMouseListener(this)
        component.addMouseMotionListener(this)
        component.addMouseWheelListener(this)
    }

    fun unsubscribeFromMouseEvents(component: Component) {
        component.removeMouseListener(this)
        component.removeMouseMotionListener(this)
        component.removeMouseWheelListener(this)
    }

    private fun dispatchToCompose(e: MouseEvent) {
        when (e.id) {
            MouseEvent.MOUSE_ENTERED, MouseEvent.MOUSE_EXITED -> return
        }

        // WARNING: it depends on implementation details that might be changed in the future without notice
        parent.dispatchEvent(e)
    }
}

⚠️ Please note that this code sample depends on implementation details that might be changed in the future without notice

[m-sasha]

Mixing Compose and JavaFX is even more complicated than just Compose and Swing. See JetBrains/compose-multiplatform#4094

[ahamilt]

Thanks again @MatkovIvan, our use case works well with the InteropPanel example you provided, we can show and interact with a Compose dialog over our JavaFX map view. Health warnings understood.

@m-sasha we are working the other way around, our app is a Compose Multiplatform app with a Compose UI, but we need to use one JavaFX map component, embedded using a SwingPanel and working well. Where we were having problems was with showing a Compose dialog/bottom sheet over the map, which is now working well with this PR and the InteropPanel example.