How to detect transparent UI elements with template matching

[ghjcfrt]

🏷️ Discussion Type

Question

Body

Question: How to Detect and Match Semi-Transparent UI Elements?

We are currently trying to improve the automatic game dialogue skipping feature, but we have encountered some difficulties.

The skip button UI in the game is fully semi-transparent (the entire UI area is rendered with transparency, rather than only having a transparent background), as shown in the image. In addition, the skip button does not support keyboard input events and can only be triggered through mouse clicks.

We initially tried using traditional template matching, but the results were extremely poor. The main reason seems to be that the appearance of this UI element is affected by the underlying game scene. Since the entire UI is rendered with transparency, its final pixel values may change depending on the background behind it.

We also considered using alpha channel information for extracting or matching the UI element, but the results were not good. However, I do not have a deep understanding of alpha channels or transparency-based image processing, so my understanding of this approach may be limited. From what I have researched, it seems that a customized matching method may be required, such as an NCC (Normalized Cross-Correlation) algorithm with alpha-channel weighting or other alpha-aware template matching approaches.

Unfortunately, we do not have sufficient experience in developing or modifying these types of algorithms, and the exploration and implementation cost may be too high for us at this stage.

Furthermore, even when using the skip button UI image itself as the template, the matching accuracy is still very low. The reason is that the visual features of this UI element are relatively simple, and similar shapes or patterns frequently appear in other parts of the game interface, resulting in many false positives.

Another approach we are considering is to detect a different opaque UI element as an anchor point, and then locate the skip button through relative coordinate offsets before triggering the mouse click. However, this approach may not work reliably across different resolutions or window sizes.

Therefore, we would like to ask:

• Are there any other algorithms or approaches that are more suitable for detecting semi-transparent UI elements?
• Is there a better way to handle UI elements affected by transparency rendering?
• Is there a resolution-independent method for locating and interacting with such UI elements?

Any suggestions or guidance would be greatly appreciated.

Guidelines

• I have read and understood this category's guidelines before making this post.