How to refer an gameobject as "THIS" or "THAT"

[Efe-Oral]

I am working on a real time VR application where I want to look at a game object and refer to it as "THIS" or "THAT". How do I achieve this functionality? At first, I thought of adding the looked game object's name alongside with the natural language user prompt to send it to LLM. But that was a bit unstable. So do you have any idea for a more reliable method to address game objects in VR setting as "THIS" or "THAT" without having to specify its name?

[CoderGamester]

This is an interesting idea and a valuable improvement.
Can be done with a image object identification with raycasts BUT just important to remind that AI at the moment is really bad with 3D space and positioning context in non 2D axis.

@Efe-Oral I wonder if you have 100 objects in the screen, how would you recognize THAT?

[Efe-Oral]

At first, I tried the raycast approach where I shoot it from the VR headset to the object, as well as drawing an outline around it so that I know which object I am looking at.

So far so good, but the problem starts when I actually try to include the object I am looking at to the natural language prompt with THIS or THAT words. Maybe the system prompt for the LLM can be modified so that it expects these words. Or maybe a word matching can be made for the fords "THIS" and "THAT" which would change the words to the object's name extracted from the raycast, and add it to the natural language prompt. Or maybe the MCP tool's parameters can be adjusted to accept these words. What do you think?

I wanted to ask for opinions before I actually commit to an idea and waste time building it.

[CoderGamester]

Thank you @Efe-Oral
Will look into your idea. Sounds like something valuable to other users too

[Efe-Oral]

Thank you, please keep me updated