Fullscreen / Chatting with the tool wont change widget or update it

[dustfire]

When in full mode, when asked a follow up question -- even it figures out that it can have the same widget but with different tool, it will create an inline version of the Widget and the current full screen widget wont update.. and user needs to exit full screen to see the new results. I can't seem to be interactively update the current widget.. Unless my query calls the same widget.. this is super limiting.

I want to say open search in full mode.. but then maybe i want to display an individual item by chat, it wont i have to exit full screen.. is there no way to tell GPT that during chat if it produces a new widget OR even the same widget HTML url.. is should swap?

If there is a chat put in there it should be smart enough to transition between widgets in same screen mode? Or am I doing something wrong here.

[henritoivar]

Having the same issue. One possible solution I tried is to have a “universal” widget which shows a different component based on the structuredOutput. It works to an extent, but it seems like an anti pattern. It also seems that even if you return the same widget it doesn’t always update predictably. I haven’t quite figured out what is the exact criteria.

[dustfire]

Exactly the same issue -- the universal approach I tried too, but even with that tool calls even wired up to use the same Widget HTMLs cause an inline new version to be created and wont update the full screen mode one -- sometimes it does, sometimes wont. It is very unreliable. Unless we are doing something wrong, would be super useful if OpenAI:

• Informed all other widgets about creation of new widget as global event -- this way we at least could take some action
• OR at least inform full screen widget of the new structuredOutput/state change -- this will help us decide what todo with UI

Currently user asks something on the chat box --> things happen on the background --> UI is stuck as if nothing happened unless user exists the full screen. This defeats the purpose of an interactive chat experience given developer has zero control over what chat does. At the very least developer can be informed when state changes.

[hmdoganjf]

I was able to solve this issue by using a BroadcastChannel between my widgets to send events to the fullscreen widgets on render. My App.tsx React code was something like this so this would call on widget render:

  useEffect(() => {
    const broadcastChannel = new BroadcastChannel('gpt-app-widget-channel');
    broadcastChannel.postMessage({ type: 'update-fullscreen-code' });
  }, []);

And on the component I want to update on fullscreen:

useEffect(() => {
  const broadcastChannel = new BroadcastChannel('gpt-app-widget-channel');
      const handleMessage = (event: MessageEvent) => {
        if (event.data?.type === 'update-fullscreen-code') {
          if (displayMode === 'fullscreen') {
            // trigger your update here
          }
        }
      };
      broadcastChannel.addEventListener('message', handleMessage);
  
      return () => {
        broadcastChannel.removeEventListener('message', handleMessage);
      };
}, [displayMode]);

Where displayMode is from the window.openai.

That said, I agree that this can be easily done by informing all widgets on new widget render, as @dustfire mentioned. Hope this helps!

[dustfire]

Ahaa! this is really interesting @hmdoganjf -- So I can use a channel even these widgets are in different iFrames to communicate between them as long as they are in the same page? thats quite nice, didn't know about this.

Will certainly try it, thank you 🙏 -- but agreed, would be much nicer if OpenAI informed the dev instead.

[hmdoganjf]

For the sake of completeness, there is a possible bug in this code involving two tabs of "fullscreened" GPT widgets, as BroadcastChannel API actually enables frames to communicate between tabs and windows too (in the same browser). It can be fixed by adding some string unique to the window to the channel name, but I did not bother with it.

[dustfire]

Thank you, at the very least learning something new here from you :)

@hmdoganjf did you try this on ChatGPT app, does it work there on mobile experience as well? I am worried even we do implement this, in iOS client this might fail given the limitations of the sandboxed web browser experiences Safari/WKWebView or however OpenAI team is rendering. (Same in Android)