AI in Microsoft Teams: J.A.R.V.I.B.

Your custom Microsoft Teams AI Copilot to build interactive dynamic 3D worlds

JARVIB (Just A Rather Very Intelligent Bot) is your personnal Copilot to help you designing 3D worlds in a collaborative way inside a personnal conversation or in a Teams meeting. Ask him to create various Babylon.js primitive or to search for specific models to load them into the scene. You can also ask to the AI to visually describe the scene for accessibility reasons. Look at possible prompts in promptsExamples.txt

• AI in Microsoft Teams: J.A.R.V.I.B.
  • View it in action
  • Setting up the sample
  • Interacting with the bot
  • Further reading

View it in action

JARVIB is using the Teams AI library to map users' intent to 3 different type of actions:

• codeToExecute which will take the Babylon.js code generated by the LLM and send it to the web page containing the 3D canvas to execute it via WebSocket
• listAvailableModel which will use the same API as PowerPoint to find for existing 3D models and return a list of available models to load through an Adaptive Card list.
• loadThisModel which will load one of the models exposed in the list displayed before

You have also 3 commands available:

• /reset will clear the conversation state, clean the various objects states and reload the web page containing the 3D canvas to start from scratch
• /describe will ask to the AI, via the text-davinci-003 LLM model to visually describe the scene
• /fullcode will return all Babylon.js code executed so far so you can copy/paste it in your project or in the Babylon.js Playground

Look at the /src/prompts in /ChatGPT/skprompt.txt to check how the AI is prompted and check the /ChatGPT/actions.json file to understand how the various actions are described for the 3 Teams AI actions.

The prompt configuration:

Pretend you're an expert in Babylon.js, the JavaScript WebGL 3D engine. 

Assume there is already an existing Babylon.js scene and engine so you don't have to create them, just generate the code to add into an existing program.
Use the scene and engine objects directly. 

Pay attention when trying to access previously created Meshes by getting access to them via their name rather than assuming the associated variable is already created.
When writing a new code, consider all the previous one you've generated to be sure the new code will be consistent with the previous one.
Remember about the already created meshes, animations or any other specific ressources before trying to create them or reuse them.

The JSON schema to help the LLM to map to the defined actions:

[
    {
        "name": "codeToExecute",
        "description": "Returns the Babylon.js JavaScript code matching the user intent, building the next code working in a consistent way with the previous code already executed",
        "parameters": {
            "type": "object",
            "properties": {
                "code": {
                    "type": "string",
                    "description": "The JavaScript code to execute next"
                }
            },
            "required": [
                "code"
            ]
        }
    },
    {
        "name": "listAvailableModel",
        "description": "List the available 3D models we can load from the library",
        "parameters": {
            "type": "object",
            "properties": {
                "nameOfTheModel": {
                    "type": "string",
                    "description": "The name of the model to search inside the library"
                }
            },
            "required": [
                "nameOfTheModel"
            ]
        }
    },
    {
        "name": "loadThisModel",
        "description": "Load the 3D model specified by the user",
        "parameters": {
            "type": "object",
            "properties": {
                "nameOfTheModel": {
                    "type": "string",
                    "description": "The name of the model to load from the library"
                }
            },
            "required": [
                "nameOfTheModel"
            ]
        }
    }
]

Setting up the sample

1. Clone the repository

   git clone https://github.com/davrous/JARVIB.git

2. Install the Teams Toolkit: https://learn.microsoft.com/en-us/microsoftteams/platform/toolkit/install-teams-toolkit

3. Set your Azure OpenAI key, endpoints and model names or your OpenAI key in the .env file.

   For Azure OpenAI, pay attention to use the "Deployment name" and not the "Model name" as the value for the properties in the .env file. If you've got an error 404 during execution, it's because you either entered the wrong endpoint or the wrong deployment name.

4. Press F5!

   The Team Toolkit will do a lot of magic for you by:

   • installing all dependencies and building the TypeScript code
   • registering the bot in the Azure Active Directory
   • building the Teams manifest file, zipping the resources to upload them in the Teams Developer Portal
   • creating 2 tunnels to expose your localhost on public URLs using VS Dev Tunnels, you can find the details in /.vscode/tasks.json
     • 1 for the bot itself mapped on port 3978
     • 1 for the tab to be used inside a Teams meeting mapped on port 3000
   • launching the Teams web app to let you install the bot / meeting extension

5. If everything goes well, you should first be able that the 2 tunnels have been created:

6. Then, the webserver will be launched and you'll be asked to install JARVIB in your developer tenant:

You can either simply press the "Add" button and this will install the app to let you discuss with it directly as a classical bot.

You can also add it to an existing meeting to be able to replicate the full experience showcased in the video.

Interacting with the bot

The easiest way to try it is to install it using the "Add" button as a bot, then open http://localhost:3000/debug.html on the side open a simple canvas 3D websocket client. Then ask to the bot to create 3D content, load some models and so on:

The other option to replicate the full experience is to load the app inside an existing meeting:

You need to accept connecting to the VS Dev Tunnel redirecting to your localhost developer machine:

Finally, save JARVIB as an app part of this chosen Teams meeting:

Now, launch the Teams meeting alone or with someone else to collaborate with:

Click on the JARVIB icon and then click on the "Share to meeting" button:

Finally, call the AI bot via @JARVIB-local following your order:

Further reading

• Teams AI library
• Azure OpenAI
• Teams Toolkit overview
• How Microsoft Teams bots work