Data Sources

Deutsch: Datenquellen

Data sources let the AI answer from your own documents: a file, all files of a folder, or the data of your organization on an ERI server. The AI searches them when a question calls for it, and names the passages it used.

Data sources are a beta feature: almost ready, but expect some bugs.

Switching data sources on

1. Open "Settings" → "App Options".
2. Under "Preview feature visibility", choose "Also show features in beta: these are almost ready for release; expect some bugs".
3. Under "Select preview features", choose "RAG: Preview of our RAG implementation where you can refer your files or integrate enterprise data within your company".

The navigation bar then shows "Data sources", and the chat gets a button for choosing data sources.

What you need

Files and folders on your computer need an embedding provider. It turns your texts into numbers that AI Studio can search. Set one up in "Settings" → "Configure Embedding Providers"; see Providers and Models.

With an embedding provider in the cloud, your documents are sent to that provider, and embedding can cost money, also repeatedly after changes. AI Studio asks you to confirm this. An embedding provider on your own computer, for example, with ollama, keeps your documents with you.

Data sources on an ERI server need no embedding provider: the server searches its data itself.

Adding a data source

Open "Settings" → "Configure Data Sources", or the gear icon on the "Data sources" page, and click "Add Data Source":

• "Local File": one file. Audio and video files are transcribed first, which needs a transcription provider.
• "Local Directory": all files of a folder, and with "Include subfolders?" also those of its subfolders.
• "External Data (ERI-Server v1)": the data of your organization on an ERI server.

Files and folders

• "Data Source Name": at most 40 characters.
• "Description": what the data source contains. It helps the AI choose the right data source.
• "Embedding": the embedding provider.
• "Required provider confidence level": only providers at this level or higher may see the content.
• "Where a chat may send data after reading this data source": see Where a chat may send data. The default keeps your content out of web searches.

For a folder, "Files to include" decides which files AI Studio reads, by default all documents and text files. Separate patterns with semicolons, for example, *.pdf;*.docx. The expert settings offer more: hidden files and folders, files and folders to leave out, such as node_modules/, a size limit of 1,024 MB by default, and how AI Studio cuts the documents into pieces.

ERI servers

Enter the "ERI v1 Server Hostname" and the "Port", then click "Test connection & read available metadata". After a successful test, choose the "Authentication Method" that the server offers, enter your credentials, and choose a retrieval process. Under "Your security policy", decide whether the content may go to any provider or only to self-hosted ones.

Your IT department gives you the address and the way to sign in. To build an ERI server yourself, see ERI Server.

Indexing

AI Studio prepares files and folders in the background. The "Data sources" page shows the progress of each data source: how many files are indexed, which are waiting, and which were skipped and why, for example, because a PDF has no readable text, a file is too large, or a recording needs a transcription provider. Images aren't indexed.

AI Studio notices when files change, and updates the index by itself. You can switch this off with "Automatic local data source refresh" and update a data source with "Refresh" instead. A new folder can be searched once its first indexing is finished.

Using data sources in the chat

1. In the chat, click "Select the data you want to use here." in the toolbar.
2. Switch on "Are data sources enabled?".
3. Choose how the AI uses them:
   • "AI-based data source selection" on: the AI picks the fitting data sources itself. Off: you choose them under "Available Data Sources".
   • "Semantic Search" on, the default: the AI searches your data sources whenever a question calls for it. Off: AI Studio searches them with every message before the AI answers, which is called classic RAG.

Semantic Search needs a model that can use tools. Otherwise, AI Studio uses classic RAG, and the selection says why.

Data sources that the selected provider may not see are greyed out or missing. To start every new chat with certain data sources, set them in the "Chat Options" under "Data Source Selection".

Sources in the answer

The passages the AI received are listed below its answer, as "Sources provided by the data providers". The eye button shows exactly what the AI received from a source. For documents with pages, the page numbers open the document at that page.

Where to go next

• Tools in the Chat: Semantic Search and the other tools.
• Provider Confidence: keep confidential documents away from some providers.