[AI Bundle][Store] Document Indexing Pipeline

[chr-hertel]

A common scenario is to

• load documents from a specific source (e.g. folder of PDFs, set of entities, URLs, ...)
• transform them (e.g. split into chunks, normalize language, summarize, ...)
• vectorize them with an embedding model
• store them in a vector store
• query for them incl. keeping track of original sources while retrieval

A configuration for this could for example look like this:

ai:
    indexer:
         my_indexer:
              loader: 'service_id'
              source: 'string' or ['array', 'of', 'string']
              transformer:
                  - 'service_id'
                  - 'service_id'
              embeddings:
                   class: 'Symfony\AI\Platform\Bridge\OpenAi\Embeddings'
                   name: 'text-embedding-ada-002'
              store: 'ai.store.chroma_db.my_store'

And we could also add a command here:

$ bin/console ai:store:index my_indexer

---

This should sit on top / rework

• Symfony\AI\Store\Document\LoaderInterface
• Symfony\AI\Store\Document\TransformerInterface
• Symfony\AI\Store\Document\Vectorizer
• Symfony\AI\Store\Indexer

[OskarStark]

The title is a bit misleading, as its not about documentation but about real Documents 😄

[OskarStark]

Can you clarify a bit more what source should contain? So maybe a dummy example of the yaml? Thanks

[chr-hertel]

scenarios i could think of:

• A DirectoryLoader with an path as source
• A PdfLoader with a path as source
• An UrlLoader with an url as source
• A YoutubeTranscriptLoader with an ID as source
• A RssFeedLoader with an URL as source

[OskarStark]

so loader and source are combined option? the source is used by a loader service?

[OskarStark]

Implementation Plan

Phase 1: Core Interfaces & Directory Loader

Core Components:

// LoaderInterface - loads documents from sources
interface LoaderInterface {
    /** @return Document[] */
    public function load(string|array $source): array;
}

// Document - represents a loaded document
class Document {
    public function __construct(
        public readonly string $content,
        public readonly array $metadata = []
    ) {}
}

// TransformerInterface - processes documents
interface TransformerInterface {
    /** @return Document[] */
    public function transform(array $documents): array;
}

DirectoryLoader Implementation:

• Recursively scan directories for text files
• Support file filtering by extension
• Capture file path, modified time in metadata

Basic Transformers:

• TextSplitterTransformer - chunk documents by size/overlap
• MetadataEnricherTransformer - add custom metadata

Phase 2: Bundle Integration

Configuration Schema:

ai:
    indexers:
        docs:
            loader: 'ai.loader.directory'
            source: '%kernel.project_dir%/docs'
            transformers: ['ai.transformer.text_splitter']
            embeddings:
                platform: openai
                model: text-embedding-3-small
            store: ai.store.memory

Services:

• Auto-register loaders/transformers with tags
• IndexerService orchestrates pipeline
• Console command: ai:index docs

Phase 3: Examples

Basic Usage:

$indexer = $container->get('ai.indexer.docs');
$indexer->index(); // Process & store documents

Store Integration:

$store = $container->get('ai.store.memory');
$results = $store->query('How to configure AI bundle?');

This approach proves the architecture with minimal complexity while staying extensible for PDF/URL loaders later.

[OskarStark]

I propose to rename source to sources, like transfomers

[OskarStark]

@chr-hertel what about making the source/sources optional? This way I can implement my FunkyDatabaseFooLoader without any "arguments"(sources)?

[OskarStark]

We are lacking the platform in your example, right?