[Store] Indexer is nice, but hard to use directly

[lyrixx]

Hello.

In my previous POC, I did everything manually. And it worked well! Thanks for that.

With the bundle, I discover the Indexer stuff.

It looks good.

But It seems I cannot use it. The indexer need a loader to work. I can override it with a specific source. But my use case does not fit.

I consume messages from Rabbitmq through messenger. So the fetching documents from loader part is not useful for me. But the next bits are really cool (filters, transformers, vectoriser, and store).

So I would expect a service where I'm able to do:

#[AsMessageHandler()]
final readonly class VectorizeCrawlUrlHandler
{
    public function __construct(
        // #[Target('crawl_url')]
        private IndexerInterface $indexer,
    ) {
    }

    public function __invoke(VectorizeCrawlUrl $crawlUrl): void
    {
        $this->indexer->index(new TextDocument(
            id: new Uuid($crawlUrl->id),
            content: $crawlUrl->body,
            metadata: new Metadata([
                // My stuff here
            ]),
        ));
    }
}

What do you think about this use case?

Could we consider splitting the indexer? Or we could change a bit the API to accept a document, or a collection of document. And in such situation, we skip the loader?

[chr-hertel]

yup, that seems like a valid flaw to me - not have a good answer from the top of my head but @OskarStark is deeper into this anyways.

i just feel like we have already quite some concepts here with indexer, loader, filters, transformers, vectorizer and store, and we should be careful to only add and add more ... so i'd be in favor of having maybe two methods on the indexer? indexDocuments and indexSource maybe?

or again that overloading that all scenarios would work on one method?
index(string|string[]|TextDocument|TextDocument[] $indexable)

[lyrixx]

I did not finished, but I started that last week:

diff --git a/src/store/src/Indexer.php b/src/store/src/Indexer.php
index d797c05f..21186fe4 100644
--- a/src/store/src/Indexer.php
+++ b/src/store/src/Indexer.php
@@ -26,12 +26,18 @@ use Symfony\AI\Store\Document\VectorizerInterface;
 class Indexer implements IndexerInterface
 {
     /**
-     * @var array<string|null>
+     * @var array<string>
      */
     private array $sources = [];
 
+    /**
+     * @var array<EmbeddableDocumentInterface>
+     */
+    private array $documents = [];
+
     /**
      * @param string|array<string>|null $source       Source identifier(s) for data loading (file paths, URLs, etc.)
+     * @param EmbeddableDocumentInterface|array<EmbeddableDocumentInterface>|null $document       Document or array of documents to process directly, bypassing the loading step
      * @param FilterInterface[]         $filters      Filters to apply after loading documents to remove unwanted content
      * @param TransformerInterface[]    $transformers Transformers to mutate documents after filtering (chunking, cleaning, etc.)
      */
@@ -40,32 +46,47 @@ class Indexer implements IndexerInterface
         private VectorizerInterface $vectorizer,
         private StoreInterface $store,
         string|array|null $source = null,
+        EmbeddableDocumentInterface|array|null $document = [],
         private array $filters = [],
         private array $transformers = [],
         private LoggerInterface $logger = new NullLogger(),
     ) {
         $this->sources = null === $source ? [] : (array) $source;
+        if (null === $document) {
+            $this->documents = [];
+        } elseif ($document instanceof EmbeddableDocumentInterface) {
+            $this->documents = [$document];
+        } else {
+            $this->documents = $document;
+        }
     }
 
-    public function withSource(string|array $source): self
+    public function withSource(EmbeddableDocumentInterface|string|array $source): self
     {
-        return new self($this->loader, $this->vectorizer, $this->store, $source, $this->filters, $this->transformers, $this->logger);
+        return new self($this->loader, $this->vectorizer, $this->store, $source, [], $this->filters, $this->transformers, $this->logger);
+    }
+
+    public function withDocument(EmbeddableDocumentInterface|array $document): self
+    {
+        return new self($this->loader, $this->vectorizer, $this->store, null, $document, $this->filters, $this->transformers, $this->logger);
     }
 
     public function index(array $options = []): void
     {
         $this->logger->debug('Starting document processing', ['sources' => $this->sources, 'options' => $options]);
 
-        $documents = [];
-        if ([] === $this->sources) {
-            $documents = $this->loadSource(null);
-        } else {
-            foreach ($this->sources as $singleSource) {
-                $documents = array_merge($documents, $this->loadSource($singleSource));
+        $documents = $this->documents;
+        if (!$documents) {
+            if (!$this->sources) {
+                $documents = $this->loadSource(null);
+            } else {
+                foreach ($this->sources as $singleSource) {
+                    $documents = array_merge($documents, $this->loadSource($singleSource));
+                }
             }
         }
 
-        if ([] === $documents) {
+        if (!$documents) {
             $this->logger->debug('No documents to process', ['sources' => $this->sources]);
 
             return;

I won't be able to work on this projet until Wednesday. I'll keep you posted with my progress