Silient Failures in Web Crawling

[PeteHalsted]

I don't want to post this as a issue because it is nonspecific. But I got Archon up and running last night and have had some success playing with it but I seem to be having a lot of issues with silent failures when trying to add to the knowledge base with web crawling. Our current project is leverage TanStackStart + Clerk and the combination is both new enough, and subtly similar to normal Clerk, TanStack Router + Query, that sometimes Claude Code struggles to use the right patterns.

We have create our own TanStack Start Bible, with out established patterns which has helped but I am trying to pull in https://tanstack.com/start/latest/docs/framework/react/overview so I can have a more extensive RAG and hopefully even expand and correct somethings that perhaps our bible is missing.

I did the same thing with the Clerk docs, which there is some specific TanStack Start docs and I managed to get one Clerk Knowledge base create for that smaller section but not for the main Clerk docs.

The issue am am having and time and again, it runs for a while says its import pages, but at some point just goes away and never actually create an entry in the knowledge base. No error or anything I just end up back at the knowledge base dashboard with no new entries. And no indication if something is still trying to process in the background (spoiler alert it isn't), I have also seen the client show disconnecting for a few minutes and same thing when I return back at the dashboard.

Is the processing actually attached and depending on the client and not running on the server in the background? Did I miss something in my setup. I know its early on, but my impression was that the knowledge base was the piece of Archon that was the maturest. So I am thinking I must be doing something wrong.

Is there an error interface, logs, something I can explore to figure out where the failures are coming from and how to resolve them?

FYI - I am using Google Gemini API key (I am Gemini Pro), I have the models set to gemini-2.5-flash and gemini-embedding-001, I enabled Context Embedding and 3 parallel processes for embedding. I am choosing 3 levels as the level for crawling.

I believe the rest of defaults

[PeteHalsted]

I found a tip on one of the issues about using 127.0.0.1 instead of localhost. I changed that, and I also turned off Use Context Embedding and my import started working. I suspect the Context Embedding was the real culprit but I am not sure.

[coleam00]

That is good to know, thanks Pete! Contextual embedding is slow so it's hard to test with it all the time, but I will look into it.

[coleam00]

Then as far as the silent errors around the crawling, this happens sometimes when switching between providers which is in our board to fix!

[DigiNomadi]

I had this issue - run this when you're starting a new crawl and watch the output:

docker compose logs -f archon-server

You'll probably see a lot of "dimension" errors...and "insert" failures.

After a couple of hours working on this error with Claude (with Supabase MCP enabled), claude did fix the issue.

Root Cause Found!
The database schema has embedding VECTOR(1536) but Google's text-embedding-004 produces 768-dimensional vectors. The .env change isn't enough - we need to fix the database schema.

Note, this happened to me after I pulled the latest changes. Everything was fine before that.