Difficulty building a GitHub-based knowledge base in OpenSearch through OpenRAG/Langflow

[gabrielz06]

Specifications

OpenRAG version: 0.5.1
Langflow version: 1.9.0

Context

I am trying to build a RAG system with OpenRAG to answer questions grounded in a GitHub repository.

My goal is to:

• Access repository files through the GitHub MCP Server
• Retrieve the file contents
• Index those contents into OpenSearch
• Use that index as the knowledge base for RAG

Essentially, the key idea is to build a reliable ingestion pipeline that can index file contents from a GitHub repository into OpenSearch and use that data as a knowledge base in OpenRAG.

I am not using Docling, because the ingestion is not local.

Since OpenRAG is built on top of Langflow and OpenSearch, I thought this community would be the right place to ask about the best architecture for this workflow.

What I could do

I was able to build a flow that indexes one file at a time.

In that flow:

• an Agent component retrieves the complete text of a specific file
• the Language Model is used only to identify/extract the file title or name
• the document is then indexed successfully into OpenSearch

So, single-file ingestion works correctly.

What I could not do

I was not able to build a successful flow to index all files in a repository, or even all files inside a given directory.

In this second flow:

• the first Agent generates a list of the target files in the repository
• the second Agent retrieves the full content of each file
• the Language Model is used only to identify/extract the title of each file before indexing

When I try the multi-file ingestion flow, I get the following error:
Error code: 429 - {'error': {'message': 'Request too large for gpt-5-nano in organization ******** on tokens per min (TPM): Limit 200000, Requested 1348009. The input or output tokens must be reduced in order to run successfully. Visit https://platform.openai.com/account/rate-limits to learn more.', 'type': 'tokens', 'param': None, 'code': 'rate_limit_exceeded'}}.

My question

I am not sure whether this is:

• an architecture limitation
• a configuration mistake
• a problem in how I designed the Langflow/OpenRAG flow
• or simply the wrong approach for repository-wide ingestion

What is the recommended way to implement this kind of ingestion in OpenRAG?

If anyone has implemented something similar, or can suggest the recommended architecture, I would be very grateful.

Thank you!

[thangldw]

The 429 is caused by the shape of the flow, not by OpenSearch. Agent 2 is effectively concatenating the repository into an LLM request, so the 1.35M requested TPM is expected. Repository ingestion should be a bounded ETL pipeline; the LLM should only see retrieved chunks at query time.

For OpenRAG 0.5.1, I would use this pipeline:

Git clone / GitHub Tree API
  -> deterministic file filtering
  -> one file per ingestion unit (bounded worker pool)
  -> Docling -> SplitText -> Embedding -> OpenSearch

Concretely:

1. For a one-time import, clone/download the repository and use Knowledge -> Add Knowledge -> Folder. OpenRAG already processes folders in the background through the OpenSearch Ingestion flow.
2. For a continuously synchronized repository, use GitHub only as the data source: enumerate the Git tree, fetch changed blobs, and materialize each blob as a file. Do not ask an Agent/LLM to enumerate files or aggregate their contents.
3. Submit files independently with bounded concurrency. In v0.5.1, OpenRAG uploads a file through POST /api/v2/files, then passes its path to run_ingestion_flow; that flow performs Docling parsing, splitting, embedding, and OpenSearch indexing. No title-extraction LLM is needed.
4. Derive the title deterministically from the relative path (optionally the first Markdown heading). Store metadata such as repo, ref, path, blob_sha, and html_url.
5. Use a stable document key such as repo:ref:path, skip files whose blob SHA has not changed, re-index changed files, and delete paths removed from the repository.
6. Exclude binaries and generated/vendor directories, cap individual file size, and apply retry/backoff to the ingestion worker pool. Start with 4-8 concurrent files and tune from observed ingestion latency/rate limits.

If you need to keep the MCP server, use it only to return one file (or one small page of file metadata) per call and immediately hand that file to ingestion. It should not be placed behind an Agent that accumulates every tool result in the model context.

Relevant v0.5.1 implementation/docs:

• Folder ingestion and background tasks
• upload_user_file and run_ingestion_flow
• Default ingestion flow

So the main fix is architectural: remove both Agents and the title LLM from the ingestion path. Changing the chat model or increasing its TPM limit would only postpone the same failure.