[Proposal] Stream-Aware RAG Plugin: Dynamic retrieval with hybrid fallback #37320
Replies: 4 comments 12 replies
-
|
Please check the |
Beta Was this translation helpful? Give feedback.
-
|
@dosu Would you help me to check my proposal's viability and offer me some further suggestions? Thank you |
Beta Was this translation helpful? Give feedback.
-
|
Hi @KERVIN-FARMER! I'm Dosu and I'm helping the Dify team. I've reviewed your Stream-Aware RAG proposal against the current Dify plugin architecture. Here's my assessment: Viability SummaryThe core V2 concept (chunk-level mid-stream token interception and injection) is not feasible with the current Plugin SDK. Dify's streaming architecture is append-only and designed for observability/replay via Redis Streams — there are no plugin hooks to intercept, pause, or inject context during token generation [1] [2]. However, your Fallback 1–4 chain is partially achievable using an iterative Agent Strategy plugin. What Works Today (No Core Mods)
What Does NOT Work
Suggestions
Revised Architecture RecommendationThis multi-turn pattern achieves your functional goals without core modifications. The tradeoff is latency (full generation cycles between retrieval steps) rather than true streaming interruption. Would you like me to look into specific implementation details of the Agent Strategy plugin interface, or explore the Datasource plugin type for your retrieval layer? To reply, just mention @dosu. Share context across your team and agents. Try Dosu.
|
Beta Was this translation helpful? Give feedback.
-
|
@dosu Thank you for replying! Your comment is truly inspiring, though I don't know if your modified plan is still worthwhile to do and can add value to the Dify programme. stream-aware-rag.md If you fell my plan is not as worthwhile to deal with, please read the following file to suggest me some new projects: |
Beta Was this translation helpful? Give feedback.
-
|
Good news — you don't need ECS or AutoDL to contribute meaningful plugins to Dify. The entire plugin development workflow runs locally with Docker + Python. Let me refine the two options for your situation. Updated Assessment: Option 2 vs Option 3Option 2: Agent Evaluation FrameworkReality check: Agent evaluation is currently deeply embedded in Dify's core test infrastructure ( Verdict: Option 3: Intelligent Retrieval Router (Datasource Plugin)This is your best bet. Here's why:
Concrete Plan for Option 3 (No Cloud Required)What you'd build: A plugin that intelligently routes incoming queries to the most relevant knowledge source based on query classification — instead of blindly searching all datasets. Local tech stack:
Step-by-Step RoadmapWeek 1: Setup & Study
Week 2: MVP
Week 3: Polish & Ship
Key Tips
Resume Framing
Alternative Low-Cost Option: Agent Strategy PluginIf you find the Datasource approach too narrow, consider building a custom Agent Strategy plugin instead. The existing ReAct [8] and Function Calling [9] strategies in
Both work locally with any LLM API. My recommendation: Start with the Intelligent Retrieval Router (Option 3). It's the most scoped, most feasible, and directly builds the RAG/retrieval skills relevant to an AI career. Would you like me to dig into any specific part of this plan? To reply, just mention @dosu. Share context across your team and agents. Try Dosu.
|
Beta Was this translation helpful? Give feedback.
-
|
@dosu I guess the Alternative Low cost option may be better for me since they're not as narrow and can be presented on CV and accepted by the reviewers; Correct me if I'm wrong |
Beta Was this translation helpful? Give feedback.
-
|
I'd actually push back slightly on that assumption. Here's why: Agent Strategy Plugin: Honest Risk AssessmentThe situation is less favorable than it might appear:
What Actually Looks Good on a CVA merged contribution to a 100k+ star project > an unmerged ambitious PR. For "accepted by reviewers" + "CV-worthy," your best options are ranked:
My Revised RecommendationIf your goal is "merged contribution that demonstrates AI/LLM skills on a CV", I'd suggest either:
Both are more likely to actually get merged than a novel agent strategy, and a merged PR is what matters for your CV. That said — if you're passionate about building a new agent strategy and view it primarily as a learning exercise (even if it doesn't get merged), it's still valuable for understanding agent architectures. Just set expectations accordingly. What direction feels right to you? To reply, just mention @dosu. Share context across your team and agents. Try Dosu.
|
Beta Was this translation helpful? Give feedback.
-
|
@dosu Thank you for replying, and here're the reply made by Claude to me, and would you give me the new advice based on that and all informations: ● 基于对 Dify 架构的深入分析，我现在为你构思 3 🚀 需求一：Multi-Modal Chain-of-Thought 图谱推理引擎 为什么"别人没做过"（差异化分析） 创新点：
与常见方案的区别：
业务/技术痛点 痛点场景：
技术痛点：
核心功能与实现路径 架构设计 ┌─────────────────────────────────────────────────────────┐ 实现路径 Step 1: AutoDL 端实现推理服务 autodl/visual_cot_service.pyclass VisualCoTReasoner: Step 2: Dify 自定义节点 api/core/workflow/nodes/visual_cot/visual_cot_node.pyfrom graphon.nodes.base.node import Node class VisualCoTNode(Node[VisualCoTNodeData]): Step 3: 向量数据库扩展（关联图谱） api/packages/dify-vdb-graphrag/src/dify_vdb_graphrag/vector.pyclass GraphRAGVector(BaseVector): 源码修改点：
简历杀手锏亮点
潜在难点与避坑指南 难点 1: 视觉模型推理质量不稳定
难点 2: 推理图谱存储开销大
难点 3: ECS-AutoDL 网络延迟
难点 4: 与 Dify Agent 节点集成复杂
🚀 需求二：实时多模态流式 RAG 引擎（Stream-Aware RAG） 为什么"别人没做过"（差异化分析） 创新点：
与常见方案的区别：
业务/技术痛点 痛点场景：
技术痛点：
核心功能与实现路径 架构设计 ┌─────────────────────────────────────────────────────────┐ 实现路径 Step 1: AutoDL 实现不确定性估计服务 autodl/uncertainty_estimator.pyclass UncertaintyEstimator: Step 2: Dify StreamRAG 节点 api/core/workflow/nodes/stream_rag/stream_rag_node.pyclass StreamRAGNode(Node[StreamRAGNodeData]): Step 3: 异步检索任务 api/tasks/stream_rag_tasks.pyfrom celery import shared_task @shared_task(queue='dataset') 源码修改点：
简历杀手锏亮点
潜在难点与避坑指南 难点 1: LLM 流式生成无法中断和重启
难点 2: 不确定性估计延迟高
难点 3: 异步检索结果如何注入
难点 4: 如何判断需要检索什么
🚀 需求三：Agent 执行轨迹的对抗式自我改进系统（Adversarial Self-Refinement） 为什么"别人没做过"（差异化分析） 创新点：
与常见方案的区别：
业务/技术痛点 痛点场景：
技术痛点：
核心功能与实现路径 架构设计 ┌─────────────────────────────────────────────────────────┐ 实现路径 Step 1: AutoDL Critic Agent 服务 autodl/critic_agent_service.pyclass CriticAgent: Step 2: Dify SelfRefineAgent 节点 api/core/workflow/nodes/self_refine_agent/self_refine_agent_node.pyclass SelfRefineAgentNode(Node[SelfRefineAgentNodeData]): |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for sharing Claude's suggestions. I need to give you a frank reality check on all three — because Claude's proposals are designed around assumptions that don't match your situation (no ECS, no AutoDL, no GPU). Critical Problem With All Three ProposalsClaude's suggestions all share the same fundamental issues:
Proposal-by-Proposal Assessment
What's Actually SalvageableOption 3 (Self-Refinement) has the best core idea and can be adapted to work within plugin constraints: Realistic Version: "Self-Refining Agent Strategy" PluginInstead of a custom workflow node, implement this as an Agent Strategy plugin that:
Why this works:
Resume framing:
What You'd Actually BuildThe strategy's
My Updated RecommendationGiven your constraints (no cloud, AI career goals, want something novel + mergeable): Build the Self-Refining Agent Strategy plugin. It's:
The Datasource plugin (Retrieval Router) remains the safer backup if you want guaranteed acceptance with lower novelty. Would you like me to dig into the Agent Strategy plugin interface so you can see exactly how to structure the self-refinement logic? To reply, just mention @dosu. Share context across your team and agents. Try Dosu.
|
Beta Was this translation helpful? Give feedback.
-
|
The hybrid fallback chain is a solid architecture. A few thoughts on making each tier more effective based on production experience with similar patterns. On the retrieval strategy itself: rather than treating vector-first with BM25 as a fallback chain (try vector, if insufficient fall back to BM25, then keyword, then pure LLM), the real power comes from fusion — running both retrieval paths simultaneously and combining results with weighted scoring. A typical effective split is 0.3 BM25 plus 0.7 vector similarity, followed by cross-encoder reranking on the merged result set. This catches cases where vector search misses exact terminology (product names, error codes, specific numbers) that BM25 handles well, and cases where BM25 misses semantically equivalent but differently-worded content that vector search handles well. Running them in parallel rather than as a fallback chain means you get the benefit of both on every query, not just when the primary fails. The stream-aware aspect of the proposal — dynamically retrieving as the conversation evolves — is where things get interesting and where memory comes in. As the conversation progresses, the retrieval context should shift to reflect accumulated context, not just the latest user message. We handle this by maintaining a working memory of the evolving conversation context and re-ranking retrieved results against this accumulated context rather than just the initial query. Concretely: after each turn, update a running summary of what the conversation is about (topics, entities mentioned, constraints established), and use that summary as an additional signal when scoring retrieval results. A document that was irrelevant to the first query might become highly relevant by turn three once the user has narrowed the topic. One practical consideration for the confidence-based retrieval trigger (your V2 target): perplexity/entropy thresholds need calibration per domain and per model. A model generating text about a familiar topic will have low perplexity regardless of whether it is hallucinating about specifics. In practice, we found that tracking retrieval hit rate (did the retrieved content actually get used in the response) is a better feedback signal than generation-time perplexity for deciding when retrieval adds value. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for your review! Your idea of 30% word search + 70% vector similarity + Jina reranker and parent page retrival are pretty riveting, and I agree with your point that recall rate is more important than perplexity and entropy. Though, how would you comment on my proposal, or would you suggest that the newest proposal made by @dosu the most viable and worthwhile task that I can make contribution to? Thank you |
Beta Was this translation helpful? Give feedback.
-
|
Great question — and @ferhimedamine's feedback is genuinely valuable production wisdom. Let me synthesize everything. Where Things StandYour original Stream-Aware RAG proposal has strong ideas but faces hard architectural blockers in Dify's plugin SDK (no mid-stream interception, no logprobs access, no custom nodes). @ferhimedamine's suggestions (fusion retrieval, working memory, hit-rate feedback) are excellent retrieval engineering insights, but they don't resolve the fundamental problem: Dify plugins can't intercept token streams or create new workflow node types [1]. My Honest RecommendationYes, the Self-Refining Agent Strategy plugin remains your best path given your constraints (no GPU, no cloud, intern-level scope, want something novel + mergeable). Here's why it still wins:
How to Incorporate @ferhimedamine's IdeasThe good news: you don't have to choose either the retrieval improvements or the self-refinement. You can combine them:
This way, when the agent fails on a knowledge-intensive task, the self-refinement loop can:
That's a compelling end-to-end story: "Self-refining agent with adaptive retrieval" — academically grounded, practically useful, and entirely buildable within the plugin SDK. Next StepsIf you're ready to move forward with this direction:
Would you like me to dig into the Agent Strategy interface details so you can start building? To reply, just mention @dosu. Share context across your team and agents. Try Dosu.
|
Beta Was this translation helpful? Give feedback.
-
|
@dosu I would choose the most viable and the most worthwhile direction, and also need to ensure the work can be merged. Please list me all the parts and works I need to deal with. Also, would you write me skills, hooks or prompts to do that project? I can forward your writings to my agent to check, update and code. Thank you |
Beta Was this translation helpful? Give feedback.
-
|
Here's your complete project plan and a detailed prompt/skill you can hand to your coding agent. Project: Self-Refining Agent Strategy Plugin for DifyWhy this is the most viable + mergeable direction:
All Parts & Work ItemsPhase 1: Setup (Day 1)
Phase 2: Plugin Scaffold (Day 2)Create file structure: Phase 3: Core Implementation (Days 3–5)
Phase 4: Testing (Days 6–7)
Phase 5: Polish & Submit (Days 8–9)
Prompt/Skill for Your Coding AgentHere's a complete prompt you can forward: # Self-Refining Agent Strategy Plugin for Dify
## Context
You are building a Dify Agent Strategy plugin called "self_refine_agent". This is a plugin for the Dify open-source LLM platform (https://github.com/langgenius/dify).
Agent Strategy plugins extend `BaseAgentStrategy` and implement a `_invoke()` generator method that yields `AgentInvokeMessage` objects. The plugin runs within the Dify Plugin Daemon and communicates via the `self.session` API.
## Reference Architecture
Study the existing `cot_agent` plugin at: https://github.com/langgenius/dify-official-plugins
- File: `strategies/ReAct.py` — ReAct loop implementation
- File: `strategies/function_calling.py` — Function Calling implementation
- File: `provider/agent.yaml` — Provider configuration
- File: `manifest.yaml` — Plugin manifest
## What to Build
### Strategy Logic: Self-Refining Agent
The strategy implements this loop:
1. **Execute**: Run agent task (invoke LLM + tools, like ReAct or Function Calling)
2. **Evaluate**: Check if the output is satisfactory (use LLM as judge)
3. **Critique**: If not satisfactory, generate a critique identifying specific issues
4. **Refine**: Re-execute with the critique injected as additional context
5. **Repeat** until quality threshold met or max refinements reached
6. **Stream** all intermediate steps (execution, critique, refinement) as visible log messages
### Key Interfaces Available
```python
# Base class to extend
from dify_plugin.interfaces.agent import AgentStrategy
class SelfRefineStrategy(AgentStrategy):
def _invoke(self, params, user_id, conversation_id, app_id, message_id, credentials):
# params contains: model, tools, instruction, query, context,
# maximum_iterations, max_refinements, refinement_threshold
...
yield self.create_text_message("final output")
yield self.create_log_message(message="...", status=LogStatus.START)
yield self.create_json_message({...})# LLM invocation
chunks = self.session.model.llm.invoke(
model_config=LLMModelConfig(**model_params),
prompt_messages=[
SystemPromptMessage(content="system prompt"),
UserPromptMessage(content="user message"),
AssistantPromptMessage(content="assistant response"),
],
stream=True,
stop=["Observation:", "Observation:\n"]
)
# Returns Generator[LLMResultChunk] when stream=True
# Tool invocation
tool_result = self.session.tool.invoke(
provider=tool_provider,
tool_name=tool_name,
tool_parameters={"param": "value"}
)
# Returns list of ToolInvokeMessage objectsStrategy YAML ConfigurationCreate identity:
name: self_refine
author: <your-github-username>
label:
en_US: Self-Refine
zh_Hans: 自我优化
description:
en_US: "Self-refining agent strategy that automatically critiques and improves its output through iterative refinement. Based on Self-Refine (NeurIPS 2023)."
zh_Hans: "自我优化 Agent 策略，通过迭代式自我批判和改进自动提升输出质量。基于 Self-Refine (NeurIPS 2023)。"
parameters:
- name: model
type: model-selector
scope: tool-call&llm
required: true
label:
en_US: Model
zh_Hans: 模型
- name: tools
type: array[tools]
required: true
label:
en_US: Tool list
zh_Hans: 工具列表
- name: instruction
type: string
required: true
label:
en_US: Instruction
zh_Hans: 指令
auto_generate:
type: prompt_instruction
template:
enabled: true
- name: query
type: string
required: true
label:
en_US: Query
zh_Hans: 查询
- name: context
type: any
scope: array[object]
required: false
label:
en_US: Context
zh_Hans: 上下文
- name: maximum_iterations
type: number
required: true
default: 5
min: 1
max: 500
label:
en_US: Maximum Iterations
zh_Hans: 最大迭代次数
- name: max_refinements
type: number
required: true
default: 2
min: 0
max: 5
label:
en_US: Maximum Refinements
zh_Hans: 最大优化次数
extra:
python:
source: strategies/self_refine.pyCore Implementation Pattern# strategies/self_refine.py
def _invoke(self, params, user_id, conversation_id, app_id, message_id, credentials):
model = params["model"]
tools = params["tools"]
query = params["query"]
instruction = params["instruction"]
max_iterations = params["maximum_iterations"]
max_refinements = params["max_refinements"]
refinement_count = 0
previous_critique = None
while refinement_count <= max_refinements:
# === EXECUTION PHASE ===
yield self.create_log_message(
message=f"Attempt {refinement_count + 1}: Executing agent...",
status=LogStatus.START
)
# Build prompt (inject critique from previous iteration if available)
system_prompt = instruction
if previous_critique:
system_prompt += f"\n\n[SELF-REFINEMENT CONTEXT]\nPrevious attempt had these issues:\n{previous_critique}\nPlease address them in this attempt."
# Run agent loop (ReAct or Function Calling style)
agent_output = self._execute_agent_loop(
model, tools, system_prompt, query, max_iterations
)
yield self.create_log_message(
message=f"Attempt {refinement_count + 1}: Complete",
status=LogStatus.FINISH
)
# === EVALUATION PHASE ===
if refinement_count >= max_refinements:
# No more refinements allowed, return current output
break
yield self.create_log_message(
message="Evaluating output quality...",
status=LogStatus.START
)
critique = self._evaluate_and_critique(model, query, agent_output)
if critique["is_satisfactory"]:
yield self.create_log_message(
message="Output quality: PASS",
status=LogStatus.FINISH
)
break
else:
yield self.create_log_message(
message=f"Output quality: NEEDS IMPROVEMENT\nIssues: {critique['issues']}",
status=LogStatus.FINISH
)
previous_critique = critique["issues"]
refinement_count += 1
# === FINAL OUTPUT ===
yield self.create_text_message(agent_output)
yield self.create_json_message({"refinement_count": refinement_count})
def _evaluate_and_critique(self, model, query, output):
"""Use LLM as judge to evaluate output quality"""
eval_prompt = f"""Evaluate this agent output for the given query.
Query: {query}
Output: {output}
Respond in JSON:
{{"is_satisfactory": true/false, "issues": "description of issues or empty string"}}
"""
result = self.session.model.llm.invoke(
model_config=LLMModelConfig(**model),
prompt_messages=[UserPromptMessage(content=eval_prompt)],
stream=False,
stop=[]
)
# Parse JSON from result
return parse_evaluation(result.message.content)Manifest.yamlversion: 0.0.1
type: plugin
author: "<your-github-username>"
name: "self_refine_agent"
label:
en_US: "Self-Refine Agent Strategy"
zh_Hans: "自我优化 Agent 策略"
created_at: "2025-06-12T00:00:00.000000000Z"
icon: icon.svg
description:
en_US: "Agent strategy with automatic self-critique and iterative refinement. Based on Self-Refine (NeurIPS 2023)."
zh_Hans: "支持自动自我批判和迭代优化的 Agent 策略。基于 Self-Refine (NeurIPS 2023)。"
tags:
- "agent"
resource:
memory: 1048576
permission:
tool:
enabled: true
model:
enabled: true
llm: true
plugins:
agent_strategies:
- "provider/self_refine.yaml"
meta:
version: 0.0.2
minimum_dify_version: "1.7.0"
arch:
- "amd64"
- "arm64"
runner:
language: "python"
version: "3.12"
entrypoint: "main"Implementation Requirements
Important Notes
To reply, just mention @dosu. Docs are dead. Just use Dosu.
|
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Self Checks
Content
I'm building a plugin to bring Self-RAG concepts into Dify in a practical, stable way.
The Problem
Standard RAG fails on multi-step reasoning when initial context isn't enough.
Agent RAG relies on the LLM's "self-awareness" to trigger retrieval, which causes hallucinations.
API Limits: Standard LLM APIs don't allow mid-stream prompt injection. We need a robust fallback strategy, not a fragile single path.
The Solution: Hybrid Triggers & Fallbacks
Instead of one method, the plugin dynamically checks model confidence with a built-in fallback chain:
Target (V2): Chunk-level Confidence Check. Evaluate Perplexity/Entropy every N tokens via an external GPU service.
Fallback 1 (MVP): Prompt-Guided. LLM outputs a special tag (e.g., [RETRIEVE: query]). Plugin intercepts, fetches data, and continues.
Fallback 2: Keyword Catch. Triggers on words like "specifically" or "explain".
Fallback 3: Standard RAG. One-shot retrieval before generation.
Fallback 4: Pure LLM. If the vector DB is down, LLM answers from memory to prevent errors.
Tech Stack & Architecture
Zero-Intrusion: 100% Dify Plugin System. No core code modifications.
Async & Fast: Uses asyncio + httpx. No Celery (keeps streaming latency low).
State: Redis for caching retrieval results and tracking external GPU health.
Roadmap
Phase 1 (MVP): Build Fallback 1-4. Works perfectly with the current Plugin SDK via multi-turn orchestration.
Phase 2: Integrate the V2 Chunk-level Confidence Check using an external GPU service.
I've also added my skills.md here, and it's in Chinese to better suit the community's requirements
stream-aware-rag.md
dify-innovation.md
Beta Was this translation helpful? Give feedback.
All reactions