Repository navigation
MNN Complex Task Skill Decomposition Experiment #4352
wangzhaode
announced in
Technical Blog
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Background
In the era of large models, new open-source models are released every day. As developers of the MNN inference framework, how can we provide low-level support at the fastest pace? This is typically a hardcore engineering task spanning Python model decomposition, C++ operator implementation, and multimodal encoding/decoding. I tried handing this task entirely to Claude Opus 4.6, currently the strongest code model in the industry, and it failed. But when I changed the structure of how I "communicated" with AI, the results improved significantly...
Supporting a new LLM model in the MNN inference framework is a complex, time-consuming, and tedious engineering task. The entire pipeline involves 7 layers of knowledge, each indispensable:
These knowledge areas span both Python and C++ sides, scattered across 10+ key files. Any mistake in any link leads to final inference failure—and errors often manifest as "model talking nonsense" or "C++ crashes," making it difficult to locate the root cause from surface symptoms. I've tried multiple times directly asking AI to complete this task, with unsatisfactory results—AI either goes astray during the architecture analysis phase, or cannot effectively debug after writing code.
Turning Point: Opus 4.6 + Task Decomposition
In the process of supporting the Qwen3.5 model with Claude Opus 4.6, I discovered a key pattern: when I decompose the task well and define testing checkpoints, Opus 4.6 basically performs very well. Its powerful code comprehension ability is sufficient to read complex HF model implementations; the problem is that it lacks a structured execution framework.
This led me to an idea: Can we decompose this complex task through Skills, while defining testing standards for each step (TDD-driven), so that an AI Agent can independently complete the entire workflow?
Skill Design
The "Skill" discussed here is not an external API calling interface, but rather an instruction system based on the AI Native Repository concept. We establish a main instruction file (e.g.,
CLAUDE.md) in the project root directory, and create structured Markdown step guides under theskills/directory, so that the Agent (such as Claude Code) can read and execute them on its own while working.Core Concepts
Step Structure
Each step file contains: objectives, prerequisites, specific operations, testing standards, common errors and fixes.
Experiment Design
To validate the effectiveness of Skills, I designed a controlled experiment:
support-new-llmSkill instruction chainGLM-OCR is a good test subject because it contains multiple technical challenges:
Input Prompt
Both experimental groups use the exact same startup method and initial prompt:
Startup Command (including initial prompt):
To authorize the Agent for fully autonomous execution and avoid permission confirmation interruptions midway, we use the following command line to start with one click:
claude --dangerously-skip-permissions "请帮我支持 GLM-OCR 模型(/home/yanxing/data/models/GLM-OCR),支持模型导出和推理。"After pressing Enter, humans completely let go and provide no further guidance on architecture or business logic, until the model determines on its own that the task is complete or falls into a loop of errors.
Experiment Results
Code Output Comparison
AutoModelForImageTextToText(generic class)GlmOcrForConditionalGeneration(precise class name)Key Issues Discovered and Handled
object.__setattr__)Final Status
To intuitively demonstrate the gap, let's directly look at the real-machine test output from MNN C++:
❌ Without Skill Version Test Results (Model Crash and Numerical Hallucination):
Although the export phase appeared normal, when starting inference on the C++ side, due to ignoring underlying structural errors, the engine crashes directly, throwing a core dump:
Moreover, even if a human intervenes to forcibly fix the template code (Jinja template), without going through Hook alignment testing, the deep RoPE alignment error in the model still persists. When testing again, the model falls into a terrifying "text hallucination" infinite loop:
✅ With Skill Version Test Results (Perfect OCR and High-Performance Inference):
After Skill's Hook alignment and C++ Jinja/RoPE fixes, multimodal inference runs completely. We input an original photo of a supermarket receipt named
xp.jpg, and the model not only accurately read various messy product names, unit prices, and serial numbers:Note: Although the
/costcommand shows a higher theoretical billing amount ($6 vs $19), the actual final settlement charged by the backend was around 90 RMB and 140 RMB respectively. While the large amount of long-context validations triggered by the TDD verification loop inflated the billed token usage (22.7m cached tokens), this extra 50 RMB (less than the cost of two Starbucks coffees) solidly saved core developers hundreds to thousands of times that amount in hourly wages that would have been wasted diving into tensors and C++ crash logs in futile debugging.Analysis
The Greatest Value of Skills: Hook Alignment Testing Catches Hidden Bugs
The version without Skill completely failed to notice that GLM-OCR uses interleaved RoPE rotation (adjacent element pairing (0,1),(2,3)...), rather than the standard half-half rotation (front-back half pairing (0,64),(1,65)...). After seeing garbage text as inference output, it simply attributed it to "OCR models are not good at pure text" and skipped validation.
The version with Skill, during Step 3's Hook alignment testing, immediately discovered a deviation of 0.014 in the layer0 output. This is like building a skyscraper where the first layer of rebar is off by 1 centimeter. Intercepted at this checkpoint, the AI did not continue blindly building, but immediately traced back to the implementation difference in the
rotate_halffunction, accurately discovering that GLM-OCR's Rotary Positional Encoding (RoPE) is interleaved, and ultimately fixed it successfully.Without this TDD checkpoint, this tiny floating-point deviation would propagate all the way to the final output, manifesting as the model frantically spitting out random characters. For a human developer to trace back through error results to debug low-level tensors would be a nightmare.
Fix Quality: Root Cause vs. Patch
The comparison of how the embed_ dtype cascading issue was fixed is very illustrative:
Without Skill: In
model.py, first saveembed_dtype, construct Vision, then restoremodel.embed.embed.to(embed_dtype). This is a "knows where the problem is but not why" patch-style fix, requiring a restoration operation every time the model loads.With Skill: In the
vision.pybase class, useobject.__setattr__(self, 'embed_', base.embed)to prevent PyTorch from registering embed_ as a child Module, fundamentally avoiding the.float()cascade. This is a one-time-permanent fix that requires no additional operations from upper-level code.The version with Skill produced higher quality fixes—even better than what I had considered when manually supporting this before. I had also handled this issue with a patch-style approach.
Limitations of Skills
The experiment also revealed several blind spots in the Skill:
rotate_half", the location speed would have been faster.Based on these findings, I iteratively updated the Skill: added a
common-pitfalls.mddocument, and supplemented RoPE variant detection, Decoder residual pattern quick reference, Jinja limitations, C++ multimodal testing guidelines in step1/step2/step4/step5.Additional Comparison: Base Model Capability and Long Context Trap
To verify whether a complete Skill framework can compensate for deficiencies in base model capabilities, I temporarily added another control group: using the GLM-5 model with all Skill constraints enabled to re-execute the task. This run failed after 5.5 hours, outputting the following execution log:
(Note: The $189.97 here is the theoretical price converted from token consumption by the terminal tool. Since this test used a monthly subscription service, this fee was not actually incurred, but the data objectively reflects the computational resources consumed by the entire process.)
This failed attempt revealed several hidden thresholds that need focused attention in actual Agent engineering deployment:
Through this comparison, we can see that for hardcore engineering tasks, Skills and TDD can serve as guardrails to raise the floor, but whether an Agent system can ultimately deliver usable results still depends on the underlying model's own code reasoning capability and stable understanding of long contexts.
Self-Evolution of Skills: Every Invocation Is an Iteration
An interesting discovery: AI executing the Skill process is itself the best reviewer of the Skill.
After completing this experiment, I had the AI review the entire execution process, reflecting on where the Skill helped it, where it caused confusion, and where improvements were needed. The AI gave very specific feedback:
rotate_half"Based on this feedback, I had the AI directly modify the Skill files:
common-pitfalls.md(new file)step1-analyze.mdstep2-mapping.mdstep4-export.mdstep5-multimodal.mdThis forms a positive feedback loop:
Every AI invocation both "uses" and "improves" the Skill. Traditional process documentation requires manual maintenance and easily becomes outdated; in this model, the Skill is automatically iterated by its user (AI). The AI is both the executor, the quality reviewer, and the document maintainer.
More importantly, these improvements are not generic—they come from real issues the AI encountered during actual execution, with concrete error symptoms, root cause analysis, and fix plans. For example, each entry in
common-pitfalls.mdincludes: problem description, how to identify it, and code-level fix. This is much more precise than documentation written from human memory.As more models are supported, this Skill will accumulate more and more practical experience, covering more and more edge cases. It can be anticipated:
This is essentially an "AI-native" knowledge management approach: rather than humans writing documentation for AI to read, AI accumulates experience through practice and feeds it back to the next AI invocation. The human role shifts from "writing process documentation" to "reviewing AI's improvement suggestions"—a much more efficient collaboration model.
The form of code repositories is undergoing irreversible change. Future excellent open-source projects, in addition to providing
README.mdandCONTRIBUTING.mdfor developers, aSKILLinstruction network facing AI Agents with even self-iterating capabilities, will perhaps become standard low-level infrastructure.Conclusion
Reviewing this engineering practice around GLM-OCR support, we arrived at a clear comparison:
Core Conclusion: For complex tasks like low-level architecture parsing, the base model's own code reasoning ability often has bottlenecks. Task decomposition and Skill-ification based on TDD constraints is a necessary means to ensure AI output reaches engineering-grade usability. Models lacking Skill guardrails are highly prone to missing tiny numerical deviations (such as RoPE heterogeneity) in long chains; while models with built-in Skill constraints can leverage structured validation checkpoints to achieve precise bug interception and root-cause fixes.
This leads to a more fundamental takeaway: When AI handles complex tasks, what it often lacks is not stronger model capability, but reasonable task structuring. Skills essentially encode our developers' "implicit experience" and "debugging intuition" into instruction chains that AI can understand, forcing it to do the right checks at the right time. For other hardcore development work—such as custom operator optimization, inference migration to new hardware—this Skill-guided approach is likewise a universal solution.
Related Resources
support-new-llmandarm-cpu-optimizehave been opened to the community and AI Agents in a highly structured manner.All reactions