Chopper: a small open model that cuts only the needed sentences out of retrieved text

[wiiiktor]

Hi! Sharing a small open-source tool that may be useful here.

                 .-~-.-~-.-~-.
                (  ~   ~   ~  )
                 (___________)
                 |___________|
                 |  ^     ^  |
                 |  o     o  |
                 |     >     |
                 | ,~~~^~~~, |
                  \  '---'  /
              _____\_______/_____         ________________________________
             /    C H O P P E R  \==[####]|                                ``--.._
            50 ms, consumer GPU           |__________________________________..--'
  .-~~~~~~~~~~-.     ┌─────────────────────────────────────────────┐     ┌───────┐     ┌─────────┐
 (   DATABASE   )    │ 20 retrieved snippets · 2,620 tokens        │     │ chops │     │   LLM   │
 |~-.________.-~|    │ ░░░░░░░░░░░█░░░░░░░░░░░░░░░░░░░░░░█░░░░░░░░ │     │       │     │         │
 |  ▓▓▓▓▓▓▓▓▓▓  | ──>│ ░░░░░░░░░░░░░░░░░░░░░█░░░░░░░░░░░░░░░░░░░░░ │ ──> │  ███  │ ──> │  fast,  │
 |~-.________.-~|    │ ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░█░░░░░░░░░░░ │     │  ███  │     │ correct │
 |  ▓▓▓▓▓▓▓▓▓▓  |    │ ░░░░░░█░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ │     │       │     │ answer  │
  `-.________.-'     └─────────────────────────────────────────────┘     └───────┘     └─────────┘
  or a long chat                   100% of the text                    2-5%: only the
                                                                       useful bits,
                                                                       word for word

https://github.com/wiiiktor/chopper_demo

A new, ultra-fast method (50 ms on a consumer-grade GPU): a per-token decision [useful? yes/no] for pages of text. Created in November 2026 in Poland, EU. A tiny edge model offering 82% of the quality of Claude Opus 5.5.

70-second video: https://youtu.be/KMvmdXCEOJU

A retriever brings back 20 text snippets, but usually only one or two sentences in them answer the question. The chopper is not an LLM summarising them: a small encoder (0.3B parameters) reads the question and all 20 snippets together in one pass and decides for every token whether it is useful. No text generation, so it takes about 50 ms per question (~2,400 tokens), 36 ms with the 0.14B version.

What it keeps are exact quotes ("chops"), never rewritten, so every chop can be traced back to its source. If the answer is not there at all, it says so, so you can search again.

Test made with Qwen3-14B answering 400 questions, one retrieval each:

• The LLM reads about 10x less text: 228 tokens instead of 2,620.
• Simple questions that need ONE fact: same accuracy, 83% vs 84%.
• Very complex questions that chain FOUR facts: 29% correct from the chops vs 12% from all snippets.

pip install chopper-rag works with LangChain, LlamaIndex, Haystack or plain Python. Models (MIT): https://huggingface.co/wiiiktor/chopper_mmbert

Feedback is very welcome, especially where it fails.