Before reporting review percentages from RAGFlow: separate retrieval evidence from the review denominator

[LMSlay]

Disclosure: I maintain TubeText. This note was prepared with AI assistance. It shares a manual review-analysis checklist, not a tested RAGFlow integration, benchmark or configuration recipe.

If you already have a permitted app-review export in a RAGFlow dataset, asking for examples of a theme and measuring its frequency are different tasks.

The RAGFlow 1.0.0-rc2 Retrieval Testing documentation describes a query-dependent result set: metadata conditions restrict the scope and Top caps the candidate results. It also says test settings are not automatically copied to Chat or Agent. The Chat Configuration documentation separately describes Top N and citations that let a reader inspect supporting chunks and their sources.

Here is the distinction I would preserve in a review brief:

• Retrieval check: Can I find relevant passages, and do those passages support the proposed theme? Follow the cited source back to the original review row. A source document, a retrieved chunk and a review are not automatically the same counting unit.
• Sample accounting: Which review rows were included in the analysis, after the chosen app, time, language and exclusion rules? Keep that list in a separate working sheet tied to the unchanged export. Do not use the number of returned chunks or citations as the review denominator.
• Theme calculation: Apply a consistent codebook to that defined review sample. Count a review at most once per theme; do not infer unique authors or the share of all users from those counts.

For a clearly fictional example, suppose an analyst has retained 80 review rows. A query returns 10 chunks about login, and several may refer to the same review. That does not establish “10 of 80 reviews mention login,” or “12.5% of users have a login problem.” The analyst must map the evidence back to review rows and code the defined sample before calculating a share. Increasing Top or adding more citations is not, by itself, proof of complete coverage.

I made this manual Google Play review-theme comparison guide for the sample-accounting side: scope, exclusions, a small codebook and an explicitly fictional worked example. It can be used with an existing permitted export; using a TubeText Actor is not required. TubeText's collection product is paid, but this note does not require collecting again or running any software.

This checklist is about the claim in the final brief. It does not assert that RAGFlow cannot perform a broader analysis, that one review always produces one chunk, or that citations alone validate a percentage. I have not run this workflow in RAGFlow.

[Yannnnnnny]

Thanks for sharing these insights and referencing our documentation! We appreciate your thoughtful explanation of the distinction between retrieval results and statistical sample counts.