Retrieval score differs between UI upload (.docx) and API upload (.txt) despite identical chunk content and embedding model

[d5devgodai-blip]

Self Checks

• I have read the Contributing Guide and Language Policy.
• This is only for bug report, if you would like to ask a question, please head to Discussions.
• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report, otherwise it will be closed.
• 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：）
• Please do not modify this template :) and fill in all the required fields.

Dify version

1.14.2

Cloud or Self Hosted

Cloud

Steps to reproduce

Environment

• Dify version: Cloud
• Embedding model: text-embedding-3-large (same for both)
• API endpoint: POST /v1/datasets/{dataset_id}/document/create-by-file

Steps to Reproduce

1. Extract text from a Japanese PDF using PyMuPDF
2. Fix spacing artifacts with GPT-4.1
3. Method A (UI) — paste cleaned text into .docx, upload via Dify UI
4. Method B (API) — upload same text as .txt via create-by-file with identical process_rule

data = {
    "name": "manual",
    "indexing_technique": "high_quality",
    "doc_form": "hierarchical_model",
    "process_rule": {
        "mode": "hierarchical",
        "rules": {
            "parent_mode": "paragraph",
            "segmentation":          {"separator": "###", "max_tokens": 4000, "chunk_overlap": 0},
            "subchunk_segmentation": {"separator": "$$$", "max_tokens": 400,  "chunk_overlap": 50},
            "pre_processing_rules": [
                {"id": "remove_extra_spaces", "enabled": True},
                {"id": "remove_urls_emails",  "enabled": False}
            ]
        }
    }
}

5. Query both datasets with the same question
6. Compare retrieval scores

✔️ Expected Behavior

Same chunk content + same embedding model + same query = same retrieval score, regardless of upload method or file format.

❌ Actual Behavior

Method	Format	Score
Dify UI	.docx	0.99
API create-by-file	.txt	0.69

Chunk content is confirmed identical in the Dify chunk viewer. Only the upload method differs.

Suspected Cause

The embedding vector stored for each chunk differs between the two paths. Likely causes:

• remove_extra_spaces pre-processing behaves differently for .txt vs .docx before the embedding call — the stored vector differs from the text visible in the UI.
• Different internal text normalization pipelines for the two file types produce different vectors for visually identical text.
• Hierarchical mode may not be fully applied via API (related: The creat_fy_file API silently fails to process the hierarchical model (parent-child segmentation) #22270, API: Parent-child chunking mode is not applied when creating documents using /create-by-text and /create-by-file endpoints #22118, fix(api): reuse hierarchical process rule when API upload uses mode=automatic #35860), causing a fallback to general chunking with different embedding behavior.

Related Issues

#22270, #22118 — hierarchical chunking broken via API
#35860 — partial fix for automatic modeers.

[dosubot]

This is a documented issue with two contributing factors:

1. Format-Specific Text Extraction

Even though your chunk content appears identical in the viewer, .docx and .txt files go through different extraction pipelines before preprocessing [1] [2]. The WordExtractor (.docx) preserves structural information (paragraphs, hyperlinks, tables) while the TextExtractor (.txt) reads the file as linear text with encoding detection. After extraction, both pass through the same CleanProcessor.clean() and remove_extra_spaces logic [3], so the preprocessing rules themselves are not the culprit. However, subtle differences introduced during extraction (e.g., whitespace handling, newline patterns) could produce slightly different text fed to the embedding model — differences that are normalized away in the chunk viewer display but still present in the actual embedded content.

2. Hierarchical Mode API Bug (Most Likely Cause)

There's a documented bug (#35858) where hierarchical chunking via the API can silently fail or behave differently [4]. The ParentChildIndexProcessor lacks automatic-mode fallback logic that exists in the ParagraphIndexProcessor. If anything in your API request causes a mismatch in how the hierarchical rules are applied, the parent-child chunk structure may differ from what the UI produces, directly impacting retrieval scoring.

Recommended Workaround

In your API call, ensure you are explicitly setting process_rule.mode to "hierarchical" (not "automatic" or "custom"), and that all sub-fields (parent_mode, segmentation, subchunk_segmentation) exactly match what the UI sends [5]. You can verify what the UI actually sends by checking the dataset's latest process rule. If you want to guarantee identical behavior, you could also try uploading .docx via the API instead of .txt to eliminate the extraction-path variable.

Your suspicion about different normalization pipelines producing different vectors for visually identical text is plausible — the chunk viewer may display post-cleaned text while the embedding was computed on a slightly different pre-cleaned version. This would explain the 0.99 vs 0.69 score gap.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.

[d5devgodai-blip]

I applied both recommended workarounds:

• Switched upload format from .txt to .docx to route through WordExtractor instead of TextExtractor
• Set remove_extra_spaces: true to match the UI's CleanProcessor path
• Confirmed process_rule.mode: "hierarchical" is explicitly set with all sub-fields (parent_mode, segmentation, subchunk_segmentation)

API payload used:

"process_rule": {
    "mode": "hierarchical",
    "rules": {
        "parent_mode": "paragraph",
        "segmentation": {
            "separator": "###",
            "max_tokens": 4000,
            "chunk_overlap": 0
        },
        "subchunk_segmentation": {
            "separator": "$$$",
            "max_tokens": 400,
            "chunk_overlap": 50
        },
        "pre_processing_rules": [
            {"id": "remove_extra_spaces", "enabled": true},
            {"id": "remove_urls_emails", "enabled": false}
        ]
    }
}

Result: score is still significantly lower than the UI upload.

Method	Format	Score
Dify UI	.docx	0.99
API create-by-file	.docx (same content)	~0.54

The screenshots below show the chunk content from both methods for the same query.

Dify UI upload — Score 0.99 (proper parent-child structure, clean chunk content)

API create-by-file upload — Score 0.54 (same content, same file format)

Same file format, same embedding model, same content — only the upload path differs.

This points to factor 2 (hierarchical API bug #35858) as the primary cause. The chunk content visible in the viewer is identical, but the embedding vectors stored for API-uploaded documents are producing lower cosine similarity scores than UI-uploaded ones.

---

Questions:

1. Is there a way to confirm that the hierarchical ParentChildIndexProcessor is actually being invoked for API uploads? Is there a log or response field that confirms hierarchical processing ran successfully?

2. You mentioned: "You can verify what the UI actually sends by checking the dataset's latest process rule." — Which API endpoint returns the stored process_rule for an existing dataset? (GET /datasets/{id} does not seem to include it.)

3. Is there a known fix or workaround for API upload completes with 0 segments, while GUI upload indexes correctly (same dataset, self-hosted 1.13.3) #35858 beyond the automatic-mode fallback in fix(api): reuse hierarchical process rule when API upload uses mode=automatic #35860? Specifically for explicit mode: "hierarchical" payloads via create-by-file?

[dosubot]

Thanks for the detailed follow-up. Here's what I found for your three questions:

---

1. Confirming ParentChildIndexProcessor is invoked

Unfortunately, there are no explicit logs emitted when this processor runs [1]. However, you can confirm it ran by checking the indexing status response for your document:

• doc_form should be "hierarchical_model" [2]
• total_segments should be > 0 (this is the count of parent segments created)
• If total_segments == 0, the processor's transform() silently returned an empty list — meaning hierarchical chunking failed

You can poll GET /v1/datasets/{dataset_id}/documents/{batch}/indexing-status after upload to check these fields.

---

2. Retrieving the stored process_rule

The process rule is exposed through the document-level endpoint, not the dataset endpoint [3]:

GET /v1/datasets/{dataset_id}/documents/{document_id}

The response includes both dataset_process_rule (the latest rule for the dataset) and document_process_rule (the rule used for that specific document). By default both are returned; you can control this with a metadata query parameter (all, without, only).

---

3. Fix/workaround for explicit mode: "hierarchical" still producing lower scores

Since you're already explicitly setting mode: "hierarchical" with all sub-fields and still seeing lower scores (0.54 vs 0.99), the issue likely isn't the empty-segments bug. A few things to investigate:

• Verify segment counts: Use the indexing-status endpoint above to confirm total_segments > 0 and completed_segments == total_segments. If segments are being created, the hierarchical processor is running.

• Summary score tracking bug: A fix was merged that introduced a summary_score_map to track the highest score for segments retrieved via summaries. In older versions, lower-scoring summaries could override relevant segment scores, reducing retrieval accuracy. Check if your Dify Cloud version includes this fix.

• Architecture insight: In hierarchical mode, only child chunks are embedded in the vector store for retrieval [4]. Parent segments are stored in the database for context but are not searchable. If the child chunk segmentation differs between your UI and API uploads (even subtly), that would directly impact the cosine similarity score.

• Compare the actual stored rules: Use the document endpoint above to retrieve the document_process_rule for both your UI-uploaded and API-uploaded documents. Compare them byte-for-byte — any difference in how the rules were stored would explain different chunking behavior.

Given that chunk content appears identical in the viewer but scores differ significantly, the most likely remaining explanation is that the child chunk boundaries differ between the two uploads, resulting in different embedding vectors. The chunk viewer may be showing the parent segment text while the actual retrieval operates on differently-segmented child chunks underneath.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.

[crazywoola]

Hi @d5devgodai-blip, thanks for the detailed report.

The .docx UI path and .txt API path can still differ before embedding because parser and normalization behavior may not be identical, even if the visible chunk text later looks the same. To investigate this on Cloud, we need enough concrete identifiers to compare the actual stored document/chunk/indexing state.

Please provide the dataset IDs, document IDs, one example query, and the exact chunk IDs or chunk text involved for both uploads. If possible, also include the API response for document creation and confirm whether both documents use the same retrieval settings, process rule, embedding provider/model, and rerank settings.