Markdown files ignore custom separator ---\n for document chunking, while TXT files respect it

[yang2233]

Self Checks

• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:)
• Please do not modify this template :) and fill in all the required fields.

Content

Issue Description

I'm experiencing inconsistent behavior with document chunking in Dify. The same document content, using the same custom separator ---\n, produces different chunking results depending on the file format.

• TXT format: Works correctly — splits into 3 chunks as expected (headers and two vulnerability fix sections)

• MD format: Does NOT respect the separator — splits into 10+ small chunks

Document Structure Example

# MySQL Database Vulnerability Fix Guide

This document covers vulnerability fixes for MySQL/MariaDB servers.

---

## Oracle Database (MySQL) Security (OpenSSL) Vulnerability

### x1

### x2

---

## MySQL Server DoS Vulnerability (CVE-2024-21177)

### x1

### x2

Expected Behavior

Both MD and TXT files should chunk identically when using the same custom separator ---\n.

Actual Behavior

• TXT file: correctly chunks into 3 sections based on ---\n

• MD file: ignores ---\n and creates many smaller chunks (10+)

Question
Is there a way to make Markdown files respect the custom separator ---\n the same way TXT files do?

Why This Matters
I want to avoid maintaining two separate copies of the same document locally (one .md and one .txt), which leads to synchronization issues and duplicated effort.

Environment
Dify version: 1.13.2

Deployment method: Docker

Additional Context
The separator --- is also standard Markdown horizontal rule syntax. It seems Dify might be interpreting it differently in Markdown parsing vs. plain text parsing.

[dosubot[bot]]

Hi @yang2233! I'm Dosu and I’m helping the dify team.

This is a confirmed behavior caused by how Dify's extraction pipeline handles .md files differently from .txt files.

What's happening: When you upload a .md file, Dify routes it to MarkdownExtractor [1], which pre-segments the content by headers (any #, ##, ###, etc.) before your custom separator logic ever runs [2]. Each header-delimited section becomes a separate Document object, and then the custom separator ---\n is applied to each of those smaller documents individually [3]. This is why you end up with 10+ chunks instead of 3.

In contrast, .txt files go through TextExtractor [4], which returns the entire file as a single Document object [5] — so your custom separator works on the full content as expected.

The file routing is determined purely by file extension in ExtractProcessor [6].

Workaround: Rename your .md files to .txt before uploading. This bypasses MarkdownExtractor and routes the file to TextExtractor, allowing your custom separator to work correctly. Multiple users have confirmed this works [7] [8].

This is admittedly not ideal since it forces you to change file formats. The underlying issue is that markdown header-based pre-segmentation runs unconditionally before custom chunking — there's currently no option to disable it when using custom separators.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.

[yang2233]

Thank you for your detailed explanation. I'll go with the file format workaround for now. @dosu

[dosubot[bot]]

You're welcome! The .txt rename workaround should work reliably for your use case. If you run into any other chunking issues, feel free to ask. 👍

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.