첨부용 config yaml 옵션 정리 - #313
Merged
Merged
Conversation
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
📝 WalkthroughWalkthroughThe attachment processor's recursive chunking is refactored to pure character-based splitting via a new ChangesRecursive chunking simplification and config restructuring
Estimated code review effort: 4 (Complex) | ~60 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@genon/preprocessor/facade/attachment_processor.py`:
- Around line 2016-2020: The PPT chunking path in attachment_processor.__call__
is unintentionally enabling page merging because recursive_chunk_size from
_default_kwargs is being used as a fallback, so PPT requests no longer stay at 1
page = 1 chunk by default. Update the chunk_size selection logic in this branch
so PPT behavior remains opt-in: only use an explicit kwargs['chunk_size'] for
merging, and ignore recursive_chunk_size for PPT unless the request explicitly
opts in. Keep the fix localized around _parse_optional_int and the chunk_size
fallback in __call__.
- Around line 1267-1273: The `_char_split_text` logic in
`attachment_processor.py` can pass an invalid overlap into
`RecursiveCharacterTextSplitter` when `chunk_size` is positive but smaller than
the default `chunk_overlap`. Update the `cs`/`co` handling so `co` is clamped to
at most `cs - 1` before constructing the splitter, while still keeping it
non-negative, and apply this just before the `RecursiveCharacterTextSplitter`
call.
In `@genon/preprocessor/facade/gitbook_doc/attachment_processor.md`:
- Around line 187-194: Clarify the `chunk_size` behavior in the
`attachment_processor` documentation by explicitly distinguishing `recursive`
and `hybrid` modes: the current description suggests `0` or negative values
disable size-based splitting universally, but in `hybrid` mode it still falls
back to `_DEFAULT_HYBRID_MAX_TOKENS`. Update the `chunk_size` row and the
warning text near the chunking section to state that `chunk_size<=0` produces a
single chunk only for the `recursive` path, while `hybrid` uses its own default
token limit unless `chunker_type` and the mode-specific chunking settings are
specified.
In `@genon/preprocessor/resource_dev/attachment_processor_config.yaml`:
- Around line 44-49: The `chunking` comments in
`attachment_processor_config.yaml` are misleading because `chunk_size=0` does
not mean “single chunk” for `hybrid` mode; update the documentation near
`chunker_type` and `chunk_size` to reflect that `recursive` uses 0 as no
size-based split while `hybrid` falls back to `_DEFAULT_HYBRID_MAX_TOKENS`, and
mirror the same wording in
`genon/preprocessor/resource/attachment_processor_config.yaml` so both config
files stay consistent.
In `@genon/preprocessor/resource/attachment_processor_config.yaml`:
- Around line 44-49: The chunk_size comment in attachment_processor_config.yaml
is ambiguous because it mixes recursive and hybrid behavior; update the
documentation near chunker_type/chunk_size to describe each mode separately.
Clarify that in recursive mode 0 means no size-based splitting and a single
chunk, while in hybrid mode values <= 0 fall back to the default token limit
instead of disabling splitting. Keep the wording aligned with the existing
chunking section so the behavior is clear at a glance.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 856a7a27-afd2-4040-a4a1-4b9c8142f1bd
📒 Files selected for processing (6)
genon/preprocessor/facade/attachment_processor.pygenon/preprocessor/facade/gitbook_doc/attachment_processor.mdgenon/preprocessor/resource/attachment_processor_config.yamlgenon/preprocessor/resource_dev/attachment_processor_config.yamlgenon/preprocessor/tests/unit/test_attachment_chunk_config_unit.pygenon/preprocessor/tests/unit/test_attachment_chunking.py
Comment on lines
+1267
to
+1273
| cs = int(chunk_size) if chunk_size is not None else 0 | ||
| co = max(int(chunk_overlap), 0) if chunk_overlap is not None else 100 | ||
|
|
||
| if cs > 0: | ||
| raw_chunks = RecursiveCharacterTextSplitter( | ||
| chunk_size=cs, chunk_overlap=co, | ||
| ).split_text(text) |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
For LangChain RecursiveCharacterTextSplitter/TextSplitter, does chunk_overlap greater than chunk_size raise ValueError?
💡 Result:
Yes, in LangChain, initializing a TextSplitter (including RecursiveCharacterTextSplitter) with a chunk_overlap that is greater than the chunk_size will raise a ValueError [1][2][3]. The base TextSplitter class includes a validation check within its init method that explicitly prevents this configuration [1][3]. If chunk_overlap is greater than chunk_size, the following error is raised: "Got a larger chunk overlap (X) than chunk size (Y), should be smaller." [1][3]. This validation applies across various text splitter implementations in LangChain [4]. top_results: [1][2][3][4]
Citations:
- 1: https://github.com/langchain-ai/langchain/blob/9ef2feb6747f5a69d186bd623b569ad722829a5e/libs/langchain/langchain/text_splitter.py
- 2: https://lagnchain.readthedocs.io/en/latest/_modules/langchain/text_splitter.html
- 3: https://github.com/langchain-ai/langchain/blob/c871c04270e18ee17121faf5ef5df6ec1009ecb2/langchain/text_splitter.py
- 4: https://github.com/langchain-ai/langchain/blob/0e97e85a/libs/text-splitters/tests/integration_tests/test_nlp_text_splitters.py
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the relevant function and nearby defaults
sed -n '1248,1285p' genon/preprocessor/facade/attachment_processor.py
# Find where chunk_size / chunk_overlap defaults are defined or passed through
rg -n "chunk_size|chunk_overlap|RecursiveCharacterTextSplitter|TextSplitter" genon/preprocessor/facade/attachment_processor.pyRepository: genonai/doc_parser
Length of output: 7832
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Show the recursive chunker helper and its direct callers
sed -n '1280,1310p' genon/preprocessor/facade/attachment_processor.py
printf '\n---\n'
sed -n '2168,2195p' genon/preprocessor/facade/attachment_processor.py
printf '\n---\n'
sed -n '1988,2058p' genon/preprocessor/facade/attachment_processor.py
# Find every callsite of _char_split_text
printf '\n=== callsites ===\n'
rg -n "_char_split_text\(" genon/preprocessor/facade/attachment_processor.pyRepository: genonai/doc_parser
Length of output: 5398
Clamp chunk_overlap before building the splitter
RecursiveCharacterTextSplitter rejects chunk_overlap > chunk_size, so _char_split_text can raise for any positive chunk_size below the default 100 when overlap isn’t set. Cap the overlap to chunk_size - 1 (or pass a smaller default) before constructing the splitter.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@genon/preprocessor/facade/attachment_processor.py` around lines 1267 - 1273,
The `_char_split_text` logic in `attachment_processor.py` can pass an invalid
overlap into `RecursiveCharacterTextSplitter` when `chunk_size` is positive but
smaller than the default `chunk_overlap`. Update the `cs`/`co` handling so `co`
is clamped to at most `cs - 1` before constructing the splitter, while still
keeping it non-negative, and apply this just before the
`RecursiveCharacterTextSplitter` call.
Comment on lines
+2016
to
+2020
| # chunk_size 우선순위: kwargs['chunk_size'] > chunking.recursive.chunk_size(recursive_chunk_size). | ||
| # 값이 없거나 <=0 이면 1 page = 1 chunk, 있으면 연속 페이지를 그 길이까지 결합. | ||
| chunk_size = _parse_optional_int(kwargs.get('chunk_size'), 'chunk_size') | ||
| if chunk_size is None: | ||
| chunk_size = _parse_optional_int(kwargs.get('generic_chunk_size'), 'generic_chunk_size') | ||
| chunk_size = _parse_optional_int(kwargs.get('recursive_chunk_size'), 'recursive_chunk_size') |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Keep PPT page chunking opt-in.
Because __call__ merges _default_kwargs, recursive_chunk_size is present for PPT requests; with a positive common/recursive chunk size, Line 2020 makes PPT merge pages by default instead of keeping 1 page = 1 chunk.
🐛 Proposed fix
- # chunk_size 우선순위: kwargs['chunk_size'] > chunking.recursive.chunk_size(recursive_chunk_size).
- # 값이 없거나 <=0 이면 1 page = 1 chunk, 있으면 연속 페이지를 그 길이까지 결합.
+ # PPT는 명시적인 kwargs['chunk_size']가 있을 때만 페이지를 병합한다.
+ # 값이 없거나 <=0 이면 1 page = 1 chunk.
chunk_size = _parse_optional_int(kwargs.get('chunk_size'), 'chunk_size')
- if chunk_size is None:
- chunk_size = _parse_optional_int(kwargs.get('recursive_chunk_size'), 'recursive_chunk_size')📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| # chunk_size 우선순위: kwargs['chunk_size'] > chunking.recursive.chunk_size(recursive_chunk_size). | |
| # 값이 없거나 <=0 이면 1 page = 1 chunk, 있으면 연속 페이지를 그 길이까지 결합. | |
| chunk_size = _parse_optional_int(kwargs.get('chunk_size'), 'chunk_size') | |
| if chunk_size is None: | |
| chunk_size = _parse_optional_int(kwargs.get('generic_chunk_size'), 'generic_chunk_size') | |
| chunk_size = _parse_optional_int(kwargs.get('recursive_chunk_size'), 'recursive_chunk_size') | |
| # PPT는 명시적인 kwargs['chunk_size']가 있을 때만 페이지를 병합한다. | |
| # 값이 없거나 <=0 이면 1 page = 1 chunk. | |
| chunk_size = _parse_optional_int(kwargs.get('chunk_size'), 'chunk_size') |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@genon/preprocessor/facade/attachment_processor.py` around lines 2016 - 2020,
The PPT chunking path in attachment_processor.__call__ is unintentionally
enabling page merging because recursive_chunk_size from _default_kwargs is being
used as a fallback, so PPT requests no longer stay at 1 page = 1 chunk by
default. Update the chunk_size selection logic in this branch so PPT behavior
remains opt-in: only use an explicit kwargs['chunk_size'] for merging, and
ignore recursive_chunk_size for PPT unless the request explicitly opts in. Keep
the fix localized around _parse_optional_int and the chunk_size fallback in
__call__.
Comment on lines
187
to
+194
| | 키 | 기본값 | 설명 | | ||
| |----|--------|------| | ||
| | `chunk_size` | `1000` | 일반 텍스트 분할 청크 크기(문자 단위). 0 이하면 1000으로 폴백 | | ||
| | `chunk_overlap` | `100` | 청크 간 오버랩(문자 단위). 음수면 100으로 폴백 | | ||
| | `chunker_type` | `"recursive"` | 청킹 모드. `recursive`(문자수 기반, 전 포맷 공용) 또는 `hybrid`(layout 기반, hwp/hwpx/docx 에만 적용). 그 외 값은 `recursive`로 폴백. 구버전 호환: 없으면 `defaults.chunker_type` 폴백 | | ||
| | `chunk_size` | `1000000` | **공통 청크 크기.** recursive 모드=문자 수, hybrid 모드=토큰 수로 해석. `0`(또는 음수)=크기 기반 분할 안 함 → 전체 문서를 1청크로 둠. recursive/hybrid 는 `chunker_type`으로 택일되므로 값 하나를 활성 모드가 자기 단위로 사용 | | ||
| | `tokenizer_path` | `/models/...all-MiniLM-L6-v2` | 청킹용 토크나이저 로컬 경로(hybrid `huggingface` 모드에서 사용). 경로가 실제 존재하면 그 경로 사용 | | ||
| | `tokenizer_id` | `sentence-transformers/all-MiniLM-L6-v2` | `tokenizer_path` 가 없을 때 폴백할 HF ID (외부 네트워크 차단 환경 대비) | | ||
|
|
||
| #### chunking.recursive (hwp/hwpx/docx + recursive 분기, 기본 청커) | ||
| > `chunk_size`/`chunker_type`/`tokenizer_*` 는 하위 블록이 아닌 `chunking` 공통 레벨에 둡니다. per-block 로 `chunking.recursive.chunk_size` / `chunking.hybrid.chunk_size` 를 지정하면 해당 모드에 한해 공통값을 덮어씁니다(선택). |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== attachment_processor.md around lines 170-260 ==\n'
nl -ba genon/preprocessor/facade/gitbook_doc/attachment_processor.md | sed -n '170,260p'
printf '\n== search for chunk_size/chunker_type resolution and hybrid default ==\n'
rg -n "_DEFAULT_HYBRID_MAX_TOKENS|chunker_type|chunk_size|hybrid" genon -g '!**/*.md'
printf '\n== search for line 246 warning in docs ==\n'
rg -n "chunk_size|chunker_type|tokenizer_|hybrid" genon/preprocessor/facade/gitbook_doc/attachment_processor.mdRepository: genonai/doc_parser
Length of output: 248
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== attachment_processor.md around lines 170-260 ==\n'
awk 'NR>=170 && NR<=260 {printf "%6d %s\n", NR, $0}' genon/preprocessor/facade/gitbook_doc/attachment_processor.md
printf '\n== search for chunk_size/chunker_type resolution and hybrid default ==\n'
rg -n "_DEFAULT_HYBRID_MAX_TOKENS|chunker_type|chunk_size|hybrid" genon -g '!**/*.md'
printf '\n== search for warning text in docs ==\n'
rg -n "0.*전체 문서를 1청크|chunk_size|chunker_type|tokenizer_|hybrid" genon/preprocessor/facade/gitbook_doc/attachment_processor.mdRepository: genonai/doc_parser
Length of output: 50376
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== attachment_processor.py chunking resolution ==\n'
awk 'NR>=1786 && NR<=1820 {printf "%6d %s\n", NR, $0}' genon/preprocessor/facade/attachment_processor.py
printf '\n== attachment_processor.py hybrid branch ==\n'
awk 'NR>=1410 && NR<=1432 {printf "%6d %s\n", NR, $0}' genon/preprocessor/facade/attachment_processor.py
printf '\n== unit tests around chunk_size zero ==\n'
awk 'NR>=58 && NR<=75 {printf "%6d %s\n", NR, $0}' genon/preprocessor/tests/unit/test_attachment_chunking.pyRepository: genonai/doc_parser
Length of output: 4556
Clarify chunk_size=0 behavior for hybrid
chunk_size<=0 only collapses to a single chunk on the recursive path. In hybrid, it falls back to _DEFAULT_HYBRID_MAX_TOKENS, so 0 does not disable chunking there. Qualify the table row and the warning at line 246 with the recursive/hybrid distinction.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@genon/preprocessor/facade/gitbook_doc/attachment_processor.md` around lines
187 - 194, Clarify the `chunk_size` behavior in the `attachment_processor`
documentation by explicitly distinguishing `recursive` and `hybrid` modes: the
current description suggests `0` or negative values disable size-based splitting
universally, but in `hybrid` mode it still falls back to
`_DEFAULT_HYBRID_MAX_TOKENS`. Update the `chunk_size` row and the warning text
near the chunking section to state that `chunk_size<=0` produces a single chunk
only for the `recursive` path, while `hybrid` uses its own default token limit
unless `chunker_type` and the mode-specific chunking settings are specified.
Comment on lines
+44
to
+49
| chunking: | ||
| # 청킹 모드: "recursive"(문자수 기반, 기본) | "hybrid"(layout 구조 기반; hwp/hwpx/docx 에만 적용) | ||
| chunker_type: "recursive" | ||
|
|
||
| # 청크 크기(공통): recursive 모드=문자 수 · hybrid 모드=토큰 수. 0=크기 기반 분할 안 함(전체 1청크) | ||
| chunk_size: 1000000 |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Locate relevant files and symbols
git ls-files | rg 'attachment_processor_config\.yaml|resource/attachment_processor_config\.yaml|chunk_size|hybrid|max_tokens|DEFAULT_HYBRID'
rg -n "_DEFAULT_HYBRID_MAX_TOKENS|chunk_size|chunker_type|hybrid" genon -g'*.py' -g'*.yaml' -g'*.yml' -g'*.md'Repository: genonai/doc_parser
Length of output: 50374
🏁 Script executed:
#!/bin/bash
set -euo pipefail
git ls-files | rg 'attachment_processor_config\.yaml|resource/attachment_processor_config\.yaml|chunk_size|hybrid|max_tokens|DEFAULT_HYBRID'
rg -n "_DEFAULT_HYBRID_MAX_TOKENS|chunk_size|chunker_type|hybrid" genon -g'*.py' -g'*.yaml' -g'*.yml' -g'*.md'Repository: genonai/doc_parser
Length of output: 50374
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Show the exact shipped comment and the resolver logic around hybrid chunk_size.
sed -n '44,70p' genon/preprocessor/resource_dev/attachment_processor_config.yaml
printf '\n---\n'
sed -n '1786,1820p' genon/preprocessor/facade/attachment_processor.py
printf '\n---\n'
sed -n '44,70p' genon/preprocessor/resource/attachment_processor_config.yamlRepository: genonai/doc_parser
Length of output: 3885
Clarify chunk_size=0 for hybrid mode 0 means “single chunk” only for recursive; hybrid falls back to _DEFAULT_HYBRID_MAX_TOKENS, so split this comment to match the actual behavior here and in genon/preprocessor/resource/attachment_processor_config.yaml.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@genon/preprocessor/resource_dev/attachment_processor_config.yaml` around
lines 44 - 49, The `chunking` comments in `attachment_processor_config.yaml` are
misleading because `chunk_size=0` does not mean “single chunk” for `hybrid`
mode; update the documentation near `chunker_type` and `chunk_size` to reflect
that `recursive` uses 0 as no size-based split while `hybrid` falls back to
`_DEFAULT_HYBRID_MAX_TOKENS`, and mirror the same wording in
`genon/preprocessor/resource/attachment_processor_config.yaml` so both config
files stay consistent.
Comment on lines
+44
to
+49
| chunking: | ||
| # 청킹 모드: "recursive"(문자수 기반, 기본) | "hybrid"(layout 구조 기반; hwp/hwpx/docx 에만 적용) | ||
| chunker_type: "recursive" | ||
|
|
||
| # 청크 크기(공통): recursive 모드=문자 수 · hybrid 모드=토큰 수. 0=크기 기반 분할 안 함(전체 1청크) | ||
| chunk_size: 1000000 |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Locate the hybrid fallback constant and its exact resolution logic
rg -n '_DEFAULT_HYBRID_MAX_TOKENS' genon/preprocessor/facade/attachment_processor.pyRepository: genonai/doc_parser
Length of output: 539
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the config resolution logic around chunk_size and chunker_type.
sed -n '1000,1095p' genon/preprocessor/facade/attachment_processor.py
printf '\n---\n'
sed -n '1395,1610p' genon/preprocessor/facade/attachment_processor.py
printf '\n---\n'
sed -n '1790,1845p' genon/preprocessor/facade/attachment_processor.py
# Inspect the config file context around the commented lines.
printf '\n=== attachment_processor_config.yaml ===\n'
sed -n '35,60p' genon/preprocessor/resource/attachment_processor_config.yaml
# Find the troubleshooting note about the 60K-token safety net / embedding limits.
printf '\n=== troubleshooting references ===\n'
rg -n "60K|60,000|embedding input limit|token limit|safety-net|safety net|chunk_size: 1000000" genon -SRepository: genonai/doc_parser
Length of output: 19113
genon/preprocessor/resource/attachment_processor_config.yaml:44-49 — chunk_size 설명을 hybrid/recursive로 분리하세요.
chunk_size: 0은 recursive에서만 전체 1청크이고, hybrid는 <= 0이면 _DEFAULT_HYBRID_MAX_TOKENS로 대체됩니다. 지금 문구는 chunker_type: hybrid에서도 “분할 안 함”으로 읽혀 오해를 부릅니다.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@genon/preprocessor/resource/attachment_processor_config.yaml` around lines 44
- 49, The chunk_size comment in attachment_processor_config.yaml is ambiguous
because it mixes recursive and hybrid behavior; update the documentation near
chunker_type/chunk_size to describe each mode separately. Clarify that in
recursive mode 0 means no size-based splitting and a single chunk, while in
hybrid mode values <= 0 fall back to the default token limit instead of
disabling splitting. Keep the wording aligned with the existing chunking section
so the behavior is clear at a glance.
HeechanKim-Genon
approved these changes
Jul 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
refactor(#312): 첨부용 전처리기 청킹/설정(config yaml) 정리
개요
첨부용 전처리기(
attachment_processor.py)의 청킹 방식이generic/recursive/hybrid3갈래로config·코드에 중복·분산돼 있었다. 실제로
generic·recursive는 둘 다 문자수 기반(
RecursiveCharacterTextSplitter)이라, 이를 문자수 기반 단일 모드로 통합하고 설정 구조를정리했다. 옵션의 위치·의미를 일관되게 재편하고, 매뉴얼과 단위테스트를 함께 갱신했다.
주요 변경
1) generic + recursive → 문자수 단일 모드로 통합
_char_split_text(text, chunk_size, chunk_overlap)신설 — pdf/txt/md/img(평문)와hwp/hwpx/docx(markdown export)가 동일한 문자 분할 로직을 공유.
chunk_size <= 0이면 분할하지 않고 문서 전체를 1청크로 둔다(PPT는 예외로1 page = 1 chunk유지).chunking.generic블록 및 관련generic_*파싱 제거.2) 청킹 옵션을
chunking공통 레벨로 승격chunker_type:defaults→chunking.chunker_type로 이동(recursive기본 |hybrid).chunk_size: 공통chunking.chunk_size로 승격. recursive=문자 수, hybrid=토큰 수로 해석(두 모드는
chunker_type으로 택일 → 값 하나를 활성 모드가 자기 단위로 사용).chunking.recursive={ chunk_overlap },chunking.hybrid={ tokenizer_type, merge_peers }로 축소.3) 토크나이저 설정 단일화
chunking.recursive.tokenizer_id·chunking.hybrid.tokenizer_id제거.chunking.tokenizer_path/tokenizer_id하나로 일원화(hybridhuggingface모드용).4) HWP 전용 옵션을
formats.hwp로 이동use_hwp_sdk·dump_sdk_output·save_images를defaults→formats.hwp로 이동(기존
formats.ppt패턴과 일관).use_pdf_sdk는 전 변환 경로 공용이라defaults유지.defaults에는 전역 옵션(log_level,use_pdf_sdk)만 남음.설정 변화 (before → after)
하위 호환
chunking.chunker_type없으면defaults.chunker_type,formats.hwp.*없으면defaults.*,chunking.chunk_size없으면chunking.recursive|generic.chunk_size순으로 폴백해 읽는다.
HwpProcessor/DocxProcessor내부 인터페이스는 변경 없음.legacy/BOK_첨부용.py는 자체 상수를 보유해 영향 없음.테스트
pytest tests/unit/test_attachment_chunking.py→ 19 passed(config 구조 검증 +
_char_split_text로직 검증).test_attachment_chunk_config_unit.py포함 — 공통 chunk_size 반영,chunker_type/HWP 옵션 이동·폴백, token cap 키 부재 검증.
변경 파일
facade/attachment_processor.pyresource/attachment_processor_config.yaml,resource_dev/attachment_processor_config.yamlfacade/gitbook_doc/attachment_processor.mdtests/unit/test_attachment_chunking.py,tests/unit/test_attachment_chunk_config_unit.pySummary by CodeRabbit
Bug Fixes
New Features
Documentation
Tests