chunk size 1000 일 때 청크결과 안나오는 오류 수정 - #337
Conversation
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
|
Warning Review limit reached
Next review available in: 44 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
📝 WalkthroughWalkthroughThe chunking processors now bound recursive splitter overlap values, use recursive-specific parse-format configuration, and avoid creating ChangesChunking Safety
Possibly related issues
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
genon/preprocessor/facade/legacy/BOK_첨부용.py (1)
1680-1685: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winResolve legacy defaults using the recursive option names first.
This path still falls back to
generic_chunk_sizeandgeneric_chunk_overlapon Lines 1678-1681, so therecursive_*values supplied by the current configuration/defaults are ignored wheneverchunk_sizeorchunk_overlapis absent. Userecursive_*first, retaininggeneric_*only as a backward-compatible fallback.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@genon/preprocessor/facade/legacy/BOK_첨부용.py` around lines 1680 - 1685, Update the legacy chunk-size and chunk-overlap default resolution before the clamping logic: when either value is absent, read the corresponding recursive_chunk_size or recursive_chunk_overlap option first, then fall back to generic_chunk_size or generic_chunk_overlap for compatibility. Preserve the existing integer conversion and bounds handling after resolution.
🧹 Nitpick comments (1)
genon/preprocessor/facade/chunking_processor.py (1)
975-976: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winUse
itertools.pairwise()for adjacent cut ranges.Ruff reports B905/RUF007 on these successive-pair iterations. Replace each
zip(cuts[:-1], cuts[1:])withpairwise(cuts)after importingpairwisefromitertools.
genon/preprocessor/facade/chunking_processor.py#L975-L976: usepairwise(cuts).genon/preprocessor/facade/convert_processor.py#L1287-L1288: usepairwise(cuts).genon/preprocessor/facade/intelligent_processor.py#L1292-L1293: usepairwise(cuts).genon/preprocessor/facade/legacy/BOK_적재용_규정.py#L775-L776: usepairwise(cuts).genon/preprocessor/facade/legacy/BOK_적재용_내부.py#L716-L717: usepairwise(cuts).genon/preprocessor/facade/legacy/BOK_적재용_외부.py#L771-L772: usepairwise(cuts).🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@genon/preprocessor/facade/chunking_processor.py` around lines 975 - 976, Replace each adjacent-cut iteration with itertools.pairwise(cuts), importing pairwise in all affected modules: genon/preprocessor/facade/chunking_processor.py lines 975-976, genon/preprocessor/facade/convert_processor.py lines 1287-1288, genon/preprocessor/facade/intelligent_processor.py lines 1292-1293, genon/preprocessor/facade/legacy/BOK_적재용_규정.py lines 775-776, genon/preprocessor/facade/legacy/BOK_적재용_내부.py lines 716-717, and genon/preprocessor/facade/legacy/BOK_적재용_외부.py lines 771-772; preserve each existing a < b filtering behavior.Source: Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@genon/preprocessor/facade/chunking_processor.py`:
- Around line 2586-2602: Update the chunk-size and overlap resolution in the
shown preprocessing logic to distinguish absent kwargs from explicit values:
preserve an explicit chunk_size=0 and convert it to the effective no-split
default of 1,000,000 rather than falling back to common_size, while retaining
configured fallback behavior when the key is absent. Handle an explicitly null
recursive_chunk_overlap safely by falling back to overlap_default before int
conversion, and keep the existing final clamping behavior.
---
Outside diff comments:
In `@genon/preprocessor/facade/legacy/BOK_첨부용.py`:
- Around line 1680-1685: Update the legacy chunk-size and chunk-overlap default
resolution before the clamping logic: when either value is absent, read the
corresponding recursive_chunk_size or recursive_chunk_overlap option first, then
fall back to generic_chunk_size or generic_chunk_overlap for compatibility.
Preserve the existing integer conversion and bounds handling after resolution.
---
Nitpick comments:
In `@genon/preprocessor/facade/chunking_processor.py`:
- Around line 975-976: Replace each adjacent-cut iteration with
itertools.pairwise(cuts), importing pairwise in all affected modules:
genon/preprocessor/facade/chunking_processor.py lines 975-976,
genon/preprocessor/facade/convert_processor.py lines 1287-1288,
genon/preprocessor/facade/intelligent_processor.py lines 1292-1293,
genon/preprocessor/facade/legacy/BOK_적재용_규정.py lines 775-776,
genon/preprocessor/facade/legacy/BOK_적재용_내부.py lines 716-717, and
genon/preprocessor/facade/legacy/BOK_적재용_외부.py lines 771-772; preserve each
existing a < b filtering behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: f5f1b969-029a-45c8-8d36-1ce5e2be7b89
📒 Files selected for processing (10)
genon/preprocessor/facade/attachment_processor.pygenon/preprocessor/facade/chunking_processor.pygenon/preprocessor/facade/convert_processor.pygenon/preprocessor/facade/intelligent_processor.pygenon/preprocessor/facade/legacy/BOK_적재용_규정.pygenon/preprocessor/facade/legacy/BOK_적재용_내부.pygenon/preprocessor/facade/legacy/BOK_적재용_외부.pygenon/preprocessor/facade/legacy/BOK_첨부용.pygenon/preprocessor/resource/chunking_processor_config.yamlgenon/preprocessor/resource_dev/chunking_processor_config.yaml
2 participants
fix(#336): chunk_size 전달 시 청크 결과 안 나오는 오류 수정 + 청킹 옵션 정리
개요
chunking_processor(코드서빙 /chunker)를 호출할 때chunk_size(예: 1000)를 kwargs 로 넘기면결과가 안 나오는(0청크로 보이는) 오류를 수정한다. 원인 분석 과정에서 발견한 동일 부류의 크래시와,
청킹 옵션 config 네이밍/중복도 함께 정리했다.
원인 (root cause)
chunk_size를 넘기면max_tokens > 0이 되어 토큰 분할 경로(split_only 5.5단계)가 켜진다.doc_items가 빈 그룹이 생기는데,get_current_chunk()가빈 텍스트만 방어하고 빈 items 는 방어하지 않아
DocMeta(doc_items=[])로 생성 → pydanticmin_length=1ValidationError→__call__밖으로 전파 → "결과 없음".chunk_overlap > chunk_size일 때 langchainRecursiveCharacterTextSplitter가ValueError로 크래시하는 동일 부류 문제도 존재했다.주요 변경
1) 크래시 수정 (핵심)
get_current_chunk()가드를if not merged_texts or not merged_items:로 확장(빈 items 그룹은 청크로 만들지 않음). 모든 DocChunk 생성이 이 단일 지점을 거치고 호출부가 이미
None을 스킵하므로 크래시가 사라진다.split_items_evenly_by_tokens()가 폭 0 범위(a,a)를 만들지 않도록if a < b필터 추가.chunk_overlap = min(max(overlap,0), chunk_size-1)로 클램프해
overlap >= size크래시(ValueError) 방지.(
chunking/intelligent/convert) + BOK 적재·첨부 legacy 사본에 lockstep 반영.attachment_processor문자 splitter(_char_split_text단일 지점)에도 overlap 클램프 반영.2) 청킹 옵션 config 정리 (attachment 정렬)
attachment_processor와 동일한recursive네이밍으로 통일(기존
generic→recursive).recursive_chunk_overlap별칭 정리.3) chunk_size 통합
chunking.chunk_size로 통합(호출 kwargs
chunk_size는 원래부터 두 경로를 모두 제어).recursive.chunk_size및 관련 sizekwargs 별칭(
recursive_chunk_size/generic_chunk_size/generic_chunk_overlap) 제거.recursive:블록에는 docling 대응 개념이 없는chunk_overlap만 남김.chunk_size: 10000유지(docling 불변). parse-format 은 0/미설정 시 코드 기본값1000000(사실상 미분할)로 대체.
영향 파일
facade/chunking_processor.py— 크래시 가드/클램프, config 읽기(recursive),_chunk_text_elements(공통 크기).facade/intelligent_processor.py,facade/convert_processor.py— get_current_chunk 가드 + split 폭0 필터.facade/attachment_processor.py—_char_split_textoverlap 클램프.facade/legacy/BOK_적재용_{규정,내부,외부}.py,facade/legacy/BOK_첨부용.py— 동일 가드/클램프 lockstep.resource/chunking_processor_config.yaml,resource_dev/chunking_processor_config.yaml— chunk_size 공통화,generic→recursive,recursive.chunk_size제거.하위 호환 / 동작 변경
chunk_size(기본 10000자) 기준으로분할된다(직전엔 사실상 미분할). docling 경로는 불변.
기존 정상 호출/정상 overlap 케이스는 결과 불변.
검증 (preprocessor venv, in-process)
chunk_size=1000→ValidationError(doc_items ... at least 1 item)로 실패.sample1.docling.json+chunk_size=1000→ 42청크 정상 출력(빈 items 그룹만 드롭).chunk_size=10000적용 확인,chunk_size=500시 더 잘게, overlap>size 도 크래시 없이 클램프.chunk_overlap단독/정상 overlap, docling 기본 경로 결과 불변. 코드 컴파일 + 두 YAML 파싱 OK.커밋
00e15434Summary by CodeRabbit