Feature/298 table description - #325
Conversation
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
|
Warning Review limit reached
Next review available in: 11 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (13)
📝 WalkthroughWalkthroughThe PR adds ChangesChunking and enrichment pipeline
Estimated code review effort: 5 (Critical) | ~120 minutes Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 12
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
genon/preprocessor/facade/intelligent_processor.py (1)
2504-2570: 📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick winSame
_configure_runtime_image_modeearly-return coupling asconvert_processor.py.Identical structure to
convert_processor.py#L2625-L2664:table_desc/table_refine/doc_summaryruntime reconfiguration lives after theif base is None: returnguard that's only meant to gate image-description reconfiguration. See the consolidated comment for details/fix suggestion.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@genon/preprocessor/facade/intelligent_processor.py` around lines 2504 - 2570, Update _configure_runtime_image_mode so the base image-description guard only skips image_description_options and image_description_enricher reconstruction; do not return from the entire method when _base_image_description_options is missing. Keep table_desc/table_refine handling under its own _base_table_description_options check and doc_summary handling under its own _base_doc_summary_options check, allowing both to be reconfigured independently.
🧹 Nitpick comments (4)
genon/preprocessor/facade/chunking_processor.py (1)
1323-1323: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueFormat multiple statements on separate lines.
For better readability and PEP-8 compliance, consider splitting these statements onto separate lines rather than using semicolons.
♻️ Proposed refactor
- gi.append(x[0]); gh.append(x[1]); gs.append(x[2]) + gi.append(x[0]) + gh.append(x[1]) + gs.append(x[2])🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@genon/preprocessor/facade/chunking_processor.py` at line 1323, In the loop containing gi.append, gh.append, and gs.append, place each append statement on its own line and remove the semicolon-separated formatting while preserving the existing order and behavior.Source: Linters/SAST tools
genon/preprocessor/tests/unit/test_chunk_size_config.py (1)
28-160: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winLGTM overall — the spy/config helpers and new clamp/chunk_mode assertions are correct and well-targeted.
One coverage gap:
genon/preprocessor/facade/chunking_processor.pyhas the identical_clamp_chunk_size/chunk_modeimplementation (per the provided graph context) but isn't covered by_DEFAULT_CONFIG/_MODULEShere, so a regression there wouldn't be caught by this suite. Consider adding"chunking_processor": "chunking_processor_config.yaml"to_DEFAULT_CONFIG(and to_MODULES) to reuse these same parametrized tests.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@genon/preprocessor/tests/unit/test_chunk_size_config.py` around lines 28 - 160, Extend the test module’s shared coverage lists by adding the chunking_processor module and its chunking_processor_config.yaml mapping to _MODULES and _DEFAULT_CONFIG. Ensure the existing parametrized tests and helpers exercise genon/preprocessor/facade/chunking_processor.py without introducing separate test logic.genon/preprocessor/resource/convert_processor_config.yaml (1)
98-99: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueExplicitly define
compact_tablesin the output section.The documentation correctly states that
output.compact_tables: trueis the default setting to remove padding from Markdown tables. However, this option is missing from the YAML templates, which limits discoverability for operators trying to configure output formats.
genon/preprocessor/resource/convert_processor_config.yaml#L98-L99: Addcompact_tables: trueunderoutput.genon/preprocessor/resource/intelligent_processor_config.yaml#L100-L101: Addcompact_tables: trueunderoutput.💡 Proposed refactor to add the configuration key
output: table_format: "html" # "html"(default) | "markdown" + compact_tables: true # markdown 표 컬럼 정렬 패딩 제거(대형 표 축소). html 포맷엔 무관🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@genon/preprocessor/resource/convert_processor_config.yaml` around lines 98 - 99, Add compact_tables: true under the output section in genon/preprocessor/resource/convert_processor_config.yaml (lines 98-99) and genon/preprocessor/resource/intelligent_processor_config.yaml (lines 100-101), alongside table_format, so both templates explicitly expose the documented default.genon/preprocessor/facade/parser_processor.py (1)
1356-1375: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueLog a warning if
table_descis enabled at runtime butgenerate_page_imagesis disabled.If
table_descis dynamically enabled via runtimekwargswhilegenerate_page_imagesisFalsein the pipeline configuration, the VLM will fail to crop the table images. Consider adding a warning log here to alert operators, similar to the documented limitation forimage_description.💡 Proposed refactor to add a warning log
self.table_description_options = resolve_runtime_table_options( tbase, table_desc=table_desc, table_refine=table_refine, ) + if self.table_description_options.enabled and not getattr(self.pipe_line_options, "generate_page_images", False): + _log.warning("[runtime_feature] table_desc is enabled at runtime, but generate_page_images is False. Table images will not be cropped.") self.table_description_enricher = TableDescriptionEnricher( self.table_description_options )🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@genon/preprocessor/facade/parser_processor.py` around lines 1356 - 1375, Add a warning in the runtime table-options block after resolving table_desc when table_desc is enabled but the pipeline’s generate_page_images configuration is false. Use the existing _log warning pattern and match the documented image_description limitation, without changing option resolution or enrichment behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@genon/preprocessor/facade/convert_processor.py`:
- Around line 2625-2664: Remove the early return in
_configure_runtime_image_mode that is triggered when
_base_image_description_options is missing. Scope image-option resolution and
enrichment updates to the image base-options check, then allow the existing
_base_table_description_options and _base_doc_summary_options branches to run
independently when their own base options are available.
- Around line 983-988: Update _table_item_to_texts so the early return guarded
by TableDescriptionExtractor only applies when refined HTML is present, not when
only a summary exists. For summary-only tables, preserve row-based splitting and
append the summary text only to the final split chunk, maintaining the
configured chunk_size behavior and avoiding duplicate summary content.
In `@genon/preprocessor/facade/enrichment/table_description.py`:
- Around line 379-387: Update TableItem.extract_summary to return only the
description annotation produced by the configured table-description source,
rather than the first DescriptionAnnotation from any enricher. Filter
annotations using the existing table-description provenance or dedicated
annotation key, while preserving the current text normalization and empty-string
fallback.
- Around line 319-324: Update the table validation around
GenosVlmHTMLDocumentBackend.parse_table_data so the first row is accepted only
when at least one normalized cell contains non-whitespace text, not merely when
grid[0] has cells. Preserve the existing checks for missing data, empty grids,
and missing data rows, and add a test covering a header row whose cells are all
empty.
- Around line 261-265: Update the resolver’s replace call so table_refine=1 also
sets enabled=True, allowing refinement to activate table enrichment
independently of table_desc. Preserve refine_enabled behavior and the existing
disabled state when neither option is enabled.
- Line 190: Update the table-description configuration parsing around
_parse_optional_bool to read the documented “enable” key instead of “enabled”,
so supplied enable: true values are recognized and enablement is preserved.
- Around line 539-564: Update the prompt-building method containing the
tpl.render call so max_context_chars is applied to individual context values,
especially doc_summary and the other safe_* inputs, before rendering. Remove the
final self._truncate_context(prompt) call and return the fully rendered prompt
unchanged, preserving the refine template’s trailing instructions and markers.
In `@genon/preprocessor/facade/intelligent_processor.py`:
- Around line 961-966: Update the table-splitting condition in the relevant
intelligent processor flow so row splitting is skipped only when refined HTML is
present, not when only a summary annotation exists. Preserve the single-chunk
behavior for tables with refined HTML while allowing summary-only large tables
to continue through the normal chunk-size-aware row splitting path.
In `@genon/preprocessor/resource_dev/convert_processor_config.yaml`:
- Line 173: Remove the plaintext API credential from both enrichment blocks in
genon/preprocessor/resource_dev/convert_processor_config.yaml at lines 173-173
and 198-198, replacing each api_key value with the project’s environment/secret
configuration reference. Use the same secret reference for document summaries
and table descriptions, and rotate the exposed credential.
In `@genon/preprocessor/resource_dev/intelligent_processor_config.yaml`:
- Line 190: Remove the plaintext api_key values from
intelligent_processor_config.yaml at lines 190-190 and 215-215, and
parser_processor_config.yaml at lines 152-152 and 177-177. Replace each with the
project’s supported environment-variable reference or a secure empty
placeholder, without committing credentials.
In `@genon/preprocessor/resource/convert_processor_config.yaml`:
- Around line 164-165: Update the doc_summary URL placeholder from
IMAGE_DESCRIPTION_SERVING_ID to ENRICHMENT_SERVING_ID in
genon/preprocessor/resource/convert_processor_config.yaml lines 164-165 and
genon/preprocessor/resource/intelligent_processor_config.yaml lines 163-164.
- Around line 186-187: Use <IMAGE_DESCRIPTION_SERVING_ID> consistently for
table_description because it relies on VLM image processing. Keep the existing
YAML URLs in convert_processor_config.yaml (lines 186-187) and
intelligent_processor_config.yaml (lines 185-186); update the corresponding
placeholders in convert_processor.md (lines 286-287), intelligent_processor.md
(lines 286-287), and parser_processor.md (lines 279-280). In parser_processor.md
(line 697), move table_description.url from the <ENRICHMENT_SERVING_ID> group
into the <IMAGE_DESCRIPTION_SERVING_ID> group.
---
Outside diff comments:
In `@genon/preprocessor/facade/intelligent_processor.py`:
- Around line 2504-2570: Update _configure_runtime_image_mode so the base
image-description guard only skips image_description_options and
image_description_enricher reconstruction; do not return from the entire method
when _base_image_description_options is missing. Keep table_desc/table_refine
handling under its own _base_table_description_options check and doc_summary
handling under its own _base_doc_summary_options check, allowing both to be
reconfigured independently.
---
Nitpick comments:
In `@genon/preprocessor/facade/chunking_processor.py`:
- Line 1323: In the loop containing gi.append, gh.append, and gs.append, place
each append statement on its own line and remove the semicolon-separated
formatting while preserving the existing order and behavior.
In `@genon/preprocessor/facade/parser_processor.py`:
- Around line 1356-1375: Add a warning in the runtime table-options block after
resolving table_desc when table_desc is enabled but the pipeline’s
generate_page_images configuration is false. Use the existing _log warning
pattern and match the documented image_description limitation, without changing
option resolution or enrichment behavior.
In `@genon/preprocessor/resource/convert_processor_config.yaml`:
- Around line 98-99: Add compact_tables: true under the output section in
genon/preprocessor/resource/convert_processor_config.yaml (lines 98-99) and
genon/preprocessor/resource/intelligent_processor_config.yaml (lines 100-101),
alongside table_format, so both templates explicitly expose the documented
default.
In `@genon/preprocessor/tests/unit/test_chunk_size_config.py`:
- Around line 28-160: Extend the test module’s shared coverage lists by adding
the chunking_processor module and its chunking_processor_config.yaml mapping to
_MODULES and _DEFAULT_CONFIG. Ensure the existing parametrized tests and helpers
exercise genon/preprocessor/facade/chunking_processor.py without introducing
separate test logic.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 97464393-836b-4ab1-a1ea-aee881944c0d
📒 Files selected for processing (32)
genon/preprocessor/examples/parse_chunk/parse_chunk_test.pygenon/preprocessor/facade/chunking_processor.pygenon/preprocessor/facade/convert_processor.pygenon/preprocessor/facade/enrichment/__init__.pygenon/preprocessor/facade/enrichment/doc_summary.pygenon/preprocessor/facade/enrichment/enrichment_config.pygenon/preprocessor/facade/enrichment/image_description.pygenon/preprocessor/facade/enrichment/table_description.pygenon/preprocessor/facade/gitbook_doc/convert_processor.mdgenon/preprocessor/facade/gitbook_doc/intelligent_processor.mdgenon/preprocessor/facade/gitbook_doc/intro.mdgenon/preprocessor/facade/gitbook_doc/parser_processor.mdgenon/preprocessor/facade/intelligent_processor.pygenon/preprocessor/facade/legacy/BOK_적재용_규정.pygenon/preprocessor/facade/legacy/BOK_적재용_내부.pygenon/preprocessor/facade/legacy/BOK_적재용_외부.pygenon/preprocessor/facade/legacy/BOK_첨부용.pygenon/preprocessor/facade/parser_processor.pygenon/preprocessor/resource/chunking_processor_config.yamlgenon/preprocessor/resource/convert_processor_config.yamlgenon/preprocessor/resource/intelligent_processor_config.yamlgenon/preprocessor/resource/parser_processor_config.yamlgenon/preprocessor/resource/prompt_table_description_default.mdgenon/preprocessor/resource/prompt_table_refine_combined.mdgenon/preprocessor/resource_dev/chunking_processor_config.yamlgenon/preprocessor/resource_dev/convert_processor_config.yamlgenon/preprocessor/resource_dev/intelligent_processor_config.yamlgenon/preprocessor/resource_dev/parser_processor_config.yamlgenon/preprocessor/resource_dev/prompt_table_description_default.mdgenon/preprocessor/resource_dev/prompt_table_refine_combined.mdgenon/preprocessor/tests/unit/test_chunk_size_config.pygenon/preprocessor/tests/unit/test_table_refine_validation.py
| ) | ||
| return None | ||
|
|
||
| enabled = _parse_optional_bool(table_desc_cfg.get("enabled"), "enabled") |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Read the documented enable configuration key.
All supplied table-description YAML blocks use enable, but this parser only reads enabled, so enable: true still resolves to disabled.
Proposed fix
- enabled = _parse_optional_bool(table_desc_cfg.get("enabled"), "enabled")
+ enabled = _parse_optional_bool(table_desc_cfg.get("enable"), "enable")
+ if enabled is None:
+ enabled = _parse_optional_bool(table_desc_cfg.get("enabled"), "enabled")📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| enabled = _parse_optional_bool(table_desc_cfg.get("enabled"), "enabled") | |
| enabled = _parse_optional_bool(table_desc_cfg.get("enable"), "enable") | |
| if enabled is None: | |
| enabled = _parse_optional_bool(table_desc_cfg.get("enabled"), "enabled") |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@genon/preprocessor/facade/enrichment/table_description.py` at line 190,
Update the table-description configuration parsing around _parse_optional_bool
to read the documented “enable” key instead of “enabled”, so supplied enable:
true values are recognized and enablement is preserved.
| - doc_summary: | ||
| enable: false # 기본 off. true 또는 런타임 doc_summary=1 로 활성화. image/table 이 공유 | ||
| url: "https://genos.genon.ai/api/gateway/rep/serving/776/v1/chat/completions" | ||
| api_key: "30241fdbb58d42e48a36495da34a6de7" |
There was a problem hiding this comment.
🔒 Security & Privacy | 🔴 Critical | ⚡ Quick win
Remove and rotate the committed API credential.
The new enrichment blocks duplicate a plaintext API key in version control. Load it from environment/secret configuration and rotate the exposed credential.
genon/preprocessor/resource_dev/convert_processor_config.yaml#L173-L173: replace the document-summary key with a secret reference.genon/preprocessor/resource_dev/convert_processor_config.yaml#L198-L198: replace the table-description key with the same secret reference.
🧰 Tools
🪛 Betterleaks (1.6.1)
[high] 173-173: Detected a Generic API Key, potentially exposing access to various services and sensitive operations.
(generic-api-key)
📍 Affects 1 file
genon/preprocessor/resource_dev/convert_processor_config.yaml#L173-L173(this comment)genon/preprocessor/resource_dev/convert_processor_config.yaml#L198-L198
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@genon/preprocessor/resource_dev/convert_processor_config.yaml` at line 173,
Remove the plaintext API credential from both enrichment blocks in
genon/preprocessor/resource_dev/convert_processor_config.yaml at lines 173-173
and 198-198, replacing each api_key value with the project’s environment/secret
configuration reference. Use the same secret reference for document summaries
and table descriptions, and rotate the exposed credential.
Source: Linters/SAST tools
| - doc_summary: | ||
| enable: false # 기본 off. true 또는 런타임 doc_summary=1 로 활성화. image/table 이 공유 | ||
| url: "https://genos.genon.ai/api/gateway/rep/serving/776/v1/chat/completions" | ||
| api_key: "30241fdbb58d42e48a36495da34a6de7" |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
Avoid hardcoding API keys in configuration files.
These newly added configuration blocks contain a hardcoded API key (30241fdbb58d42e48a36495da34a6de7). Even in development environments, committing credentials to source control poses a security risk. Consider removing the plaintext keys and sourcing them securely using environment variables or a secrets manager.
genon/preprocessor/resource_dev/intelligent_processor_config.yaml#L190-L190: Remove the hardcoded API key and replace it with an environment variable reference or secure placeholder (e.g.,"").genon/preprocessor/resource_dev/intelligent_processor_config.yaml#L215-L215: Remove the hardcoded API key here as well.genon/preprocessor/resource_dev/parser_processor_config.yaml#L152-L152: Remove the hardcoded API key here.genon/preprocessor/resource_dev/parser_processor_config.yaml#L177-L177: Remove the hardcoded API key here.
🧰 Tools
🪛 Betterleaks (1.6.1)
[high] 190-190: Detected a Generic API Key, potentially exposing access to various services and sensitive operations.
(generic-api-key)
📍 Affects 2 files
genon/preprocessor/resource_dev/intelligent_processor_config.yaml#L190-L190(this comment)genon/preprocessor/resource_dev/intelligent_processor_config.yaml#L215-L215genon/preprocessor/resource_dev/parser_processor_config.yaml#L152-L152genon/preprocessor/resource_dev/parser_processor_config.yaml#L177-L177
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@genon/preprocessor/resource_dev/intelligent_processor_config.yaml` at line
190, Remove the plaintext api_key values from intelligent_processor_config.yaml
at lines 190-190 and 215-215, and parser_processor_config.yaml at lines 152-152
and 177-177. Replace each with the project’s supported environment-variable
reference or a secure empty placeholder, without committing credentials.
Source: Linters/SAST tools
| enable: false # 기본 off. true 또는 런타임 doc_summary=1 로 활성화. image/table 이 공유 | ||
| url: "http://llmops-gateway-api-service:8080/rep/serving/<IMAGE_DESCRIPTION_SERVING_ID>/v1/chat/completions" |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
Correct the serving ID placeholder for doc_summary.
The doc_summary enricher performs text-based LLM summarization, but the YAML configuration templates incorrectly use <IMAGE_DESCRIPTION_SERVING_ID> (a VLM endpoint). According to the documentation, this should use <ENRICHMENT_SERVING_ID>.
genon/preprocessor/resource/convert_processor_config.yaml#L164-L165: Replace<IMAGE_DESCRIPTION_SERVING_ID>with<ENRICHMENT_SERVING_ID>.genon/preprocessor/resource/intelligent_processor_config.yaml#L163-L164: Replace<IMAGE_DESCRIPTION_SERVING_ID>with<ENRICHMENT_SERVING_ID>.
📍 Affects 2 files
genon/preprocessor/resource/convert_processor_config.yaml#L164-L165(this comment)genon/preprocessor/resource/intelligent_processor_config.yaml#L163-L164
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@genon/preprocessor/resource/convert_processor_config.yaml` around lines 164 -
165, Update the doc_summary URL placeholder from IMAGE_DESCRIPTION_SERVING_ID to
ENRICHMENT_SERVING_ID in
genon/preprocessor/resource/convert_processor_config.yaml lines 164-165 and
genon/preprocessor/resource/intelligent_processor_config.yaml lines 163-164.
2 participants
feat(#298): 표 description(요약/재구성) + 청킹 옵션(
chunk_mode/split_only) +doc_summary단계화개요
TableItem 단위 표 description(요약 병기 + refine HTML 재구성)을 표준 enrichment 로 도입하고,
청크의 표 출력 포맷/크기 제어 옵션과
doc_summary단계화를 함께 정리했다.intelligent_processor중심으로convert/parser/BOK 적재용까지 확산했다.표 description 은 표를 이미지로 VLM 에 보내 두 가지를 얻는다.
\n---\n[표 설명]\n<요약>로 병기.주요 변경
1) 표 description (
table_description) — 신규 enrichmentfacade/enrichment/table_description.py(신규, 공용 모듈 — intelligent/convert/parser 가 import).DescriptionAnnotation, 재구성 HTML 은MiscAnnotation(content={"refined_html": ...})로 TableItem 에 부착. 청킹 시
intelligent_processor._extract_table_text()가 소비(refine 있으면 표 본체 교체, 요약 있으면
[표 설명]병기).table_desc(→enable),table_refine(→refine.enable) — 요청 단위 오버라이드.prompt_table_description_default.md(요약),prompt_table_refine_combined.md(
[[[TABLE_HTML]]]/[[[TABLE_SUMMARY]]]마커 규약으로 재구성 HTML + 요약 통합 응답).2) refine 결과 유효성 검증 — 잘못된 재구성/요약이면 원본 표로 폴백
VLM 재구성이 깨지거나(잘림/degeneration) 요약에 마크업이 새어들어도 그대로 쓰이지 않도록 다단 검증 추가.
is_valid_refined_html(): 재구성 HTML 구조 검증 — 원문에서<table>/</table>태그쌍 개수 일치(bs4 자동보정으로 인한 잘린 표 통과 방지) + docling grid 복원 + 2행 이상 + 첫 행 non-empty.
실패 시 annotation 부착 안 함 → 다운스트림이 원본 표로 폴백.
_parse_refine_output()강건화:[[[TABLE_SUMMARY]]]마커 누락 등 마커 구조가 불완전하면(응답 잘림/토큰 반복 폭주) 전체를 요약으로 덤프하지 않고
("", "")반환 → 폐기.is_valid_table_summary(): 요약에[[[TABLE_HTML]]]마커나 원문<table>잔재가 섞이면 폐기(요약 경로로 깨진 표가 새어드는 사례 차단).
3) 표 출력 포맷 옵션 (
output)markdown선택 시 doclingMarkdownDocSerializer(compact_tables=…)로 직렬화(네이티브 표와 동일 경로).4) 청킹 옵션 (
chunking)chunk_mode:split_only=구조 기반 청크를 유지하고chunk_size초과 청크만 분할(작은 청크 병합 안 함) /resize_all=모든 청크를chunk_size에 맞게 병합·분할(기존 방식).chunk_size최소 1024 보정(_clamp_chunk_size): 0 초과이면서 1024 미만이면 1024 로 상향(과도한 문맥 절단 방지). 0/None 은 그대로.intelligent/convert/chunking+ BOK 적재용내부/외부/규정.구현: 2.5단계(긴 청크 분할)·4단계(토큰 병합 트리거)·5단계(greedy 병합)를
resize_all전용으로 게이트하고,split_only전용 5.5단계(초과 그룹만 균등 분할) 추가.chunk_size,chunk_mode. 예제 러너parse_chunk_test.py에--chunk-mode추가.5)
doc_summary단계화facade/enrichment/doc_summary.py(신규). 문서 본문요약을 요청당 1회 계산해_enrichment_context로image/table description 이
{{doc_summary}}로 공유(중복 LLM 호출 제거), 결과는 출력 metadata 에도 노출.실제 요약 계산은 기존
body_summary.summarize_body()에 위임. 런타임 kwargsdoc_summary.6) 확산 / 배선
convert/parser: 표 description·출력 포맷 옵션·런타임 토글 배선.CHUNK_MODE) +chunk_size보정 반영.7) config / 프롬프트 / 문서 / 테스트
resource·resource_dev, intelligent/convert/chunking/parser):table_description·output·chunking블록.prompt_table_description_default.md,prompt_table_refine_combined.md(각resource·resource_dev).intelligent/convert/parser/intro) 갱신.tests/unit/test_table_refine_validation.py(신규),tests/unit/test_chunk_size_config.py(clamp/mode 확장).하위 호환 / 동작 변경
table_description/refine/doc_summary전부 기본enable:false→ 켜지 않으면 회귀 없음.chunk_size출고 기본이 0 → 10000,chunk_mode기본 split_only 로 바뀐다.기존 "무분할(0)" 대비 문서가 10000 초과 시 분할된다. 기존 동작 재현은
chunk_size: 0또는 (병합 방식은)chunk_mode: resize_all.CHUNK_MAX_TOKENS 0→10000,CHUNK_MODE split_only→ BOK 청크 출력이 실제로 바뀐다.output.table_format기본html유지,compact_tables는 markdown 포맷에만 영향 → 기본(html) 출력 불변.table_description/refine/doc_summary는 각각 추가 LLM 호출을 유발(활성화 시).테스트 / 검증
python -m py_compile: 표 enrichment 모듈 + 6개 청커 facade 통과.test_table_refine_validation.py16 passed(구조/태그쌍/마커/summary 검증),test_chunk_size_config.py(min-clamp·chunk_mode 우선순위) 통과, enrichment 스위트 회귀 없음(126 passed).docling_parse.ContentConfig) 충돌로 facade import 기반 테스트가 skip 되는 관례 →실서빙 E2E 는
shkim_labs/test.py/ 게이트웨이 스크립트로 수행.(
refined table invalid/refine 응답 마커 불완전 → 폐기로그) 확인.변경 파일
facade/enrichment/table_description.py,facade/enrichment/doc_summary.py,tests/unit/test_table_refine_validation.py,resource/·resource_dev/의prompt_table_description_default.md,prompt_table_refine_combined.mdfacade/intelligent_processor.py,convert_processor.py,chunking_processor.py,facade/legacy/BOK_적재용_{내부,외부,규정}.py(+BOK_첨부용.py소규모)facade/enrichment/__init__.py,enrichment_config.py,image_description.pyfacade/parser_processor.pyresource/·resource_dev/의intelligent/convert/chunking/parser_processor_config.yamlfacade/gitbook_doc/{intelligent,convert,parser}_processor.md,intro.mdexamples/parse_chunk/parse_chunk_test.py,tests/unit/test_chunk_size_config.pySummary by CodeRabbit
split_onlyand resizingresize_all.