Feature/339 monimo demo - #340
Merged
Merged
Conversation
|
Warning Review limit reached
Next review available in: 33 seconds You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThe PR adds document-type custom-field routing, FAQ spreadsheet mapping, row-level vector generation, metadata preservation, serving-repository packaging, deployment documentation, configuration, tests, and processing examples. ChangesCustom fields and serving
Estimated code review effort: 5 (Critical) | ~120 minutes Sequence Diagram(s)sequenceDiagram
participant Client
participant DocumentProcessor
participant TabularCustomFieldsMapper
participant xlsx_processor
participant ChunkingProcessor
Client->>DocumentProcessor: submit CSV/XLSX with doc_type
DocumentProcessor->>TabularCustomFieldsMapper: map matching rows
TabularCustomFieldsMapper-->>DocumentProcessor: return custom_fields_row elements
DocumentProcessor->>xlsx_processor: build row vectors
xlsx_processor-->>ChunkingProcessor: provide row metadata and content
ChunkingProcessor-->>Client: return row-level vectors
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 10
🧹 Nitpick comments (1)
genon/preprocessor/facade/convert_processor.py (1)
303-316: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDuplicated inline
normalize_doc_typefallback in two facades. Both files define their own fallback normalizer when the enrichment module import fails. The shared root cause is a copied definition that can drift from the canonicalcustom_fields_enricher.normalize_doc_type, which would make the stampeddoc_typedisagree with the value that tabular mappers match on.
genon/preprocessor/facade/convert_processor.py#L303-L316: confirm the fallback matches the canonical rules, or import the normalizer from a module that has no optional dependency.genon/preprocessor/facade/intelligent_processor.py#L321-L333: apply the same change so both facades use one definition.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@genon/preprocessor/facade/convert_processor.py` around lines 303 - 316, Remove the duplicated inline normalize_doc_type fallbacks so both facades use one canonical normalizer whose rules match custom_fields_enricher.normalize_doc_type. Update genon/preprocessor/facade/convert_processor.py lines 303-316 and genon/preprocessor/facade/intelligent_processor.py lines 321-333, importing the normalizer from a module without optional dependencies or otherwise reusing the canonical definition; preserve the existing enrichment and mapper fallback behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@build-script/code-serving-README.md`:
- Around line 46-49: Update the GitHub authentication instructions immediately
above the git clone command so they match the SSH URL: document the required
SSH-key setup and remove the misleading GitHub ID/token guidance. Alternatively,
switch the command to an HTTPS URL and state that Git prompts for the username
and personal access token.
- Around line 42-44: Update the git clone command in the code-serving setup
instructions to use the verified HTTPS Gitea URL or the intended repository’s
SSH clone URL instead of the current HTTP URL, while preserving the repository
and destination placeholders.
In `@build-script/sync-serving-repo.sh`:
- Around line 71-74: Update the submodule detection around SERVING_DIR and DEST
so IS_SUBMODULE is set only when the exact serving path is registered in the
root repository’s .gitmodules, not merely when git rev-parse finds any
repository. If the path is unregistered, fail safely or force dry-run behavior
before any deletion or staging can occur.
In `@genon/preprocessor/converters/xlsx_processor.py`:
- Line 553: Update the reg_date fallback in the XLSX processing flow to generate
a timezone-aware current UTC timestamp before appending the UTC designator.
Preserve any existing reg_date value and the current seconds-level ISO
formatting.
- Around line 525-526: Validate the headers in the table-loading flow before
constructing data_rows, rejecting any duplicate exact header names. Apply this
check before dict(zip(headers, values)) so
TabularCustomFieldsMapper._header_index receives no already-collapsed columns,
while preserving normal conversion for unique headers.
In `@genon/preprocessor/examples/parse_chunk/parse_chunk_test.sh`:
- Around line 43-45: Remove the duplicated second cache-step command and its
associated “2)” comment from the cache section in parse_chunk_test.sh,
preserving the single original cache step and the intervening FAQ block.
In `@genon/preprocessor/facade/chunking_processor.py`:
- Around line 2705-2715: Update the page normalization used by the chunking flow
around n_page, page_chunk_counts, and the per-element page assignment to safely
handle non-numeric values, matching _chunk_text_elements’ try/except fallback
behavior by using page 1. Reuse the normalized value consistently for counting
and processing without changing valid numeric-page behavior.
In `@genon/preprocessor/facade/convert_processor.py`:
- Around line 3401-3402: Preserve the original exceptions when the tabular
custom-fields handlers re-raise GenosServiceException by chaining with “from
exc”. Apply this in genon/preprocessor/facade/convert_processor.py lines
3401-3402 and genon/preprocessor/facade/intelligent_processor.py lines
3289-3290, keeping each facade’s existing constructor argument type unchanged.
In `@genon/preprocessor/resource_dev/custom_field_card.yaml`:
- Line 12: Remove the hardcoded api_key from custom_field_card.yaml and update
CustomFieldsEnricher to obtain it through the repository’s secret-injection
mechanism before constructing the Bearer Authorization header. Rotate and revoke
the exposed credential, then remove any copied or seeded instances of the old
key from repository history and configuration.
In `@genon/preprocessor/tests/unit/test_custom_fields_routing.py`:
- Around line 104-114: Update the monkeypatched store_metadata_in_document
lambda in the enrichment test to accept the new preserve_nulls argument while
continuing to append metadata to stored. Keep the existing FAQ and CARD
assertions unchanged.
---
Nitpick comments:
In `@genon/preprocessor/facade/convert_processor.py`:
- Around line 303-316: Remove the duplicated inline normalize_doc_type fallbacks
so both facades use one canonical normalizer whose rules match
custom_fields_enricher.normalize_doc_type. Update
genon/preprocessor/facade/convert_processor.py lines 303-316 and
genon/preprocessor/facade/intelligent_processor.py lines 321-333, importing the
normalizer from a module without optional dependencies or otherwise reusing the
canonical definition; preserve the existing enrichment and mapper fallback
behavior.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 10f27806-40e6-4ec3-af28-62ee1a8b5aa1
⛔ Files ignored due to path filters (5)
genon/preprocessor/examples/sample_files/pptx.pdfis excluded by!**/*.pdfgenon/preprocessor/examples/sample_files/pptx예시.pdfis excluded by!**/*.pdfgenon/preprocessor/examples/sample_files/txt예시.pdfis excluded by!**/*.pdfgenon/preprocessor/examples/sample_files/데이터분석.pdfis excluded by!**/*.pdfgenon/preprocessor/sample_files/doc_sample.docis excluded by!**/*.doc
📒 Files selected for processing (42)
.gitignorebuild-script/build-docling-wheel.shbuild-script/code-serving-README.mdbuild-script/code-serving-doc-parser/Dockerfilebuild-script/code-serving-doc-parser/Dockerfile.gpubuild-script/code-serving-doc-parser/pyproject.tomlbuild-script/doc-parser-build.configbuild-script/sync-serving-repo.shgenon/.gitignoregenon/preprocessor/converters/xlsx_processor.pygenon/preprocessor/examples/.gitignoregenon/preprocessor/examples/code_serving/serving_gateway_test.pygenon/preprocessor/examples/code_serving/serving_gateway_test.shgenon/preprocessor/examples/parse_chunk/parse_chunk_test.pygenon/preprocessor/examples/parse_chunk/parse_chunk_test.shgenon/preprocessor/examples/sample_files/txt예시.txtgenon/preprocessor/examples/sample_files/데이터분석.jsongenon/preprocessor/facade/chunking_processor.pygenon/preprocessor/facade/convert_processor.pygenon/preprocessor/facade/enrichment/custom_fields_enricher.pygenon/preprocessor/facade/enrichment/field_transforms.pygenon/preprocessor/facade/enrichment/tabular_custom_fields.pygenon/preprocessor/facade/intelligent_processor.pygenon/preprocessor/facade/legacy/BOK_적재용_규정.pygenon/preprocessor/facade/legacy/BOK_적재용_내부.pygenon/preprocessor/facade/legacy/BOK_적재용_외부.pygenon/preprocessor/facade/parser_processor.pygenon/preprocessor/resource/convert_processor_config.yamlgenon/preprocessor/resource/custom_field_card.yamlgenon/preprocessor/resource/custom_field_faq.yamlgenon/preprocessor/resource/intelligent_processor_config.yamlgenon/preprocessor/resource/parser_processor_config.yamlgenon/preprocessor/resource/prompt_custom_fields_card_system.mdgenon/preprocessor/resource/prompt_custom_fields_card_user.mdgenon/preprocessor/resource_dev/convert_processor_config.yamlgenon/preprocessor/resource_dev/custom_field_card.yamlgenon/preprocessor/resource_dev/custom_field_faq.yamlgenon/preprocessor/resource_dev/intelligent_processor_config.yamlgenon/preprocessor/resource_dev/parser_processor_config.yamlgenon/preprocessor/resource_dev/prompt_custom_fields_card_system.mdgenon/preprocessor/resource_dev/prompt_custom_fields_card_user.mdgenon/preprocessor/tests/unit/test_custom_fields_routing.py
Comment on lines
+42
to
+44
| # 코드서빙 생성 시 만들어진 gitea repo clone (gitea id 는 코드서빙 페이지에서 확인) | ||
| # id와 pass는 GenOS의 id와 pass를 입력해줍니다. | ||
| git clone http://llmops-gitea-service:3000/llmops/<코드서빙 gitea id>.git <gitea_dir> |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== repo files =="
git ls-files | sed -n '1,200p'
echo
echo "== target readme excerpt =="
if [ -f build-script/code-serving-README.md ]; then
cat -n build-script/code-serving-README.md | sed -n '1,120p'
else
echo "build-script/code-serving-README.md not found"
fi
echo
echo "== related gitea references =="
rg -n "llmops-gitea-service|code-serving|gitea|HTTP|HTTPS|git clone|LLMOPS|GenOS" . -SRepository: genonai/doc_parser
Length of output: 50376
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== init.sh excerpt =="
cat -n build-script/code-serving-doc-parser/scripts/init.sh | sed -n '1,80p'
echo
echo "== git config references =="
rg -n "extraheader|credential|git config|http\.extraHeader|extraHeader" build-script genon/preprocessor/build-script -S || true
echo
echo "== URL behavior verifier (parsing only) =="
python3 - <<'PY'
from urllib.parse import urlparse, urlunparse
urls = [
"http://llmops-gitea-service:3000/llmops/<코드서빙 gitea id>.git",
"git@github.com:genonai/doc_parser_code_serving.git",
"https://github.com/genonai/doc_parser.git",
]
for url in urls:
p = urlparse(url)
print(f"URL={url}")
print(f" scheme={p.scheme!r} netloc={p.netloc!r} path={p.path!r} has_credential={bool(p.username) or bool(p.password)}")
PYRepository: genonai/doc_parser
Length of output: 5003
Sensitive Data Exposure (CWE-319): Cleartext Transmission of Sensitive Information
Reachability: Internal
Use an HTTPS Gitea clone URL before requesting credentials.
Line 44 shows git clone http://llmops-gitea-service:3000/... while lines 43 and 64 require GenOS credentials, so credentials can be sent over unencrypted HTTP. Replace this with the verified HTTPS Gitea URL or SSH clone for the intended repository.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@build-script/code-serving-README.md` around lines 42 - 44, Update the git
clone command in the code-serving setup instructions to use the verified HTTPS
Gitea URL or the intended repository’s SSH clone URL instead of the current HTTP
URL, while preserving the repository and destination placeholders.
|
|
||
| # ── LLM 연결 ──────────────────────────────────────────────────────────────── | ||
| url: "https://genos.genon.ai/api/gateway/rep/serving/752/v1/chat/completions" | ||
| api_key: "d1a9e0acab6243019008a96cd8af868e" |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== file excerpt =="
cat -n genon/preprocessor/resource_dev/custom_field_card.yaml | sed -n '1,60p'
echo
echo "== repository locations for custom field card and api_key =="
rg -n --hidden --glob '!*.lock' --glob '!dist/**' --glob '!build/**' --glob '!node_modules/**' 'custom_field_card|api_key|api-key|api-key|base_url|models|serving' . | head -200
echo
echo "== tracked status =="
git ls-files --error-unmatch genon/preprocessor/resource_dev/custom_field_card.yaml || trueRepository: genonai/doc_parser
Length of output: 27346
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== parser_processor_config references =="
rg -n 'custom_fields|custom_field_card|config_file' genon/preprocessor parser_processor_config.yaml pyproject.toml docs 2>/dev/null || true
echo
echo "== custom_field_card config usage =="
rg -n 'custom_field_card|custom_fields|CustomFieldsEnricher|resource_dev' genon docling tests main.py 2>/dev/null | head -200
echo
echo "== build/docker COPY paths for genon/preprocessor/resource_dev files =="
rg -n 'COPY|genon/preprocessor/resource_dev|custom_field_card|resource_dev' Dockerfile build-script pyproject.toml genon/preprocessor 2>/dev/null | head -200Repository: genonai/doc_parser
Length of output: 50374
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== custom_fields_enricher auth/header implementation =="
sed -n '60,140p' genon/preprocessor/facade/enrichment/custom_fields_enricher.py
sed -n '380,430p' genon/preprocessor/facade/enrichment/custom_fields_enricher.py
echo
echo "== main.py config selection with env override context =="
sed -n '120,155p' main.py
echo
echo "== parse_chunk test env/config context =="
sed -n '1,90p' genon/preprocessor/examples/parse_chunk/parse_chunk_test.py
sed -n '60,90p' genon/preprocessor/examples/parse_chunk/parse_chunk_test.sh
echo
echo "== build/package exclude/include references for resource_dev =="
rg -n 'include.*resource_dev|exclude.*resource_dev|resource_dev|packages\.include|package_data|MANIFEST|COPY .*resource_dev|COPY .*genon/preprocessor' pyproject.toml setup.cfg MANIFEST.in Dockerfile .dockerignore gitignore 2>/dev/null || trueRepository: genonai/doc_parser
Length of output: 11221
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== .gitignore relevant lines =="
cat -n .gitignore | sed -n '1,80p'
echo
echo "== resource_dev custom field card files =="
git ls-files 'genon/preprocessor/resource_dev/custom_field*.yaml'
echo
echo "== resource custom field card files =="
git ls-files 'genon/preprocessor/resource/custom_field*.yaml'
echo
echo "== live request code absence/source =="
sed -n '110,260p' genon/preprocessor/facade/enrichment/custom_fields_enricher.pyRepository: genonai/doc_parser
Length of output: 9538
Sensitive Data Exposure (CWE-798): Use of Hard-coded Credentials
Exploitability: Moderate
Remove and rotate the committed API credential.
genon/preprocessor/resource_dev/custom_field_card.yaml stores a reusable api_key, and CustomFieldsEnricher sends that value as Authorization: Bearer ... for matching custom_fields runs. Move the key to secret injection, rotate the exposed value, and remove copied/seeded history instances from the repository.
🧰 Tools
🪛 Betterleaks (1.7.3)
[high] 12-12: Detected a Generic API Key, potentially exposing access to various services and sensitive operations.
(generic-api-key)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@genon/preprocessor/resource_dev/custom_field_card.yaml` at line 12, Remove
the hardcoded api_key from custom_field_card.yaml and update
CustomFieldsEnricher to obtain it through the repository’s secret-injection
mechanism before constructing the Bearer Authorization header. Rotate and revoke
the exposed credential, then remove any copied or seeded instances of the old
key from repository history and configuration.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@build-script/sync-serving-repo.sh`:
- Around line 206-213: Update the VERSION generation block to produce valid JSON
by encoding or escaping VERSION and every other interpolated field, including
source commit metadata and wheel name. Preserve the existing field names and
output location while ensuring values containing quotes or other JSON-special
characters remain parseable.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: a8e586a3-01bf-4dbd-aedc-2ea4ae07df87
📒 Files selected for processing (9)
build-script/code-serving-README.mdbuild-script/sync-serving-repo.shgenon/preprocessor/converters/xlsx_processor.pygenon/preprocessor/examples/parse_chunk/parse_chunk_test.shgenon/preprocessor/facade/chunking_processor.pygenon/preprocessor/facade/convert_processor.pygenon/preprocessor/facade/intelligent_processor.pygenon/preprocessor/tests/unit/test_custom_fields_routing.pygenon/preprocessor/tests/unit/test_parser_processor_unit.py
💤 Files with no reviewable changes (1)
- genon/preprocessor/examples/parse_chunk/parse_chunk_test.sh
🚧 Files skipped from review as they are similar to previous changes (6)
- build-script/code-serving-README.md
- genon/preprocessor/facade/chunking_processor.py
- genon/preprocessor/converters/xlsx_processor.py
- genon/preprocessor/facade/convert_processor.py
- genon/preprocessor/facade/intelligent_processor.py
- genon/preprocessor/tests/unit/test_custom_fields_routing.py
HeechanKim-Genon
approved these changes
Aug 5, 2026
This was referenced Aug 11, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
개요
카드/FAQ 문서 custom fields 추출 기능과, 이를 실제 배포하기 위한
코드서빙(GenOS Code Serving) 빌드·배포 환경을 추가합니다. 함께 xlsx/csv 직접 처리와
좌표 없는 표 처리 오류 등 파이프라인 버그를 수정합니다.
주요 변경사항
1. Custom Fields 추출 (카드/FAQ 메타데이터)
facade/enrichment/custom_fields_enricher.py— LLM 기반 custom fields 추출기.doc_type(card/faq)별외부 config(yaml)로 프롬프트·출력 필드를 정의.
facade/enrichment/tabular_custom_fields.py— 행 기반(표) 문서용tabular_mapping추출기. 행별metadata element 생성.
facade/enrichment/field_transforms.py— conf yaml에 지정한 필드가 최종 청크 메타데이터에 모두 출력되도록 수정.resource/custom_field_card.yaml,custom_field_faq.yaml,prompt_custom_fields_card_{system,user}.md(resource / resource_dev 양쪽).tests/unit/test_custom_fields_routing.py.2. XLSX/CSV 직접 처리
converters/xlsx_processor.py— PDF 변환 없이 xlsx/csv를 직접 처리.build_docling_document()(시트=1페이지)및
build_tabular_vectors()(행별 1청크, 병합셀 unmerge + forward-fill) 제공. Weaviate property 명명 제약 대응.3. 코드서빙 배포 환경
code-serving/를doc_parser_code_serving서브모듈로 추가 (빌드 산출물 repo).build-script/build-docling-wheel.sh— docling 폴더를 wheel로 빌드해packages/에 동봉(도커 빌드 불필요).build-script/sync-serving-repo.sh— genon 코드 +main.py+ docling wheel만 whitelist 복사, 배포본 동기화.build-script/code-serving-README.md— GenOS 배포·등록 및 호출 매뉴얼. 독자 repo에서 로컬 테스트 가능한 환경 추가.4. 버그 수정
5. 테스트/기타
examples/code_serving/serving_gateway_test.{py,sh},parse_chunk_test.{py,sh}정비 및 샘플 파일 추가..gitignore정리.검증
pytest genon/preprocessor/tests/unit/test_custom_fields_routing.pyexamples/code_serving/serving_gateway_test.sh— 게이트웨이 엔드포인트(적재/파싱/청킹) 통합 확인.examples/parse_chunk/parse_chunk_test.sh— 카드/FAQ 샘플로 custom fields가 청크 메타데이터에 출력되는지 확인.Summary by CodeRabbit