Feature/133 config model path

[inoray]

feat(#133): 모델 경로(토크나이저·artifacts) yaml 설정화 + 미치환 플레이스홀더 경고

배경 / 문제

• 모델 경로 하드코딩: 청킹용 토크나이저 경로(/models/doc_parser_models/...)와
  HF fallback id, docling 모델(TableFormer 등) artifacts 경로가 코드에 하드코딩되어
  있어, site별로 경로가 다르거나 외부 네트워크가 차단된 환경에서 재배포 없이 변경할 수
  없었다.
• 플레이스홀더 치환 누락: site 배포 시 config yaml의 <WHISPER_ENDPOINT> 등
  플레이스홀더를 실제 값으로 치환해야 하는데, 치환을 빠뜨려도 기동 시점에 드러나지 않고
  해당 기능을 실제로 호출할 때서야 오류가 발생했다.

변경 사항

1. 모델 경로 yaml 설정화 (models: 섹션 신설)

config yaml에 models: 섹션을 추가하여 토크나이저/artifacts 경로를 코드 수정 없이 조정
가능하도록 노출. 미지정 시 현행 하드코딩 기본값과 동일하게 동작(하위 호환).

models:
  # 청킹용 토크나이저. tokenizer_path 가 실제 존재하면 그 경로,
  # 없으면 tokenizer_id(HF) 로 폴백 (외부 네트워크 차단 환경 대비)
  tokenizer_path: "/models/doc_parser_models/sentence-transformers-all-MiniLM-L6-v2"
  tokenizer_id: "sentence-transformers/all-MiniLM-L6-v2"
  # docling 모델(TableFormer 등) 로컬 경로. 비우면 docling 기본 캐시 사용(현행).
  artifacts_path: ""

• _resolve_tokenizer(models_cfg) 헬퍼: tokenizer_path 가 실제 존재하면 로컬
  경로(Path)를, 없으면 tokenizer_id(HF) 로 폴백. GenosSmartChunker.tokenizer
  기본값도 동일 로직의 모듈 상수(_DEFAULT_TOKENIZER_LOCAL_PATH /
  _DEFAULT_TOKENIZER_ID)로 정리.
• artifacts_path: config에 값이 있을 때만 pipe_line_options.artifacts_path
  를 설정하고, 비어 있으면 설정하지 않아 docling 기본 캐시 동작을 그대로 유지
  (ocr_pipe_line_options 는 deep copy라 자동 전파). convert/intelligent에 적용.
• DocumentProcessor 에서 cfg.get("models") 를 읽어 self._tokenizer 를 결정하고,
  split_documents() 의 청킹 시 해당 토크나이저를 전달.

2. 미치환 플레이스홀더 경고

• _warn_unresolved_placeholders(cfg, config_path): _load_config() 에서 config
  전체를 재귀 스캔해 <[A-Z0-9_]+> 패턴(미치환 UPPER_SNAKE 플레이스홀더)을 탐지.
  발견 시 경로와 함께 WARNING 로그를 남긴다.
• fail-fast 아님: 기동을 막지 않고 경고만 남겨, 해당 플레이스홀더가 사용되지 않는
  경우(예: 음성 미사용 site의 <WHISPER_ENDPOINT>)에도 정상 기동.

3. whisper 설정 주석 정리

• whisper 설정이 음성(.wav/.mp3/.m4a) 처리 시에만 필요한 선택 항목임을 주석으로
  명확화하고, <WHISPER_ENDPOINT> 가 미사용 시 무시됨을 명시.

영향 범위 / 호환성

• models: 섹션 / 신규 키를 생략하면 기존과 동일하게 코드 기본값으로 동작(하위 호환).
• artifacts_path 가 비어 있으면 docling 기본 캐시 경로를 그대로 사용.
• 플레이스홀더 경고는 기동을 차단하지 않음(WARNING 로그만).

변경 파일

facade 4종

• genon/preprocessor/facade/parser_processor.py
• genon/preprocessor/facade/convert_processor.py
• genon/preprocessor/facade/intelligent_processor.py
• genon/preprocessor/facade/attachment_processor.py

config yaml 7종 (models: 섹션 추가 / whisper 주석 정리)

• resource/{parser,convert,intelligent,attachment}_processor_config.yaml
• resource_dev/{convert,intelligent,attachment}_processor_config.yaml

테스트

• 각 processor를 기본 config로 기동 → 신규 키 없이도 현행과 동일 동작 확인.
• tokenizer_path 가 존재하지 않는 환경에서 tokenizer_id(HF) 폴백 확인.
• artifacts_path 지정 시 docling 모델이 해당 로컬 경로에서 로딩되는지 확인.
• <WHISPER_ENDPOINT> 등 플레이스홀더가 남은 config 로드 시 경로와 함께 WARNING
  로그가 출력되는지 확인.

Summary by CodeRabbit

Release Notes

• New Features

  • Configurable tokenizer selection: specify local model paths or use HuggingFace-hosted alternatives for document chunking operations
  • Configurable artifacts directory for managing local model storage and resources
  • Configuration validation with startup warnings for unresolved placeholders, allowing startup to proceed with advisory alerts

• Documentation

  • Improved configuration documentation clarifying audio transcription and endpoint configuration guidance

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR adds configuration-driven tokenizer resolution and runtime validation for unresolved YAML placeholders across document processors. Tokenizers are now resolved from local paths with HuggingFace fallback, injected through processor constructors, and stored for chunking. Configuration files introduce new models sections defining tokenizer and artifact paths.

Changes

Configuration-driven tokenization and placeholder validation

Layer / File(s)	Summary
Placeholder warning system across facades genon/preprocessor/facade/attachment_processor.py, convert_processor.py, intelligent_processor.py, parser_processor.py	A new _warn_unresolved_placeholders(cfg, config_path) function scans loaded YAML configs for unresolved <UPPER_SNAKE> patterns and logs WARNINGs listing each placeholder with its location, invoked after config validation without failing startup.
Tokenizer resolution constants and functions genon/preprocessor/facade/attachment_processor.py, convert_processor.py, intelligent_processor.py	Introduces _DEFAULT_TOKENIZER_LOCAL_PATH and _DEFAULT_TOKENIZER_ID constants and implements _resolve_tokenizer(models_cfg) to select local tokenizer paths when present, otherwise falling back to HuggingFace tokenizer ids.
DocumentProcessor tokenizer initialization from config genon/preprocessor/facade/attachment_processor.py, convert_processor.py, intelligent_processor.py	DocumentProcessor.__init__ now reads the models config section, resolves tokenizers via _resolve_tokenizer, stores the result on self._tokenizer, and optionally sets pipe_line_options.artifacts_path when provided in config.
GenosSmartChunker default tokenizer updates genon/preprocessor/facade/convert_processor.py, intelligent_processor.py	GenosSmartChunker.tokenizer default initialization now uses the new tokenizer constants instead of inline hardcoded path/id existence checks.
Tokenizer injection through processor constructors and chunkers genon/preprocessor/facade/attachment_processor.py, convert_processor.py, intelligent_processor.py	DocxProcessor and HwpProcessor constructors now accept optional tokenizer arguments and store them on self._tokenizer. Their split_documents methods pass this tokenizer to HybridChunker, and DocumentProcessor injects self._tokenizer into both processor constructors and GenosSmartChunker during chunking.
Configuration file models and endpoint sections genon/preprocessor/resource/*.yaml, genon/preprocessor/resource_dev/*.yaml	All processor YAML configs now include models blocks specifying tokenizer_path, tokenizer_id, and optionally artifacts_path. Parser config clarifies the Whisper endpoint (whisper.url) and its purpose for audio processing, with corresponding updates in resource_dev variants.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Poem

🐰 A tokenizer's tale, no paths left astray,
Local first, then Hugging Face comes to play,
Config now whispers where models reside,
Placeholders warned before they can hide,
Chunkers and processors, in harmony they decide!

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (1 warning, 1 inconclusive)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 43.33% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Title check	❓ Inconclusive	The pull request title 'Feature/133 config model path' is partially related to the changeset. While it references 'config' and 'model path', it is vague and branch-like in structure; it does not clearly describe the main changes (making model paths configurable and adding placeholder warnings).	Consider revising the title to be more descriptive, such as 'Make tokenizer and artifacts paths configurable via config' or 'Add configurable model paths and placeholder warnings'.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feature/133-config-model-path

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

🧹 Nitpick comments (1)

genon/preprocessor/facade/parser_processor.py (1)

143-169: 💤 Low value

Consider extracting _warn_unresolved_placeholders to a shared utility module.

This function is duplicated across multiple facade processors (intelligent_processor, parser_processor, convert_processor, attachment_processor). Extracting it to a shared module (e.g., genon/preprocessor/facade/config_utils.py) would reduce duplication and simplify future maintenance.

That said, if the facade architecture intentionally keeps processors self-contained for deployment flexibility, the duplication is acceptable.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@genon/preprocessor/facade/parser_processor.py` around lines 143 - 169, The
_warn_unresolved_placeholders function in parser_processor.py is duplicated
across multiple facade processors; extract it into a shared utility module
(e.g., create genon/preprocessor/facade/config_utils.py) as a single function
named _warn_unresolved_placeholders (or warn_unresolved_placeholders)
implemented exactly once, then replace the local implementations in
parser_processor, intelligent_processor, convert_processor, and
attachment_processor with imports from that module and update their calls;
ensure the regex, traversal logic, signature (cfg: dict, config_path: str) and
logging via _log remain unchanged and run linters/tests to validate imports.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@genon/preprocessor/facade/parser_processor.py`:
- Around line 143-169: The _warn_unresolved_placeholders function in
parser_processor.py is duplicated across multiple facade processors; extract it
into a shared utility module (e.g., create
genon/preprocessor/facade/config_utils.py) as a single function named
_warn_unresolved_placeholders (or warn_unresolved_placeholders) implemented
exactly once, then replace the local implementations in parser_processor,
intelligent_processor, convert_processor, and attachment_processor with imports
from that module and update their calls; ensure the regex, traversal logic,
signature (cfg: dict, config_path: str) and logging via _log remain unchanged
and run linters/tests to validate imports.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 59b682a8-5263-4ad9-a774-baad379d8533

📥 Commits

Reviewing files that changed from the base of the PR and between 1fe91bf and 6c18c28.

📒 Files selected for processing (11)

• genon/preprocessor/facade/attachment_processor.py
• genon/preprocessor/facade/convert_processor.py
• genon/preprocessor/facade/intelligent_processor.py
• genon/preprocessor/facade/parser_processor.py
• genon/preprocessor/resource/attachment_processor_config.yaml
• genon/preprocessor/resource/convert_processor_config.yaml
• genon/preprocessor/resource/intelligent_processor_config.yaml
• genon/preprocessor/resource/parser_processor_config.yaml
• genon/preprocessor/resource_dev/attachment_processor_config.yaml
• genon/preprocessor/resource_dev/convert_processor_config.yaml
• genon/preprocessor/resource_dev/intelligent_processor_config.yaml