Skip to content

[Fix] 평가 Runner 및 NLG judge 검증 보강#161

Merged
whc9999 merged 2 commits into
devfrom
fix/evaluation-review-valid-issues
Jul 23, 2026
Merged

[Fix] 평가 Runner 및 NLG judge 검증 보강#161
whc9999 merged 2 commits into
devfrom
fix/evaluation-review-valid-issues

Conversation

@whc9999

@whc9999 whc9999 commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator

✨ 어떤 이유로 PR를 하셨나요?

  • feature 병합
  • 버그 수정(아래에 issue #를 남겨주세요)
  • 코드 개선
  • 코드 수정
  • 배포
  • 기타(아래에 자세한 내용 기입해주세요)

📋 세부 내용 - 왜 해당 PR이 필요한지 작업 내용을 자세하게 설명해주세요

  • 평가 CSV rejectionCodeCounts에서 NONE 제외
  • 빈 questionAnalyses에 대한 judge 문장 평가 hallucination 검증 추가
  • NLG judge OpenAI usage 기반 input/output token 기록
  • EvaluationExitCoordinator 종료 실패 시 System.exit 보장
  • 종료 Coordinator 중복 exit 요청 멱등성 테스트 추가
  • 관련 평가 회귀 테스트 보강

📸 작업 화면 스크린샷

⚠️ PR하기 전에 확인해주세요

  • 로컬테스트를 진행하셨나요?
  • 머지할 브랜치를 확인하셨나요?
  • 관련 label을 선택하셨나요?

🚨 관련 이슈 번호 [ ]

Summary by CodeRabbit

  • 버그 수정
    • 평가 결과의 거절 사유 통계에서 의미 없는 NONE 항목이 제외됩니다.
    • 입력된 질문이 없는데 평가 응답이 생성된 경우, 잘못된 결과로 저장되지 않고 검증 실패로 처리됩니다.
    • 평가 결과에 AI 호출의 입력·출력 토큰 사용량이 표시됩니다.
    • 평가 종료 과정에서 오류가 발생해도 종료 처리가 안정적으로 완료되며, 중복 종료 요청을 방지합니다.

- 평가 CSV rejectionCodeCounts에서 NONE 제외
- 빈 questionAnalyses에 대한 judge 문장 평가 hallucination 검증 추가
- NLG judge OpenAI usage 기반 input/output token 기록
- EvaluationExitCoordinator 종료 실패 시 System.exit 보장
- 종료 Coordinator 중복 exit 요청 멱등성 테스트 추가
- 관련 평가 회귀 테스트 보강
@whc9999 whc9999 self-assigned this Jul 23, 2026
@coderabbitai

coderabbitai Bot commented Jul 23, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@whc9999, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 33 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 8afc66ce-5522-4c25-abcb-3526b2566662

📥 Commits

Reviewing files that changed from the base of the PR and between 54f0590 and bccf989.

📒 Files selected for processing (1)
  • src/test/java/com/jobdri/jobdri_api/domain/analysis/evaluation/EvaluationAnalysisRunnerSafetyTest.java
📝 Walkthrough

Walkthrough

평가 결과의 토큰 사용량 기록, 질문 평가 검증, 후보 결정 통계 집계가 수정되었습니다. 종료 연산은 주입 가능한 구현으로 분리되었으며, 중복 호출과 Spring 종료 예외를 검증하는 테스트가 추가되었습니다.

Changes

평가 분석 처리

Layer / File(s) Summary
NLG 결과 메타데이터 및 검증
src/main/java/com/jobdri/jobdri_api/domain/analysis/evaluation/NlgEvaluationAiClient.java, src/main/java/com/jobdri/jobdri_api/domain/analysis/evaluation/NlgEvaluationBatchService.java, src/test/java/com/jobdri/jobdri_api/domain/analysis/evaluation/NlgEvaluationBatchServiceTest.java
OpenAI 응답의 사용량을 입력·출력 토큰으로 기록하고, 빈 질문 분석에 평가 응답이 포함되면 judge_validation_failed로 처리합니다.
후보 결정 통계 집계
src/main/java/com/jobdri/jobdri_api/domain/analysis/evaluation/EvaluationAnalysisBatchService.java, src/test/java/com/jobdri/jobdri_api/domain/analysis/evaluation/EvaluationAnalysisBatchServiceTest.java
후보 2개와 수락·거절 결정 통계를 검증하며, NOT_ACTIONABLE은 집계하고 NONE은 제외합니다.
종료 연산 위임 및 예외 처리
src/main/java/com/jobdri/jobdri_api/domain/analysis/evaluation/EvaluationExitCoordinator.java, src/test/java/com/jobdri/jobdri_api/domain/analysis/evaluation/EvaluationAnalysisRunnerSafetyTest.java
Spring 종료와 시스템 종료를 별도 연산으로 주입하고, 종료 코드 해결 실패 시에도 시스템 종료를 수행하며 중복 종료 요청을 한 번만 처리합니다.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Suggested labels: 🐛 fix

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed 제목이 평가 Runner와 NLG judge 검증 보강이라는 핵심 변경을 잘 요약합니다.
Description check ✅ Passed 필수 섹션은 유지되었고 작업 이유와 변경 내용이 구체적으로 적혀 있어 템플릿을 대부분 충족합니다.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/evaluation-review-valid-issues

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@src/test/java/com/jobdri/jobdri_api/domain/analysis/evaluation/EvaluationAnalysisRunnerSafetyTest.java`:
- Around line 360-377: Update the EvaluationExitCoordinator test to make the
Spring exit operation return a nonzero code such as 17, then assert that the
SystemExitOperation receives that returned code rather than the originally
requested code. Preserve the existing duplicate-exit assertions ensuring both
Spring and system exit operations execute only once.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 324b9ab8-9f7d-4811-9931-85c1d190fdfd

📥 Commits

Reviewing files that changed from the base of the PR and between 53db15e and 54f0590.

📒 Files selected for processing (7)
  • src/main/java/com/jobdri/jobdri_api/domain/analysis/evaluation/EvaluationAnalysisBatchService.java
  • src/main/java/com/jobdri/jobdri_api/domain/analysis/evaluation/EvaluationExitCoordinator.java
  • src/main/java/com/jobdri/jobdri_api/domain/analysis/evaluation/NlgEvaluationAiClient.java
  • src/main/java/com/jobdri/jobdri_api/domain/analysis/evaluation/NlgEvaluationBatchService.java
  • src/test/java/com/jobdri/jobdri_api/domain/analysis/evaluation/EvaluationAnalysisBatchServiceTest.java
  • src/test/java/com/jobdri/jobdri_api/domain/analysis/evaluation/EvaluationAnalysisRunnerSafetyTest.java
  • src/test/java/com/jobdri/jobdri_api/domain/analysis/evaluation/NlgEvaluationBatchServiceTest.java

- Spring exit 반환 코드와 요청 코드를 다르게 설정
- SystemExitOperation이 resolved exit code를 받는지 검증
- 중복 exit 요청 멱등성 검증 유지
@whc9999
whc9999 merged commit 33f05d3 into dev Jul 23, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant