feat: 가정통신문 라벨링 데이터 평가 스크립트 추가#8
Merged
Merged
Conversation
|
Warning Review limit reached
Your plan includes 1 review of capacity. Refill in 41 minutes and 5 seconds. Your organization has run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After more review capacity refills, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than trial, open-source, and free plans. In all cases, review capacity refills continuously over time. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (7)
📝 WalkthroughWalkthrough뉴스레터 분석 품질을 평가하는 완전한 시스템을 추가합니다. 30개 라벨링 JSON 데이터셋, 평가 스크립트, 실행 문서를 포함하며, baseline 및 OpenAI 모드를 지원하고 정밀도/재현율/F1 지표를 계산합니다. Changes가정통신문 분석 평가 프레임워크
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Suggested labels
Suggested reviewers
🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 8
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
scripts/evaluate_newsletter_labels.py (1)
1-562:⚠️ Potential issue | 🔴 Critical | ⚡ Quick win파이프라인 실패: ruff 포맷팅 필수
파이프라인에서
ruff format --check검사가 실패했습니다. 머지 전에 다음 명령을 실행해 포맷을 적용해야 합니다.ruff format scripts/evaluate_newsletter_labels.py🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/evaluate_newsletter_labels.py` around lines 1 - 562, The file fails the ruff formatting check; run the project formatter and commit the changes: run `ruff format` on this module (which will reformat imports, spacing, line breaks and any style issues in functions like main, load_samples, parse_sample, evaluate_samples, etc.), verify no remaining formatting errors by re-running `ruff format --check` (or `ruff check`), and commit the resulting changes before pushing.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@data/newsletter-labels/newsletter-005.json`:
- Around line 1-98: The file's top-level is an array causing inconsistent
structure; update newsletter-005.json to match others by removing the
surrounding array and make the top-level a single object (the existing object
containing "documentId": "doc_005", "documentTitle", "labels", etc.), ensuring
the document object remains intact so downstream loaders expecting an object
(not an array) will parse it consistently.
In `@data/newsletter-labels/newsletter-014.json`:
- Line 2: documentId 값이 다른 파일들과 형식 불일치(doc_0014 vs. doc_008/doc_009 등)를 일으키므로
newsletter-014.json의 "documentId" 값을 패턴에 맞게 수정하세요: 변경 대상은 JSON 키 "documentId"이며
값 "doc_0014"를 "doc_014"가 아닌 프로젝트의 기존 형식(예: "doc_014"이 아니라 다른 파일들과 동일한 자리수 규칙—대부분
"doc_014" 대신 "doc_14"가 아니라면 프로젝트 관례에 맞춘 "doc_014"→"doc_014" 대신 실제 패턴에 일치하도록
수정)으로 통일해야 합니다; 파일 내 "documentId" 값을 프로젝트의 다른 파일들(예:
"doc_008","doc_009","doc_010")과 동일한 숫자 자리수 형식으로 일괄 변경해 주세요.
In `@data/newsletter-labels/newsletter-019.json`:
- Line 4: The documentDate field in this label is "2024-09-04" which differs
from most other labels (2025–2026); verify whether this is intentional or a typo
and, if it’s a typo, update the documentDate value to the correct year (e.g.,
"2026-09-04") in newsletter-019.json; to validate across the dataset, run the
year-distribution check suggested in the review (the provided script that
extracts .documentDate from all JSON labels) and correct any other inconsistent
dates found.
In `@data/newsletter-labels/newsletter-021.json`:
- Around line 14-18: The labels file has mismatched time granularity:
dateCandidates[].resolved includes time (e.g., "2025-06-19T08:00") while
labels[].date is date-only ("2025-06-19"), and
scripts/evaluate_newsletter_labels.py currently truncates values with value[:10]
when comparing; update the project guideline/schema to state a single canonical
rule (either: resolved may include time but labels[].date must be date-only, or
require both fields to be date-only) and make a corresponding change to the
dataset or docs so they match that rule; reference the keys
dateCandidates[].resolved and labels[].date in newsletter-021.json and the
comparison logic in scripts/evaluate_newsletter_labels.py to ensure consistency
(and, if choosing the date-only rule, normalize resolved to date-only across
files or update the evaluator to parse ISO datetimes consistently).
In `@data/newsletter-labels/newsletter-025.json`:
- Around line 6-19: The dateCandidates IDs do not match the documentId: update
the two entries with "id": "dc_024_1" and "id": "dc_024_2" to "dc_025_1" and
"dc_025_2" to align with "documentId": "doc_025", and also update any other
references to those IDs (the reference mentioned around line 36) so all usages
of dc_024_1/dc_024_2 are replaced with dc_025_1/dc_025_2 (ensure you update both
the dateCandidates array and any pointer/lookup that references those IDs).
In `@data/newsletter-labels/newsletter-027.json`:
- Around line 6-13: The date candidate ID "dc_028_1" is inconsistent with the
documentId "doc_027"; rename the dateCandidates entry id to "dc_027_1" and
update any other occurrences or references of "dc_028_1" in this JSON (e.g., the
reference later in the file) to "dc_027_1" so the ID naming is consistent with
documentId ("doc_027").
In `@data/newsletter-labels/newsletter-029.json`:
- Around line 6-19: The dateCandidates IDs are using the wrong document prefix
("dc_027_1", "dc_027_2") while documentId is "doc_029"; update these to
"dc_029_1" and "dc_029_2" respectively and also update every place that
references those old IDs (the references tied to the same date candidate IDs
elsewhere in this JSON: the three other fields that currently point to
"dc_027_1"/"dc_027_2") so all references match the new "dc_029_*" IDs; ensure
you change the values for the id fields inside dateCandidates and any references
that previously pointed to "dc_027_1" or "dc_027_2".
In `@data/newsletter-labels/README.md`:
- Around line 22-31: Add a concrete example object inside the labels array in
the README schema showing all expected fields so validators and devs know the
shape; update the example JSON under "labels" to include a sample label object
with keys: type, title, evidenceText, selectedDateCandidateId, dateStatus, date,
target, actionRequired, and schoolContext, and ensure their sample values
illustrate formats (e.g., ISO date string for date, enum-like strings for
type/dateStatus) so the schema users can implement and validate correctly.
---
Outside diff comments:
In `@scripts/evaluate_newsletter_labels.py`:
- Around line 1-562: The file fails the ruff formatting check; run the project
formatter and commit the changes: run `ruff format` on this module (which will
reformat imports, spacing, line breaks and any style issues in functions like
main, load_samples, parse_sample, evaluate_samples, etc.), verify no remaining
formatting errors by re-running `ruff format --check` (or `ruff check`), and
commit the resulting changes before pushing.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: cee034ea-d0f9-4efe-9940-10367ff32a03
📒 Files selected for processing (37)
.gitignoreREADME.mdapp/config.pydata/newsletter-labels/README.mddata/newsletter-labels/newsletter-001.jsondata/newsletter-labels/newsletter-002.jsondata/newsletter-labels/newsletter-003.jsondata/newsletter-labels/newsletter-004.jsondata/newsletter-labels/newsletter-005.jsondata/newsletter-labels/newsletter-006.jsondata/newsletter-labels/newsletter-007.jsondata/newsletter-labels/newsletter-008.jsondata/newsletter-labels/newsletter-009.jsondata/newsletter-labels/newsletter-010.jsondata/newsletter-labels/newsletter-011.jsondata/newsletter-labels/newsletter-012.jsondata/newsletter-labels/newsletter-013.jsondata/newsletter-labels/newsletter-014.jsondata/newsletter-labels/newsletter-015.jsondata/newsletter-labels/newsletter-016.jsondata/newsletter-labels/newsletter-017.jsondata/newsletter-labels/newsletter-018.jsondata/newsletter-labels/newsletter-019.jsondata/newsletter-labels/newsletter-020.jsondata/newsletter-labels/newsletter-021.jsondata/newsletter-labels/newsletter-022.jsondata/newsletter-labels/newsletter-023.jsondata/newsletter-labels/newsletter-024.jsondata/newsletter-labels/newsletter-025.jsondata/newsletter-labels/newsletter-026.jsondata/newsletter-labels/newsletter-027.jsondata/newsletter-labels/newsletter-028.jsondata/newsletter-labels/newsletter-029.jsondata/newsletter-labels/newsletter-030.jsondocs/env.mddocs/newsletter-evaluation.mdscripts/evaluate_newsletter_labels.py
| { | ||
| "documentId": "doc_019", | ||
| "documentTitle": "학교폭력 예방을 위한 안내문", | ||
| "documentDate": "2024-09-04", |
There was a problem hiding this comment.
문서 날짜가 과거 연도를 참조하는지 확인이 필요합니다.
documentDate가 "2024-09-04"로 설정되어 있어 대부분의 다른 문서(2025~2026년)와 연도가 다릅니다. 이것이 의도된 과거 데이터인지, 아니면 2026-09-04의 오타인지 확인이 필요합니다.
다음 스크립트로 모든 라벨 데이터의 연도 분포를 확인하세요:
#!/bin/bash
# Description: 모든 newsletter 라벨 파일의 documentDate 연도 분포 확인
fd -e json . data/newsletter-labels/ -x jq -r '.documentId + ": " + .documentDate' | sort🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@data/newsletter-labels/newsletter-019.json` at line 4, The documentDate field
in this label is "2024-09-04" which differs from most other labels (2025–2026);
verify whether this is intentional or a typo and, if it’s a typo, update the
documentDate value to the correct year (e.g., "2026-09-04") in
newsletter-019.json; to validate across the dataset, run the year-distribution
check suggested in the review (the provided script that extracts .documentDate
from all JSON labels) and correct any other inconsistent dates found.
Comment on lines
+14
to
+18
| "id": "dc_020_2", | ||
| "raw": "2025.06.19.(목) 08:00~17:30", | ||
| "resolved": "2025-06-19T08:00", | ||
| "note": "체험학습 당일 일정" | ||
| } |
There was a problem hiding this comment.
🧹 Nitpick | 🔵 Trivial | 💤 Low value
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Description: resolved 필드에 시간이 포함된 dateCandidate 찾기
fd -e json . data/newsletter-labels/ -x jq -r 'select(.dateCandidates != null) | .documentId + ": " + (.dateCandidates[] | select(.resolved | contains("T")) | .id + " -> " + .resolved)'Repository: GACHI-Project/GACHI-AI
Length of output: 870
🏁 Script executed:
#!/bin/bash
set -euo pipefail
FILE="data/newsletter-labels/newsletter-021.json"
echo "---- file header (first 80 lines) ----"
nl -ba "$FILE" | sed -n '1,90p'
echo "---- around lines 10-30 ----"
nl -ba "$FILE" | sed -n '10,30p'
echo "---- around lines 40-60 ----"
nl -ba "$FILE" | sed -n '40,70p'
echo "---- occurrences of dc_020_2 ----"
rg -n "dc_020_2" "$FILE" || trueRepository: GACHI-Project/GACHI-AI
Length of output: 166
🏁 Script executed:
#!/bin/bash
set -euo pipefail
FILE="data/newsletter-labels/newsletter-021.json"
echo "---- lines 1-90 (with line numbers via awk) ----"
awk 'NR>=1 && NR<=90 {printf "%5d:%s\n", NR, $0}' "$FILE"
echo "---- exact lines 14-18 ----"
awk 'NR>=14 && NR<=18 {printf "%5d:%s\n", NR, $0}' "$FILE"
echo "---- exact lines 47-49 ----"
awk 'NR>=47 && NR<=49 {printf "%5d:%s\n", NR, $0}' "$FILE"
echo "---- occurrences of dc_020_2 (with surrounding lines) ----"
rg -n -C 3 "dc_020_2" "$FILE" || trueRepository: GACHI-Project/GACHI-AI
Length of output: 3610
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "---- search for usage of labels.date ----"
rg -n --hidden --glob='!**/node_modules/**' '"date"\s*:\s*' data/newsletter-labels -S || true
rg -n --hidden --glob='!**/node_modules/**' 'labels[^\\n\\r]{0,80}\\.(date|selectedDateCandidateId|dateStatus)' . -S || true
rg -n --hidden --glob='!**/node_modules/**' 'selectedDateCandidateId' . -S || true
rg -n --hidden --glob='!**/node_modules/**' 'dateCandidates' . -S || true
echo "---- search for usage of dateCandidates[].resolved ----"
rg -n --hidden --glob='!**/node_modules/**' 'resolved' . -S || true
echo "---- search for any date parsing/validation patterns ----"
rg -n --hidden --glob='!**/node_modules/**' '(ISO|datetime|T\\d\\d:\\d\\d|parse.*date|날짜|시간|timezone|date\\s*format|strict)' . -S || trueRepository: GACHI-Project/GACHI-AI
Length of output: 46797
🏁 Script executed:
#!/bin/bash
set -euo pipefail
files=(
"app/schemas.py"
"scripts/evaluate_newsletter_labels.py"
"docs/newsletter-labeling-guide.md"
"app/services/newsletter_prompt.py"
"app/services/newsletter_extractor.py"
)
for f in "${files[@]}"; do
echo "---- $f (line count) ----"
wc -l "$f"
done
echo "---- app/schemas.py: around Date/Label definitions ----"
awk 'NR>=1 && NR<=140 {print NR ":" $0}' app/schemas.py
echo "---- scripts/evaluate_newsletter_labels.py: around where datetime/dateCandidates are normalized (lines ~240-360) ----"
awk 'NR>=160 && NR<=380 {print NR ":" $0}' scripts/evaluate_newsletter_labels.py
echo "---- docs/newsletter-labeling-guide.md: date format rules ----"
awk 'NR>=1 && NR<=120 {print NR ":" $0}' docs/newsletter-labeling-guide.md
echo "---- app/services/newsletter_prompt.py: dateCandidates/date usage hints ----"
awk 'NR>=1 && NR<=180 {print NR ":" $0}' app/services/newsletter_prompt.pyRepository: GACHI-Project/GACHI-AI
Length of output: 20258
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Show helper date parsing / normalization and matching logic
awk 'NR>=240 && NR<=340 {print NR ":" $0}' scripts/evaluate_newsletter_labels.py
echo "---- find and show _date_part definition ----"
rg -n "def _date_part" scripts/evaluate_newsletter_labels.py
# print around the match
line=$(rg -n "def _date_part" scripts/evaluate_newsletter_labels.py | head -n1 | cut -d: -f1)
start=$((line-20)); end=$((line+80))
awk -v s=$start -v e=$end 'NR>=s && NR<=e {print NR ":" $0}' scripts/evaluate_newsletter_labels.py
echo "---- show _normalize_date and datetime_match logic (match_items) ----"
rg -n "def _normalize_date|datetime_match|_match_accuracy|normalize_date" scripts/evaluate_newsletter_labels.py
# print around key areas roughly
awk 'NR>=380 && NR<=520 {print NR ":" $0}' scripts/evaluate_newsletter_labels.py
echo "---- show prediction_to_label_item which fills expected datetime from raw label ----"
rg -n "def prediction_to_label_item" scripts/evaluate_newsletter_labels.py
line=$(rg -n "def prediction_to_label_item" scripts/evaluate_newsletter_labels.py | head -n1 | cut -d: -f1)
start=$((line-20)); end=$((line+140))
awk -v s=$start -v e=$end 'NR>=s && NR<=e {print NR ":" $0}' scripts/evaluate_newsletter_labels.py
echo "---- show label item -> expected datetime usage when comparing ----"
rg -n "expected\.datetime|predicted.*datetime|datetime_match" scripts/evaluate_newsletter_labels.py
awk 'NR>=520 && NR<=620 {print NR ":" $0}' scripts/evaluate_newsletter_labels.pyRepository: GACHI-Project/GACHI-AI
Length of output: 21876
🏁 Script executed:
#!/bin/bash
set -euo pipefail
awk 'NR>=1 && NR<=140 {print NR ":" $0}' scripts/evaluate_newsletter_labels.pyRepository: GACHI-Project/GACHI-AI
Length of output: 4289
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n "resolved|normalizedDate|datetime|T\\d\\d:\\d\\d|ISO|format.*date" docs scripts app -SRepository: GACHI-Project/GACHI-AI
Length of output: 3255
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n '"date"\s*:\s*"\d{4}-\d{2}-\d{2}T' data/newsletter-labels -S || true
rg -n '"resolved"\s*:\s*"\d{4}-\d{2}-\d{2}T' data/newsletter-labels -S || trueRepository: GACHI-Project/GACHI-AI
Length of output: 1056
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n '"date"\s*:\s*"\d{4}-\d{2}-\d{2}T' data/newsletter-labels -S || trueRepository: GACHI-Project/GACHI-AI
Length of output: 48
newsletter-021.json 날짜/시간 포맷 규칙 정리
dateCandidates[].resolved는 "2025-06-19T08:00"처럼 시간 포함인데, 이를 참조하는 labels[].date는 "2025-06-19"처럼 날짜만 들어갑니다(14-18 vs 47-49).
다만 평가 로직(scripts/evaluate_newsletter_labels.py)은 resolved/date 모두 value[:10]로 잘라 시간은 비교에 쓰지 않으므로, 이 불일치가 “평가로 인한 데이터 손실”로 이어지진 않습니다.
스키마/가이드에 “resolved는 시간 포함 문자열이어도 라벨 date는 날짜만 사용한다(또는 resolved도 date-only 권장)” 같은 규칙을 명확히 해 혼동을 줄이는 쪽을 권장합니다.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@data/newsletter-labels/newsletter-021.json` around lines 14 - 18, The labels
file has mismatched time granularity: dateCandidates[].resolved includes time
(e.g., "2025-06-19T08:00") while labels[].date is date-only ("2025-06-19"), and
scripts/evaluate_newsletter_labels.py currently truncates values with value[:10]
when comparing; update the project guideline/schema to state a single canonical
rule (either: resolved may include time but labels[].date must be date-only, or
require both fields to be date-only) and make a corresponding change to the
dataset or docs so they match that rule; reference the keys
dateCandidates[].resolved and labels[].date in newsletter-021.json and the
comparison logic in scripts/evaluate_newsletter_labels.py to ensure consistency
(and, if choosing the date-only rule, normalize resolved to date-only across
files or update the evaluator to parse ISO datetimes consistently).
Hminkyung
approved these changes
May 25, 2026
Collaborator
Hminkyung
left a comment
Hminkyung
left a comment
There was a problem hiding this comment.
확인 완료했씁니다!!! 고생하셨습니다~~
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
📌 작업 요약
/ai/newsletters/analyze평가 스크립트 추가.gitignore정리gpt-4.1-mini로 변경🌿 브랜치 정보
feat/#7-newsletter-label-evaldevelop(기본)✅ 체크리스트
feat/refac/hotfix/chore/design/bugfix)feat/fix/refactor/docs/style/chore)🧪 테스트 결과
All checks passed!
Summary by CodeRabbit
릴리스 노트
새로운 기능
문서
기타