문제 검토 파이프라인 추가 (export, review, 판정) - #55
Merged
Merged
Conversation
scripts/export-challenges pulls SQL challenge definitions from a CTFd through the admin API (token from the environment), and scripts/review-challenges starts a disposable MySQL judge from the checkout, grades every definition with scripts/regrade-challenges --review, prints authoring findings, and removes the containers. The review findings cover execution failures, missing ORDER BY, repeated sort-key values, ORDER BY expressions the tool cannot check, unrounded AVG/division results, empty or near-limit results, and slow requests. REVIEW.md documents the procedure and how to act on each finding so a future TA or an agent can run it unchanged.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
목적
"문제 검토해 줘"라고 하면 조교나 에이전트가 그대로 돌릴 수 있는 한 줄짜리 파이프라인과, 다음 학기 조교가 문서만 보고 따라 할 수 있는 절차를 만듭니다.
변경
scripts/export-challenges: 관리자 access token(환경 변수)으로 CTFd에서 SQL 문제 정의를 JSON으로 내보냅니다.scripts/review-challenges: 일회용 MySQL 8.4 + 이 checkout의 judge를 Compose로 띄워 전 문제를 채점하고 검토 판정을 출력한 뒤 정리합니다. judge는 localhost에만 노출됩니다(docker-compose.review.yml).scripts/regrade-challenges --review: 실행 실패, ORDER BY 없음, 정렬 키 동률 행 수, 자동 확인 불가한 정렬 식, ROUND 없는 AVG·나눗셈, 0행·500행 이상, 느린 요청을 문제별로 판정합니다.REVIEW.md: 절차와 판정별 조치 표. README와 AUTHORING.md에서 링크합니다.tests/test_regrade_review.py: 판정 함수 단위 테스트.검증
지난 학기 문제 61개에 실행한 결과 34개가 확인 대상으로 분류됐습니다. ORDER BY 없음 8, 정렬 키 동률 4(수동 분석에서 옛 엔진과 순서가 달랐던 문제와 일치), 정렬 식 확인 불가 3, ROUND 없는 집계 14, 실행 실패 1(
#32ambiguous GROUP BY)입니다. 문제 SQL 자체는 저장소에 넣지 않았습니다.