-
Notifications
You must be signed in to change notification settings - Fork 183
Open
Description
Description
When running utils/validate_questions.py, if a questions/<id>_<slug>/meta.json is missing a required field (per schemas/question.schema.json), the script outputs a stack-trace but the error message is not clearly pointing to the exact file and field causing the failure.
Steps to Reproduce
- Create a new question folder
questions/999_test_problem/withmeta.jsonmissing thedifficultyfield. - Run
python utils/validate_questions.py. - Observe the script fails with a generic “validation error” stack-trace, but not the specific “meta.json field ‘difficulty’ missing” for folder 999_test_problem.
Expected Behaviour
The error message should clearly indicate:
- the path of the offending folder (e.g.,
questions/999_test_problem/meta.json) - the exact missing or invalid field (e.g.,
difficulty) - optionally a link or pointer to the schema definition.
Actual Behaviour
Only a generic error is shown in the console/log and one has to dig through the stack-trace to find which question folder caused it.
Environment
- Repository commit: latest main branch as of opening this issue
- Python version: (e.g., 3.11)
- OS: (e.g., macOS/Linux)
Suggestion
Modify validate_questions.py to catch schema validation exceptions and wrap them in a custom error message that includes file path, offending field, and perhaps suggestions for fix.
Metadata
Metadata
Assignees
Labels
No labels