Skip to content

Improve error message when schema validation fails in validate_questions.py #573

@tahamuzammil100

Description

@tahamuzammil100

Description

When running utils/validate_questions.py, if a questions/<id>_<slug>/meta.json is missing a required field (per schemas/question.schema.json), the script outputs a stack-trace but the error message is not clearly pointing to the exact file and field causing the failure.

Steps to Reproduce

  1. Create a new question folder questions/999_test_problem/ with meta.json missing the difficulty field.
  2. Run python utils/validate_questions.py.
  3. Observe the script fails with a generic “validation error” stack-trace, but not the specific “meta.json field ‘difficulty’ missing” for folder 999_test_problem.

Expected Behaviour

The error message should clearly indicate:

  • the path of the offending folder (e.g., questions/999_test_problem/meta.json)
  • the exact missing or invalid field (e.g., difficulty)
  • optionally a link or pointer to the schema definition.

Actual Behaviour

Only a generic error is shown in the console/log and one has to dig through the stack-trace to find which question folder caused it.

Environment

  • Repository commit: latest main branch as of opening this issue
  • Python version: (e.g., 3.11)
  • OS: (e.g., macOS/Linux)

Suggestion

Modify validate_questions.py to catch schema validation exceptions and wrap them in a custom error message that includes file path, offending field, and perhaps suggestions for fix.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions