Make validation errors less verbose

[JosephMarinier]

Description:

Here is an example of a response to a PATCH /config with an invalid config:

• Before:

1 validation error for AzimuthConfig
uncertainty -> iterations
ensure this value is greater than or equal to 1 (type=value_error.number.not_ge; limit_value=1)

• After:

AzimuthConfig['uncertainty']['iterations']: ensure this value is greater than or equal to 1

And here is an extreme example with 4 errors (a, b, c and d) in a GET /dataset_splits/c/utterances?outcome=a&outcome=b&sort=d&pipeline_index=0

• Before:

5 validation errors for Request
query -> outcome -> 0
value is not a valid enumeration member; permitted: 'CorrectAndPredicted', 'CorrectAndRejected', 'IncorrectAndPredicted', 'IncorrectAndRejected' (type=type_error.enum; enum_values=[<OutcomeName.CorrectAndPredicted: 'CorrectAndPredicted'>, <OutcomeName.CorrectAndRejected: 'CorrectAndRejected'>, <OutcomeName.IncorrectAndPredicted: 'IncorrectAndPredicted'>, <OutcomeName.IncorrectAndRejected: 'IncorrectAndRejected'>])
query -> outcome -> 1
value is not a valid enumeration member; permitted: 'CorrectAndPredicted', 'CorrectAndRejected', 'IncorrectAndPredicted', 'IncorrectAndRejected' (type=type_error.enum; enum_values=[<OutcomeName.CorrectAndPredicted: 'CorrectAndPredicted'>, <OutcomeName.CorrectAndRejected: 'CorrectAndRejected'>, <OutcomeName.IncorrectAndPredicted: 'IncorrectAndPredicted'>, <OutcomeName.IncorrectAndRejected: 'IncorrectAndRejected'>])
path -> dataset_split_name
value is not a valid enumeration member; permitted: 'eval', 'train', 'all' (type=type_error.enum; enum_values=[<DatasetSplitName.eval: 'eval'>, <DatasetSplitName.train: 'train'>, <DatasetSplitName.all: 'all'>])
path -> dataset_split_name
value is not a valid enumeration member; permitted: 'eval', 'train', 'all' (type=type_error.enum; enum_values=[<DatasetSplitName.eval: 'eval'>, <DatasetSplitName.train: 'train'>, <DatasetSplitName.all: 'all'>])
query -> sort
value is not a valid enumeration member; permitted: 'index', 'utterance', 'label', 'prediction', 'confidence' (type=type_error.enum; enum_values=[<UtterancesSortableColumn.index: 'index'>, <UtterancesSortableColumn.utterance: 'utterance'>, <UtterancesSortableColumn.label: 'label'>, <UtterancesSortableColumn.prediction: 'prediction'>, <UtterancesSortableColumn.confidence: 'confidence'>])

• After:

query parameter outcome=a: value is not a valid enumeration member; permitted: 'CorrectAndPredicted', 'CorrectAndRejected', 'IncorrectAndPredicted', 'IncorrectAndRejected'
query parameter outcome=b: value is not a valid enumeration member; permitted: 'CorrectAndPredicted', 'CorrectAndRejected', 'IncorrectAndPredicted', 'IncorrectAndRejected'
path parameter dataset_split_name=c: value is not a valid enumeration member; permitted: 'eval', 'train', 'all'
query parameter sort=d: value is not a valid enumeration member; permitted: 'index', 'utterance', 'label', 'prediction', 'confidence'

Checklist:

You should check all boxes before the PR is ready. If a box does not apply, check it to acknowledge it.

• ISSUE NUMBER. You linked the issue number (Ex: Resolve #XXX).
• PRE-COMMIT. You ran pre-commit on all commits, or else, you
  ran pre-commit run --all-files at the end.
• USER CHANGES. The changes are added to CHANGELOG.md and the documentation, if they impact
  our users.
• DEV CHANGES.
  • Update the documentation if this PR changes how to develop/launch on the app.
  • Update the README files and our wiki for any big design decisions, if relevant.
  • Add unit tests, docstrings, typing and comments for complex sections.

[gabegma]

So cool!!

[JosephMarinier]

I don't understand the problem with the documentation test failing. It has hardly anything to do with this PR, so I included a simple workaround, and we should look into it in another PR. Ready for review!

[gabegma]

LGTM!
I don't understand all the details, but thank you for the thorough testing!

It's probably outside the scope of this PR, but when I was playing with it, I put the threshold in the config page to -50%, and got these surprising warnings:

I only expected the last one.

[JosephMarinier]

@gabegma

I put the threshold in the config page to -50%, and got these surprising warnings:
I only expected the last one.

That happens with all Union types. The postprocessors is a list of Union[TemperatureScaling, ThresholdConfig, CustomObject], so Pydantic tries to parse

{
  "class_name": "azimuth.utils.ml.postprocessing.Thresholding",
  "args": [],
  "kwargs": {"threshold": -0.5},
  "remote": null,
  "threshold": -0.5
}

into those 3 classes, and gives you the reason why each one is failing. It can't be a TemperatureScaling since class_name is incompatible, it can't be a CustomObject since the field threshold doesn't exist, and finally, it can't be a ThresholdConfig because the value of threshold is below 0.

I was not able to figure out what the order of the errors means, so I didn't feel confident enough to assume that the last one is always the relevant one, for example. Also, you really could make different errors in the config, and getting all the errors back at once is pretty neat. So, we'll have to play more with it and come up with ideas on how to improve.

[JosephMarinier]

Another example in the config would be use_cuda: Union[Literal["auto"], bool].
If we try to PATCH the config with

{"use_cuda": "potato"}

we get

AzimuthConfig['use_cuda']: unexpected value; permitted: 'auto'
AzimuthConfig['use_cuda']: value could not be parsed to a boolean

which makes sense.

[gabegma]

Thank you for the thorough explanation!!