Assertion failed: this_val < max_bin_val intra_mode_search.c, line 359 when running avif_example_encode.c with depth > 8

[lumesca-neil]

Issue Description:
When attempting to run the avif_example_encode.c example with a targeted bit depth higher than 8, an assertion failure occurs. The specific error is:

Assertion failed: this_val < max_bin_val, file C:\Users\nward\Development\experimental\libavif\ext\aom\av1\encoder\intra_mode_search.c, line 359

Steps to Reproduce:

1. Use avif_example_encode.c with a bit depth higher than 8. I have tried 10 and 12
2. Set encodeYUVDirectly to 1.
3. The issue occurs with the avifEncoderCreate() is called.

Expected Behavior:
The example should run without assertion failures even when using a bit depth higher than 8.

Additional Information:

• Operating System: Win 10 - x64
• Compiler and Version: VC++ 2019
• libavif Version: main - commit f7ce882

Code Snippet:

avifImage* image = avifImageCreate(128, 128, 10, AVIF_PIXEL_FORMAT_YUV444);

Note:
The issue does not occur with a bit depth of 8, indicating that the problem might be related to the handling of higher bit depths.

---

[y-guyon]

Upstream libaom issue:
https://bugs.chromium.org/p/aomedia/issues/detail?id=3402#c3