Tests: Added integration tests for GBNF parser

[HanClinto]

Added integration tests for GBNF parser to validate correctness of grammar parsing, as well as correctness of string matching.

If we are going to increase the performance of the grammar matcher, then this will help us pin end-to-end behavior while we work on speed and memory improvements.

Might conflict a little bit with #5948, but if that one gets merged first then I'll adjust this one to match.

[HanClinto]

Okay, all cleaned up. Was able to get rid of a hacky #include "llama.cpp" that was in the first version of this test thanks to small internal API changes made in #5948, so these tests are a bit cleaner now. test-llama-grammar.cpp still has that same hacky #include:

llama.cpp/tests/test-llama-grammar.cpp

Line 5 in a74401f

#include "llama.cpp" // TODO: not great

But because that test uses more internal functions that I don't think we should expose via the internal API, I think that one is still fine for now.

Usage / expected-behavior:

./tests/test-grammar-integration
Expected error:  parse: error parsing grammar: Undefined rule identifier 'numero'
End of expected error. Test successful.

Note: This error message is a leftover from a hardcoded error message printed in the grammar parser. Rather than change that, I just added a message to alert the user that the error message is expected. Not the cleanest, but it's probably fine (?).

Otherwise, this test feels reasonably clean, and I think I'm happy with it. Ready for review / merge whenever.

Thank you!

[HanClinto]

@ochafik Pinging you on this -- if you have interest I'd love to get your feedback on the usability of the string-vs.-grammar code in here. Does this work for your JSON test needs, or is there anything else we should do to clean it up more?

I think there might be room to add some helper functions in here (or even in the core), but I didn't want to go crazy with that at first. "Duplicated code is better than the wrong abstraction", as they say.

[ochafik]

if you have interest I'd love to get your feedback on the usability of the string-vs.-grammar code in here.

@HanClinto Sorry I'm late to the party, will look into integrating your code to the json tests this week ✌️

[HanClinto]

if you have interest I'd love to get your feedback on the usability of the string-vs.-grammar code in here.

@HanClinto Sorry I'm late to the party, will look into integrating your code to the json tests this week ✌️

No rush -- whenever you get to it. Mainly wanted to make sure you knew I was following up on stuff we'd talked about earlier.

Thanks so much!