Add extensible tests to transformers #663
Conversation
riedgar-ms
changed the title
| # Inexact, but at least make sure not too much was produced | ||
| assert len(lm["answer"]) < 8, f"Output: {lm['answer']}" |
There was a problem hiding this comment.
I like the idea but model quality checks like this (when asserting against unconstrained generations) are a bit dangerous -- you'd be surprised how often a model with minor updates on HuggingFace might spit out an answer that e.g. isn't a number. I'd either add a regex constraint to the gen and check that the number produced can be cast to an integer or just check that the model generated something at all
There was a problem hiding this comment.
Shouldn't len() always return an integer?
| assert lm["answer"] in ["p", "t", "w"] |
There was a problem hiding this comment.
glad to see a forced grammar test here!
There was a problem hiding this comment.
My background idea is to have a smoke test for each of the ways a generation (be that gen(), select() etc.) can be invoked.
Add basic smoke tests to
transformerswhich are parameterised with the name of a Hugging Face model. This is to prevent things like #609 reoccurring.