You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The batch_consistency_test.py files check that behaviour is consistent on batches vs. items, but the current infrastructure does this with batches of identical items. This approach may miss bugs related to data leakage between items within a batch, for example #994. I suggest items should be different to ensure those kinds of issues are flagged.
I have an alternate implementation here which uses batches with different items (where possible). Would you like me to open a PR?
(I've just implemented it for the functional tests for now, but I can also expand it to the root batch_consistency_check.py too.)
The text was updated successfully, but these errors were encountered:
jcaw
changed the title
Batch consistency should not be tested with a batch of identical items
Should batch consistency be tested with a batch of non-identical items?
Feb 22, 2021
Thanks for the suggestion. I although thought a similar thing when reviewing #994. I briefly looked at your code and basically, your approach is to pass a batch to the helper method and run it separately and individually, correct? This looks like a good improvement. Please open a PR.
The
batch_consistency_test.py
files check that behaviour is consistent on batches vs. items, but the current infrastructure does this with batches of identical items. This approach may miss bugs related to data leakage between items within a batch, for example #994. I suggest items should be different to ensure those kinds of issues are flagged.I have an alternate implementation here which uses batches with different items (where possible). Would you like me to open a PR?
(I've just implemented it for the
functional
tests for now, but I can also expand it to the rootbatch_consistency_check.py
too.)The text was updated successfully, but these errors were encountered: