You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When submitting an answer-changing (or metadata-changing) PR, regression tests fail as expected, but in cases where the change is intentional, there should be a way to indicate the new, correct answer independently of the updated code.
Expected Behavior
Regression Tests should pass if code changes produce a different, but expected answer.
Current Behavior
Regression Tests fail due to changed values, even if changed values are deemed correct externally.
Possible Solution
Provide a file with expected new results, which also creates traceable documentation of the evolution of changes
Related Issues
#50 Testing: checks for data and metadata should be separated
The text was updated successfully, but these errors were encountered:
When submitting an answer-changing (or metadata-changing) PR, regression tests fail as expected, but in cases where the change is intentional, there should be a way to indicate the new, correct answer independently of the updated code.
Expected Behavior
Regression Tests should pass if code changes produce a different, but expected answer.
Current Behavior
Regression Tests fail due to changed values, even if changed values are deemed correct externally.
Possible Solution
Provide a file with expected new results, which also creates traceable documentation of the evolution of changes
Related Issues
#50 Testing: checks for data and metadata should be separated
The text was updated successfully, but these errors were encountered: