Join GitHub today
CalTRACK Issue: Vote on/agree to uniform testing approach #129
Article reference number in CalTRACK documentation (optional): 126.96.36.199 and 188.8.131.52 indicate this is in scope, but it's not currently directly addressed in the methods
I'd like to propose that this group agree to a uniform approach we would use to evaluate changes to modeling approaches and/or new modeling approaches. This is motivated by a desire to have a standard understanding of how versions/updates improve CalTRACK, and to take the burden of determining a testing approach off of group members who want to propose a change.
The testing approach the group settles on should include a process for testing and appropriate metrics (note: I'm talking about modeling here; you might probably need different metrics for other sections of the methods). The metrics should be able to be applied between models with data at different time resolutions (e.g., hourly and daily), and applied in out-of-sample testing. We'd also need to discuss what the counterfactual against which you'd compare would be (current CalTRACK? An older version?).
Ideally this approach would come with a standard data set (or a few), but that is not necessarily required.
Proposed test methodology
A supermajority or consensus vote of group members would choose a testing methodology.
Another way of going about this would be to submit a test methodology with a particular issue and reach consensus on the testing protocol prior to working on the issue. There are likely going to be different testing requirements and different thresholds for different issues and probably different data requirements as well. If we try to set this up beforehand, it's likely that we'll spend all of our time creating exceptions to the rules we've laid down for ourselves.