Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Metrics for assessing model fit for a portfolio of hourly-modeled buildings #99

Closed
eliotcrowe opened this issue May 24, 2018 · 2 comments

Comments

@eliotcrowe
Copy link

commented May 24, 2018

#85 At a previous meeting we saw OpenEE's work on weighted portfolio CV(RMSE), and I wanted to jot down some LBNL thinking on that. If RMSE is squared before adding up the values, then the output is a statistically valid metric, an aggregated CV(RMSE). Two cautions though, when interpreting the results:

  1. A guidance value of <25% has been suggested for CV(RMSE) for a single building, but there is no evidence that the SAME guidance value is appropriate for an aggregated value for a portfolio. The metric may be appropriate but setting an acceptable level for that metric needs more work.
  2. When discussing, we should be clear that this is a model fitness metric, not a quantification of savings uncertainty

We're happy to chat more on this (though I'm not the right person to get deeper in the weeds myself)

image

@hshaban

This comment has been minimized.

Copy link
Collaborator

commented May 24, 2018

Thanks @eliotcrowe. I think we're on the same page - here are some of the conclusions that were buried in the 50 or so threads in #71. in general, we think the aggregate CVRMSE might be a decent metric for planning and trend analysis, but we definitely can't determine the actual values of appropriate thresholds without more testing, because they would vary depending on many different factors.

CVRMSE is not savings uncertainty - CVRMSE is a measure of model fitness and can be used to evaluate how closely a model represents a building's energy use. Fractional savings uncertainty (FSU) is the bounds within which savings are expected to lie (e.g. Savings = 100 kWh +/- 10%). Lower FSU means higher confidence in your results. We are recommending portfolio FSU as the main metric to evaluate portfolio-level results in certain applications. CVRMSE can be used to screen individual buildings prior to an EE intervention, and aggregates of CVRMSE at the portfolio-level may serve as proxies for uncertainty (only the trends, not the actual values). This could be useful in some cases for planning, because calculating FSU requires knowledge of the actual or estimated savings.

@hshaban

This comment has been minimized.

Copy link
Collaborator

commented Jul 26, 2018

Closing this - Hourly model metrics will be further investigated in #92

@hshaban hshaban closed this Jul 26, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.