New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expanded balance point search range #72

Closed
hshaban opened this Issue Feb 1, 2018 · 28 comments

Comments

Projects
6 participants
@hshaban
Collaborator

hshaban commented Feb 1, 2018

Default grid search parameters (heating: 55-65 F, cooling: 66-75 F) may be restrictive in more extreme climates.

We propose the following:
Allowing non-overlapping user-defined grid search ranges.
Allowing the balance point search to be conducted in temperature increments up to 2 deg F.

@hshaban

This comment has been minimized.

Collaborator

hshaban commented Feb 1, 2018

Testing: For a test dataset calculate savings using Caltrack as well as arbitrary expanded search ranges

@margaretsheridan

This comment has been minimized.

margaretsheridan commented Feb 1, 2018

Explore balance point estimates by climate zone and building type? Document the diversity of results.

@steevschmidt

This comment has been minimized.

steevschmidt commented Feb 1, 2018

To achieve accurate regressions HEA has found it is necessary to adjust balance point temperatures for each home independently across a broad temperature range, even in mild climates like the SF Bay Area.

  • Due to the non-linearity of degree days, getting the right balance point temperature
    for each home is crucial to accurately disaggregate heating and cooling loads.
  • HEA has observed a 25F degree variation in balance point temperatures from homes in the same area, due largely to high internal gains (from occupants & plug loads) combined with increasingly well-insulated homes.
  • HEA utilizes a balance point range of 50-75F for heating and 65-90F for cooling.
@mcgeeyoung

This comment has been minimized.

Contributor

mcgeeyoung commented Feb 2, 2018

That's helpful. Are you able to provide test results that show differences in results if you constrain your balance points? For example, see some of the work that was done in CalTRACK 1.0 where we established a methodology for floating balance points for daily methods.

@mcgeeyoung

This comment has been minimized.

Contributor

mcgeeyoung commented Feb 9, 2018

@margaretsheridan I wonder if you could say more about your proposal? Are you suggesting that there might be variable ranges of balance points based on the particular climate zone or building type? So we would advise limiting balance point ranges in temperate climates, but allowing more extreme values in more extreme climates?

@danrubado

This comment has been minimized.

danrubado commented Feb 13, 2018

Energy Trust has also observed a wide range of balance points creating site-level models with good fits in various billing analyses over the years. We support increasing the range of balance point temperatures tested, but I'm not sure that it should be a user-defined. I think we should use the same ranges of balance points to be tested for every site, to be consistent. I don't have a specific recommendation for ranges of balance point temperatures to test, but we have often tested a 30 degree range of balance point temps for HDD and CDD. For example: HDD 40-70 and CDD 60-90.

The HDD and CDD balance point ranges should be allowed to overlap. You may want to test both HDD60/CDD60 and HDD65/CDD70. So long as you do not allow the CDD balance point to be less than HDD balance point, I think it is fine and maybe even desirable to have some overlap between the ranges of balance point values tested.

We have a slight preference for a one-degree increment in balance point temperatures tested, but it probably doesn't make that big of a difference.

@margaretsheridan

This comment has been minimized.

margaretsheridan commented Feb 13, 2018

I have been looking at balance points in reference to some work I have been doing here at SMUD and I thought we may see some variation based on a number of characteristics such as climate zone, state, building type, and building age. Looking at heating and cooling slopes and balance points could help to identify outliers in cohort groups. For example, looking at large samples of homes within California building standard vintages can provide quite a bit of insight into the efficiency and use of individual homes.

@hshaban

This comment has been minimized.

Collaborator

hshaban commented Feb 15, 2018

Proposed test methodology:

  • Caltrack monthly models will be fit to the baseline period usage data using the default search grid ranges.
  • Different search ranges will be established for the heating balance point (e.g. 45-65 F, 35-65 F) and cooling balance point (e.g. 66-85, 66-95).
  • An acceptable search range will be selected based on the acceptance criteria below.
  • Different temperature increments will be used with the selected search range (e.g. 1, 2, 3 F). Variable search increments will also be investigated
  • An acceptable maximum temperature increment will also be selected based on the acceptance criteria
  • Out of sample prediction errors will be calculated for all test cases.

Acceptance criteria:

  • For the balance point search range, the widest search range that does not inflate the out of sample errors and is physically realistic will be recommended for inclusion in the spec. Any noticeable trends by climate zone will be presented as general observations, but not necessarily in the spec. Overlapping cooling and heating search ranges will be allowed, but the cooling balance point will be required to be greater than the heating balance point.
  • For the balance point search increment, the largest possible search increment that does not inflate the out of sample errors will be recommended for inclusion in the spec.
@bkoran

This comment has been minimized.

bkoran commented Feb 15, 2018

FWIW, I use a multi-pass grid search so that a wider search increment can be used initially.
A fairly wide search range is needed for commercial buildings. The ranges posted by Steve and Dan have too high a lower limit for cooling for commercial buildings.

@hshaban

This comment has been minimized.

Collaborator

hshaban commented Mar 1, 2018

TEST RESULTS

Background
The CalTRACK model uses variable base degree day regression to model baseline and reporting period energy use. Part of this process involves fitting a number of models across a range of heating and cooling balance points. One decision point that is typically decided by an analyst, is the actual search range of balance points. A larger search range requires more computational time and may yield non-physical results (e.g. if the heating balance point is 100 F), especially when data quality is subpar. On the other hand, a search range that is too narrow may lead to poor model fit.

Dataset
Billing data from 1000 residential buildings in Oregon and daily data from 1000 participants in an HVAC program in California.

Tested parameters
Default Caltrack methods were used with different search ranges for the HDD and CDD balance points.

Results

Caltrack models were fit to the Oregon building usage dataset using 4 balance point search ranges:
• 10 degree spread: 55-65 heating, 65-75 cooling
• 20 degree spread: 45-65 heating, 65-85 cooling
• 30 degree spread: 40-70 heating, 60-90 cooling
• 40 degree spread: 40-80 heating, 50-90 cooling

Figure 1 shows the distribution of best-fit HDD balance points for three of those ranges. It is clear that when the search grid is constrained, many models tend to settle at either end of the search range (e.g. for the default search range, almost 30% of buildings have an HDD balance point of exactly 65 F). When the search range is expanded, the distribution of best-fit balance points tends towards a Gaussian distribution (centered at 63 F for this particular dataset). Therefore, we can conclude that a search grid that is too constrained may result in sub-optimal model fits.

Interestingly, Figure 2 demonstrates that even though many models may be sub-optimal, the effect on out-of-sample model performance is marginal. Only a handful of models see a significant change in out-of-sample mean absolute error in predictions and the change in overall performance for the sample is negligible.

image
Figure 1. Effect of balance point search range on distribution of best-fit HDD balance points.

image
Figure 2. Effect of balance point search range on out of sample mean absolute errors (MAE)

In a separate study, the effect of the balance point range on estimated savings was also analyzed using a dataset of HVAC program participants. Baseline models were fit to the full set of program participants five times, varying the search range for the HDD balance point and keeping all other parameters constant. Figure 3 shows clearly that the HDD search range has a negligible effect on calculated savings.

image
Figure 3. Effect of HDD balance point search range on program savings.

We also tested several search increments between 1 and 4 F and in general, the fits do not seem to be much affected when the best-fit balance point is off from the optimal one by up to 2 degrees (Figure 4). This implies that having search increments up to 3 degrees is acceptable as it ensures that the optimal temperature will always be within one degree of a point on the search grid.

image
Figure 4. Effect of varying heating balance point by 1-2 degrees on model fit.

Recommendations

  • Expand balance point search range to 40-80 F for heating and 50-90 F for cooling.
  • Allow variable search increments up to 3 F, including multipass grid search and non-uniform search increments.
@steevschmidt

This comment has been minimized.

steevschmidt commented Mar 1, 2018

Great data!
These results make sense to me, except for Figure 3...
Can anyone explain why all five results in this chart look absolutely identical?
I would have expected different types of energy savings (HVAC vs baseline) to produce variations in this test.

@hshaban

This comment has been minimized.

Collaborator

hshaban commented Mar 1, 2018

Good question, @steevschmidt . Most models didn't change with an expanded search grid and I think it's just a matter of the dense plot masking the points that did change. I went back and pulled out a 100 buildings where the optimal balance point and total savings did change and as you can see below the changes are very modest. It appears that when we change the balance point search grid, the savings breakdown (heating vs cooling vs baseload) does change quite a bit, however, the aggregate savings are relatively unchanged (usually only a 2-3% change). I'll dig in deeper when I get an opportunity.

image

@bkoran

This comment has been minimized.

bkoran commented Mar 1, 2018

@steevschmidt

This comment has been minimized.

steevschmidt commented Mar 1, 2018

Hassan wrote:

... savings breakdown (heating vs cooling vs baseload) does change quite a bit, however, the aggregate savings are relatively unchanged ...

If you have an inaccurate disaggregation of heating & cooling loads, and significant changes in weather between the base and reporting years, I believe your savings calculations will be inaccurate.

@hshaban

This comment has been minimized.

Collaborator

hshaban commented Mar 1, 2018

@bkoran @steevschmidt I agree that we'd ideally get an accurate load disaggregation. The results show that for some buildings, the balance points and load disaggregation change when expanding the search range, but we don't have the ground truth. The question raised by this issue is what search range we should explore when selecting models. Does using a wide search range give us a better chance at finding the best balance point or is a constrained one more appropriate? Your thoughts?

@steevschmidt

This comment has been minimized.

steevschmidt commented Mar 1, 2018

Does using a wide search range give us a better chance at finding the best balance point or is a constrained one more appropriate? Your thoughts?

I think wider is better, but for balance points outside of expected ranges more work can (& should?) be done. For example, I mentioned pool heating during today's call. There are homes with two different heating balance points throughout the year: one very high for pool heating during an extended summer swim period, and another during the "home heating only" parts of the year. Depending on whether your goal is "fit" or "accurate load disaggregation", one may chose different approaches.

@margaretsheridan

This comment has been minimized.

margaretsheridan commented Mar 1, 2018

I also would like to emphasize the quality of the temperature data going into the model - particularly for hourly estimation. I performed correlations of hourly weather station data by proximity to fill in missing data and to help identify outliers. This cleaning activity greatly improved my adjusted R squared values. I am now using a third-party product to ensure the quality of the weather data. This should be foundational and should be part of a standardized process.

@hshaban

This comment has been minimized.

Collaborator

hshaban commented Mar 1, 2018

I think wider is better, but for balance points outside of expected ranges more work can (& should?) be done.

I agree with you @steevschmidt . A wider range makes sense and we could take anomalous balance points into account with building qualification (those buildings may need custom models separate from the default Caltrack path)

@hshaban

This comment has been minimized.

Collaborator

hshaban commented Mar 1, 2018

@margaretsheridan Definitely useful input. If you don't mind, could you add some thoughts around weather data cleaning and how to treat missing values in Issue #65? Or any relevant literature/documentation would also help.

@bkoran

This comment has been minimized.

bkoran commented Mar 1, 2018

Also agree with margaretsheridan. We've seen cases where nearby NWS sites don't track each other; there are anomalous deviations for a period of time of months or more. (I've also seen where temperatures deviate dependent upon which way the wind is blowing, which is not anomalous.) This is in addition to the dropouts and bad data. Data cleaning, in general, is an important issue.

@bkoran

This comment has been minimized.

bkoran commented Mar 1, 2018

@bkoran

This comment has been minimized.

bkoran commented Mar 15, 2018

Again, I don't see a strong rationale for constraining the grid search.

Related to this, it is important that the daily methods are not limited to a single slope for heating and a single slope for cooling. This often doesn't match the physics of the building and can result in poorer fits. It may not mean that savings estimates will change greatly, but it probably will: Least squares slopes tend to aim at the points that are at the extreme cold or extreme warm temperatures. However, the majority of temperatures are in the more moderate range. So the slopes, and hence predictions will be off. Attached is a model for gas; similar situations for electricity can also occur.

You may consider that the attached gas use could be modeled with a negative coefficient for the cooling slope. That is possible, but the point is that a model limited to 2 slopes, or 2 slopes plus a flat range between the balance/change point temperatures, is not always sufficient for daily data.
gasusedailymodel

Since I think these methods are planned to also be used for groups of homes, attached is a model of ~200 homes, showing 3 slopes. This is from Figure 16 in

@bkoran

This comment has been minimized.

bkoran commented Mar 15, 2018

Since I think these methods are planned to also be used for groups of homes, attached is a daily model of ~200 homes, showing 3 slopes. This is from Figure 16 in
http://www.calmac.org/publications/AMI_Report_Vol_2_Appendices_FINAL.pdf
group of 200 homes

@mcgeeyoung

This comment has been minimized.

Contributor

mcgeeyoung commented Mar 15, 2018

@bkoran I've been thinking a lot about the "elbow" issue. There's no doubt that it's observable. There are three questions that seem important to answer about it. First, what does the poor model fit do to savings estimates? If the model errors are evenly distributed over the course of a year, would we be concerned about its effects? Second, are there quantitative criteria for determining when an "elbow" would be valuable to search for? Should it be default? Should it be configurable? Should it be based on probability of occurrence? Finally, how do we test for this? The image you showed above is a kW function, not a kWh function. Is that relevant?

It strikes me that when we get to hourly modeling that this is going to be a much more relevant conversation. Don't give up on this. We are benefitting from your insights on this issue. We just need a way to generalize it and test it so that it can be fit into the methods specification in a clear, broadly applicable fashion.

@bkoran

This comment has been minimized.

bkoran commented Mar 15, 2018

I can't answer most of your questions very well. I covered savings up above; I do think it can have an effect. It could be that it's effects could be minimized by using a different fit than least squares; not sure about that.
There are physical characteristics when an "elbow" should be searched for, but I don't know how to know when analytically except by brute force trying and testing significance.
I don't think it should be default; it is common but not the most common.
Configurable? Don't know what you mean.
kW vs. kWh is just time integration; don't think it matters

@bkoran

This comment has been minimized.

bkoran commented Mar 15, 2018

A bit more--this is more specific to hourly models, although it can affect daily models:
One of the most common measures in building commissioning is improving economizer operation, i.e. cooling with outside air. Economizers operate at moderate temperatures, from the cooling change/balance point up to somewhere between 62 and 73 degF, depending on climate and humidity. In a model they are characterized as having a steeper slope than the rest of the cooling. Since the savings are at temperatures close to the change/balance point, I believe the effect of this improvement would be all but invisible if only a single slope is used for cooling. Maybe that is good enough in a whole building project with many measures, but this is a very common measure, and the 2 slopes are often obvious when an economizer is performing well. (Corollary is that the base case often doesn't show it, but the post case does.)

The first 2 charts below are EnergyPlus models of a DOE prototype building. The first is a daily model, the second hourly. The third chart is an hourly model of a real building.

daily energyplus show economizer
hourly energyplus show economizer
hourly real bldg show economizer

@margaretsheridan

This comment has been minimized.

margaretsheridan commented Mar 15, 2018

When I was forecasting hourly system load for systems operations at SMUD years ago, I used multiple techniques including a regression with a third order polynomial for the temperature-load relationship. This would track fairly well with piecewise linear load-temp relationship. The greater simplicity of modeling would need to be examined relative to any loss in accuracy.

@hshaban hshaban moved this from In progress to Done in CalTRACK Mar 30, 2018

@hshaban

This comment has been minimized.

Collaborator

hshaban commented Jul 26, 2018

This update has been integrated in CalTRACK 2. Closing this issue

@hshaban hshaban closed this Jul 26, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment