Ehale ignore measures

[elainethale]

This branch adds the feature of being able to having the ResStock BuildExistingModel ignore measures. There will be a companion pull request to buildstockbatch that passes that argument through the project yml.

[elainethale]

Will screen out our project-specific measures to better isolate the feature I am requesting to pull in.

[shorowit]

@elainethale What is the example for which you want to exclude a given measure?

[shorowit]

The reason I ask is that it can be dangerous to remove measures from the residential workflow. Maybe there is a better way to accomplish your goal?

[elainethale]

Our use case is doing flexibility estimation. We have DR measures that we add to the workflow. Those measures have attributes whose values we want to randomly sample with the standard ResStock process, but then we want to run both a baseline and perturbed case. measures_to_ignore lets us list our added DR measures to run the baseline; running the full model produces the perturbed case. We are not proposing to ignore the main ResStock measures. (We've been discussing with @ejhw, @nmerket and @rHorsey.)

[elainethale]

@shorowit, I believe the circleci checks are not running because my branch is on a fork. I was able to push my analogous buildstockbatch branch over to the NREL/buildstockbatch repo, which let the circleci tests run there. Would you like to give me permissions to do the same here?

[codecov-io]

Codecov Report

Merging #310 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #310   +/-   ##
=======================================
  Coverage   93.12%   93.12%           
=======================================
  Files         168      168           
  Lines       48926    48926           
=======================================
  Hits        45560    45560           
  Misses       3366     3366

Impacted Files	Coverage Δ
resources/buildstock.rb	65.76% <ø> (ø)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 45f9a1d...173766f. Read the comment docs.

[codecov-io]

Codecov Report

Merging #310 into master will decrease coverage by <.01%.
The diff coverage is 10%.

@@            Coverage Diff             @@
##           master     #310      +/-   ##
==========================================
- Coverage   93.12%   93.11%   -0.01%     
==========================================
  Files         168      168              
  Lines       48926    48931       +5     
==========================================
+ Hits        45560    45561       +1     
- Misses       3366     3370       +4

Impacted Files	Coverage Δ
Rakefile	92.77% <ø> (ø)	⬆️
resources/buildstock.rb	65.76% <ø> (ø)	⬆️
resources/meta_measure.rb	69.62% <10%> (-1.63%)	⬇️
project_testing/tests/test.rb
project_singlefamilydetached/tests/test.rb	83.33% <0%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 45f9a1d...58d29c2. Read the comment docs.

[joseph-robertson]

Since you're not intending to ignore any of the main ResStock measures, I guess this seems fine. However, I'm a little concerned this leaves the door open for other users to try to ignore main ResStock measures, which could be problematic.

Perhaps users shouldn't be permitted to ignore any measures found in https://github.com/NREL/OpenStudio-BuildStock/blob/master/resources/measure-info.json (the main ResStock measures)? Though it looks like this would probably break some of your buildstockbatch tests...

[joseph-robertson]

Good catch.

[joseph-robertson]

Another good catch. Move this down where it's actually used.

[elainethale]

I understand your concern about people ignoring the main ResStock measures. Some ideas:

• Don't include measures_to_ignore in the example project .yml file at all. (Currently it is there but commented out.)
• Output a warning if any of the measures in the .json are ignored
• Add a class attribute to the workflow generator that lists un-ignorable measures. Maybe provide a __test flag to skip those checks for testing purposes.
• I'm also open to testing in a different way, if you have any particular suggestions for how to do that.

On the pythonic principle of expecting people to act responsibly, I suggest implementing the first two bullets. What would you prefer?

[shorowit]

I have to admit that I'm not convinced this is needed. It seems trivial to handle the needed DR workflows via TSV files without adding this capability (that can be misused).

For example, create an Enable DR.tsv with options Yes and No. Set all other DR TSVs to use the Enable DR.tsv as a dependency. Then you just have to set the probability to 1.0 for Yes or No to achieve the desired output.

My two cents.

[ejhw]

I haven't been following this closely, but I had a similar thought to Scott's example. Is there a reason that won't work?

[joseph-robertson]

We used a similar approach to what Scott is mentioning for @jonwinkler's installation quality project: https://github.com/NREL/OpenStudio-BuildStock/blob/install-qual_both_fault/project_singlefamilydetached/housing_characteristics/HVAC%20Airflow%20Is%20Faulted.tsv.

[rHorsey]

@shorowit and @ejhw - This will also be useful for us in commercial. We don't have the same measure workflow dependence that y'all do and we don't use TSV files in the same way as you do analytically. For us, being able to turn on and off measures in the build existing model will be both useful and reasonable, and not having it right now leads to me doing stupid things manually with a high risk of transcription-type error introduction. If people turn off measures (particularly if the first two bullets in @elainethale's suggestion are implemented) they are making an affirmative decision that they will need to be responsible for, and so long as @elainethale makes sure that the documentation of the feature warns in red bolded text I don't see any issue. Again, if this was just a residential workflow issue I'd defer, but this will be useful for commercial usage and a reasonable and appropriate feature for us to use in the future.

[elainethale]

Steps to prevent feature misuse:

• BuildExistingModel will error out if ResStock core measures are ignored
• Add appropriately cautionary language to the argument description (check how it reads in PAT)
• Add a clear warning to the buildstockbatch documentation
• Update the three PAT projects (new version of the measure and re-populate argument values)
• Add write up and new PAT graphic to OpenStudio-BuildStock documentation
• Remove measures_to_ignore from example yml file

[elainethale]

Example output when you try to ignore a core ResStock measure (from singularity_output.log):

[22:37:19.831250 ERROR] Core ResStock measure ResidentialApplianceClothesDryer cannot be ignored
[22:37:19.831314 ERROR] Core ResStock measure ResidentialApplianceClothesDryer cannot be ignored
	/measures/BuildExistingModel/measure.rb:180:in `block in run'
	/measures/BuildExistingModel/measure.rb:175:in `each'
	/measures/BuildExistingModel/measure.rb:175:in `run'
	:/ruby/2.2.0/gems/openstudio-workflow-1.3.3/lib/openstudio/workflow/util/measure.rb:467:in `apply_measure'
	:/ruby/2.2.0/gems/openstudio-workflow-1.3.3/lib/openstudio/workflow/util/measure.rb:74:in `block in apply_measures'
	:/ruby/2.2.0/gems/openstudio-workflow-1.3.3/lib/openstudio/workflow/util/measure.rb:32:in `each_index'
	:/ruby/2.2.0/gems/openstudio-workflow-1.3.3/lib/openstudio/workflow/util/measure.rb:32:in `apply_measures'
	:/ruby/2.2.0/gems/openstudio-workflow-1.3.3/lib/openstudio/workflow/jobs/run_os_measures.rb:52:in `perform'
	:/ruby/2.2.0/gems/openstudio-workflow-1.3.3/lib/openstudio/workflow/run.rb:269:in `step'
	:/ruby/2.2.0/gems/openstudio-workflow-1.3.3/lib/openstudio/workflow/run.rb:216:in `run'
	:/openstudio_cli.rb:975:in `execute'
	:/openstudio_cli.rb:763:in `execute'
	:/openstudio_cli.rb:1746:in `<main>'
	eval:110:in `eval'
	eval:110:in `require_embedded_absolute'
	eval:95:in `block in require'
	eval:89:in `each'
	eval:89:in `require'
	eval:3:in `<main>'
[22:37:19.831477 ERROR] :/ruby/2.2.0/gems/openstudio-workflow-1.3.3/lib/openstudio/workflow/util/measure.rb failed with message Runner error :/ruby/2.2.0/gems/openstudio-workflow-1.3.3/lib/openstudio/workflow/util/measure.rb failed with Core ResStock measure ResidentialApplianceClothesDryer cannot be ignored, /measures/BuildExistingModel/measure.rb:180:in `block in run'

Also, failed.job does show up in the data point run folder.

[elainethale]

@joseph-robertson @nmerket -- I just resolved the conflict that showed up in NREL/buildstockbatch#102. Once the checks are re-run there, assuming that comes out well, who should pull both of these code changes in?

[nmerket]

I'll merge NREL/buildstockbatch#102. @joseph-robertson can do this one.

[elainethale]

Looks like NREL/buildstockbatch#102 is good now, except for having lost its "approved review" status.

[elainethale]

Thanks @joseph-robertson!