Fix #5147 and generally modernize measure templates + get openstudio to run pytest #5149

jmarrec · 2024-04-16T16:37:57Z

Pull request overview

Fixes Issue with runner.lastEnergyPlusSqlFile #5147

Pull Request Author

Model API Changes / Additions
Any new or modified fields have been implemented in the EnergyPlus ForwardTranslator (and ReverseTranslator as appropriate)
Model API methods are tested (in src/model/test)
EnergyPlus ForwardTranslator Tests (in src/energyplus/Test)
If a new object or method, added a test in NREL/OpenStudio-resources: Add Link
If needed, added VersionTranslation rules for the objects (src/osversion/VersionTranslator.cpp)
Verified that C# bindings built fine on Windows, partial classes used as needed, etc.
All new and existing tests passes
If methods have been deprecated, update rest of code to use the new methods

Labels:

If change to an IDD file, add the label IDDChange
If breaking existing API, add the label APIChange
If deemed ready, add label Pull Request - Ready for CI so that CI builds your PR

Review Checklist

This will not be exhaustively relevant to every PR.

Perform a Code Review on GitHub
Code Style, strip trailing whitespace, etc.
All related changes have been implemented: model changes, model tests, FT changes, FT tests, VersionTranslation, OS App
Labeling is ok
If defect, verify by running develop branch and reproducing defect, then running PR and reproducing fix
If feature, test running new feature, try creative ways to break it
CI status: all green or justified

* make it clear that if you use runner.lastEnergyPlusSQLFile you need to reopen it potentially * Use a chdir context

…ated functions to get warnings/errors (stepWarnings, not warnings)

…) + upgrade openstudio measure --examples

It's painfully slow to have to translate it from 1.13.0 to 3.7.0 everytime. Maybe we could consider using 3.2.0 or something like that, but there's no point keeping it so old

…====================== test session starts ============================== platform linux -- Python 3.8.18, pytest-7.4.2, pluggy-1.3.0 -- /home/julien/Virtualenvs/py39/bin/python3 cachedir: .pytest_cache rootdir: /home/julien/Software configfile: setup.cfg collecting ... collected 0 items -- generated xml file: /home/julien/Software/Others/OpenStudio/ruby/junit.xml -- ============================ no tests ran in 0.05s =============================

…un the minitest/pytest on it

…ed, missed this one ``` E:145: 56: [Correctable] Style/StringLiterals: Prefer single-quoted strings when you don't need string interpolation or special symbols. E:162: 56: [Correctable] Style/StringLiterals: Prefer single-quoted strings when you don't need string interpolation or special symbols. E:171: 35: [Correctable] Style/StringLiterals: Prefer single-quoted strings when you don't need string interpolation or special symbols. NoMethodError: undefined method `taint' for "/home/julien/.cache/rubocop_cache":String ```

…ed you, so I forgive you) https://godbolt.org/z/Wo7dPqvs3 ```c++ #include <fmt/compile.h> #include <fmt/format.h> #include <type_traits> #include <variant> using VariantType = std::variant<int, unsigned int>; template <class> inline constexpr bool always_false_v = false; constexpr void test_type(auto&& arg) { using T = std::decay_t<decltype(arg)>; if constexpr (std::is_same_v<T, unsigned int>) { fmt::print(FMT_COMPILE("Hey there unsigned! cppcheck thought you wouldn't be here\n")); } else if constexpr (std::is_same_v<T, int>) { fmt::print(FMT_COMPILE("Hello integer, how are you?\n")); } else { static_assert(always_false_v<T>, "non-exhaustive visitor!"); } } constexpr void test_variant_type(const VariantType& arg) { std::visit( [](auto&& arg) { using T = std::decay_t<decltype(arg)>; if constexpr (std::is_same_v<T, int>) { fmt::print(FMT_COMPILE("Hello integer, how are you?\n")); } else if constexpr (std::is_same_v<T, unsigned int>) { fmt::print(FMT_COMPILE("Hey there unsigned! cppcheck thought you wouldn't be here\n")); } else { static_assert(always_false_v<T>, "non-exhaustive visitor!"); } }, arg); } int main() { int i = 10; test_type(i); unsigned u = 10; test_type(u); fmt::print("Variant\n"); auto iv = VariantType(i); auto uv = VariantType(u); test_variant_type(iv); test_variant_type(uv); } ```

jmarrec · 2024-04-17T22:12:47Z

src/cli/MeasureUpdateCommand.cpp

+      auto canonicalTestDir = openstudio::filesystem::canonical(opt.directoryPath);
+
+      // Pytest
+      fmt::print(fmt::fg(fmt::color::yellow),
+                 "┌{0:─^{2}}┐\n"
+                 "│{1: ^{2}}│\n"
+                 "└{0:─^{2}}┘",
+                 "", "Starting Python Tests", 80);
+      fmt::print("\n");
+      auto pytestOutDir = canonicalTestDir / "test_results" / "pytest";
+
+      // -o junit_familar=legacy is to mimic having a pytest.ini with:
+      // [pytest]
+      // junit_family=legacy
+      // This uses xunit1 format, and adds the line + file at which error occured, something we don't have with xunit2 (default)
+
+      auto runPytestCmd = fmt::format(
+        R"python(
+import pytest
+
+pytest.main([
+  "--junit-xml={junit}",
+  "--cov={test_dir}",
+  "--cov-report=term-missing",
+  "--cov-report=json:{json}",
+  "--cov-report=lcov:{lcov}",
+  "--cov-report=html:{html}",
+  "--cov-report=xml:{xml}",
+  "-o", "junit_family=legacy",
+  "{test_dir}",
+])
+)python",
+        fmt::arg("junit", (pytestOutDir / "junit.xml").generic_string()),  //
+        fmt::arg("test_dir", canonicalTestDir.generic_string()),           //
+        fmt::arg("json", (pytestOutDir / "python_coverage.json").generic_string()),
+        fmt::arg("lcov", (pytestOutDir / "python_coverage.lcov").generic_string()),
+        fmt::arg("html", (pytestOutDir / "python_coverage_html").generic_string()),
+        fmt::arg("xml", (pytestOutDir / "python_coverage.xml").generic_string()));
+
+      fmt::print("runPytestCmd={}\n", runPytestCmd);
+      pythonEngine->exec(runPytestCmd);
+
+      fmt::print(fmt::fg(fmt::color::red),
+                 "┌{0:─^{2}}┐\n"
+                 "│{1: ^{2}}│\n"
+                 "└{0:─^{2}}┘",
+                 "", "Starting ruby tests", 80);
+      fmt::print("\n");


openstudio measure --run-tests will now run Pytests with coverage

this won't be on the dashboard yet.

@DavidGoldwasser @kbenne will need to figure out who should trigger pytest, the C++ CLI or the measure tester gem.

Anyways, for now, I think like this we could just tweak measure-tester-gem to consume the pytest xmls if they are present, since the ruby code runs after the python one.

But if you use measure-tester as a gem and use the rake task, then you'd have to manually run pytest.

This is great. We can try to do work in measure tester gem shortly after 3.8.0 is out to include in 3.8.x or 3.9.0. Many Python measures may also use additional python libraries. In the case when their is extra libraries, I wouldn't expect tests to run with out of the box install without some extra configuration for the additional libraries.

it should work as long as you pass --python_path options to the CLI.

--python_path DIR Add additional directory to add to front of PYTHONPATH environment variable (may be used more than once)

jmarrec · 2024-04-17T22:13:34Z

python/requirements.txt

+jinja2 == 3.1.3
 numpy == 1.24.4
 pandas == 2.0.3
-pytest == 7.4.2
+pytest == 8.1.1
+coverage == 7.4.4
+pytest-cov == 5.0.0


Bumped pytest and added coverage / pytest-cov.

I bumped jinja2 too.

pandas and numpy are frozen because we use python 3.8

jmarrec · 2024-04-17T22:13:56Z

ruby/engine/embedded_help.rb

@@ -806,7 +806,7 @@ def self.find(*paths, ignore_error: true)
      ps = [path]
      while file = ps.shift
        catch(:prune) do
-          yield file.dup.taint
+          yield file.dup # .taint


That was leading to a nasty error when running rubocop via measure tester

jmarrec · 2024-04-17T22:14:25Z

src/utilities/bcl/BCLMeasure.cpp

@@ -59,7 +59,7 @@ static constexpr std::array<std::pair<std::string_view, std::string_view>, 3> ap
  {"resources", "resource"},
 }};

-static constexpr std::array<std::string_view, 1> ignoredSubFolders{"tests/output"};
+static constexpr std::array<std::string_view, 3> ignoredSubFolders{"tests/output", "__pycache__", "tests/__pycache__"};


When updating BCLMeasure / copying it, do not grab __pycache__

At some point we may want to also not grab anything .gitignored

jmarrec · 2024-04-17T22:15:51Z

src/utilities/bcl/templates/ModelMeasure/tests/model_measure_test.rb

+    assert_equal(1, result.stepInfo.size)
+    assert_empty(result.stepWarnings)


Throughout the tests, I modernized the templates to

Use non-deprecated methods (warnings is deprecated since 2.x, replaced with stepWarnings)

Use minitest assertions that provide much better failure messages

jmarrec · 2024-04-17T22:16:01Z

src/utilities/bcl/templates/ModelMeasure/tests/model_measure_test.rb

+    # Ensure the model did not start with a space named like requested
+    refute_includes(model.getSpaces.map(&:nameString), "New Space")


another example

jmarrec · 2024-04-17T22:16:45Z

src/utilities/bcl/templates/ReportingMeasure/tests/reporting_measure_test.rb

+    sqlFile = runner.lastEnergyPlusSqlFile.get
+    if !sqlFile.connectionOpen
+      sqlFile.reopen
+    end
+    hours = sqlFile.hoursSimulated
+    refute_empty(hours)
+    assert_equal(8760.0, hours.get)


That makes it clear that #5147 shouldn't happen if you do things correctly.

The reporting measure.rb does sql_file.close at the end.

jmarrec · 2024-04-17T22:17:50Z

src/utilities/bcl/templates/EnergyPlusMeasure/tests/test_energyplus_measure.py

+CURRENT_DIR_PATH = Path(__file__).parent.absolute()
+sys.path.insert(0, str(CURRENT_DIR_PATH.parent))
 from measure import EnergyPlusMeasureName
+sys.path.pop(0)
+del sys.modules['measure']


To be able to load several pytest files when --run-tests is happening, and because every measure is measure.py, we need del sys.modules['measures'] or it remembers the first one it found

jmarrec · 2024-04-17T22:18:22Z

src/utilities/bcl/templates/EnergyPlusMeasure/tests/test_energyplus_measure.py

+# This allows running openstudio CLI on this file (`openstudio test_measure.py`, maybe with extra args)
+if __name__ == "__main__":
+    import sys
+    if len(sys.argv) > 1:
+        pytest.main([__file__] + sys.argv[1:])
+    else:
+        pytest.main()


This bit is really optional, but I think it's nice to be able to do this, like you could do openstudio measure_test.rb

jmarrec · 2024-04-17T22:19:26Z

src/cli/test/test_bcl_measure_templates.py

+@pytest.mark.parametrize("measure_language", MEASURE_LANGUAGES)
+@pytest.mark.parametrize("measure_type", MEASURE_TYPES)
+def test_create_measure_and_run_tests(osclipath: Path, measure_type: str, measure_language: str, verbose: bool = VERBOSE):
+    measure_dir_path = get_measure_dir(measure_type=measure_type, measure_language=measure_language)
+    if measure_dir_path.is_dir():
+        shutil.rmtree(measure_dir_path)
+
+    cmd = get_cmd(osclipath=osclipath, measure_type=measure_type, measure_language=measure_language)
+
+    if verbose:
+        print(f"Instantiating new measure {measure_language} - {measure_type}:")
+        print(f"cmd = {shlex.join(cmd)}")
+    output = _run_subprocess_and_capture(cmd)
+    if verbose:
+        print(output)
+
+    if measure_language == 'Ruby':
+        run_minitest(osclipath=osclipath, measure_dir_path=measure_dir_path, verbose=verbose)
+    else:
+        run_pytest(osclipath=osclipath, measure_dir_path=measure_dir_path, verbose=verbose)


This pytest will:

Instantiate every Measure Type + Measure Language combination

Calll pytest or minitest on it

This ensures we have WORKING templates.

https://github.com/jmarrec/openstudio-gems/releases/tag/v3.2.2-6 https://github.com/jmarrec/openstudio-gems/actions/runs/8729423756/job/23951340514#step:7:63

…sure_templates

jmarrec · 2024-04-17T22:28:45Z

src/cli/CMakeLists.txt

+    file(MAKE_DIRECTORY "${PROJECT_BINARY_DIR}/Testing/")
+    add_test(NAME OpenStudioCLI.test_bcl_measure_templates
+      COMMAND ${Python_EXECUTABLE} -m pytest --verbose --os-cli-path $<TARGET_FILE:openstudio> "${CMAKE_CURRENT_SOURCE_DIR}/test/test_bcl_measure_templates.py"
+      WORKING_DIRECTORY "${PROJECT_BINARY_DIR}/Testing/"
+    )


register the pytest as a cmakelists one.

Voluntarily not using "-n auto" (pytest-xdist) so I check for loading errors (the thing del sys.modules['measure'] fixes).

I used system pytest (because it wasn't working with the CLI at first) but we could replace it with openstudio, or do it with both.

Mind that the ReportingMeasure tests are quite slow (there's a simulation happening)

And maybe another one for openstudio measure --run_tests Testing/bcl_templates once all measures have been instantiated too.

We can figure that out later

https://github.com/NREL/openstudio-gems/releases/tag/v3.8.0-RC1

…sure_templates

close #5155

…Measure 5155 ruby engine fixture.wrong method measure

Find system ruby 3.2

…sure_templates

Not sure why copy is even overriden... It's an alias to cp anyways. Read https://www.ruby-lang.org/en/news/2019/12/12/separation-of-positional-and-keyword-arguments-in-ruby-3-0/

jmarrec · 2024-04-19T13:08:02Z

Lol windows runner is always so fun

[189/9099] Updating EnergyPlus simulation in D:/git/OS-build-release-v2/resources/energyplus/Daylighting_School/, this may take a while

FAILED: resources/energyplus/Daylighting_School/eplusout.err D:/git/OS-build-release-v2/resources/energyplus/Daylighting_School/eplusout.err 

cmd.exe /C "cd /D D:\git\OS-build-release-v2\resources && "C:\Program Files\CMake\bin\cmake.exe" -E copy energyplus/Daylighting_School/in.idf energyplus/Daylighting_School/in.idf && "C:\Program Files\CMake\bin\cmake.exe" -E copy D:/git/OS-build-release-v2/EnergyPlus-24.1.0-9d7789a3ac-Windows-x86_64/Energy+.idd energyplus/Daylighting_School/Energy+.idd && "C:\Program Files\CMake\bin\cmake.exe" -E copy D:/git/OS-build-release-v2/EnergyPlus-24.1.0-9d7789a3ac-Windows-x86_64/WeatherData/USA_CO_Golden-NREL.724666_TMY3.epw energyplus/Daylighting_School/in.epw && "C:\Program Files\CMake\bin\cmake.exe" -E chdir energyplus/Daylighting_School D:/git/OS-build-release-v2/EnergyPlus-24.1.0-9d7789a3ac-Windows-x86_64/energyplus.exe > energyplus/Daylighting_School/screen.out"

Program terminated: EnergyPlus Terminated--Error(s) Detected.

ci-commercialbuildings · 2024-04-19T13:25:43Z

CI Results for 6525992:

Ubuntu 20.04:
- OpenStudio-3.8.0-beta+65259923b3-Ubuntu-20.04-x86_64.deb
- OpenStudio-3.8.0-beta+65259923b3-Ubuntu-20.04-x86_64.tar.gz
Ubuntu 22.04:
- OpenStudio-3.8.0-beta+65259923b3-Ubuntu-22.04-x86_64.deb
- OpenStudio-3.8.0-beta+65259923b3-Ubuntu-22.04-x86_64.tar.gz
Darwin:
Windows:
Code Coverage:

jmarrec added 7 commits April 16, 2024 10:37

Fix #5147 - Adjust templates

2fa6841

* make it clear that if you use runner.lastEnergyPlusSQLFile you need to reopen it potentially * Use a chdir context

Modernize templates: use specific Minitest assertions, use non-deprec…

49eeb17

…ated functions to get warnings/errors (stepWarnings, not warnings)

Reject __pycache__ in BCLMeasure files (where you run pytest manually…

f09ba02

…) + upgrade openstudio measure --examples

Update the Measure templates example_model.osm to be 3.7.0

40f2077

It's painfully slow to have to translate it from 1.13.0 to 3.7.0 everytime. Maybe we could consider using 3.2.0 or something like that, but there's no point keeping it so old

Add ability to run a pytest file from the openstudio CLI.

ba6371c

Add a ctest (via pytest) that instantiates measures from template + r…

c2fc7e5

…un the minitest/pytest on it

jmarrec added component - Measures Pull Request - Ready for CI This pull request if finalized and is ready for continuous integration verification prior to merge. component - Ruby bindings component - Python bindings labels Apr 16, 2024

jmarrec requested a review from kbenne April 16, 2024 16:37

jmarrec self-assigned this Apr 16, 2024

jmarrec added 5 commits April 17, 2024 14:02

Use the new openstudio-gems that as NREL/openstudio-gems#77

80913d8

NEW: openstudio measure --run-tests will run Pytest tests with coverage!

dbdd2ed

tell pytest to use xunit1 format for junit so we have file + line

f8f4f7f

jmarrec force-pushed the 5147_modernize_measure_templates branch from e94ba7e to 5bf4fe2 Compare April 17, 2024 13:52

jmarrec commented Apr 17, 2024

View reviewed changes

jmarrec added 2 commits April 18, 2024 00:24

Update openstudio-gems to maybe 3.8.0-RC1

249bcee

https://github.com/jmarrec/openstudio-gems/releases/tag/v3.2.2-6 https://github.com/jmarrec/openstudio-gems/actions/runs/8729423756/job/23951340514#step:7:63

Merge remote-tracking branch 'origin/develop' into 5147_modernize_mea…

890fc2c

…sure_templates

jmarrec commented Apr 17, 2024

View reviewed changes

jmarrec and others added 7 commits April 18, 2024 01:24

Update gems package to one without pycall.rb

27c03dc

Remove pycall from EMBEDDED_EXT_INITS

4369e03

Use the v3.8.0-RC gems package from NREL

8dc27cf

https://github.com/NREL/openstudio-gems/releases/tag/v3.8.0-RC1

We retagged 3.8.0-rc1 gems

4b54051

Merge remote-tracking branch 'origin/develop' into 5147_modernize_mea…

2bc2502

…sure_templates

Fix RubyEngineFixture.WrongMethodMeasure test

b9f63e0

close #5155

Merge pull request #5156 from NREL/5155-RubyEngineFixture.WrongMethod…

7726e86

…Measure 5155 ruby engine fixture.wrong method measure

Find System ruby 3.2 and validate interpreter & version

0ccf03b

jmarrec mentioned this pull request Apr 19, 2024

Find system ruby 3.2 #5158

Merged

19 tasks

jmarrec added 5 commits April 19, 2024 09:59

Merge pull request #5158 from NREL/find_system_ruby

0f4b9d6

Find system ruby 3.2

Merge remote-tracking branch 'origin/develop' into 5147_modernize_mea…

12d5bab

…sure_templates

Fix a test that had an harcoded value (StackLevelTooDeepMeasure)

61f6085

Track HEREDOC workflow PR

d4e6381

Fixup the ugly patching of FileUtils.cp / cp_r

6525992

Not sure why copy is even overriden... It's an alias to cp anyways. Read https://www.ruby-lang.org/en/news/2019/12/12/separation-of-positional-and-keyword-arguments-in-ruby-3-0/

jmarrec merged commit 8d12eff into develop Apr 19, 2024
3 of 6 checks passed

kbenne deleted the 5147_modernize_measure_templates branch April 19, 2024 13:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix #5147 and generally modernize measure templates + get openstudio to run pytest #5149

Fix #5147 and generally modernize measure templates + get openstudio to run pytest #5149

jmarrec commented Apr 16, 2024

jmarrec Apr 17, 2024

DavidGoldwasser Apr 17, 2024

jmarrec Apr 19, 2024 •

edited

Loading

jmarrec Apr 17, 2024

jmarrec Apr 17, 2024

jmarrec Apr 17, 2024

jmarrec Apr 17, 2024

jmarrec Apr 17, 2024

jmarrec Apr 17, 2024

jmarrec Apr 17, 2024

jmarrec Apr 17, 2024

jmarrec Apr 17, 2024

jmarrec Apr 17, 2024

jmarrec Apr 17, 2024

jmarrec commented Apr 19, 2024

ci-commercialbuildings commented Apr 19, 2024 •

edited

Loading

		assert_equal(1, result.stepInfo.size)
		assert_empty(result.stepWarnings)

		# Ensure the model did not start with a space named like requested
		refute_includes(model.getSpaces.map(&:nameString), "New Space")

Fix #5147 and generally modernize measure templates + get openstudio to run pytest #5149

Fix #5147 and generally modernize measure templates + get openstudio to run pytest #5149

Conversation

jmarrec commented Apr 16, 2024

Pull request overview

Pull Request Author

Review Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jmarrec Apr 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jmarrec commented Apr 19, 2024

ci-commercialbuildings commented Apr 19, 2024 • edited Loading

jmarrec Apr 19, 2024 •

edited

Loading

ci-commercialbuildings commented Apr 19, 2024 •

edited

Loading