Skip to content

Test case creation: add helper function to create test case #1017

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jun 23, 2025

Conversation

nitbharambe
Copy link
Member

@nitbharambe nitbharambe commented Jun 12, 2025

Changes proposed in this PR include:

A helper function is added to make adding validation test cases easier.

  • Serializes data to file
  • Update data can be optionally provided to make batch test case
  • It automatically generates license files.

Signed-off-by: Nitish Bharambe <nitish.bharambe@alliander.com>
@mgovers mgovers changed the title Helper funciton for test case creation Test case creation: add helper function to create test case Jun 12, 2025
@nitbharambe nitbharambe added feature New feature or request labels Jun 12, 2025
Signed-off-by: Nitish Bharambe <nitish.bharambe@alliander.com>
@nitbharambe nitbharambe marked this pull request as ready for review June 13, 2025 06:59
nitbharambe and others added 2 commits June 13, 2025 09:02
Co-authored-by: Martijn Govers <martijn.govers@alliander.com>
Signed-off-by: Nitish Bharambe <78108900+nitbharambe@users.noreply.github.com>
Signed-off-by: Nitish Bharambe <nitish.bharambe@alliander.com>
Copy link
Member

@TonyXiang8787 TonyXiang8787 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nitbharambe @mgovers
My proposal is to move the validation test function (not the test case finder and parametrize) to the PGM package.

The validation test function should read a path to a test case, and some meta data (like calculation method, calculation type, etc.). And it should run the validation tests.

In this way, user can easily run more validation test just by install PGM self without going into developer mode.

@TonyXiang8787 TonyXiang8787 requested a review from Copilot June 15, 2025 18:16
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Adds a new helper _make_test_case to generate and serialize end-to-end test artifacts (input, optional update batch, output, params) along with license files, and provides unit tests to verify its behavior.

  • Introduce LICENSE_TEXT constant and _make_test_case function in utils.py.
  • Update test_utils.py to import the helper and add test__make_test_case covering batch and non-batch flows.
  • Ensure serialization and license-writing calls are made for each generated file.

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
src/power_grid_model/utils.py Add LICENSE_TEXT and implement _make_test_case to write JSON and license files.
tests/unit/test_utils.py Import LICENSE_TEXT, DatasetType, _make_test_case; add parametrized test for the new helper.
Comments suppressed due to low confidence (3)

tests/unit/test_utils.py:208

  • Add an assertion to verify that params.json itself is written (e.g., write_text_mock.assert_any_call(save_path / "params.json", data=json.dumps(params, indent=2), encoding="utf-8")) to ensure full coverage of all output files.
write_text_mock.assert_any_call(save_path / "params.json.license", data=LICENSE_TEXT, encoding="utf-8")

tests/unit/test_utils.py:185

  • [nitpick] Consider renaming test__make_test_case to test_make_test_case for consistency with typical pytest naming conventions.
def test__make_test_case(

tests/unit/test_utils.py:172

  • Add a negative test case to cover invalid output_dataset_type and assert that it raises ValueError to fully test the error path in _make_test_case.
@pytest.mark.parametrize(

@mgovers
Copy link
Member

mgovers commented Jun 16, 2025

@nitbharambe @mgovers My proposal is to move the validation test function (not the test case finder and parametrize) to the PGM package.

The validation test function should read a path to a test case, and some meta data (like calculation method, calculation type, etc.). And it should run the validation tests.

In this way, user can easily run more validation test just by install PGM self without going into developer mode.

While indeed useful, I think that's a separate feature that is independent of this PR. In the first place, we want users to be able to provide us with something that is debugable (a test case). If we want to allow them to debug themselves by publishing the test loader, then we should do some serious cleanup work on it first.

We would also need to decide whether to publish the test data as well (or maybe a subset of the data), and a design on how to do that without making the PGM package too big

@TonyXiang8787
Copy link
Member

@nitbharambe @mgovers My proposal is to move the validation test function (not the test case finder and parametrize) to the PGM package.
The validation test function should read a path to a test case, and some meta data (like calculation method, calculation type, etc.). And it should run the validation tests.
In this way, user can easily run more validation test just by install PGM self without going into developer mode.

While indeed useful, I think that's a separate feature that is independent of this PR. In the first place, we want users to be able to provide us with something that is debugable (a test case). If we want to allow them to debug themselves by publishing the test loader, then we should do some serious cleanup work on it first.

We would also need to decide whether to publish the test data as well (or maybe a subset of the data), and a design on how to do that without making the PGM package too big

I think this is the same feature from user-story perspective. If user has the function to create a validation dataset, the user wants to be able to run the validation without installing PGM in developer mode or cloning the source code (including the test scripts).

So from user perspective merging only this function alone does not make sense.

@mgovers
Copy link
Member

mgovers commented Jun 16, 2025

@nitbharambe @mgovers My proposal is to move the validation test function (not the test case finder and parametrize) to the PGM package.
The validation test function should read a path to a test case, and some meta data (like calculation method, calculation type, etc.). And it should run the validation tests.
In this way, user can easily run more validation test just by install PGM self without going into developer mode.

While indeed useful, I think that's a separate feature that is independent of this PR. In the first place, we want users to be able to provide us with something that is debugable (a test case). If we want to allow them to debug themselves by publishing the test loader, then we should do some serious cleanup work on it first.
We would also need to decide whether to publish the test data as well (or maybe a subset of the data), and a design on how to do that without making the PGM package too big

I think this is the same feature from user-story perspective. If user has the function to create a validation dataset, the user wants to be able to run the validation without installing PGM in developer mode or cloning the source code (including the test scripts).

So from user perspective merging only this function alone does not make sense.

Still, i would argue that running validation cases is a separate PR at the minimum. I think it's good to follow an incremental improvement approach here. I think cleaning up the test case runner needs to be prioritized with the team and would cause needless delays for merging this PR

Signed-off-by: Nitish Bharambe <nitish.bharambe@alliander.com>
Copy link

Copy link
Member

@TonyXiang8787 TonyXiang8787 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test runner into package will be done in a separate PR. I hereby approve this PR.

@nitbharambe nitbharambe added this pull request to the merge queue Jun 23, 2025
Merged via the queue into main with commit 8944883 Jun 23, 2025
31 checks passed
@nitbharambe nitbharambe deleted the feature/make-test-dataset-helper-fn branch June 23, 2025 08:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature or request
Projects
Status: No status
Development

Successfully merging this pull request may close these issues.

4 participants