Feature/experimental validation tests

[mgovers]

Add experimental features option for validation tests that are not yet finalized

The usage is as follows:

• the extra_params dict contains an update to the params.json which may overload certain parameters for a specific calculation type. E.g.:
• the fail expected failure feature is extended
  • made into a dict
  • Python
    • made mandatory (i.e.: it should be removed if the expected failure is ever fixed)
    • now also supports exceptions other than AssertionError
  • C++
    • remains [skipped] (unchanged)
    • all cases marked fail are now skipped (not only "fail": true)
• the experimental_features parameter may be set to enabled or disabled to enable experimental features from the Python API

params.json

Expected failures

No issue

{
  "calculation_method": ["newton_raphson", "iterative_current"],
  "rtol": 1e-5,
  "atol": 1e-5
}

Global specification

All cases should result in expected fail (xfail)

{
  "calculation_method": ["newton_raphson", "iterative_current"],
  "rtol": 1e-5,
  "atol": 1e-5,
  "fail": {
    "reason": "Experimental feature under development",
    "raises": "PowerGridError"  // may be omitted for AssertionError
  }
}

Extra params: expected fail

Only iterative_current results in expected fail (xfail)

  "calculation_method": ["newton_raphson", "iterative_current"],
  "rtol": 1e-5,
  "atol": 1e-5,
  "extra_params": {
    "iterative_current": {
      "fail": {
        "reason": "Experimental feature under development",
        "raises": "PowerGridError"  // may be omitted for AssertionError
      }
    }
  }
}

Both specified

All calculation_methods should result in expected fail (xfail)

{
  "calculation_method": ["newton_raphson", "iterative_current"],
  "rtol": 1e-5,
  "atol": 1e-5,
  "fail": "failure reson"
  "extra_params": {
    "iterative_current": {
      "fail": {
        "reason": "Experimental feature under development",
        "raises": "PowerGridError"  // may be omitted for AssertionError
      }
    }
  }
}

experimental features

For all calculation_methods

{
  "calculation_method": ["newton_raphson", "iterative_current"],
  "rtol": 1e-5,
  "atol": 1e-5,
  "experimental_features": "enabled"
}

As calculation_method-specific parameter

{
  "calculation_method": ["iterative_linear", "newton_raphson"],
  "rtol": 1e-6,
  "atol": {
    "q.*": 1e-1,
    "default": 1e-6
  },
  "extra_params": {
    "newton_raphson": {
      "experimental_features": "enabled"
    }
  }
}

All specified

{
  "calculation_method": ["iterative_linear", "newton_raphson"],
  "rtol": 1e-6,
  "atol": {
    "q.*": 1e-1,
    "default": 1e-6
  },
  "extra_params": {
    "newton_raphson": {
      "experimental_features": "enabled",
      "fail": {
        "reason": "Experimental feature under development",
        "raises": "PowerGridError"
      }
    }
  }
}

[mgovers]

Quality Gate passed

The SonarCloud Quality Gate passed, but some issues were introduced.

2 New issues 0 Security Hotspots 89.3% Coverage on New Code 0.0% Duplication on New Code

See analysis details on SonarCloud

newly introduced sonar cloud issues are not issues but actual API design choices consistent with existing code base

[TonyXiang8787]

Looks good. One change needed in the C API docstring.

[sonarcloud]

Quality Gate passed

The SonarCloud Quality Gate passed, but some issues were introduced.

2 New issues
0 Security Hotspots
89.3% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud