WIP : Handle dual and reduced cost#220
Closed
tbittar wants to merge 180 commits into
Closed
Conversation
…ime/scenario-dependent values optest4 is identical to optest1 except minimum_generation_modulation is set to non-zero constant values (0.5/0.3/0.4) for unique_prod/unique_prod2/unique_prod3, so the lower-bound expression min(p_max_cluster, min_gen_mod*unit_count*p_max_unit) actually constrains generation instead of trivially evaluating to 0. Closes #9 https://claude.ai/code/session_01VoihrWyHoXtpxyBHrwmwCA
…io-varying bounds - All scenario-dependent TSV files now have 2 columns (2 scenarios, 168 timesteps) - minimum_generation_modulation alternates (0.0, 0.5) / (0.3, 0.1) per timestep for [scenario0, scenario1], making it genuinely time- AND scenario-dependent (non-trivial) - New test function test_model_behaviour_scenario_and_time_dependent_bounds runs with scenarios=2, verifying that the min/ceil bound expressions in test_lib.thermal are correctly handled when the parameter varies across both time and scenarios - Removes the reference CSV (no longer needed; OPTIMAL status + gap check suffices) Closes #9 https://claude.ai/code/session_01VoihrWyHoXtpxyBHrwmwCA
* refactor: replace OR-Tools pipeline with vectorized linopy solver Replace the scalar OR-Tools based build_problem pipeline with a fully vectorized linopy/xarray pipeline. A single AST traversal now produces linopy LinearExpressions covering all components × time steps × scenarios, adding all constraints in one add_constraints() call per constraint type. Key changes: - Add linopy_linearize.py: VectorizedLinopyBuilder visitor that maps VariableNodes → linopy.Variable, ParameterNodes → xr.DataArray, and handles time shifts, time sums, scenario expectations, and port fields via vectorized xarray/linopy operations. - Add linopy_problem.py: LinopyOptimizationProblem and build_problem() using 4-phase construction (params, variables, ports, constraints). - Fix time_shift for per-component shifts: masked accumulation over unique shift values replaces scalar extraction (fixes d_min_down≠1). - Fix _apply_time_shift: assign_coords after isel resets time coordinates so subsequent xarray arithmetic does not silently re-align. - Rewrite output_values.py to extract results from linopy_model.solution. - Remove OR-Tools test files; update all e2e tests to linopy API. https://claude.ai/code/session_01CXGhwjptqV25QGYb56CFdt * chore: delete OR-Tools dead code after linopy migration Remove all files that were part of the old scalar OR-Tools pipeline and are no longer referenced by any active code: Source: - simulation/optimization.py (929 lines) — OR-Tools OptimizationProblem - simulation/linearize.py (333 lines) — scalar expression linearizer - simulation/linear_expression.py (438 lines) — LinearExpression/Term/TermKey - simulation/benders_decomposed.py — depended on optimization.py; out of scope Tests: - tests/unittests/lib_parsing/test_objective_contribution.py — patched optimization.py internals and used ortools directly - tests/e2e/functional/test_investment_pathway.py — tested build_benders_decomposed_problem (out of scope) - tests/e2e/integration/test_benders_decomposed.py — was already fully @pytest.mark.skip; benders is out of scope Dependencies: - Remove ortools from pyproject.toml dependencies - Remove [mypy-ortools.*] stanza from mypy.ini https://claude.ai/code/session_01CXGhwjptqV25QGYb56CFdt * style: apply black 23.7.0 formatting to new/modified files https://claude.ai/code/session_01CXGhwjptqV25QGYb56CFdt * fix: resolve mypy errors in linopy_linearize.py and linopy_problem.py linopy_linearize.py: - time_shift: split the DataArray check so mypy narrows the type before passing to _da_to_int; raise ValueError if shift evaluates to a linopy Variable/LinearExpression (which should never happen for a valid model) - time_sum slow path: assert both bounds are DataArrays before calling .astype(int), replacing the unreachable else-int() branches that mypy flagged as type errors linopy_problem.py: - remove unused get_solution(model_id: str, ...) which looked up _linopy_vars keyed by (int, str) using a str key — wrong type and logically broken; no callers existed https://claude.ai/code/session_01CXGhwjptqV25QGYb56CFdt * chore: remove dead imports and OR-Tools package dependencies Python imports: - simulation/__init__.py: drop re-exports of BendersSolution, BendersRunner, MergeMPSRunner — nothing imports them anymore - simulation/output_values.py: remove BendersSolution, BendersMergedSolution, BendersDecomposedSolution classes (only used by the deleted benders_decomposed.py) and the now-unused `import math` Dependencies: - requirements.in: remove ortools==9.9.3963 direct dependency - requirements.txt: remove ortools==9.9.3963 and its transitive-only dependents absl-py, immutabledict, protobuf; scrub ortools from comment annotations on numpy and pandas entries https://claude.ai/code/session_01CXGhwjptqV25QGYb56CFdt * docs: rewrite module docstrings to describe current design only Remove comparative phrasing ("instead of", "replaces OR-Tools") from the module docstrings of linopy_linearize.py and linopy_problem.py. Both docstrings now describe the module's own behaviour and structure without requiring the reader to know any prior implementation. https://claude.ai/code/session_01CXGhwjptqV25QGYb56CFdt * refactor: remove unused params from constraint/objective builders Drop `components` and `total_obj` from `_create_constraints_for_model` (neither was used), remove `components` from `_add_objectives_for_model`, and iterate over `self.model_components.keys()` in the phase-4 build loop. Closes tbittar#5 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Formatting * Refacto --------- Co-authored-by: Claude <noreply@anthropic.com>
* refactor: replace OR-Tools pipeline with vectorized linopy solver Replace the scalar OR-Tools based build_problem pipeline with a fully vectorized linopy/xarray pipeline. A single AST traversal now produces linopy LinearExpressions covering all components × time steps × scenarios, adding all constraints in one add_constraints() call per constraint type. Key changes: - Add linopy_linearize.py: VectorizedLinopyBuilder visitor that maps VariableNodes → linopy.Variable, ParameterNodes → xr.DataArray, and handles time shifts, time sums, scenario expectations, and port fields via vectorized xarray/linopy operations. - Add linopy_problem.py: LinopyOptimizationProblem and build_problem() using 4-phase construction (params, variables, ports, constraints). - Fix time_shift for per-component shifts: masked accumulation over unique shift values replaces scalar extraction (fixes d_min_down≠1). - Fix _apply_time_shift: assign_coords after isel resets time coordinates so subsequent xarray arithmetic does not silently re-align. - Rewrite output_values.py to extract results from linopy_model.solution. - Remove OR-Tools test files; update all e2e tests to linopy API. https://claude.ai/code/session_01CXGhwjptqV25QGYb56CFdt * chore: delete OR-Tools dead code after linopy migration Remove all files that were part of the old scalar OR-Tools pipeline and are no longer referenced by any active code: Source: - simulation/optimization.py (929 lines) — OR-Tools OptimizationProblem - simulation/linearize.py (333 lines) — scalar expression linearizer - simulation/linear_expression.py (438 lines) — LinearExpression/Term/TermKey - simulation/benders_decomposed.py — depended on optimization.py; out of scope Tests: - tests/unittests/lib_parsing/test_objective_contribution.py — patched optimization.py internals and used ortools directly - tests/e2e/functional/test_investment_pathway.py — tested build_benders_decomposed_problem (out of scope) - tests/e2e/integration/test_benders_decomposed.py — was already fully @pytest.mark.skip; benders is out of scope Dependencies: - Remove ortools from pyproject.toml dependencies - Remove [mypy-ortools.*] stanza from mypy.ini https://claude.ai/code/session_01CXGhwjptqV25QGYb56CFdt * style: apply black 23.7.0 formatting to new/modified files https://claude.ai/code/session_01CXGhwjptqV25QGYb56CFdt * fix: resolve mypy errors in linopy_linearize.py and linopy_problem.py linopy_linearize.py: - time_shift: split the DataArray check so mypy narrows the type before passing to _da_to_int; raise ValueError if shift evaluates to a linopy Variable/LinearExpression (which should never happen for a valid model) - time_sum slow path: assert both bounds are DataArrays before calling .astype(int), replacing the unreachable else-int() branches that mypy flagged as type errors linopy_problem.py: - remove unused get_solution(model_id: str, ...) which looked up _linopy_vars keyed by (int, str) using a str key — wrong type and logically broken; no callers existed https://claude.ai/code/session_01CXGhwjptqV25QGYb56CFdt * chore: remove dead imports and OR-Tools package dependencies Python imports: - simulation/__init__.py: drop re-exports of BendersSolution, BendersRunner, MergeMPSRunner — nothing imports them anymore - simulation/output_values.py: remove BendersSolution, BendersMergedSolution, BendersDecomposedSolution classes (only used by the deleted benders_decomposed.py) and the now-unused `import math` Dependencies: - requirements.in: remove ortools==9.9.3963 direct dependency - requirements.txt: remove ortools==9.9.3963 and its transitive-only dependents absl-py, immutabledict, protobuf; scrub ortools from comment annotations on numpy and pandas entries https://claude.ai/code/session_01CXGhwjptqV25QGYb56CFdt * docs: rewrite module docstrings to describe current design only Remove comparative phrasing ("instead of", "replaces OR-Tools") from the module docstrings of linopy_linearize.py and linopy_problem.py. Both docstrings now describe the module's own behaviour and structure without requiring the reader to know any prior implementation. https://claude.ai/code/session_01CXGhwjptqV25QGYb56CFdt * refactor: remove unused params from constraint/objective builders Drop `components` and `total_obj` from `_create_constraints_for_model` (neither was used), remove `components` from `_add_objectives_for_model`, and iterate over `self.model_components.keys()` in the phase-4 build loop. Closes tbittar#5 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Formatting * feat: evaluate extra outputs vectorized over xarray (issue #4) Replace the scalar (time, scenario) loop in extra output evaluation with a vectorized xarray-based approach, consistent with the Linopy refactoring. Key changes: - `extra_output.py`: add VectorizedExtraOutputBuilder (ExpressionVisitor returning xr.DataArray), supporting all AST nodes including nonlinear ops (floor/ceil/min/max, var*var) and sum_connections via port arrays; add _build_port_arrays_xarray / _build_slave_port_array_xarray helpers that replicate the incidence-matrix logic of _LinopyProblemBuilder but with post-solve DataArrays; remove ExtraOutputValueProvider (obsolete). - `linopy_problem.py`: expose param_arrays, model_components and models on LinopyOptimizationProblem so OutputValues can access them post-solve; remove expand_operators_for_extra_output (no longer needed). - `output_values.py`: replace _evaluate_single_extra_output scalar loop with a model-level vectorized pass using VectorizedExtraOutputBuilder; add _fill_extra_output_from_da to unpack a DataArray into Dict storage. - `test_output_values.py`: update mocks for new problem fields; add test_extra_output_with_sum_connections and test_extra_output_nonlinear. https://claude.ai/code/session_017jpeajLjMMyPuTeQpvWtBA * refactor: extract shared port-connection helpers to avoid duplication _group_port_connections_by_master and _build_incidence_matrix are now module-level functions in linopy_problem.py. Both _LinopyProblemBuilder._build_slave_port_array (linopy path) and _build_slave_port_array_xarray (extra-output path) call them instead of duplicating the connection-grouping + incidence-matrix logic. _involves is also no longer copied in extra_output.py (was already in linopy_problem.py). https://claude.ai/code/session_017jpeajLjMMyPuTeQpvWtBA * style: apply black 23.7.0 formatting to linopy_problem.py https://claude.ai/code/session_017jpeajLjMMyPuTeQpvWtBA * refactor: import _eval_int, _da_to_int, _has_dim from linopy_linearize These three helpers were duplicated verbatim from linopy_linearize.py. Remove the local copies in extra_output.py and import them directly. Also drop the now-unused EvaluationContext/EvaluationVisitor imports. https://claude.ai/code/session_017jpeajLjMMyPuTeQpvWtBA * refactor: unify port-array building via build_port_arrays factory function Both the linopy problem builder (pre-solve) and the extra-output evaluator (post-solve) now share a single build_port_arrays(model, components, all_models, all_model_components, network, make_builder) function in linopy_problem.py. The caller supplies a make_builder(model_key, model) factory that returns the appropriate ExpressionVisitor: - _LinopyProblemBuilder._build_port_arrays_for_model → one-liner calling build_port_arrays with a VectorizedLinopyBuilder factory. - OutputValues._evaluate_extra_outputs → calls build_port_arrays with a VectorizedExtraOutputBuilder factory (lambda capturing var_solution_arrays and problem after Optional narrowing). Consequently, _build_port_arrays_xarray and _build_slave_port_array_xarray are removed from extra_output.py, together with the now-unused imports of _build_incidence_matrix and _group_port_connections_by_master. _build_slave_port_array is removed from _LinopyProblemBuilder. https://claude.ai/code/session_017jpeajLjMMyPuTeQpvWtBA * refactor: extract shared time-operator logic to module-level functions _apply_time_shift, _eval_int_expr, _time_shift, _time_eval, _time_sum, and _all_time_sum are now module-level functions in linopy_linearize.py. Both VectorizedLinopyBuilder and VectorizedExtraOutputBuilder delegate their time_* methods to these shared functions, reducing each to a one-liner. _linopy_add is also moved from linopy_problem.py to linopy_linearize.py (where LinopyExpression is defined) and imported back. Using _linopy_add for accumulation in the shared functions makes them work correctly for both linopy and pure-xarray visitors. https://claude.ai/code/session_017jpeajLjMMyPuTeQpvWtBA * Vectorize OutputVariable and ExtraOutput in OutputModel Replace per-component scalar Dict[TimeScenarioIndex, float] storage with vectorized xr.DataArray[component, time, scenario] held in a new OutputModel class (one per GEMS model, covering all its components). - output_values_base.py: replace BaseOutputValue with two independent dataclasses OutputVariable and ExtraOutput, each storing an Optional[xr.DataArray]; remove _value, _size, _set, get. - extra_output.py: remove ExtraOutput(BaseOutputValue) subclass and its _set() method; import ExtraOutput from output_values_base. - output_values.py: introduce OutputModel, ComponentOutputView, VarOutputView and ExtraOutputView; update OutputValues to use _models/comp_to_model_key; drop _fill_extra_output_from_da and the scalar unpacking loops; preserve the component().var().value API for all 51 existing e2e call sites via thin backward-compat views. - simulation_table.py: iterate _models DataArrays (isel over component, time, scenario) instead of _components scalar dicts. - Tests updated to the new internal API; all 63 tests pass. https://claude.ai/code/session_017jpeajLjMMyPuTeQpvWtBA * Refacto * Formatting --------- Co-authored-by: Claude <noreply@anthropic.com>
…problems from standard library and a pypsa-eur based test case (data not included)
Added performance tests of optimization problem building with simple …
Adding an advanced test with time-dep and scenario-dep var bounds
Removed the test for model behaviour with time-dependent bounds, which verified complex bound expressions under varying scenarios.
Neither antares_craft nor scipy are imported anywhere in the codebase. Removes them from pyproject.toml and requirements.in, and cleans up their transitive-only packages (antares-study-version, antares-timeseries-generation, psutil, requests, charset-normalizer, idna, urllib3) from the generated requirements files. scipy remains as a transitive dep via linopy. https://claude.ai/code/session_01HZtmQxUDepFWW4TxWBZyv4
…line The scalar expansion pipeline (OperatorsExpansion, ApplyTimeShift, ApplyTimeStep, ApplyScenario) and its associated AST node types (ProblemVariableNode, ProblemParameterNode, TimeIndex/ScenarioIndex subclasses) are unused since the refactorization to vectorized linopy+xarray problem building. Remove them entirely: - Delete src/gems/expression/operators_expansion.py - Delete tests/unittests/expressions/visitor/test_operators_expansion.py - Remove ProblemVariableNode, ProblemParameterNode, problem_var(), problem_param() from expression.py - Remove TimeIndex, NoTimeIndex, TimeShift, TimeStep, ScenarioIndex, NoScenarioIndex, CurrentScenarioIndex, OneScenarioIndex from expression.py - Remove pb_variable()/pb_parameter() abstract methods from ExpressionVisitor and their dispatch from visit() in visitor.py - Remove pb_variable()/pb_parameter() stub/error methods from all visitor implementations: copy, context_adder, evaluate, indexing, degree, print, equality, linopy_linearize, extra_output, model/port https://claude.ai/code/session_019cUZinfENNLYbqw8cVeSTp
…SGig Remove dead dependencies: antares_craft and scipy
Remove dead code: ProblemVar/Param nodes and operators_expansion pipe…
Build simulation table from linopy problem
- Add pandas and attrs to pyproject.toml dependencies (both are imported in src/ but were absent from the declared deps) - Add linopy, xarray, highspy, pandas, attrs to requirements.in to sync it with pyproject.toml - Remove pandas from requirements-dev.in (it belongs in production deps) - Add comment on highspy clarifying it is the HiGHS solver backend used indirectly via linopy https://claude.ai/code/session_0163cQuKsfWRgmgXkNiRFtbE
Remove the OutputValues class and all its helper classes (OutputModel, ComponentOutputView, VarOutputView, ExtraOutputView). Migrate all usages to SimulationTableBuilder.build(problem) which returns a flat pandas DataFrame with columns: block, component, output, absolute-time-index, block-time-index, scenario-index, value, basis-status. - Delete src/gems/simulation/output_values.py - Remove OutputVariable from output_values_base.py (keep ExtraOutput for extra_output.py) - Export SimulationTableBuilder and SimulationColumns from gems.simulation.__init__ - Replace OutputValues usage in 9 test files with DataFrame queries - Migrate extra output unit tests to test_simulation_table_extra_outputs.py - Update docs/user-guide/outputs.md and AGENTS.md to reflect new API https://claude.ai/code/session_012R79FcYYAwT4BRo6dAz5Zv
…for_model in linopy_problem.py to speed up problem building
Fix missing and undeclared production dependencies
…ationtable-7ib6V Replace OutputValues with SimulationTable across codebase
* Delete reference to build strategy * Remove unseless libs
#40) * Remove ComponentParameterNode/ComponentVariableNode and output_values_base.py Delete ComponentParameterNode, ComponentVariableNode, comp_param(), comp_var() from the expression AST — these were produced by ContextAdder (also deleted) which was never called in production. All vectorized-pipeline visitors already raised ValueError on encountering these nodes. Cascading cleanup: - Remove comp_parameter / comp_variable abstract methods and dispatch from ExpressionVisitor / visit() in visitor.py - Strip the method pairs and their imports from all 9 concrete visitors: print, copy, evaluate, evaluate_parameters, equality, degree, indexing, linopy_linearize, extra_output, port - Remove get_component_{variable,parameter}_value from ValueProvider / EvaluationContext (evaluate.py) and get_component_parameter_value from ParameterValueProvider (evaluate_parameters.py) - Remove get_component_{variable,parameter}_structure from IndexingStructureProvider (indexing.py) and the NotImplementedError stubs in model.py's Provider inner class - Update 4 test files: drop test_comp_parameter(), remove ComponentParameterNode from test_copy_ast(), remove comp_param/comp_var parametrize cases Also inline ExtraOutput from output_values_base.py into extra_output.py and delete output_values_base.py (its only remaining consumer). https://claude.ai/code/session_01E1avUHPpjDTp3id3n7zyC5 * Apply black==23.7.0 formatting to modified files https://claude.ai/code/session_01E1avUHPpjDTp3id3n7zyC5 --------- Co-authored-by: Claude <noreply@anthropic.com>
- data.py: fix DataBase.get_value to wrap timestep in a list and broaden return type to Union[float, ndarray]; change dataframe_to_time_series to return pd.Series instead of Dict[TimeIndex, float] to match TimeSeriesData.time_series field type - linopy_linearize.py: add return-value to type: ignore on multiplication (left * right can yield QuadraticExpression) - linopy_problem.py: suppress spurious ndarray assignment errors on ScenarioSeriesData and fallback get_value calls inside for-loop - simulation_table.py: replace unavailable attr.dataclass with dataclasses.dataclass - Add CLAUDE.md with project commands and architecture overview Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Enforce model id uniqueness * Remove usage of id(model)
Add LinopyModel alias to distinguish from gems.model.Model
- ResolutionConfig.horizon → block_length (length of one optimization window) - ResolutionConfig.overlap → block_overlap (timesteps shared between consecutive windows) - SimulationSession.total_timesteps → study_length (overall study duration) - load_session parameter renamed accordingly block_length is now consistent with OptimizationProblem.block_length, which already used that name for the same concept. https://claude.ai/code/session_01HTa79fVcbZmsLaNLXdG6ay
feat: add SimulationSession with new resolution modes
Add unit test for SimulationTable correctness on partial TimeBlock (i…
Parse binary variable
* Support for out-of-bounds-processing * Remove BlockBorderManagement * Handle parametrized shifts within sums * Cleaner implementation * Remove unrelevant description * Remove unused imports * Remove unused imports * Fix test * Raise with time or scenario dependency in shift
…nSession, SimulationTable. (#110) * Harmonize CLI/folder/session/table interactions (issue #106) + parameters.yml - New src/gems/study/parameters.py: StudyParameters pydantic model reading first-time-step, last-time-step, nb-scenarios, solver, solver-logs, solver-parameters from study_dir/parameters.yml (optional, defaults to frontal/highs/1-scenario if absent) - folder.py: remove dead optim-config ref in load_study; rewrite run_study to delegate entirely to SimulationSession via load_parameters, returns None and auto-exports SimulationTable to study_dir/output/ - session.py: add first_timestep, solver_name, solver_logs, solver_parameters fields; fix _run_frontal/_run_sequential/_run_parallel to start from first_timestep; stamp run_id as table_id on all produced SimulationTables - simulation_table.py: add table_id to SimulationTable; add to_csv/to_parquet/ to_netcdf methods with auto-naming from table_id; remove SimulationTableWriter - runner.py: rename CommandRunner -> AntaresXpansionCommandRunner; add module docstring clarifying this wraps AntaresXpansion external binaries - parsing.py: simplify CLI to --study (required) + --optim-config (optional); remove --duration and --scenario (now driven by parameters.yml) - main.py: delegate main_cli to run_study; keep input_libs/input_system/ input_database/_write_structure_txt for E2E test compatibility - Update tests to use new SimulationTable export API (to_csv/to_parquet/ to_netcdf) instead of deleted SimulationTableWriter https://claude.ai/code/session_01HyPLW7cBK5kugdX8Tqi9YE * Apply black 23.7.0 and isort 5.12.0 formatting Fix formatting issues found by CI linters (same versions as remote repo). https://claude.ai/code/session_01HyPLW7cBK5kugdX8Tqi9YE * Fix mypy call-arg errors in StudyParameters Remove redundant explicit alias= from Field() calls — ModifiedBaseModel already applies _to_kebab via alias_generator, so re-declaring the same hyphenated aliases caused the pydantic mypy plugin to treat the fields as required even though they had defaults. https://claude.ai/code/session_01HyPLW7cBK5kugdX8Tqi9YE * Fix circular import between folder.py and session.py folder.py imports SimulationSession from session.py at module level, and session.py imported load_study from folder.py at module level, creating a cycle. Move the load_study import inside load_session() where it is used. https://claude.ai/code/session_01HyPLW7cBK5kugdX8Tqi9YE * Fix test_run_study and related issues for new run_study API - StudyParameters: override extra="ignore" so unknown fields in parameters.yml (e.g. no-output, export-mps from other tools) are silently dropped instead of raising a validation error. - folder.py: remove kwargs that SimulationSession does not accept (first_timestep, solver_name, solver_logs, solver_parameters). - test_study_from_folder: update test_run_study to call the new run_study(study_dir) signature (no TimeBlock or scenario count args), copy the study to tmp_path to avoid polluting the source tree, and assert the output CSV is created with an objective-value row instead of inspecting a returned OptimizationProblem. https://claude.ai/code/session_01HyPLW7cBK5kugdX8Tqi9YE * Remove low-level helpers comment section Removed commented section for low-level helpers. * harmonizing time * Merge parameters.yml into optim-config: add time-scope, solver-options, scenario-scope sections - Add TimeScopeConfig (start-timestep / end-timestep), SolverOptionsConfig (solver, solver-logs, solver-parameters) and ScenarioScopeConfig (nb-scenarios) to OptimConfig in optim_config/parsing.py. - Update session.py to read time range from optim_config.time_scope and solver settings from optim_config.solver_options; remove the now-redundant solver_name / solver_logs / solver_parameters fields from SimulationSession. - Update folder.py to drop parameters.yml loading entirely; scenario_ids are now derived from optim_config.scenario_scope. - Delete study/parameters.py (StudyParameters / load_parameters). - Migrate all test parameters.yml values into the corresponding input/optim-config.yml files and delete the parameters.yml files. https://claude.ai/code/session_01G29xWf8E1XyzkMWQ7YPYXo * style: apply black formatting to session.py https://claude.ai/code/session_01G29xWf8E1XyzkMWQ7YPYXo * Compute scenario_ids from optim_config instead of passing as constructor arg scenario_ids is now a property on SimulationSession derived from optim_config.scenario_scope.nb_scenarios, removing it as a required constructor parameter. load_session() drops its scenario_ids argument for the same reason. https://claude.ai/code/session_01QaaRTupB2WQ528NDgKs4Mh * Add e2e consistency test for frontal, parallel, and sequential resolution modes Tests that run_study with three different optim-config files (frontal, parallel- subproblems with block-length=168, and sequential-subproblems with block-length=168 and block-overlap=1) produces identical per-timestep simulation table values for a fully time-separable LP problem (andromede_v1 DSR study, base028 variant without thermal clusters). The study uses 504 timesteps (end-timestep=503). Parallel mode runs 3 blocks of 168 timesteps; sequential mode also uses block-length=168 and block-overlap=1, which produces 4 blocks (3 full + 1 partial). Duplicate rows from the sequential overlap are deduplicated before comparison. The frontal vs parallel test also asserts that the summed block objectives are equal. Closes #105. https://claude.ai/code/session_01TQZE7z7wJfDDq794Te37tR * Add simple_generator LP model and restore gas/oil/coal to dsr_3_blocks study Replaces the MIP antares-historic.thermal model with a new continuous LP model (simple_generator) in the local andromede_v1_models library, and adds gas_base_zone, oil_base_zone, and coal_base_zone to the test system. This keeps the test system realistic (thermal dispatch in merit order) while avoiding MIP degeneracy: with zero startup/fixed costs the MIP solver finds multiple equivalent integer solutions across block boundaries, making per-timestep comparison between resolution modes unreliable. https://claude.ai/code/session_01TQZE7z7wJfDDq794Te37tR * Merge study libraries into test_lib and simplify test_optim_modes - Replace andromede_v1_models.yml + antares_historic.yml with a single test_lib.yml (id: test-lib) containing only the 5 models used by the dsr_3_blocks study: area, load, renewable, dsr, simple_generator - Update system.yml model references to use test-lib.* - Remove _DEGENERATE_OUTPUTS and the integer-variable comment from test_optim_modes.py; simplify _per_timestep_df accordingly https://claude.ai/code/session_01TQZE7z7wJfDDq794Te37tR * Update system.yml model references to test-lib Follows the library consolidation: replace antares-historic.* and andromede-v1-models.* prefixes with test-lib.* throughout. https://claude.ai/code/session_01TQZE7z7wJfDDq794Te37tR * Apply black 23.7 formatting to test_optim_modes https://claude.ai/code/session_01TQZE7z7wJfDDq794Te37tR * Rename timescope keys: start/end-timestep → first/last-time-step https://claude.ai/code/session_01WWT1fjYx11X3uUbzuXY3NT * Rename solver-options fields: solver->name, solver_logs->logs, solver_parameters->parameters (#122) https://claude.ai/code/session_01DJAihUc9MLUGXimU25Rnu3 Co-authored-by: Claude <noreply@anthropic.com> * Remove load_session function from session.py Removed the load_session function and its associated docstring. * Remove load_session from __all__ exports * refactor: move run_study to runner.py to eliminate circular dependency (#123) * refactor: move run_study to runner.py to eliminate circular dependency folder.py was importing SimulationSession from session.py, which in turn needed to import load_study from folder.py (worked around with a local import). Moving run_study to a new runner.py breaks the cycle: folder.py now only handles loading, session.py can import load_study at module level, and runner.py owns the orchestration between the two. https://claude.ai/code/session_01GnAgJKNP5b8D8SzH1TXRMU * style: apply black formatting to runner.py https://claude.ai/code/session_01GnAgJKNP5b8D8SzH1TXRMU * fix: update test imports to use gems.study.runner for run_study https://claude.ai/code/session_01GnAgJKNP5b8D8SzH1TXRMU --------- Co-authored-by: Claude <noreply@anthropic.com> * Remove load_session function to resolve circular dependency Removed the load_session function to fix circular dependency issues. * feat: use timestamp run_id and per-run output_dir in run_study (#124) The session now receives a run_id (minute-granularity timestamp, e.g. 20260427T1430) and an output_dir of study_dir/output/{run_id}/, so each run is isolated in its own subdirectory. Results are written via session.output_dir instead of a hardcoded study_dir/output path. https://claude.ai/code/session_01ShR8EaCPWNrspYqLNFfV4X Co-authored-by: Claude <noreply@anthropic.com> * fix: resolve mypy type error for Path | None passed to to_csv (#125) Extract output_dir as a local Path variable before constructing the SimulationSession so the non-optional Path is passed directly to table.to_csv() instead of session.output_dir (typed Optional[Path]). https://claude.ai/code/session_014B2YPGb79WZ9boNhFfaaKV Co-authored-by: Claude <noreply@anthropic.com> * fix: update output globs to search subdirectories after per-run output_dir change (#127) runner.py now writes to output/{run_id}/simulation_table_*.csv (introduced in #124), but the e2e tests were still globbing the flat output/ directory. Switch to a recursive glob (**/) so tests find the file regardless of depth. https://claude.ai/code/session_01PXs3nz8srGDfQPGHG8xp5m Co-authored-by: Claude <noreply@anthropic.com> * Integrate PR 92 (non-cyclic constraints) into gemspy-issue-106-strategy (#128) * Integrate PR 92 (non-cyclic constraints) into gemspy-issue-106-strategy Source changes: - Rename ValueType.BOOLEAN → BINARY in model/common.py and variable.py - Remove BlockBorderManagement enum from optimization.py and __init__.py; replace with OutOfBoundsFilter that reads per-constraint mode from optim-config out-of-bounds-processing section (cyclic is the implicit default when a constraint is not listed) - Add ShiftValidityVisitor and _ShiftAmountEvaluator to vectorized_builder.py for computing per-(component, time) DROP validity masks - Add export_lp() helper to OptimizationProblem - Update build_problem() and build_decomposed_problems() signatures: drop border_management param, accept optional optim_config instead - Add OutOfBoundsMode / OutOfBoundsConstraintConfig / OutOfBoundsProcessingConfig to optim_config/parsing.py alongside the existing TimeScopeConfig / SolverOptionsConfig / ScenarioScopeConfig; add _check_oob_constraint_ids validation; extend validate_optim_config to check both model_decomposition and out_of_bounds_processing Test / study changes: - Drop border_management=BlockBorderManagement.CYCLE from all test call sites (test_component_dependent_time_shift, test_libs_*, poc) - Add test_out_of_bounds_processing.py and test_shift_validity_visitor.py from PR 92 - Add 4 new study directories from PR 92 (simple_system_cyclic, simple_system_drop, system_cyclic_with_param_in_shift, system_drop_with_param_in_shift); update their optim-config.yml to include time-scope (first-time-step: 0, last-time-step: 2) https://claude.ai/code/session_01KdDTbQgHkM7DrB9nEZ8ZvR * Delete out-of-bounds processing documentation Removed out-of-bounds processing section from the documentation. --------- Co-authored-by: Claude <noreply@anthropic.com> * Fix mypy errors: duplicate function definition and duplicate keyword argument (#129) - Remove duplicate `_check_oob_constraint_ids` definition in parsing.py (line 297 was identical to line 181) - Remove duplicate `oob_filter` keyword argument in optimization.py `_OptimizationProblemBuilder` call https://claude.ai/code/session_01YHkEmwVVZmYh5aZ4KJZitP Co-authored-by: Claude <noreply@anthropic.com> * Refactor simulation table writing in tests (remove TableWriter) * Update study folder docstring Removed optional optim-config.yml description from docstring. --------- Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: OUSTRY Antoine <aoustry@gmail.com>
#132) * Add E2E test for rolling-horizon suboptimality (Issue #102) and fix session optim_config propagation Adds a new E2E test that demonstrates rolling-horizon suboptimality by comparing frontal (obj=10) vs sequential (obj=406) modes on a 6-step storage+oscillating-load system. The test pins down carry-over SoC divergence at t=2 and t=4, and the resulting unserved energy at t=3 and t=5 in sequential mode. Also fixes a bug in SimulationSession._run_block where optim_config was not passed to build_problem, causing out-of-bounds constraint modes (e.g. 'drop') to be silently ignored in all resolution strategies. https://claude.ai/code/session_01EazRk7MY1LDFKD5crgEpzE * Fix black formatting in test_rolling_horizon_suboptimality.py https://claude.ai/code/session_01EazRk7MY1LDFKD5crgEpzE --------- Co-authored-by: Claude <noreply@anthropic.com>
#130) * Consolidate project config into pyproject.toml and update dependencies - Remove all requirements*.txt/in files; move dependencies into pyproject.toml - Add full [dependency-groups] dev section (mypy, black, isort, pytest-cov, pre-commit, pyarrow, types-PyYAML, antlr4-tools, pandas-stubs) - Add [project.optional-dependencies] doc section (mkdocs, mkdocs-material, mkdocstrings-python) - Update main dependency minimum versions (numpy>=1.26, anytree>=2.13, antlr4-python3-runtime>=4.13.2, highspy>=1.14, xarray>=2025.3, pandas>=2.2, attrs>=26.0, PyYAML>=6.0.2) - Migrate pytest config from pytest.ini to [tool.pytest.ini_options] - Migrate mypy config from mypy.ini to [tool.mypy] and [[tool.mypy.overrides]] - Delete pytest.ini and mypy.ini - Update CI workflow to use uv (astral-sh/setup-uv) instead of pip+requirements - Update .readthedocs.yml to install package with doc optional-dependencies - Regenerate uv.lock with all new dev and doc packages resolved Closes #115 https://claude.ai/code/session_01TsNpaZDGnFEC5boG5LzadV * Fix black config * Formatting * Fix tests import --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Thomas Bittar <thomas.bittar@rte-france.com>
* Fix import sorting in main.py to satisfy isort check https://claude.ai/code/session_01JW9i1Antr2DTLXZ1PJcNN7 * Fix black formatting in e2e test files for textwrap.dedent calls https://claude.ai/code/session_01JW9i1Antr2DTLXZ1PJcNN7 --------- Co-authored-by: Claude <noreply@anthropic.com>
#133) * Implement scenario builder: vectorized MC scenario → column dispatch (issue #101) - ScenarioBuilder.load() parses scenariobuilder.dat (group, mc_scenario = time_serie_number format; 1-based columns stored as 0-based internally) into per-group numpy arrays for loop-free resolution via resolve_vectorized(). - DataBase.get_values() resolves MC scenarios → col_idx at use time (vectorized, no Python loop over S) and dispatches to the underlying data object in one call. - get_value() on all data classes vectorized: scenario arg is now np.ndarray of col_idx; ScenarioSeriesData stores scenario_series as np.ndarray for O(1) indexing. - Scenarization removed from data objects (resolution now lives in DataBase). - build_data_base() accepts optional ScenarioBuilder and records scenario_group per parameter; build_scenarized_data_base() and _resolve_scenarization() deleted. - load_study() passes ScenarioBuilder to build_data_base() so the full study path is wired. - optimization.py is now unaware of ScenarioBuilder/scenario_group: the isinstance + for-s_pos loops are replaced by a single database.get_values() call per component. - Tests: existing unit/e2e tests updated to new API; new dispatch beacon test added. https://claude.ai/code/session_018xbwEtc3QCZ7jGVBoi8fMe * Fix mypy, black and isort issues - folder.py: load ScenarioBuilder before passing it to build_data_base() (fixes used-before-definition mypy error) - parsing.py, optimization.py, test_scenario_builder_dispatch.py: apply black 23.7 and isort formatting https://claude.ai/code/session_018xbwEtc3QCZ7jGVBoi8fMe * Fix test_data_consistency: update ScenarioSeriesData construction to np.ndarray ScenarioSeriesData.scenario_series changed from Mapping[ScenarioIndex, float] to np.ndarray in the scenario builder refactor. Update the two direct instantiations in test_data_consistency.py to pass np.array([100, 50]) and remove the now-unused ScenarioIndex import. https://claude.ai/code/session_018xbwEtc3QCZ7jGVBoi8fMe * Fix shape broadcasting when data type doesn't cover all requested dimensions TimeSeriesData returns (T,) but optimization.py may request (T, S) when a time-only parameter is used in a time+scenario problem. Likewise ScenarioSeriesData returns (S,) but may be assigned into a (T, S) slot. Broadcast to the correct shape so optimization.py can unconditionally write data[i, :, :] = v. https://claude.ai/code/session_018xbwEtc3QCZ7jGVBoi8fMe * Return broadcast view instead of copy to avoid materialising (T,S) array The broadcast view is read-only but optimization.py only reads from it (data[i,:,:] = v), so no copy is needed. Memory cost drops from T*S elements to T or S elements respectively. https://claude.ai/code/session_018xbwEtc3QCZ7jGVBoi8fMe * Use np.asarray and direct .values[] access to avoid unnecessary copies - TimeSeriesData: replace iloc[array].to_numpy() with .values[asarray()]; the iloc path allocated an intermediate pandas Series before copying to numpy; direct .values[] indexing skips that object entirely - TimeScenarioSeriesData / DataBase.get_values(): np.array() forces a copy even when the input is already an ndarray; np.asarray() avoids that The np.ix_ fancy-index copy (TimeScenarioSeriesData) and the scenario_series fancy-index copy (ScenarioSeriesData) are unavoidable for arbitrary index selection and are left as-is. https://claude.ai/code/session_018xbwEtc3QCZ7jGVBoi8fMe * Apply black formatting to data.py https://claude.ai/code/session_018xbwEtc3QCZ7jGVBoi8fMe * Raise ValueError for unknown scenario group instead of silently falling back resolve_vectorized() previously returned the identity mapping when a scenario_group was named but not present in the builder, masking misconfiguration (typo in system.yml, missing entry in scenariobuilder.dat). Now: None → identity (parameter has no group, correct); non-None but missing → ValueError with the group name and list of known groups. https://claude.ai/code/session_018xbwEtc3QCZ7jGVBoi8fMe * Fix test_optim_modes CI failure: remove spurious scenario-group declarations (#135) The dsr_3_blocks study declared scenario-group: sg on all 7 components but had no scenariobuilder.dat file to define that group. This was silently ignored by the old identity-fallback behaviour, but now raises a ValueError after 70c526e enforced strict validation. Since scenario mapping is irrelevant to these optimisation-mode tests (single scenario, single-column data series), simply drop the declarations. https://claude.ai/code/session_01EDyJsnw5zAQGkWAVAruCjc Co-authored-by: Claude <noreply@anthropic.com> * scenario_builder: raise ValueError on incomplete MC scenario mapping (#136) Validates at load time that every integer in 0..max_mc is explicitly present in the scenariobuilder.dat for each group. Previously a missing entry would silently fall through to the identity default (col_idx == mc_scenario), hiding misconfigured files. https://claude.ai/code/session_016CVWfhLdoDHaiMbTM45iwv Co-authored-by: Claude <noreply@anthropic.com> * Guard resolve_vectorized against out-of-bounds MC scenario indices; add tests A numpy IndexError on out-of-bounds access gave no context about which group or which indices were problematic. Add an explicit check that raises ValueError naming the offending indices and the defined range. Tests added: - playlist (subset) resolves correctly - out-of-bounds index raises ValueError - unknown group raises ValueError https://claude.ai/code/session_018xbwEtc3QCZ7jGVBoi8fMe --------- Co-authored-by: Claude <noreply@anthropic.com>
* Implement scenario builder: vectorized MC scenario → column dispatch (issue #101) - ScenarioBuilder.load() parses scenariobuilder.dat (group, mc_scenario = time_serie_number format; 1-based columns stored as 0-based internally) into per-group numpy arrays for loop-free resolution via resolve_vectorized(). - DataBase.get_values() resolves MC scenarios → col_idx at use time (vectorized, no Python loop over S) and dispatches to the underlying data object in one call. - get_value() on all data classes vectorized: scenario arg is now np.ndarray of col_idx; ScenarioSeriesData stores scenario_series as np.ndarray for O(1) indexing. - Scenarization removed from data objects (resolution now lives in DataBase). - build_data_base() accepts optional ScenarioBuilder and records scenario_group per parameter; build_scenarized_data_base() and _resolve_scenarization() deleted. - load_study() passes ScenarioBuilder to build_data_base() so the full study path is wired. - optimization.py is now unaware of ScenarioBuilder/scenario_group: the isinstance + for-s_pos loops are replaced by a single database.get_values() call per component. - Tests: existing unit/e2e tests updated to new API; new dispatch beacon test added. https://claude.ai/code/session_018xbwEtc3QCZ7jGVBoi8fMe * Fix mypy, black and isort issues - folder.py: load ScenarioBuilder before passing it to build_data_base() (fixes used-before-definition mypy error) - parsing.py, optimization.py, test_scenario_builder_dispatch.py: apply black 23.7 and isort formatting https://claude.ai/code/session_018xbwEtc3QCZ7jGVBoi8fMe * Add unit tests for untested code in recent commits Covers gaps identified in the scenario-builder vectorized dispatch (issue #101), binary variable parsing, and the data-layer refactor: ScenarioBuilder (test_scenario_builder.py): - Empty ScenarioBuilder() returns identity for any group - resolve_vectorized with None or unknown group returns mc_scenarios unchanged - ScenarioBuilder.load() correctly skips blank lines and # comments Data layer (tests/unittests/data/test_data.py — new): - load_ts_from_file: .txt path, .tsv path, missing-file error, None-input error - dataframe_to_time_series / dataframe_to_scenario_series: ValueError on wrong shape - Data structure get_value error paths (KeyError when required index is None) for TimeSeriesData, ScenarioSeriesData, and TimeScenarioSeriesData - _build_data error paths: str value with no dependency flags, float value with time/scenario flag set - build_data_base: param-level scenario_group overrides component-level group - DataBase.get_values: no-builder identity pass-through; mc_scenarios=None - DataBase.get_value: ConstantData scalar path (not an ndarray) - DataBase.get_data: KeyError on unknown key Variable factories (test_model.py): - bool_var() produces BINARY type with [0, 1] bounds and default structure - int_variable() produces INTEGER type - Variable.__eq__ returns False for non-Variable comparands https://claude.ai/code/session_012LXLvK8LD4CxfmwdyBq7ZR * Fix test_data_consistency: update ScenarioSeriesData construction to np.ndarray ScenarioSeriesData.scenario_series changed from Mapping[ScenarioIndex, float] to np.ndarray in the scenario builder refactor. Update the two direct instantiations in test_data_consistency.py to pass np.array([100, 50]) and remove the now-unused ScenarioIndex import. https://claude.ai/code/session_018xbwEtc3QCZ7jGVBoi8fMe * Fix isort failures that blocked CI - test_data.py: remove extra blank line after last import block - main.py: reorder run_study/parsing/resolve_components imports into alphabetical order (broken since commit 202f60c) https://claude.ai/code/session_012LXLvK8LD4CxfmwdyBq7ZR * Fix shape broadcasting when data type doesn't cover all requested dimensions TimeSeriesData returns (T,) but optimization.py may request (T, S) when a time-only parameter is used in a time+scenario problem. Likewise ScenarioSeriesData returns (S,) but may be assigned into a (T, S) slot. Broadcast to the correct shape so optimization.py can unconditionally write data[i, :, :] = v. https://claude.ai/code/session_018xbwEtc3QCZ7jGVBoi8fMe * Return broadcast view instead of copy to avoid materialising (T,S) array The broadcast view is read-only but optimization.py only reads from it (data[i,:,:] = v), so no copy is needed. Memory cost drops from T*S elements to T or S elements respectively. https://claude.ai/code/session_018xbwEtc3QCZ7jGVBoi8fMe * Use np.asarray and direct .values[] access to avoid unnecessary copies - TimeSeriesData: replace iloc[array].to_numpy() with .values[asarray()]; the iloc path allocated an intermediate pandas Series before copying to numpy; direct .values[] indexing skips that object entirely - TimeScenarioSeriesData / DataBase.get_values(): np.array() forces a copy even when the input is already an ndarray; np.asarray() avoids that The np.ix_ fancy-index copy (TimeScenarioSeriesData) and the scenario_series fancy-index copy (ScenarioSeriesData) are unavoidable for arbitrary index selection and are left as-is. https://claude.ai/code/session_018xbwEtc3QCZ7jGVBoi8fMe * Apply black formatting to data.py https://claude.ai/code/session_018xbwEtc3QCZ7jGVBoi8fMe * Raise ValueError for unknown scenario group instead of silently falling back resolve_vectorized() previously returned the identity mapping when a scenario_group was named but not present in the builder, masking misconfiguration (typo in system.yml, missing entry in scenariobuilder.dat). Now: None → identity (parameter has no group, correct); non-None but missing → ValueError with the group name and list of known groups. https://claude.ai/code/session_018xbwEtc3QCZ7jGVBoi8fMe * Fix test_optim_modes CI failure: remove spurious scenario-group declarations (#135) The dsr_3_blocks study declared scenario-group: sg on all 7 components but had no scenariobuilder.dat file to define that group. This was silently ignored by the old identity-fallback behaviour, but now raises a ValueError after 70c526e enforced strict validation. Since scenario mapping is irrelevant to these optimisation-mode tests (single scenario, single-column data series), simply drop the declarations. https://claude.ai/code/session_01EDyJsnw5zAQGkWAVAruCjc Co-authored-by: Claude <noreply@anthropic.com> * scenario_builder: raise ValueError on incomplete MC scenario mapping (#136) Validates at load time that every integer in 0..max_mc is explicitly present in the scenariobuilder.dat for each group. Previously a missing entry would silently fall through to the identity default (col_idx == mc_scenario), hiding misconfigured files. https://claude.ai/code/session_016CVWfhLdoDHaiMbTM45iwv Co-authored-by: Claude <noreply@anthropic.com> * Guard resolve_vectorized against out-of-bounds MC scenario indices; add tests A numpy IndexError on out-of-bounds access gave no context about which group or which indices were problematic. Add an explicit check that raises ValueError naming the offending indices and the defined range. Tests added: - playlist (subset) resolves correctly - out-of-bounds index raises ValueError - unknown group raises ValueError https://claude.ai/code/session_018xbwEtc3QCZ7jGVBoi8fMe --------- Co-authored-by: Claude <noreply@anthropic.com>
Wrap `time_series.values` in `np.asarray()` so `result` is always `np.ndarray`, eliminating the invalid ExtensionArray tuple-index and incompatible return-value errors. https://claude.ai/code/session_016diUF6LYFhJdn2JrJyj9Di Co-authored-by: Claude <noreply@anthropic.com>
* update pre-commit * Update
Collaborator
Author
|
Superseded by a clean cherry-pick onto main — only functional changes included. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WIP : Handle dual and reduced cost
Ported from tbittar#145. Closes #182