Skip to content

Conversation

@pabloitu
Copy link
Collaborator

@pabloitu pabloitu commented Nov 1, 2025

  • Implementation to run an experiment concurrently, with tasks have associated dependencies, and are placed into a directed acyclic graph. This mode is only available for time dependent models, whose source code is encapsulated in a docker image. In practice, catalog and evaluations tasks are sequential, whereas forecasts creation is run in parallel. User can set run_mode in the configuration file as sequential (default) or parallel.
  • Decoupled engine runner from profiling. Profiler prints (in debug) a timing summary grouped by task class. Added catch for keyboard interrupts, so all running processes can be killed before exiting (critical for parallel running and container deployment).
  • Minor rework of Task.str, such that it prints its instance name by default.
  • Docker environments now properly catch errors and logs. Added also a static method, called from TaskGraph, that effectively kills all containers through the python SDK.
  • To handle parallelization, models require to have a unique input folder for each time window. Therefore, a volume is set between a folder from the host filesystem (other than model/input), to /app/model/input within the container. Contents are located dynamically before running the container, by the methods CatalogRepository.set_input_cats, Model.prepare_args and Model.prepare_extra_input.
  • Important fix on time_config parsing, which sometimes caused that time-dependent experiments were not recognized as such. Also, catalog repository now parses existing dropped catalog, rather than filtering it for each task.
  • Updated all of the existing unit tests, extended engine and environments tests. created engine-model and engine-experiment integration tests. Added remaining tutorial cases (h,i,j) for end-to-end tests, which can only be run locally for processing time and space limitations in the CI.
  • Added run_mode and force_rerun parameters in experiment configurations. Updated all refactored classes from last commits

…tasks have associated dependencies, and are placed into a directed acyclic graph. This mode is only available for time dependent models, whose source code is encapsulated in a docker image. In practice, catalog and evaluations tasks are sequential, whereas forecasts creation is run in parallel. User can set run_mode in the configuration file as sequential (default) or parallel.

ft: decoupled engine runner from profiling. Profiler prints (in debug) a timing summary grouped by task class. Added catch for keyboard interrupts, so all running processes can be killed before exiting (critical for parallel running and container deployment).
ft: minor rework of Task.__str__, such that it prints its instance name by default.
ft: docker environments now properly catch errors and logs. Added also a static method, called from TaskGraph, that effectively kills all containers through the python SDK.
ft: to handle parallelization, models require to have a unique input folder for each time window. Therefore, a volume is set between a folder from the host filesystem (other than model/input), to /app/model/input within the container. Contents are located dynamically before running the container, by the methods CatalogRepository.set_input_cats, Model.prepare_args and Model.prepare_extra_input.
fix: important fix on time_config parsing, which sometimes caused that time-dependent experiments were not recognized as such.
tests: updated all of the existing unit tests, extended engine and environments tests. created engine-model and engine-experiment integration tests. Added remaining tutorial cases (h,i,j) for end-to-end tests, which can only be run locally for processing time and space limitations in the CI.
docs: Added run_mode and force_rerun parameters in experiment configurations. Updated all refactored classes from last commits
@pabloitu pabloitu linked an issue Nov 1, 2025 that may be closed by this pull request
@pabloitu pabloitu merged commit 777ae3b into main Nov 1, 2025
7 checks passed
@pabloitu pabloitu deleted the 56-decide-on-and-implement-dag-solving-strategy branch November 1, 2025 18:22
@pabloitu pabloitu mentioned this pull request Nov 12, 2025
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Decide on and implement DAG solving strategy

2 participants