Version 2.0.0b1

[2.0.0b1]

Added

• New merlin database command to interact with new database functionality
  • When running locally, SQLite will be used as the database. Otherwise your current results backend will be used
  • merlin database info: prints some basic information about the database
  • merlin database get: allows you to retrieve and print entries in the database
  • merlin database delete: allows you to delete entries in the database
• Added db_scripts/ folder containing several new files all pertaining to database interaction
  • data_models: a module that houses dataclasses that define the format of the data that's stored in Merlin's database.
  • db_commands: an interface for user commands of merlin database to be processed
  • merlin_db: houses the MerlinDatabase class, used as the main point of contact for interactions with the database
  • entities/: A folder containing modules that define a structured interface for interacting with persisted data.
  • entity_managers/: A folder containing classes responsible for managing high-level database operations across all entities.
• Added backends/ folder containing a new OOP way to interact with results backend databases
  • results_backend: houses an abstract class ResultsBackend that defines what every supported backend implement in Merlin
  • redis/: A folder containing the RedisBackend class that defines specific interactions with the Redis database
  • sqlite/: A folder containing the SQLiteBackend class that defines specific interactions with the SQLite database
  • backend_factory: houses a factory class MerlinBackendFactory that initializes an appropriate ResultsBackend instance
• Added monitors/ folder containing a refactored, OOP approach to handling the merlin monitor command
  • celery_monitor: houses the CeleryMonitor class a concrete subclass of TaskServerMonitor for monitoring Celery task servers
  • monitor_factory: houses a factory class MonitorFactory that initializes an appropriate TaskServerMonitor instance
  • monitor: houses the Monitor class, used as the top-level point of interaction for the monitor command
  • task_server_monitor: houses the TaskServerMonitor ABC class, which serves as a common interface for monitoring task servers
• New worker related classes:
  • MerlinWorker: base class for defining task server workers
  • CeleryWorker: implementation of MerlinWorker specifically for Celery workers
  • WorkerFactory: to help determine which task server worker to use
  • MerlinWorkerHandler: base class for managing launching, stopping, and querying multiple workers
  • CeleryWorkerHandler: implementation of MerlinWorkerHandler specifically for manager Celery workers
  • WorkerHandlerFactory: to help determine which task server handler to use
• A new celery task called mark_run_as_complete that is automatically added to the task queue associated with the final step in a workflow
• Ability to filter database queries for the get all-* and delete all-* commands
• New MerlinBaseFactory class to help enable future plugins for backends, monitors, status renderers, etc.
• Ability to turn off the auto-restart functionality of the monitor with --no-restart
• Tests for the monitor files

Changed

• Maestro version requirement is now at minimum 1.1.10 for status renderer changes
• The BackendFactory, MonitorFactory, and StatusRendererFactory classes all now inherit from MerlinBaseFactory
• Launching workers is now handled through worker classes rather than functions in the celeryadapter.py file