DEVELOPMENT

Development Guide

pyproject.toml

pyproject.toml is the project's standard Python packaging and tool configuration file. It tells installers how to build the project, which dependencies it needs, how to expose the command-line program, and where the source code, templates, and tests are located.

Build system

[build-system]
requires = ["setuptools>=68"]
build-backend = "setuptools.build_meta"

The project uses setuptools as its build backend. When building or installing the package, Python's build frontend first makes setuptools version 68 or newer available, then asks setuptools.build_meta to build it.

Project metadata and dependencies

The [project] table defines the distribution name (playbook-scheduler), version, short description, README used as package documentation, and minimum Python version (3.10).

The dependencies list contains packages installed for every use of Playbook Scheduler:

• APScheduler>=3.10,<4 provides scheduled execution. The upper bound keeps the project on the APScheduler 3.x API.
• Jinja2>=3.1 renders the HTML report.
• PyYAML>=6.0 reads the YAML configuration.

Version constraints specify a minimum acceptable version. APScheduler also has an explicit maximum; the Jinja2 and PyYAML dependencies do not.

Optional development dependencies

[project.optional-dependencies]
dev = [
    "pylint>=4.1.2",
    "pytest>=9.1.1",
    "ruff==0.16.10",
    "yamllint==1.38.0",
]

The dev extra adds pytest for tests, Ruff for Python linting, and yamllint for YAML linting. Install the package and development tools in the active environment with:

python -m pip install -e ".[dev]"

The -e flag installs the package in editable mode, so changes under src/ are used without reinstalling after every edit. ansible-core provides the external ansible-playbook executable and remains separate from the development tools. It is pinned in requirements.txt for the default Ansible executable setup; jobs using their own ansible_venv can use Ansible installed in that project environment instead.

Command-line entry point

[project.scripts]
playbook-scheduler = "playbook_scheduler.cli:main"

This creates the playbook-scheduler command when the package is installed. It calls the main function in src/playbook_scheduler/cli.py.

Source layout and package data

The setuptools configuration maps the Python package root to src/ and tells setuptools to discover packages there. Consequently, the importable package is playbook_scheduler, even though the project is laid out using the common src/ directory convention.

The package-data setting includes *.j2 files under src/playbook_scheduler/templates/ in built distributions. These files contain the Jinja2 HTML report template and must be present for report generation to work after installation.

pytest configuration

The [tool.pytest.ini_options] table makes tests/ the default test directory and adds src/ to pytest's import path. This lets tests import playbook_scheduler directly from the source tree, including when the project has not been installed.

Run the tests with:

python -m pytest

Tests

The files under tests/ are automated pytest tests for Playbook Scheduler itself. They check that the code still behaves correctly after changes to the report template, the configuration handling, the scheduler, and so on. They do not test your playbooks or hosts: every test runs in a temporary directory, and Ansible is never actually started.

File	Covers
test_config.py	Loading the configuration: path resolution, invalid cron expressions, missing playbooks, ansible_venv
test_runner.py	Running a job: success, non-zero exit code, timeout and stopping Ansible, virtual environment
test_reporting_retention.py	HTML report (escaping, filters, skipped runs, shortened output) and removal of expired runs
test_cli.py	Error output of the command-line interface, ansible-playbook check in validate
test_shutdown.py	Stopping running jobs and the scheduler on SIGTERM/SIGINT
test_scheduler.py	Automatic reload of the configuration while serve is running
test_concurrency.py	Lock files, skipping a job that is already running, atomic writes, max_parallel_jobs

Run in the terminal

From the project directory, with the virtual environment activated and the dev extra installed:

source .venv/bin/activate
pytest                          # all tests
pytest -v                       # show the name of every test
pytest tests/test_config.py     # a single file
pytest -k reload                # only tests whose name contains "reload"
pytest -x                       # stop at the first failure

Dots or PASSED mean the tests succeeded. On FAILED, pytest shows which assert failed and the values involved.

Run in VS Code

1. Select .venv as the Python interpreter (Ctrl+Shift+P → "Python: Select Interpreter").
2. Open the Testing view (flask icon) in the sidebar and, when prompted, choose pytest and the tests directory.
3. Run all tests or individual ones from there. Every def test_… line in a test file also gets a ▶ button; right-click it and choose "Debug Test" to stop at breakpoints.

Write new tests

pytest automatically collects every test_* function in tests/test_*.py. A test usually consists of one or more assert statements. Add a tmp_path parameter to receive an empty temporary directory for files the test creates. tests/test_config.py is a good starting point.

Run pytest before every commit.

Concurrency and locking

Several Ansible runs can be active at the same time: the scheduler runs different jobs in parallel, and playbook-scheduler run can be started while serve is running. All of them share the same runs_directory and reports_directory. The rules below keep them from getting in each other's way.

Thread pools in the scheduler

run_scheduler in src/playbook_scheduler/scheduler.py creates the APScheduler BlockingScheduler with two thread pools (executors):

Executor	Threads	Runs
default	max_parallel_jobs (default 10)	The configured Ansible jobs
config_watcher	1	The configuration check every 30 seconds

Every job runs ansible-playbook as a child process from one thread of the default pool, so up to max_parallel_jobs different jobs run at once. When all threads are busy, a due job waits for a free thread. If it waits longer than misfire_grace_time (one hour), it is dropped for this fire time. The watcher has its own pool so that long playbook runs cannot delay reloading the configuration. Thread pools cannot be resized while running, so a changed max_parallel_jobs only logs a warning and takes effect after a restart.

Each job is registered with these APScheduler options:

• max_instances=2: one instance runs Ansible. The second instance exists only so that a fire time during a running job reaches execute_job, where the job lock rejects it and a skipped run is recorded. With max_instances=1, APScheduler would drop that fire time itself and only log a warning, so the report would not show it.
• coalesce=True and misfire_grace_time=3600: fire times that were missed, for example because the process was blocked or restarting, are combined into one run if they are at most one hour late.

The scheduled function is run_scheduled_job. It logs the start of the job, calls execute_job, and logs the result: INFO for success, ERROR for any other status. JobAlreadyRunning becomes a single WARNING line Skipped scheduled run: … instead of a traceback.

The configuration watcher has misfire_grace_time set to its interval. With APScheduler's default of one second, a check that started slightly late would log a "was missed" warning.

Lock files

APScheduler's max_instances only works inside one process and would drop the second run without a record. To prevent parallel runs of a job in all cases, including playbook-scheduler run and a second serve process, src/playbook_scheduler/files.py provides file_lock, an exclusive lock on a file under runs_directory/.locks/:

Lock file	Mode	Held while
<job name>.lock	Non-blocking	ansible-playbook is running for that job
_report.lock	Blocking	Retention and report generation run

• Job lock (execute_job in service.py): if the lock is already held, file_lock(..., blocking=False) raises LockBusy immediately, and execute_job records a run with status skipped (record_skipped_run in runner.py), refreshes the report, and raises JobAlreadyRunning. The run is not queued. playbook-scheduler run exits with code 3.
• Report lock (refresh_report in service.py): when two jobs finish at the same time, the second one waits until the first has finished removing expired runs and writing the report. Without it, one thread could delete an old run file while another is reading it, which would fail with FileNotFoundError. The job lock is released before the report lock is taken, so a run never holds both.

The lock name _report starts with an underscore and can therefore not collide with a job name, which must start with a letter or number.

How file_lock works:

• On Linux and macOS it uses fcntl.flock, on Windows msvcrt.locking on the first byte of the file. Waiting for a blocking lock polls every 0.1 seconds.
• Each call opens its own file handle. These locks belong to the handle, not to the process, so they also exclude other threads of the same process.
• The operating system releases the lock when the process ends, including on a crash or kill -9. A lock file left behind therefore does not block later runs.
• Lock files are intentionally never deleted. If one process deleted the file while another waited on it, a third process would create a new file and receive a second, independent lock.
• flock can be unreliable on network file systems such as NFS. Keep runs_directory on a local file system if several processes use it.

Atomic writes

atomic_write_text in files.py first writes into a temporary file in the target directory and then replaces the target with os.replace. Readers, such as a browser opening reports/index.html or load_runs reading run JSON files, therefore always see either the complete old file or the complete new file, never a partially written one. Run records (runner.py) and the HTML report (reporting.py) both use it. Temporary files start with a dot and end in .tmp, so *.json globs never pick them up.

Timeout and stopping Ansible

run_job in src/playbook_scheduler/runner.py starts ansible-playbook with subprocess.Popen and waits with communicate(timeout=timeout_seconds). When the timeout expires, _stop_process stops the process in two steps:

1. Send SIGTERM (process.terminate()) and wait up to STOP_GRACE_SECONDS (30 seconds).
2. Only if Ansible has not exited by then, send SIGKILL (process.kill()).

The order matters. Ansible runs every worker process in its own session, so the workers and the tasks they started are not part of ansible-playbook's process group. On SIGTERM, ansible-playbook stops its workers and their tasks itself. On SIGKILL it cannot clean up, and the workers keep running the current task on the managed hosts after the run has already been recorded as timed out. subprocess.run(timeout=...), which Playbook Scheduler used before, sends SIGKILL immediately and therefore left such processes behind.

The output produced until the stop is kept in stdout and stderr of the run record, return_code stays null, and error says Timed out after … seconds.

The same stop sequence runs if waiting is interrupted, for example by Ctrl+C during playbook-scheduler run; the interruption is then raised again.

Stopping the scheduler

run_scheduler registers StopSignalHandler (scheduler.py) for SIGTERM and SIGINT. Python runs signal handlers in the main thread, which in serve only runs the APScheduler loop; the jobs run in the thread pool.

On the first signal the handler:

1. Calls stop_running_jobs() in runner.py. It sets the stop flag and sends SIGTERM to every ansible-playbook process in _running_processes. As with the timeout, SIGTERM lets Ansible end its workers and tasks.
2. Calls scheduler.shutdown(wait=False), so no new runs start and scheduler.start() returns.

Back in run_scheduler, jobs_executor.shutdown(wait=True) waits until every job thread has finished. Each interrupted run returns from communicate(), sees the stop flag, is recorded as failed with Interrupted by shutdown., and refreshes the report. Then serve exits with code 0. A run that exits with code 0 at the same moment is recorded as successful as usual.

A second signal calls stop_running_jobs(force=True), which sends SIGKILL to processes that ignored SIGTERM. That may leave Ansible's workers behind, which is why it is not the first step.

Details:

• run_job adds every started process to _running_processes under _running_lock and removes it when it ends. A process that starts just after stop_running_jobs() went through the set sees the stop flag when it is added and is terminated right away.
• The handler does not wait itself. Waiting in a signal handler would block the main thread and could be interrupted by the next signal.
• With systemd, KillMode=mixed (see README.md) makes sure only Playbook Scheduler receives SIGTERM. With the default control-group, systemd also sends SIGTERM to Ansible directly. Ansible may then exit before Playbook Scheduler has set its stop flag, and the run is recorded as a normal failure with exit code -15.

Checking the ansible-playbook executable

find_ansible_playbook in config.py returns the executable a job would run: with ansible_venv the one inside that environment, otherwise the first ansible-playbook on PATH, found with shutil.which. which also checks that the file is executable. validate in cli.py prints the path per job, or an error, and exits with code 2 if any job has no executable.

load_config does not run this check. Otherwise every test and every configuration reload would need an Ansible installation, and serve would reject a configuration only because PATH differs.

Tests

tests/test_concurrency.py covers the lock behavior, a second run of a busy job in execute_job including its skipped record, the CLI and the scheduler, logging, atomic writes, and the max_parallel_jobs and report_output_lines settings.

Logging

serve calls configure_logging() in scheduler.py before the scheduler starts. It configures the root logger with logging.basicConfig:

• Format %(asctime)s %(levelname)s %(message)s, date format %Y-%m-%d %H:%M:%S%z, which is local time with UTC offset.
• Level INFO, output to standard error.
• The apscheduler logger is set to WARNING. At INFO, APScheduler logs every execution, including the configuration check every 30 seconds. Warnings such as missed fire times still appear.

The modules log through logging.getLogger(__name__). Only serve configures logging. validate, run, and report are interactive and keep writing plain text to standard output and standard error with print.

The signal handler also logs. Python runs it in the main thread between two bytecode instructions; logging handlers use a reentrant lock, so this cannot deadlock in the main thread.

Report size

The run records keep the complete output. Only the HTML report is shortened, by tail_output in reporting.py:

1. Keep the last report_output_lines lines (default 200).
2. Cut every remaining line after MAX_LINE_CHARS (1000) characters and mark it with …[line cut]. With -v, Ansible prints each task result as a single JSON line, which a line limit alone would not shorten.

generate_report passes tail_output with the configured number of lines to the template as tail. The template shows a note with the number of omitted lines and the file name of the JSON record. load_runs adds that name as record_file to every loaded record; it is not stored in the JSON file.

The output is the main contributor to the report size: stdout of each run of the last retention_days days, all in one file.

Code Quality

To garanty good code quality we use:

ruff → Formatting and various style checks

pylint → Architecture and code quality checks

pytest → Tests

yamllint → YAML files

pylint

Python code linter

pylint $(git ls-files '*.py')

pytest

Run the python tests

pytest

Ruff

Check the code locally with Ruff:

ruff check .

Besides the default rules, pyproject.toml enables:

• D (pydocstyle): every module, class, and function under src/ needs a docstring following PEP 257. The summary line is in the imperative mood ("Return …", not "Returns …"); arguments, return values, yielded values, and raised exceptions are described in Google-style Args:, Returns:, Yields:, and Raises: sections.
• ANN (flake8-annotations): every function needs PEP 484 type hints for all parameters and the return value. Types that are only needed for annotations, such as Ansible's TaskResult in the callback plugin, are imported under if TYPE_CHECKING: so they are not imported at runtime.

Tests under tests/ are exempt from both rules.

yamllint

Check the YAML files with yamllint:

yamllint config/ src/

Open report in WSL

If you are working in a WSL you need to create a web server like this.

python3 -m http.server 8000 --directory reports

Then open in the Windows browser http://localhost:8000 . WSL2 does a forwarding of localhost to the Windows system.