Appendix F: Monitor Classes

Previous Chapter: Appendix E: Graph Manager Classes

---

• Introduction
• Class hierarchy
• Lifecycle
• Configuration attributes
• Built-in checks and hooks
• Stat ratios
• Reports
• Persisting stats to a collection
• Alerts
• CLI arguments
• Writing and running a monitor

Introduction

A monitor gives a broad, cross-job view of a workflow: it scans the spider and script jobs that ran in a time window, aggregates stats from them (and from their logs), derives ratios, optionally emits a report, and raises alerts when stats breach thresholds. BaseMonitor (shub_workflow/utils/monitor.py) is the base class. Unlike Spidermon (which monitors a single spider job), a BaseMonitor looks across many jobs — useful when zero items in one job is normal but a project-wide drop is not.

It is a BaseScript, not a loop manager (see Appendix B: Script Classes): it runs its checks once and exits, so it's meant to be scheduled periodically (e.g. a daily and a monthly periodic job, or as a task in a graph manager).

This appendix is the complete reference. For an example-led introduction read the Monitors tutorial chapter first.

Class hierarchy

BaseScript                           (Appendix B)
   ▲   ▲   ▲
   │   │   └── AlertSenderMixin                 message queue + senders (--subject)
   │   └────── SpiderStatsAggregatorMixin       aggregate_spider_stats(), target_spider_stats
   └────────── BaseMonitor                      checks, ratios, reports, hooks, alerts
                    ▲
        SentryMixin / SlackMixin  (mix in front of BaseMonitor to enable Sentry / Slack alerts)

Lifecycle

run() does, in order:

1. compute the time window from --period / --start-time / --end-time;
2. auto-discover and run every check_* method (so adding a check_foo(self, start_limit, end_limit) method registers a new check with no wiring). The built-ins are check_spiders, check_scripts_stats, check_script_logs;
3. stats_postprocessing(start_limit, end_limit) — your derived stats;
4. run_stats_ratios() — compute the stats_ratios;
5. generate_report() if --generate-report;
6. run_stats_hooks(start_limit, end_limit) — fire stats_hooks;
7. save_stats_to_collection(...) — persist the stats to a Scrapy Cloud collection if --collection-name was given (see Persisting stats to a collection);
8. upload_stats(), print_stats(), then close() (which sends queued alerts).

All aggregated values live in self.stats (the Scrapy stats collector). When --flow-id is set (or the monitor is scheduled by a graph manager, inheriting its flow id), the checks only consider jobs of that workflow; otherwise every job in the window is considered.

Configuration attributes

The monitor is mostly declarative — set these on your subclass:

Attribute	Type	Meaning
target_spider_classes	{SpiderClass: stats_prefix}	which spiders to scan, and the stats prefix for each group. Default {Spider: ""} (all spiders, no prefix).
target_spider_stats	tuple[str, ...]	extra stat-key regex prefixes to aggregate from spiders, on top of the always-on BASE_TARGET_SPIDER_STATS.
stats_only_total	bool	if True, only emit …/total (not per-spider). Default False on the monitor.
target_script_stats	{script: ((regex, prefix), …)}	aggregate matching stats of a script (py:foo.py) into <prefix>/<group-or-statname> (+ <prefix>/total).
target_script_logs	{script: ((regex, stat), …)}	aggregate numbers parsed from a script's log lines (for long-running scripts whose final stats aren't enough).
stats_ratios	((num_regex, den_regex, target), …)	compute ratio stats (see Stat ratios).
stats_hooks	((stat_regex, method_name), …)	call a method when a stat matches (see below).
report_table	tuple[tuple[str, …], …]	header row + data rows for generate_report().
additional_projects	tuple[int, …]	also scan these project ids (multi-project setups).
default_subject	str	default alert subject (AlertSenderMixin; --subject overrides).

BASE_TARGET_SPIDER_STATS (always aggregated from spiders): downloader/response_status_count/, downloader/response_count, item_scraped_count, spider_exceptions/, scrapy-zyte-api/429, zyte_api_proxy/response/status/429. Each is emitted as <prefix>/<statkey>/<spider> and <prefix>/<statkey>/total (per-spider entries omitted when stats_only_total).

Built-in checks and hooks

Member	Purpose
check_spiders(start, end)	scan finished jobs of target_spider_classes spiders; call spider_job_hook(jobdict) then aggregate_spider_stats.
check_scripts_stats(start, end)	aggregate target_script_stats from finished script jobs; calls script_job_hook(jobdict).
check_script_logs(start, end)	aggregate target_script_logs from running+finished script jobs' logs.
check_<name>(self, start, end)	your own check — auto-discovered and run; use for filesystem counts, API quotas, etc.
spider_job_hook(jobdict) / script_job_hook(jobdict)	override — per-job custom aggregation.
stats_postprocessing(start, end)	override — derive stats after all checks.
aggregate_spider_stats(jobdict, prefix="")	the aggregation primitive (override to customize).

stats_hooks map a stat regex to a method called as hook(start_limit, end_limit, value, *regex_groups) — one extra arg per regex group. Typical use is to compare value against a threshold and self.append_message(...) an alert.

Stat ratios

Each stats_ratios entry is (numerator_regex, denominator_regex, target_stat). Stats matching the numerator/denominator are summed (grouped by the first regex group if present) and the ratio is written to target_stat (suffixed with the group when the numerator has one), rounded to 4 decimals. The monitor also auto-adds response-status-code rate ratios (…/downloader/response_count/rate/<code>/<spider>) for each target_spider_classes prefix. Ratios run after stats_postprocessing.

Reports

Set report_table (a tuple of string tuples: header row, then data rows — usually built in stats_postprocessing from collected stats) and run with --generate-report. --report-format chooses pretty (default), pretty_with_borders, pretty_with_tabs (paste into a spreadsheet) or csv; --slack-report sends it to Slack instead of printing.

Persisting stats to a collection

With --collection-name <name>, after the stat hooks run the monitor stores the entire stats dict as a single record in that Scrapy Cloud collection (in the target project) — the built-in way to keep a history of monitor stats across runs (e.g. one record per day), instead of a separate "super-monitor". The record's _key is the window's date YYYY-MM-DD for a one-day window, or <start> to <end> otherwise.

The window must be midnight-aligned: both the start and end of the window must be at 00:00, otherwise saving is skipped with an error logged. So schedule it with day-aligned windows — e.g. a daily run with -e "today at 0:00", or a monthly run spanning whole days.

Alerts

BaseMonitor is an AlertSenderMixin: queue alert text with self.append_message(msg) (typically from a stats_hooks method), and close() flushes the queue through every registered sender. By default no sender is registered (messages just stay queued). To actually deliver:

• Slack — subclass SlackMixin (shub_workflow.contrib.slack); needs SPIDERMON_SLACK_* settings and spidermon[slack-sdk].
• Sentry — subclass SentryMixin (shub_workflow.contrib.sentry); needs SPIDERMON_SENTRY_* settings and spidermon[sentry-sdk].

Mix the alert class in front of BaseMonitor: class Monitor(SlackMixin, BaseMonitor). The alert subject comes from --subject or default_subject. Both senders honour a "fake" setting (SPIDERMON_*_FAKE) that logs instead of sending — handy for local runs.

CLI arguments

--period/-p (window length in seconds, default 86400), --start-time/-s and --end-time/-e (any dateparser string; -s overrides -p), --generate-report, --report-format, --slack-report, --collection-name (save stats to a Scrapy Cloud collection — see Persisting stats to a collection), --subject; plus BaseScript options (--project-id, --flow-id, --children-tag, -g/-v). The time window filters jobs by finish time (and, for script-log aggregation, log-line timestamps).

Writing and running a monitor

import logging

from shub_workflow.utils.monitor import BaseMonitor
from shub_workflow.contrib.slack import SlackMixin
from myproject.spiders import MyBaseSpider     # the spider class(es) to monitor


class Monitor(SlackMixin, BaseMonitor):        # alert mixin first

    default_subject = "MyProject monitor"
    target_spider_classes = {MyBaseSpider: "discovery"}
    target_spider_stats = ("dropped_items/",)
    target_script_stats = {"py:deliver.py": ((r"delivered_count/(.+)", "delivered"),)}
    stats_ratios = (("spider_exceptions/.+/(.+)", "item_scraped_count/(.+)", "exceptions_rate"),)
    stats_hooks = ((r"^discovery/item_scraped_count/total$", "no_items_hook"),)

    def no_items_hook(self, start_limit, end_limit, value, *groups):
        if value == 0:
            self.append_message("ALERT: zero items scraped in the window.")


if __name__ == "__main__":
    from shub_workflow.utils import get_kumo_loglevel

    logging.basicConfig(format="%(asctime)s %(name)s [%(levelname)s]: %(message)s", level=get_kumo_loglevel())
    Monitor().run()

A monitor runs once and exits, so schedule it periodically (e.g. a daily and a monthly periodic job, each with its own tag, or as a task in a graph manager). As with any shub-workflow script, register it in your project's setup.py to deploy and run it on Scrapy Cloud as py:monitor.py.

---

Next Chapter: Appendix G: Issuer Classes