Repository navigation
Releases: profullstack/scraper-daemon
Release list
0.3.0: cadence + daemon.feeds()
Tiered adaptive feed crawls: planNext (15 min to 1 week by rhythm, archive after 180 days silent), daemon.feeds() dispatch by tier with BullMQ priority, runNow for refresh buttons.
v0.2.2
A failed job no longer blocks its task forever: failed task jobs are removed, and enqueue clears a finished job still holding the task's id, so a queue wedged by 0.2.1 recovers on upgrade (#5).
v0.2.1
A Redis that is down never wedges the daemon or its status page: bounded status reads (storeAnswered:false), watchdog armed as soon as the connection exists, no Queue/Worker before Redis answers a PING, stop() drops connections that cannot close, one memoised connection, ioredis errors to the daemon log. New option redisCheckMs.
v0.2.0: BullMQ engine
The standard queue for every site is now BullMQ. createDaemon({ redis }) runs one queue per site on the site's own Redis: a job scheduler fires the dispatcher tick, one Worker runs due work at the site's concurrency. Same API and scheduling rules as 0.1 (error ladder, per-host serial, retire, nextIn, resumable pipelines). Run records and task state live in Redis beside the queue unless the site passes its own store; each app keeps its data where it wants.
Hardened for the fleet's BullMQ incidents: job ids never contain ':'; processors return nothing, completed jobs are removed and event streams capped on Queue and Worker; no QueueEvents; a watchdog pings Redis through the daemon's own connection and exits after three misses; status() gives up on Redis after 3 s.
Also: the local engine no longer unrefs its timer (a poller with no server exited after one round). bullmq + ioredis are optional peer dependencies.
v0.1.1
pgStore stores state as real jsonb through any driver. Through Bun's SQL client 0.1.0 wrote every state value double-encoded (a jsonb string holding JSON text); values now cross the driver as text both ways, and old rows are unwrapped on first use. A Postgres self-test now runs in CI.
v0.1.0
First release: one updater daemon for every listing site. Jobs, refresh queues (oldest due first, per-host serial, own cadence, retire after N failures) and resumable pipelines; polite and SSRF-safe fetch with robots.txt and 429 hold-back; memory, SQLite and Postgres stores; status() for /api/v1/daemon, onRun/onStall hooks and a status CLI. Zero dependencies, Node 20+ and Bun.