Skip to content

Releases: profullstack/scraper-daemon

0.3.0: cadence + daemon.feeds()

Choose a tag to compare

@sh1pt-actions-fleet sh1pt-actions-fleet released this 29 Oct 00:54
8ffacba

Tiered adaptive feed crawls: planNext (15 min to 1 week by rhythm, archive after 180 days silent), daemon.feeds() dispatch by tier with BullMQ priority, runNow for refresh buttons.

v0.2.2

Choose a tag to compare

@sh1pt-actions-fleet sh1pt-actions-fleet released this 28 Oct 16:26
103bc80

A failed job no longer blocks its task forever: failed task jobs are removed, and enqueue clears a finished job still holding the task's id, so a queue wedged by 0.2.1 recovers on upgrade (#5).

v0.2.1

Choose a tag to compare

@ralyodio ralyodio released this 19 Oct 14:15
be72110

A Redis that is down never wedges the daemon or its status page: bounded status reads (storeAnswered:false), watchdog armed as soon as the connection exists, no Queue/Worker before Redis answers a PING, stop() drops connections that cannot close, one memoised connection, ioredis errors to the daemon log. New option redisCheckMs.

v0.2.0: BullMQ engine

Choose a tag to compare

@ralyodio ralyodio released this 19 Oct 13:33
f458f6b

The standard queue for every site is now BullMQ. createDaemon({ redis }) runs one queue per site on the site's own Redis: a job scheduler fires the dispatcher tick, one Worker runs due work at the site's concurrency. Same API and scheduling rules as 0.1 (error ladder, per-host serial, retire, nextIn, resumable pipelines). Run records and task state live in Redis beside the queue unless the site passes its own store; each app keeps its data where it wants.

Hardened for the fleet's BullMQ incidents: job ids never contain ':'; processors return nothing, completed jobs are removed and event streams capped on Queue and Worker; no QueueEvents; a watchdog pings Redis through the daemon's own connection and exits after three misses; status() gives up on Redis after 3 s.

Also: the local engine no longer unrefs its timer (a poller with no server exited after one round). bullmq + ioredis are optional peer dependencies.

v0.1.1

Choose a tag to compare

@ralyodio ralyodio released this 19 Oct 12:30
7f6d305

pgStore stores state as real jsonb through any driver. Through Bun's SQL client 0.1.0 wrote every state value double-encoded (a jsonb string holding JSON text); values now cross the driver as text both ways, and old rows are unwrapped on first use. A Postgres self-test now runs in CI.

v0.1.0

Choose a tag to compare

@ralyodio ralyodio released this 19 Oct 11:52
5587e44

First release: one updater daemon for every listing site. Jobs, refresh queues (oldest due first, per-host serial, own cadence, retire after N failures) and resumable pipelines; polite and SSRF-safe fetch with robots.txt and 429 hold-back; memory, SQLite and Postgres stores; status() for /api/v1/daemon, onRun/onStall hooks and a status CLI. Zero dependencies, Node 20+ and Bun.