Repository navigation
v8.0.0 #182
Graeme22
announced in
Announcements
v8.0.0
#182
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
streaQ v8.0.0
v8 is a large, breaking release with a new dead-letter queue, reworked task pipelining, and a refreshed web UI. It contains a variety of improvements including security, bugfixes, and performance.
Not only has streaQ's API changed, but also its core data format and Lua scripts. This means v8 workers cannot run tasks enqueued with v7 and vice versa. In fact, for safety v8 uses a modified queue prefix to ensure it won't even see your v7 queues (and vice versa). This makes it easy to migrate safely:
At this point you can safely delete v7 keys and stop all v7 workers.
Pipelining redesign
then()andotherwise()are curried: Both now take the next task and return a function, which you call with that task's remaining arguments. This separates how tasks are linked from what each one receives. It lets type checkers verify the child's signature against the parent's output and allows positional arguments in addition to forwarded ones:Chaining operators removed:
|and^are gone: usethen()andotherwise()instead. The operators were a precedence trap, asawait a | bparses as(await a) | b, so every use needed extra parentheses, and since|and^have different precedence,a | b ^ cparses asa | (b ^ c)and silently builds the wrong chain.The parent's output is the child's first argument: Tuple outputs are no longer unpacked. Whatever the parent returns is passed as the child's first positional argument, and anything you pass when building the chain comes after it. So now typing is more straightforward and you get more flexibility to pass additional arguments if necessary.
forward=Falseruns a task after another without passing its output:then(task, forward=False)waits for the parent to succeed but doesn't pass its result on, so the child gets only the arguments you give it. Some chains are about ordering, not data: send a notification after something finishes, or clean up after a job. v7 had no way to express that in a chain without the child accepting, and then ignoring, the parent's output.otherwise()can use its own arguments: By default, a fallback still replays the failed task's args and kwargs, as in v7. Withforward=False, it uses the arguments you pass instead. Either way it's skipped and returns the parent's result if the parent succeeds.Built-in
map()andfilter(): v8 provides new built-in pipelining utilities for common operations:task.map(fn)(...)runsfnon each item oftask's output, in parallel, then aggregates the results into a list.task.filter(predicate)(...)runspredicateon each item oftask's output, in parallel, and keeps the items for whichpredicateis truthy.fn.map(items)(...)andpredicate.filter(items)(...)do the same, but start from an iterable directly.Extra arguments can be passed to every call and results come back in input order. If any subtask fails, the whole step fails, with the subtask's exception as the cause. The built-ins give subtasks deterministic IDs, so a retried map doesn't re-run items that already succeeded, and they use a single pub/sub connection to listen for results, regardless of how many items there are.
start(after=...)removed (no more fan-in or DAG): Dependencies are expressed only through chains. Each task has at most one parent and one child, and a task can no longer wait on arbitrary task IDs. Since fan-in needed per-task dependency sets and extra Lua to keep them consistent, and it was the main source of leaked and orphaned tasks. Linear chains can be stored on the tasks themselves, with deterministic child IDs, and this simplification makes new features possible: idempotency and retrying pipelined tasks.So this code is no longer works directly:
However, this and other complex relationships can still be expressed via orchestrators.
Only the first task in a chain can be configured:
start()raises on any link except the first. Cron schedules and tasks registered withexpirecan't be chained either.New features
Worker(dlq_ttl=...)to park failed tasks instead of discarding them:0(default) disables it,Nonekeeps them forever. Parked tasks can be inspected, requeued withrequeue_by_id()or removed withpurge_by_id(), from code or from the UI. Aborted, expired, and dependency-failed tasks aren't parked, nor are tasks with a fallback.DEADandDEPENDS.get_tasks_by_status(DONE)excludes dead tasks when the DLQ is on.1.235M), and a wider layout.Smaller breaking API changes
uniqueparameter removed: The per-task Redis lockREDIS_UNIQUEis gone from the task/cron decorators,RegisteredTask,finish_task, and the docs. Cron jobs no longer default tounique=True+ 1h timeout, and the "unique tasks must have a timeout" error is gone. Uniqueness was a clunky solution to problems that no longer exist thanks to the atomic cron scheduling script present in streaQ for a while now.streaq.utils.to_tuple()max_schedule_driftfor cron tasks is nowexpire: a payload TTL measured from when the task becomes due. It isn't compatible with idempotency or the DLQ.timeoutdefault changed from 1 hour toNone.TaskStatus.NOT_FOUNDis removed:status()simply returnsNonefor missing tasks.TaskInfo.dependencies/dependentsare replaced bychildas multiple dependencies for a single task are no longer possible.Bugfixes
refresh_timeout(nowclaim_task) andpublish_taskscripts.run_taskchecks the reclaim result before the expiry check, fixing a bug where an expired task error could be recorded for a task another worker owned.prioritieslist to several workers no longer reverses it.start(priority=...)instead of silently enqueuing to a stream that will never be read.Other changes
RTypeVar became covariantR_coeverywhere.signing_secret, since Redis write access is sufficient for an attacker to achieve arbitrary code executionstreaq webbinds to127.0.0.1by default instead of0.0.0.0Pre-merge checklist
make lint)make test)This discussion was created from the release v8.0.0.
All reactions