mu's tasks are not reliable as-is

[rsyring]

I couldn't remember the exact details of the problems involved with mu's async invocation of lambda functions. ChatGPT filled in the blanks:

---

You remembered correctly. Mu’s implementation uses Lambda’s built-in asynchronous invocation queue, which is convenient but is not a durable task queue with an exactly-once—or even strict at-least-once—guarantee.

In AsyncTask.invoke(), Mu calls the same Lambda with:

client().invoke(
    FunctionName=self.lambda_func,
    InvocationType='Event',
    Payload=...,
)

A 202 means Lambda accepted the event into its internal asynchronous queue. It does not mean the task ran or succeeded. Mu logs the request ID and otherwise has no task record or result tracking. ([GitHub]1)

Reliability semantics

AWS explicitly documents both possible duplication and possible loss:

Even if your function doesn't return an error, it's possible for it to receive the same event multiple times because the queue itself is eventually consistent.

AWS then says that if the function cannot keep up, events may be deleted without ever being sent to the function. ([AWS Documentation]2)

So the practical semantics are:

• Usually delivered.
• May be delivered more than once.
• May eventually be discarded without running.
• No ordering guarantee.
• No task status or result retrieval.
• A 202 only confirms admission to Lambda’s queue.

That makes it acceptable for best-effort work, but not by itself for jobs where losing one silently would be materially bad.

Is SQS involved?

Internally, Lambda uses an AWS-managed asynchronous event queue. It is queue-like, but it is not an SQS queue that you own or can inspect. You cannot:

• inspect pending messages,
• control visibility timeouts,
• redrive individual messages,
• inspect queue depth directly,
• choose FIFO semantics,
• retain messages for 14 days,
• manually pause or drain consumers.

SQS becomes involved only if you explicitly:

1. publish tasks to your own SQS queue and configure Lambda as its consumer, or
2. configure an SQS queue as a Lambda failure destination/DLQ.

Retries

There are two different failure classes.

Function errors

For exceptions, runtime failures, and timeouts, Lambda invokes the function initially and retries it twice by default:

• initial invocation,
• retry after roughly one minute,
• retry after roughly another two minutes.

That means a non-idempotent task may run partially three times. ([AWS Documentation]2)

Throttling and Lambda service errors

For insufficient concurrency, 429 responses, and 500-class service errors, Lambda keeps retrying with exponential backoff for up to six hours by default. A sufficiently backed-up event can expire before execution. ([AWS Documentation]2)

You can configure:

• MaximumRetryAttempts: 0–2 for function errors.
• MaximumEventAgeInSeconds: up to six hours.

DLQs and failure destinations

Neither is automatically present. Mu’s code itself does not configure one.

You can configure either:

Lambda dead-letter queue

A standard SQS queue or SNS topic receives events that exhaust retries or exceed maximum age.

The DLQ receives essentially the original event, plus some message attributes. It does not contain the rich execution response information. ([AWS Documentation]3)

On-failure destination

This is generally the better modern option. It can target:

• SQS,
• SNS,
• S3 for failures,
• another Lambda,
• EventBridge.

It receives an invocation record containing the original request, response/error details, invoke count, and failure condition. ([AWS Documentation]3)

A failure destination is still not absolute protection. AWS documents that destination delivery itself can fail because of permissions, unsupported target types, or size limits. You need alarms on:

• AsyncEventsDropped
• DestinationDeliveryFailures
• DeadLetterErrors
• function Errors
• function Throttles

Important Mu-specific caveats

Tasks must be idempotent

Lambda can invoke the same task multiple times even when the first execution appeared successful. Every meaningful task should have an idempotency key and persist completion state somewhere durable.

For example:

@task
def charge_customer(payment_id):
    payment = load_payment(payment_id)

    if payment.status == 'completed':
        return

    charge = provider.charge(
        ...,
        idempotency_key=str(payment.id),
    )

    mark_completed(payment, charge)

Simply making the Python function “mostly safe” is not enough if it calls an external API between database operations.

The enqueue operation is not transactional with your database

A common failure sequence is:

1. Commit application data.
2. Process crashes before func.invoke().
3. The task is never submitted.

The inverse can also happen:

1. Invoke the task.
2. Database transaction rolls back.
3. Task runs against data that does not exist or is not committed yet.

This is the classic dual-write problem. Lambda async invocation does not solve it. For critical tasks, an application outbox table written in the same database transaction is the usual solution.

Mu has no durable task identity

The AWS request ID logged here identifies the invocation API request, but Mu does not store:

• a logical task ID,
• submission time,
• execution status,
• retry count,
• completion status,
• failure details.

Consequently, it is difficult to answer, “Did this particular task run?”

I would add a generated task ID to the payload even for the lightweight implementation:

{
    'task-id': str(uuid.uuid4()),
    'task-path': ...,
    'args': ...,
    'kwargs': ...,
}

That ID should also be included in structured logs and used for idempotency.

Deployment/version coupling

By default, AWS_LAMBDA_FUNCTION_NAME invokes the unqualified function. An event queued under one deployment can execute against a later deployment.

Because the payload stores:

'module.path:function_name'

renaming or moving a task while old events remain queued could make them fail. Invoking a version or alias gives more controlled deployment semantics, although it introduces lifecycle considerations for old versions.

Payload serialization

The task payload is JSON, so arguments must be safely serializable and should remain small. Async Lambda invocation has its own payload limit, and large payloads also make failure-destination delivery more likely to exceed downstream limits.

There is also a questionable behavior here:

def json_dump(self, value):
    if isinstance(value, dt.date | dt.datetime):
        return value.isoformat()
    return json.dumps(value)

For an unsupported object, returning json.dumps(value) can turn it into an embedded JSON string rather than rejecting it. In many cases json.dumps(value) will simply raise anyway. I would raise TypeError explicitly for unsupported values.

Would I use this?

I would use direct InvocationType='Event' for things like:

• refreshing a disposable cache,
• sending noncritical analytics,
• triggering recomputable work,
• work where occasional manual recovery is acceptable.

I would not rely on it alone for:

• billing,
• email that must be sent,
• fulfillment,
• permanent data transformations,
• webhooks requiring auditable delivery,
• anything where “we silently missed one” is unacceptable.

For those, I would use:

application transaction
    -> outbox row
    -> publisher
    -> owned SQS queue
    -> Lambda consumer
    -> SQS DLQ

An owned SQS queue gives you much better operational visibility, retention, redrive, back-pressure behavior, and failure handling. It still provides at-least-once delivery rather than exactly-once execution, so tasks must remain idempotent.

My assessment is that Mu’s implementation is a useful minimal convenience abstraction, but it should be described as “best-effort asynchronous Lambda invocation,” not as a general-purpose reliable task queue. At minimum, I would add an on-failure SQS destination, alarms, task IDs, idempotency guidance, and explicit retry/event-age configuration.