[Bug] START_TO_CLOSE timeouts deterministically hang Fiber workers

[michaeldiscala]

Hi there! Would love some thoughts on the bug below. Thank you!

What are you really trying to do?

We are running our temporal workers / activities with the Fiber activity executor using the Async gem. Our Temporal process basically looks like:

Async do
  worker = Temporalio::Worker.new(
    client:,
    task_queue: TASK_QUEUE,
    activities: ACTIVITIES,
    workflows: WORKFLOWS
  )
  log("worker", "running on #{TASK_QUEUE} (pid #{Process.pid})")
  worker.run(shutdown_signals: %w[SIGINT SIGTERM])
end

Describe the bug

Activities and workflows will process fine, until an activity exceeds its START_TO_CLOSE timeout and receives a CanceledError. Once this happens, the reactor loop will be entirely frozen. No more activities will run. Workflow activities continue to get picked up but silently fail, until we've leaked all the workflow slots. We see this in our production app today on the latest temporal SDK 1.6.0

Minimal Reproduction

Note: this was drafted by AI, but confirmed to reproduce the issue manually

worker.rb

# frozen_string_literal: true

# Minimal reproduction: a fiber-executor activity hitting its START_TO_CLOSE
# timeout permanently bricks the Temporal Ruby worker.
#
# Gemfile:
#   gem "temporalio", "1.6.0"
#   gem "async", "2.43.0"
#
# Setup:
#   docker run -d --name temporal-dev -p 7233:7233 \
#     temporalio/temporal:latest server start-dev --ip 0.0.0.0
#
# Then, in two terminals:
#   bundle exec ruby worker.rb
#   bundle exec ruby driver.rb
#
# Expected: ping workflows keep completing forever.
# Actual:   ping workflows stop completing the moment the hang activity is
#           cancelled, and never recover.

$stdout.sync = true

require "bundler/setup"
require "async"
require "temporalio/activity"
require "temporalio/client"
require "temporalio/retry_policy"
require "temporalio/worker"
require "temporalio/workflow"

TASK_QUEUE = "min-repro"

def log(msg) = puts("[#{Time.now.strftime('%H:%M:%S')}] worker: #{msg}")

# Never returns on its own. Core's local start_to_close timer fires at 5s and
# delivers cancellation via Fiber#raise -- that is the bug trigger.
class HangActivity < Temporalio::Activity::Definition
  activity_executor :fiber

  def execute
    log("hang activity started (will be cancelled at 5s)")
    sleep(3600)
  end
end

# Liveness probe: proves whether the worker still dispatches ANY new work.
class PingActivity < Temporalio::Activity::Definition
  activity_executor :fiber

  def execute(n)
    log("ping activity #{n}")
    "pong-#{n}"
  end
end

class HangWorkflow < Temporalio::Workflow::Definition
  def execute
    Temporalio::Workflow.execute_activity(
      HangActivity,
      start_to_close_timeout: 5,
      retry_policy: Temporalio::RetryPolicy.new(max_attempts: 1)
    )
  end
end

class PingWorkflow < Temporalio::Workflow::Definition
  def execute(n)
    Temporalio::Workflow.execute_activity(
      PingActivity, n,
      start_to_close_timeout: 10,
      retry_policy: Temporalio::RetryPolicy.new(max_attempts: 1)
    )
  end
end

client = Temporalio::Client.connect("127.0.0.1:7233", "default")

# The `Async` wrapper is essential to the bug. It places the SDK's single
# dispatch loop -- Temporalio::Worker#run's `loop { runner.next_event }`, which
# handles activity tasks AND workflow activations -- onto a fiber managed by a
# transfer-based scheduler. Cancel delivery calls Fiber#raise inline on that
# loop; Fiber#raise uses resume semantics, async uses transfer semantics, so the
# call never returns and the loop is stranded forever.
#
# Control: delete the two `activity_executor :fiber` lines above (leaving this
# Async block exactly as-is) so the activities run on the default thread-pool
# executor. The same timeout then does no harm -- pings keep completing
# indefinitely. Cancellation is still delivered; only the delivery mechanism
# differs.
Async do
  worker = Temporalio::Worker.new(
    client:,
    task_queue: TASK_QUEUE,
    activities: [HangActivity, PingActivity],
    workflows: [HangWorkflow, PingWorkflow]
  )
  log("running on #{TASK_QUEUE}")
  worker.run(shutdown_signals: %w[SIGINT SIGTERM])
end

driver.rb

# frozen_string_literal: true

# Drives the reproduction. Starts one workflow whose activity will hit its
# START_TO_CLOSE timeout, then repeatedly runs a trivial workflow to show that
# the worker stops processing new work entirely.

$stdout.sync = true

require "bundler/setup"
require "temporalio/client"

TASK_QUEUE = "min-repro"

def log(msg) = puts("[#{Time.now.strftime('%H:%M:%S')}] driver: #{msg}")

client = Temporalio::Client.connect("127.0.0.1:7233", "default")
run = Time.now.to_i

log("starting HangWorkflow (its activity hits START_TO_CLOSE after 5s)")
client.start_workflow("HangWorkflow", id: "hang-#{run}", task_queue: TASK_QUEUE)

log("polling with PingWorkflow every 2s...")

n = 0
loop do
  n += 1
  handle = client.start_workflow(
    "PingWorkflow", n, id: "ping-#{run}-#{n}", task_queue: TASK_QUEUE
  )

  # Hard timeout on a thread: once the worker is bricked the workflow simply
  # never progresses, so we need our own deadline for an unambiguous verdict.
  t = Thread.new { handle.result }
  if t.join(8)
    log("ping #{n}: OK")
  else
    t.kill
    log("ping #{n}: STUCK -- worker has stopped dispatching work")
  end

  sleep(2)
end

Gemfile

# frozen_string_literal: true

source "https://rubygems.org"

# Versions pinned to those in use when the bug was reproduced.
# temporalio 1.6.0 was the latest release at the time.
gem "async", "2.43.0"
gem "temporalio", "1.6.0"

Environment/Versions

• OS and processor: Linux
• Temporal Version: 1.6.0 SDK
• Confirmed with both Temporal docker & cloud edition

Additional context

[michaeldiscala]

AI driven analysis of the issue -- I've read this pretty thoroughly and everything tracks, but please take with a grain of salt.

AI analysis of root cause (read & confirmed)

When a fiber-executor activity reaches its START_TO_CLOSE timeout, the SDK
delivers cancellation with Fiber#raise. Under a transfer-based fiber scheduler
such as async, Fiber#raise never returns to its caller. The caller is the
worker's single task-dispatch loop, so the worker permanently stops dispatching
all work — activities and workflow activations alike — while the process and
the reactor stay healthy. Workflow task slots then drain monotonically to zero
as the server retries activations that will never be picked up.

Reproduced 100% of trials (3/3 bit-identical).

The resume/transfer mismatch

Ruby has two fiber-switching disciplines, and they are not interchangeable:

• resume/yield — asymmetric and stack-like. f.resume suspends the caller
  and records it as f's resumer. Control returns to the caller only when f
  calls Fiber.yield or terminates.
• transfer — symmetric. f.transfer suspends the current fiber and switches
  to f, recording no resumer relationship. Control comes back only if someone
  explicitly transfers back.

Fiber#raise is defined in terms of resume: it resumes the target and raises
the exception at the point where the target last suspended. The caller therefore
becomes the target's resumer and blocks awaiting a yield-or-terminate that it can
only receive through the resume discipline.

async drives every fiber with transfer. Async::Scheduler#block
(scheduler.rb:248) suspends a fiber with @selector.transfer, and the selector
resumes fibers by transferring into them. Async's task fibers therefore never
Fiber.yield, and their lifecycle ends by transferring to the scheduler rather
than returning into a resumer.

So when the dispatch-loop fiber D calls A.raise(CanceledError) on activity
fiber A:

1. Ruby suspends D, records it as A's resumer, and enters A with the exception.
2. A leaves again by transfer — to the selector when it next blocks, or via
   Async::Task's completion path when it finishes.
3. Nothing ever transfers back to D. D is not in the selector's ready list,
   holds no timer, and is not blocked on any IO — from Async's point of view D
   simply never returned from the line it was executing. Async has no record that
   D is runnable, so it is never scheduled again.

D is suspended forever, mid-Fiber#raise.

This is unconditional. The caller is stranded whether the target blocks again
after rescuing, returns immediately without blocking, or is killed outright by the
exception (see minimal reproduction below). All three exit through a transfer.

Why the existing safety nets don't help

• Ruby's scheduler-aware path is never taken. Ruby 3.3+ defines
  Fiber::Scheduler#fiber_interrupt for exactly this problem, and async 2.43
  implements it (scheduler.rb:403, routing through unblock). Instrumenting
  the hook shows it is never invoked in any scenario — same-thread
  Fiber#raise does not dispatch to it.
• The cross-thread route is unavailable. Raising into the fiber from another
  thread fails with FiberError: fiber called across threads. (Notably this
  failure is safe: it raises in the caller instead of stranding it.)
• The SDK's defensive rescue cannot fire. handle_cancel_task
  (activity_worker.rb:152) wraps activity._cancel in rescue StandardError,
  which would have contained a FiberError. But Fiber#raise does not raise
  here — it simply never returns, so the rescue is unreachable.

Why the blast radius is total

Temporalio::Worker#run's dispatch loop (worker.rb:205,
loop { event = runner.next_event; ... }) runs on the calling fiber, and there
is exactly one of them for the whole worker: it handles activity task polls,
workflow activation polls, decode results, and completions. Cancel callbacks are
invoked inline on it:

worker.rb:205                  _run_all_root         <- the ONE dispatch loop
worker.rb:712                  _on_poll_bytes
activity_worker.rb:87          handle_task
activity_worker.rb:152         handle_cancel_task
cancellation.rb:156            Cancellation#on_cancel
activity_executor/fiber.rb:43  Fiber#raise           <- parks here forever

Cancellation#add_cancel_callback documents that callbacks "usually need to be
quick and thread-safe since [they are] called in the canceler's thread". The
fiber.raise callback registered by ActivityExecutor::Fiber#set_activity_context
(fiber.rb:43) is neither quick nor returning.

Note the task-tree position of the dispatch loop is irrelevant — running it as a
child or grandchild Async::Task reproduces identically.

Observed behaviour

With one activity timing out (start_to_close: 5s), instrumented worker:

evloop_iters (patched MultiRunner#next_event)	freezes at cancel, waiting=false (stuck inside dispatch, not idling)
reactor_ticks (an unrelated Async task)	keeps advancing — reactor is alive
ActivityWorker slots	released normally
WorkflowWorker slots	100 → 95 in 35s, monotonic, never recovers
New workflows	never complete again

Minimal reproduction (no Temporal)

require "async"

Async do
  target = Fiber.schedule { sleep(60) }
  sleep(0.3)
  target.raise(RuntimeError.new("boom"))
  warn("never reached")   # <-- caller is stranded here, forever
end

Full Temporal-level reproduction, with a SIGUSR1 fiber dump showing the
dispatch loop parked in Fiber#raise, is available on request.