Skip to content

Only restore a retried parent when the retry succeeds - #13474

Open
masaori335 wants to merge 1 commit into
apache:masterfrom
masaori335:asf-master-parent-healthcheck
Open

Only restore a retried parent when the retry succeeds#13474
masaori335 wants to merge 1 commit into
apache:masterfrom
masaori335:asf-master-parent-healthcheck

Conversation

@masaori335

Copy link
Copy Markdown
Contributor

A parent that accepts connections but never sends a response was marked down and then restored to the pool by the very next retry probe, even when that probe also failed, so it kept receiving traffic indefinitely while healthy peers took none of the load.

handle_response_from_parent() called markParentUp() on entry, before the switch on current.state, so a parent selected as a retry candidate was restored before ATS knew the retry's outcome. markParentUp() zeroes failedAt and failCount; the markParentDown() that follows a failed probe then takes its result->retry branch, which leaves new_fail_count at 0, so available is never re-cleared. The parent therefore returned to the pool with a cleared failure count after every retry_time window, no matter how long it stayed degraded.

The CONNECTION_ALIVE branch already calls markParentUp() for a retry that actually succeeded, which is the correct restore point. Removing the entry call restores the behavior that predates #8546: that commit backed out the retry-limiting work and replaced a retryComplete() call -- which only decremented an in-flight retry counter and never touched availability -- with markParentUp(), silently turning "the retry finished" into "the parent is healthy".

Two autests pin both outcomes of the retry path. A parent that is still silent when probed stays down; a parent that has recovered is restored. Each was negative-controlled against its own call site: reverting this change fails the first, and removing the CONNECTION_ALIVE call fails only the second.

A parent that accepts connections but never sends a response was
marked down and then restored to the pool by the very next retry
probe, even when that probe also failed, so it kept receiving traffic
indefinitely while healthy peers took none of the load.

handle_response_from_parent() called markParentUp() on entry, before
the switch on current.state, so a parent selected as a retry candidate
was restored before ATS knew the retry's outcome. markParentUp() zeroes
failedAt and failCount; the markParentDown() that follows a failed
probe then takes its result->retry branch, which leaves new_fail_count
at 0, so available is never re-cleared. The parent therefore returned
to the pool with a cleared failure count after every retry_time window,
no matter how long it stayed degraded.

The CONNECTION_ALIVE branch already calls markParentUp() for a retry
that actually succeeded, which is the correct restore point. Removing
the entry call restores the behavior that predates apache#8546: that commit
backed out the retry-limiting work and replaced a retryComplete() call
-- which only decremented an in-flight retry counter and never touched
availability -- with markParentUp(), silently turning "the retry
finished" into "the parent is healthy".

Two autests pin both outcomes of the retry path. A parent that is still
silent when probed stays down; a parent that has recovered is restored.
Each was negative-controlled against its own call site: reverting this
change fails the first, and removing the CONNECTION_ALIVE call fails
only the second.
@masaori335 masaori335 added this to the 11.0.0 milestone Aug 3, 2026
@masaori335 masaori335 self-assigned this Aug 3, 2026
Copilot AI review requested due to automatic review settings August 3, 2026 00:29

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes parent retry availability handling in ATS’s HTTP transaction path so a parent marked down is only restored when a retry probe actually succeeds, preventing a degraded-but-connecting parent from re-entering the pool after every retry_time window.

Changes:

  • Remove the unconditional markParentUp() call at entry to HttpTransact::handle_response_from_parent() for retried parents, so restore happens only on success (CONNECTION_ALIVE).
  • Add two AuTest replay scenarios that pin both retry outcomes: failed retry keeps the parent down; successful retry restores it.
  • Document the corrected retry_time semantics in the admin guide.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file
File Description
src/proxy/http/HttpTransact.cc Stops restoring a retried parent before the retry outcome is known; restore remains on the success path.
tests/gold_tests/parent_proxy/parent_retry_availability.test.py Adds a gold test that runs both retry-outcome replays.
tests/gold_tests/parent_proxy/replays/parent_retry_failure_stays_down.replay.yaml New replay asserting a failed retry probe does not restore a degraded parent.
tests/gold_tests/parent_proxy/replays/parent_retry_success_restores.replay.yaml New replay asserting a successful retry probe restores the parent to the pool.
doc/admin-guide/files/records.yaml.en.rst Clarifies retry_time behavior: restore only on successful retry, otherwise remain unavailable until next window.

@cmcfarlen
cmcfarlen requested review from cmcfarlen and traeak August 3, 2026 22:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

2 participants