Skip to content

meta: record why a resource was frozen, and let an operator unfreeze it - #75

Open
bjmeetsfo wants to merge 1 commit into
mainfrom
oss/meta-freeze-reasons
Open

meta: record why a resource was frozen, and let an operator unfreeze it#75
bjmeetsfo wants to merge 1 commit into
mainfrom
oss/meta-freeze-reasons

Conversation

@bjmeetsfo

Copy link
Copy Markdown
Collaborator

The problem

Freezing recorded no reason at all. The metaserver could not tell an operator taking a node out for maintenance from the failure detector convicting one that stopped answering — even though the two want opposite recovery behaviour.

There was also no way out of a freeze. Tables have unfreeze_table; servers and proxies had nothing. The only path back to Normal was for the frozen resource to re-register itself, and apply_register_server permits that as soon as the freeze cooldown has elapsed.

The cooldown defaults to zero. So a convicted node re-registers immediately and returns itself to service: the failure detector's decision is erased by the very node it was about, and conviction is advisory rather than binding.

What this adds

FreezeReason on servers and proxiesOperator for an explicit admin request, Unresponsive when the detector convicts for silence, Restarted when reboot detection (#64) fires. Reported through list_servers and the topology event stream, so an operator can see why a node is out. Set at every freeze site: the stale-heartbeat sweep and the adaptive detector mark their own verdicts as convictions, and the adaptive path distinguishes a restart from a silence using the plan it already computes.

unfreeze_server / unfreeze_proxy, exposed as POST /servers/unfreeze and POST /proxies/unfreeze. Always available whatever the freeze reason, because an operator has to be able to overrule the metaserver.

Unfreeze is recorded as its own mutation. It was previously the one state change that recorded nothing, so mutation-log recovery would have silently re-frozen the resource — a test covers that replay.

TS_META_FORBID_SELF_CLEARING_CONVICTION makes the metaserver refuse a re-registration from a resource it convicted, returning conviction_requires_unfreeze until an operator intervenes.

An operator freeze is deliberately not covered by the lock: that already has a human in the loop, and the freeze cooldown is the knob for holding a node out.

Off by default

Variable Default Meaning
TS_META_FORBID_SELF_CLEARING_CONVICTION 0 A convicted resource cannot re-register its way back

The automatic recovery this removes is load-bearing wherever the freeze cooldown is left at zero, so turning it on is a deployment decision. Recording the reason and offering unfreeze are unconditional — neither changes existing behaviour.

Tests

7 new tests: the reason recorded for a detector freeze and for an operator freeze, a convicted server refused re-registration and readmitted after unfreeze, the default path still recovering automatically, an operator freeze not being locked, a restart locked out under its own reason, a convicted proxy covered the same way, and an unfreeze surviving mutation-log replay.

Verification:

  • cargo test -p temporalstore-rust --lib meta::tests — 34 passed, 0 failed.
  • cargo test -p temporalstore-rust --lib meta176 passed, 0 failed.
  • cargo test -p temporalstore-rust --bin metaserver — 18 passed, 0 failed.
  • cargo build -p temporalstore-rust --bin metaserver — clean, no new warnings.

Independent of #73.

Freezing recorded no reason at all. The metaserver could not tell an operator
taking a node out for maintenance from the failure detector convicting one that
stopped answering, even though the two want opposite recovery behaviour.

There was also no way out of a freeze. Tables have unfreeze_table; servers and
proxies had nothing. The only path back to Normal was for the frozen resource to
re-register itself, and apply_register_server permits that as soon as the freeze
cooldown has elapsed. The cooldown defaults to zero, so a convicted node
re-registers immediately and returns itself to service: the failure detector's
decision is erased by the very node it was about, and conviction is advisory
rather than binding.

FreezeReason is now recorded on servers and proxies - Operator for an explicit
admin request, Unresponsive when the detector convicts for silence, Restarted
when reboot detection fires - and reported through list_servers and the topology
event stream, so an operator can see why a node is out. The reason is set at
every freeze site: the stale-heartbeat sweep and the adaptive detector mark
their own verdicts as convictions, and the adaptive path distinguishes a restart
from a silence using the plan it already computes.

unfreeze_server and unfreeze_proxy return a frozen resource to service, exposed
as POST /servers/unfreeze and /proxies/unfreeze. They are always available
whatever the freeze reason, because an operator has to be able to overrule the
metaserver. Unfreeze is recorded as its own mutation: it was previously the one
state change that recorded nothing, so mutation-log recovery would have silently
re-frozen the resource, and a test covers that replay.

With TS_META_FORBID_SELF_CLEARING_CONVICTION the metaserver refuses a
re-registration from a resource it convicted, returning
conviction_requires_unfreeze until an operator intervenes. An operator freeze is
deliberately not covered: that already has a human in the loop, and the freeze
cooldown is the knob for holding a node out.

Off by default. The automatic recovery it removes is load-bearing wherever the
freeze cooldown is left at zero, so turning it on is a deployment decision.
Recording the reason and offering unfreeze are unconditional - neither changes
existing behaviour.

7 new tests: the reason recorded for a detector freeze and an operator freeze, a
convicted server refused re-registration and readmitted after unfreeze, the
default path still recovering automatically, an operator freeze not being
locked, a restart locked out under its own reason, a convicted proxy covered the
same way, and an unfreeze surviving mutation-log replay.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants