Proposal: humus's own metrics

[Zaba505]

Note

Part of the humus re-evaluation; see Proposal: humus on the bedrock process model, wave 5, which revision 9 added. It builds on Proposal: OpenTelemetry without globals and Proposal: humustest, and on the four wave-4 clients whose breakers and retries it measures: outbound HTTP clients, outbound gRPC clients, producing to Kafka and RabbitMQ. It depends on no other wave-5 proposal: failure injection and authentication and authorization decide their own telemetry. It adds an instrument to no package outside those four clients. It answers the question outbound HTTP clients' revision 1 left for every client at once: whether humus records instruments of its own.

Summary

Each of the four wave-4 clients records four instruments of humus's own, for its circuit breaker and its retries. Nothing else in humus records one.

Instrument	Kind	Unit	Attributes, beside the dependency's	Recorded
circuit.state	asynchronous UpDownCounter	{breaker}	circuit.state: closed, open or half-open	at each collection, from Breaker.State
circuit.transitions	Counter	{transition}	circuit.state and circuit.previous_state	at each change of the breaker's state
circuit.calls	Counter	{call}	circuit.outcome: success, failure, ignored or rejected	at each call the breaker counts or rejects
retry.decisions	Counter	{decision}	retry.decision: again, succeeded, stopped, exhausted, throttled or expired	after each attempt; not by the Kafka Producer

• The names are in the circuit. namespace of the breakers' log records, and retry. beside it. Every attribute value is the String() of bedrock's own type: circuit.State, circuit.Outcome and retry.Decision.
• Each client records them in its own meter scope, its package's import path, which is the scope of its log records. Each data point carries the dependency's attributes, the same as the breaker's log records: server.address and server.port, with messaging.system too for Kafka.
• The state is a count of breakers in each state, as OpenTelemetry's Kubernetes conventions count containers in each state. Two clients of one origin add up instead of overwriting each other.
• The retry budget's tokens aren't exported. A retry the budget refused is a throttled decision, counted exactly, and the tokens mean nothing without MaxTokens. Open question 1.
• No other package records an instrument of its own: not the probes, the drain, the lifecycle, a job, or the Kafka and RabbitMQ consumers. Each is measured already, by OpenTelemetry's instrumentation or by the orchestrator, or happens once per process. See What isn't measured.
• No Go identifier is added. A test reads the instruments as it reads any metric, with humustest.Telemetry.Metrics.

Background

• The clients left it here. Outbound HTTP clients asked in its open question 4 whether the clients should export a breaker's state and a budget's tokens. Its revision 1 answered no for wave 4, because the question is wider than one client. The charter's revision 9 made it a proposal of its own, and outbound gRPC clients, producing to Kafka and RabbitMQ named it in their revisions. The doc comments of grpcclient and rabbitmq, and the docs pages of all four clients, point here.

• bedrock expects a framework to export these. circuit lists State as "whenever, such as in a metrics callback", and makes it report half-open "as soon as the open period has passed, with no traffic" so that a gauge shows it. retry lists Budget.Tokens the same way. circuit observes no results or rejections: "A framework counts them by wrapping the classifier and by checking for ErrRejected" (circuit v0.24.0). retry's Attempt.Done returns a Decision for every attempt, which says why no attempt follows: Succeeded, Stopped, Exhausted, Throttled or Expired (retry v0.24.0). queue/kafka pins circuit v0.25.0 and the other three modules v0.24.0; the two releases of circuit are identical.

• What each client records today. Each records OpenTelemetry's instruments for its protocol, and logs its breaker's transitions. Nothing records the breaker's state, and only one client counts its rejections:

  Client	Instruments	Its breaker's rejections	A retry its budget throttled
  httpclient	otelhttp's http.client.request.duration and http.client.request.body.size, per attempt	in http.client.request.duration, with error.type	nowhere
  grpcclient	otelgrpc's rpc.client.call.duration, per attempt	a span of the client's own; in no instrument, since the attempt never reaches grpc-go	nowhere
  kafka.Producer	messaging.client.sent.messages and messaging.client.operation.duration, per record, and kotel's broker counters	on each record's span; "neither instrument counts it" (producer.go#L262)	franz-go retries; nothing to count
  rabbitmq.Publisher	messaging.client.sent.messages and messaging.client.operation.duration, per attempt that reaches a channel	on the publish's span; in no instrument, since a rejected attempt never reaches a channel	nowhere

  The transitions are log records at WARN and INFO, with circuit.previous_state and the counts of the state the breaker left (httpclient/breaker.go#L72, grpcclient/breaker.go#L105, kafka/breaker.go#L68, rabbitmq/breaker.go#L69). An alert on an open breaker is a log query today.

• OpenTelemetry has no conventions for circuit breakers or retries. Each library names its own. MicroProfile Fault Tolerance, for example, records ft.circuitbreaker.calls.total with circuitBreakerResult (success, failure or circuitBreakerOpen), ft.circuitbreaker.opened.total, and ft.retry.calls.total with retryResult (Open Liberty's list). OpenTelemetry's naming rules apply all the same: a namespace isn't plural, a counter's name is plural when its unit is an annotation such as {call}, nothing ends in _total, and a metric "MAY share the same name" as an attribute. Its Kubernetes metrics give the shape for a state: k8s.container.status.state is an UpDownCounter in {container}, with an attribute of the same name, of which "only the value corresponding to the current state will be non-zero".

• Before the reset, humus recorded two instruments of its own, both in queue/kafka and both on OpenTelemetry's global meter: messaging.client.messages.processed and messaging.client.messages.committed (queue/kafka/otel.go at v0.15.0). The Kafka runtime replaced them in wave 3 with OpenTelemetry's messaging instruments. humus had no breaker or retries before wave 4.

Proposed API

No exported identifier is added, removed or changed. The instruments are the API: their names, kinds, units, descriptions and attributes, which dashboards and alerts depend on, and which change only through a proposal.

The instruments

Each client creates them from s.Telemetry.MeterProvider.Meter(<its import path>), in its constructor:

circuit.state         Int64ObservableUpDownCounter  {breaker}
                      "Circuit breakers in each state. Only the current state's value is non-zero."
circuit.transitions   Int64Counter                  {transition}
                      "Changes of state of a circuit breaker."
circuit.calls         Int64Counter                  {call}
                      "Calls through a circuit breaker, by how each counted."
retry.decisions       Int64Counter                  {decision}
                      "What retry decided after each attempt."

circuit.previous_state is on the breakers' log records already. The other three attributes are new:

Attribute	Values	Is
circuit.state	closed, open, half-open	circuit.State.String(): the state a data point counts, or the state a transition went to
circuit.previous_state	the same	the state a transition left, as on the log records
circuit.outcome	success, failure, ignored, rejected	circuit.Outcome.String() for a call the breaker counted, and rejected for one it didn't admit
retry.decision	again, succeeded, stopped, exhausted, throttled, expired	retry.Decision.String()

The dependency's attributes

Every data point carries the attributes that name the client's dependency. They are those of the breaker's log records, so a series and a record join on them:

Client	Meter scope	The dependency's attributes
httpclient	github.com/z5labs/humus/httpclient	server.address and server.port of BaseURL's origin
grpcclient	github.com/z5labs/humus/grpc/grpcclient	server.address and server.port of Addr
kafka.Producer	github.com/z5labs/humus/queue/kafka	messaging.system kafka, and server.address and server.port of the first of ProducerSettings.Brokers
rabbitmq.Publisher	github.com/z5labs/humus/queue/rabbitmq	server.address and server.port of Connection.Addr

The Kafka Producer's log records carry messaging.system alone today, so two Producers for two clusters, which producing to Kafka's revision 1 allows, can't be told apart. Its transition records gain server.address and server.port of the first broker too. CheckBrokers accepts only host:port, so the first broker always has both. It is the seed the end user named first, not the broker that answered: the breaker guards the cluster, and the seed is what names it in the settings.

The Kafka and RabbitMQ scopes already hold OpenTelemetry's messaging instruments, which these join. The HTTP and gRPC scopes hold no instrument today: otelhttp and otelgrpc record in their own.

Which client records which

	circuit.state	circuit.transitions	circuit.calls	retry.decisions
httpclient	yes	yes	one call per attempt	one per attempt
grpcclient	yes	yes	one per unary attempt, and one per stream open	one per unary attempt, and one per attempt to open a stream
kafka.Producer	yes	yes	one per Produce	no: franz-go retries inside a produce request, and exposes no decision
rabbitmq.Publisher	yes	yes	one per attempt	one per attempt

The RabbitMQ Service retries its connection with a retry.Retrier that has no budget and only ends with Serve's context. It records no decisions: it isn't on the request path, and each failed attempt is already a WARN record, connecting failed (service.go#L395).

circuit.state

At each collection, the callback reads Breaker.State() once and observes all three states: 1 for the current one and 0 for the other two. Every state is reported every time, so a series never goes stale while the client exists, and sum by (circuit_state) counts the breakers in each.

• It is read, not tracked. State reports the state "as time has left it": once OpenFor has passed, it reads half-open with no call made, although the breaker makes the transition, and the observer logs it, only on the next call. So the instrument shows half-open as soon as the breaker would admit a probe. It calls nothing of the end user's and changes nothing.
• It is an UpDownCounter in {breaker}, not a gauge. Its values add up: two clients with the same dependency attributes, which outbound HTTP clients' revision 1 allows when two dependencies share an origin, report closed 1 and open 1 rather than one hiding the other. The Go SDK sums the observations that two callbacks make for one attribute set of an asynchronous UpDownCounter, and keeps only the last for a gauge; the prototype checked both.
• It is registered with Meter.RegisterCallback, not with a callback option. The SDK keeps the callback option of an instrument's first creation only, and drops the others with a warning, "Repeated observable instrument creation with callbacks. Ignoring new callbacks" (go.opentelemetry.io/otel/sdk/metric v1.47.0). The second client of a package in an App creates the instrument again, so its callback would be lost. The prototype hit this.
• It is registered last in the constructor, once nothing after it can fail, so a constructor that returns an error leaves no callback behind.
• It stays registered until the telemetry shuts down. The Setup's telemetry belongs to the App, and App.Run shuts it down after the OnStop functions (app.go#L218-L223). So the final export carries each breaker's state as the App left it, such as a breaker that was open when the pod stopped, and a test reads it after the App has exited. State is safe to call once the client is closed.

circuit.transitions

One increment for each circuit.Transition, in the observer that writes its log record, with circuit.state set to the state it went to and circuit.previous_state to the one it left. So each log record has one increment, and the two carry the same attributes.

The state alone misses short opens. The OTLP exporters collect every minute, and OpenFor is 10s by default, so a breaker can open, go half-open and close between two collections, and circuit.state never shows it open. rate(circuit_transitions{circuit_state="open"}) counts every opening.

circuit.calls

One increment for each call the breaker counted, with the outcome the classifier gave it, and one for each call it rejected, with rejected. A call is what circuit calls a call, one Allow: an attempt for HTTP, gRPC and RabbitMQ, and one Produce for Kafka, whatever its number of records.

• Outcomes are counted by wrapping the classifier, as circuit's doc says to. Each client already passes circuit.New a classifier of its own (countVerdict, breakerOutcome, classify, classifyBreaker), and the wrapper records what it returns. Ticket.Done runs the classifier before it settles the ticket, so a late result, which the breaker then discards, is counted with its outcome too. A ticket settled by Release, such as a stream whose caller cancelled it or an attempt that panicked, reaches no classifier and counts nothing, as it counts nothing in the breaker.
• Rejections are counted where each client checks Allow's error, before it returns the rejection.
• When it is recorded follows when the ticket is settled. An HTTP response's ticket is settled by its body, when service code reads it to the end or closes it. A gRPC stream's is settled when it ends. A Kafka call's is settled once every record has its result, which may be after Produce returns.
• Why the outcomes and not only the rejections. For HTTP and gRPC, the outcome is the end user's Classify, and this is the only place its verdicts show: a 409 their classifier counts as a failure is a 4xx in otelhttp's instrument. And a rising share of failure is what trips the breaker, before it opens.
• HTTP's rejections are counted twice, once here and once in otelhttp's http.client.request.duration, whose error.type they already carry. That keeps one query for the rejections of all four clients.

retry.decisions

One increment after each Attempt.Done, with the Decision it returned. Every attempt has exactly one, rejected attempts included, which retry decides Stopped.

again is what retry decided, not what the client then did, as retry's doc says of Again. Another attempt doesn't follow when:

• the HTTP request isn't safe to repeat, or the response's body doesn't fit the 64 KiB buffer, and the client breaks out of the loop after Done;
• the call's context ends during the wait.

Done takes a token from the budget for a non-idempotent request's Retry all the same, since the budget counts the dependency's failures, so counting its again keeps the instrument in step with the budget. The span of each attempt the client did make, http.request.resend_count included, is unchanged.

A call ends on one decision other than again, unless it ends in one of the two ways above. So throttled counts the retries the budget refused, exhausted the calls that used every attempt, and expired the calls whose deadline came before the next attempt could start.

What isn't measured

An instrument of humus's own measures state that humus decides, that no OpenTelemetry instrumentation or the orchestrator measures already, and that changes many times over a process's life. Each package the charter named was decided against that rule:

• The retry budget's tokens. Not exported: throttled counts the budget refusing a retry, exactly, while a gauge sampled every minute misses a budget that empties and refills between collections. The tokens also mean nothing without MaxTokens, which would take a second instrument, retry.budget.limit. Open question 1.
• The probes. No instrument. The kubelet counts every probe's result, as prober_probe_total by probe_type and result, and the pod's Ready condition is the readiness itself. A probe attached to the end user's server is already in otelhttp's or otelgrpc's server instruments, with its 503. A probes server on a port of its own records nothing, on purpose (httphealth/service.go#L78). An asynchronous instrument would check the monitors at each collection, which costs a check and can change state: grpcclient's Connected starts the connection. A monitor's error is already an ERROR record with health.probe. Open question 3.
• The drain and the lifecycle. No instrument. Each happens once per process, and is already the lifecycle's records, serving, shutting down and stopped, its ShutdownCause and its exit status.
• Jobs. No instrument. A job runs once per process, under its job span, and its exit status is what the orchestrator counts, as a Kubernetes Job's completions and failures.
• The Kafka consumer. No new instrument. It records three of OpenTelemetry's messaging instruments, and kotel its broker counters. Consumer lag is measured outside the process, from the group's committed offsets, by a tool that also sees a group whose consumers are all down, which an instrument in the consumer can't report. Assignments and revocations are INFO records (service.go#L837).
• The RabbitMQ consumer. No new instrument. It records two messaging instruments. A lost connection is a WARN record, lost the connection (service.go#L1026), as is each failed connect, and it is rare.
• grpcclient's Connected, the servers, openapi/v3 and telemetry. No instrument. otelhttp and otelgrpc measure the servers and openapi's answers, by status code, and Connected latches once.

In a test

A test reads the instruments with humustest.Telemetry.Metrics(ctx), as it reads any metric. While the App runs, Metrics collects from its ManualReader, which runs every callback, so circuit.state reads the breaker at that moment. Once the App has exited, Metrics returns what the SDK's Shutdown collected, after the OnStop functions ran, so the state is still there. humustest gains nothing: a test filters the scope and the name itself, with a helper of its own like the prototype's points.

tel := humustest.NewTelemetry()
st := runJobWith(t, h.Host(), tel.Builder(), paymentsSettings(t, ca), func(ctx context.Context, c *http.Client) error {
	getN(t, ctx, c, 5, "https://payments:8443/") // five 500s open the breaker
	rm, err := tel.Metrics(ctx)
	require.NoError(t, err)
	require.Equal(t, map[string]int64{
		"circuit.state=closed":    0,
		"circuit.state=open":      1,
		"circuit.state=half-open": 0,
	}, points(t, rm, "circuit.state"))
	return nil
})

Design decisions

• Instruments in the four clients only. They are where humus makes decisions that nothing else measures: what the breaker counts, when it opens, and what retry does. Every other package's state is measured by OpenTelemetry's instrumentation or the orchestrator, or happens once. See What isn't measured.
• The circuit. and retry. namespaces, with bedrock's words. The charter asks for names in line with the breakers' log records, which use circuit.. The attribute values are bedrock's String()s, so a Decision in the code and in a query read the same. OpenTelemetry recommends a prefix for names it doesn't define; the instrumentation scope already names humus on every data point. Open question 4.
• Counters for what happens, one asynchronous instrument for what is. The transitions, outcomes and decisions are counted as they happen, so none is lost between collections. Only the state is read at collection, because it has no event of its own: half-open comes from time passing.
• The state is additive. As an UpDownCounter of breakers per state, it means the same for one client and for several that share attributes. That's the Kubernetes conventions' shape, and it needs no name for a client to stay correct.
• No name for a client. A client is told apart by its dependency's attributes, as its log records are. Two clients of one origin add up. The rare service that has two, such as two APIs behind one gateway, sees both in one series, and their transitions in one log stream. Open question 2.
• The dependency's attributes, and nothing per call. No rpc.method, no http.request.method, no messaging.destination.name. The breaker and the budget are one per dependency, so a method or a topic would only split one breaker's counts, and grow with the API. otelhttp, otelgrpc and the messaging instruments break calls down by method and destination already.
• Recorded in each package's own scope, where its log records and, for Kafka and RabbitMQ, its messaging instruments are. A View in the end user's Builder, which telemetry.SDK's doc allows (sdk.go#L27), drops or renames them. humus adds no setting to turn them off.
• Never unregistered. circuit.state's callback lives as long as the App's telemetry, which ends after the OnStop functions, so the last export has the state the App stopped in.
• No exported names or helpers. The instrument and attribute names are documented, not exported as constants, and humustest gains no reader for them. Neither is worth an API of its own.
• Failure injection needs nothing here. These instruments read the classifier's verdicts and retry's decisions. Whatever Proposal: failure injection decides an injected abort counts as, they count it that way, with no change to either proposal.

Testing

A prototype passed the tests below under go test -race -count=3, on main at 862a82f5, Go 1.27.0, OpenTelemetry's SDK v1.47.0 and bedrock's circuit and retry v0.24.0. It added the four instruments to httpclient: a file of its own, a wrapper around countVerdict, the transition counter beside the log record, the rejection in the breaker layer and the decision in the retry layer. Each test runs the end user's whole App inside a testing/synctest bubble, on a humustest Host, with a fake dependency served by httpserver on the same Host over TLS, certificates from humustest.NewCA, and the client's existing test helpers. OpenFor takes no real time. The prototype was in a scratchpad and isn't kept; the sketches are trimmed from it.

points collects the data points of one instrument in the client's scope, keyed by their attributes other than the dependency's, which it checks:

func points(t *testing.T, rm metricdata.ResourceMetrics, name string) map[string]int64 {
	t.Helper()
	out := map[string]int64{}
	for _, sm := range rm.ScopeMetrics {
		if sm.Scope.Name != "github.com/z5labs/humus/httpclient" {
			continue
		}
		for _, m := range sm.Metrics {
			if m.Name != name {
				continue
			}
			sum, ok := m.Data.(metricdata.Sum[int64])
			require.True(t, ok, "%s is a %T", name, m.Data)
			for _, dp := range sum.DataPoints {
				var kvs []string
				for _, kv := range dp.Attributes.ToSlice() {
					switch kv.Key {
					case "server.address":
						require.Equal(t, "payments", kv.Value.AsString())
					case "server.port":
						require.EqualValues(t, 8443, kv.Value.AsInt64())
					default:
						kvs = append(kvs, string(kv.Key)+"="+kv.Value.Emit())
					}
				}
				sort.Strings(kvs)
				out[strings.Join(kvs, ",")] += dp.Value
			}
		}
	}
	return out
}

The instruments follow the breaker through open, half-open with no call made, and closed, and the state is still there after the App has exited:

func TestClient_metrics_follow_the_breaker(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		h := humustest.NewHost(t)
		ca := humustest.NewCA(t)
		dep := &failing{} // 500 until healthy, a breaker Failure that isn't retried
		stop := startPayments(t, h, h.Host(), ca, dep)
		defer stop()

		tel := humustest.NewTelemetry()
		st := runJobWith(t, h.Host(), tel.Builder(), paymentsSettings(t, ca), func(ctx context.Context, c *http.Client) error {
			getN(t, ctx, c, 5, "https://payments:8443/")
			_, err := get(ctx, c, "https://payments:8443/")
			require.ErrorIs(t, err, circuit.ErrRejected)
			require.Equal(t, map[string]int64{"circuit.state=closed": 0, "circuit.state=open": 1, "circuit.state=half-open": 0}, stateOf(t, collect(t, ctx, tel)))

			time.Sleep(10 * time.Second) // OpenFor, and no call
			require.Equal(t, map[string]int64{"circuit.state=closed": 0, "circuit.state=open": 0, "circuit.state=half-open": 1}, stateOf(t, collect(t, ctx, tel)))

			dep.healthy.Store(true)
			getN(t, ctx, c, 3, "https://payments:8443/")
			return nil
		})
		requireExitZero(t, st)

		rm := collect(t, context.Background(), tel) // what Shutdown collected, after OnStop
		require.Equal(t, map[string]int64{"circuit.state=closed": 1, "circuit.state=open": 0, "circuit.state=half-open": 0}, stateOf(t, rm))
		require.Equal(t, map[string]int64{
			"circuit.previous_state=closed,circuit.state=open":      1,
			"circuit.previous_state=open,circuit.state=half-open":   1,
			"circuit.previous_state=half-open,circuit.state=closed": 1,
		}, points(t, rm, "circuit.transitions"))
		require.Equal(t, map[string]int64{
			"circuit.outcome=failure":  5,
			"circuit.outcome=rejected": 1,
			"circuit.outcome=success":  3,
		}, points(t, rm, "circuit.calls"))
		require.Equal(t, map[string]int64{
			"retry.decision=stopped":   6, // five 500s and the rejection
			"retry.decision=succeeded": 3,
		}, points(t, rm, "retry.decisions"))
	})
}

A retry the budget refuses is counted. With a budget of 3 tokens and a dependency that answers 503 to everything, the first call's first 503 takes a token and is retried, its second takes another and is throttled, and the second call's first 503 takes the last:

require.Equal(t, map[string]int64{
	"retry.decision=again":     1,
	"retry.decision=throttled": 2,
}, points(t, collect(t, context.Background(), tel), "retry.decisions"))

Two clients of one origin add up. One App builds two clients for https://payments:8443, opens the first one's breaker and leaves the second's closed, and reads circuit.state while its job runs:

require.Equal(t, map[string]int64{"circuit.state=closed": 1, "circuit.state=open": 1, "circuit.state=half-open": 0}, stateOf(t, collect(t, ctx, tel)))

That test first failed, with closed 0 and open 1: the prototype had given the state's callback with metric.WithInt64Callback, and the SDK had dropped the second client's. A separate test against the SDK alone showed the rest: a callback option on a repeated creation is ignored, two callbacks registered with RegisterCallback that observe one attribute set are summed for an asynchronous UpDownCounter, and a gauge keeps the last one's value. With RegisterCallback, the test passed.

The stories add the tests the prototype didn't have:

• grpcclient. A unary call and a stream open, through retries and the breaker, with the same expectations as above. A stream counts its outcome when it ends, and a stream its caller cancels counts nothing. A rejected attempt counts rejected and decides stopped.
• kafka.Producer. One circuit.calls increment per Produce, whatever its number of records. A call the breaker rejects counts rejected once. A call whose records failed for their topic's own reason counts ignored. The instruments and the transition records carry server.address and server.port of the first broker, and two Producers for two clusters have two series. No retry.decisions is recorded. These run on kfake, as the Producer's tests do.
• rabbitmq.Publisher. The same as HTTP's, as unit tests on the stand-in that *amqp.Channel satisfies, inside synctest, with no broker. The container tests are unchanged.
• The callback is registered last. A constructor that fails leaves circuit.state with no data point.
• No globals. With OpenTelemetry's global MeterProvider set to a recorder of its own, nothing is recorded in it.
• Two tests change. httpclient's TestClient_measures_each_attempt and grpcclient's telemetry test assert that "humus records no instrument of its own". The prototype's change failed the first of them, as expected. Each now checks that its package's scope holds the four instruments and nothing else.

Each package stays held to the 80% coverage gate.

Changes to other files

This proposal's stories make these, as it owns humus's own metrics:

• httpclient, grpc/grpcclient, queue/kafka and queue/rabbitmq record the instruments, in a file of their own in each package, and their doc comments' telemetry sections list them. httpclient's NewClient adds the Setup's MeterProvider to the nil check it makes of its LoggerProvider.
• queue/kafka's Producer adds server.address and server.port of the first broker to its transition records.
• The grpc, queue/kafka and queue/rabbitmq modules' go.mod list go.opentelemetry.io/otel/metric as a direct requirement. Each has it already, as an indirect one.
• The telemetry docs page gains a section, "humus's own instruments": the four instruments, their attributes, which client records which, the rule in What isn't measured, and reading them in a test.
• The four clients' docs pages, http-client.md, grpc-client.md, kafka.md and rabbitmq.md, replace "humus records no instrument of its own" and the pointers to this proposal with what each client records, and an example alert on circuit.state.

humustest, telemetry, health, the servers and the consumers don't change.

Migration from today

Before the reset	After wave 4	With this proposal
messaging.client.messages.processed and .committed in queue/kafka, on OpenTelemetry's global meter	OpenTelemetry's messaging instruments, on the Setup's MeterProvider	unchanged
no breakers	a breaker's state in WARN and INFO log records only	circuit.state and circuit.transitions, beside the records
no breakers	rejections in otelhttp's instrument for HTTP, and on spans for the other three	circuit.calls with rejected, for all four
no retries	a retry is one more attempt's span; a throttled one shows nowhere	retry.decisions, with throttled, exhausted and expired
a Kafka producer of the end user's	Producer records carry messaging.system only	and server.address and server.port of the first broker

There is nothing for the end user to migrate. The instruments appear on the next upgrade, and a View in their Builder drops any they don't want.

Open questions

1. Export the retry budget's tokens? As proposed, no: retry.decisions counts every retry the budget refuses as throttled, exactly, which is what an alert needs. A tokens gauge shows how close a budget is to refusing, but only next to MaxTokens, which would be a second instrument, retry.budget.limit, and both are sampled once a minute, while a budget can empty and refill between collections. As UpDownCounters, as the state is, two budgets with the same attributes would add up. Recommendation: no.
2. Tell two clients of one dependency apart? As proposed, a client is named by its dependency's attributes, and two clients of one origin add up, which every instrument here is built to do. A setting set in code, such as Peer string in each client's settings, could put OpenTelemetry's service.peer.name on its instruments and its breaker's records, for two APIs behind one gateway. That is a field in four settings structs for a case wave 4 allowed but no user has reported. Recommendation: no.
3. Count the probes' answers? As proposed, no: the kubelet counts every probe, an attached probe is in the server's instruments, and a monitor's error is a log record. A counter of answers by probe and result, recorded as each probe is answered, would show the probes server's answers, which nothing records today, without checking a monitor at collection. Recommendation: no.
4. Prefix the names with humus.? OpenTelemetry recommends that a name it doesn't define carry a prefix of its own, so it can't clash with a convention it defines later. As proposed, the instruments use circuit. and retry., as the breakers' landed log records do, and the instrumentation scope names humus on every data point. A prefix would rename the records' circuit. attributes too, which nothing has released yet. Recommendation: keep circuit. and retry..

Out of scope

• Instruments for failure injection, and for authentication and authorization: their own wave-5 proposals decide their telemetry.
• A retry's waits, such as the time a call spent between attempts, and attempts per call as a histogram. The spans show both.
• Consumer lag, partition counts and connection counts for the Kafka and RabbitMQ consumers, for the reasons in What isn't measured.
• The OpenTelemetry SDK's own self-observability metrics.
• A name for a client in its telemetry, unless open question 2 is answered yes.
• Dashboards and alert rules beyond the docs pages' example: Proposal: walkthroughs and a humus MIGRATING.md (wave 6), whose services show these instruments.

References

• Proposal: humus on the bedrock process model, the charter, and its revision 9, which made this proposal
• Outbound HTTP clients' revision 1, open question 4 and the shared names of the breakers' records; outbound gRPC clients' revision 1, producing to Kafka's revision 1 and RabbitMQ's revision 1
• OpenTelemetry without globals and humustest: telemetry.SDK, and humustest.Telemetry's ManualReader and final collection
• bedrock's circuit and retry v0.24.0: Breaker.State, the classifier, Transition, Attempt.Done, Decision and Budget.Tokens
• OpenTelemetry's naming rules, general metric conventions and Kubernetes metrics, for k8s.container.status.state
• The Go SDK's Meter.RegisterCallback, and its warning on repeated callback options
• MicroProfile Fault Tolerance's metrics, in Open Liberty's list
• The kubelet's prober_probe_total
• Before the reset: queue/kafka/otel.go at v0.15.0

[Zaba505]

Revision 1: the open questions answered, and every name starts with humus.

The open questions:

1. The retry budget's tokens aren't exported, as recommended.
2. A client gets no name of its own, as recommended, until users ask for one. Clients of one dependency that keep to its contract should see the same failures, so their instruments adding up is what they should do.
3. The probes' answers aren't counted, as recommended.
4. Every name this proposal defines starts with humus., against the recommendation. The names after the prefix don't change.

On the fourth, the libraries that define metrics OpenTelemetry doesn't nearly all prefix them with their own name:

• gRPC records grpc.client.attempt.started, grpc.client.call.duration and grpc.lb.outlier_detection.ejections_enforced, beside OpenTelemetry's rpc. instruments, which OpenTelemetry's compatibility page maps them to (gRPC's OpenTelemetry metrics, grpc-go v1.84.0).
• Resilience4j records resilience4j.circuitbreaker.state, resilience4j.circuitbreaker.calls and resilience4j.retry.calls (its Micrometer metrics).
• Polly records resilience.polly.strategy.events and resilience.polly.pipeline.duration (its telemetry).
• MicroProfile Fault Tolerance records ft.circuitbreaker.calls.total and ft.retry.calls.total, as the post's Background lists.
• OpenTelemetry's own conventions keep the libraries' names for metrics they took in: .NET's aspnetcore., kestrel. and signalr..
• franz-go's kotel is the exception. Its messaging.kafka.connects.count and the like sit in OpenTelemetry's messaging. namespace, which the naming rules advise against.

The instrumentation scope names humus on every data point, but most of these series end in Prometheus, where the scope is only an otel_scope_name label. circuit.state and retry.decisions are generic enough that another library, or a breaker of the end user's, could record the same names, and a query that doesn't filter on the scope would add them together. humus.circuit.state can't collide. OpenTelemetry defines no circuit. or retry. namespace, and its one open proposal for resilience, semantic-conventions#2355, uses resilience.. So the prefix guards against other libraries more than against OpenTelemetry.

The breakers' log records take the prefix too, so a series and a record still join on the same names. No release carries the records' circuit. attributes: the latest, v0.15.0, came before the wave-4 clients. Renaming them now breaks no one. After the next release it would.

Each section below replaces or adds to the part of the same name in the post. Elsewhere in the post, each name of an instrument or an attribute reads as Names shows.

Names

In the post	Now
circuit.state, the instrument and its attribute	humus.circuit.state
circuit.transitions	humus.circuit.transitions
circuit.calls	humus.circuit.calls
retry.decisions	humus.retry.decisions
circuit.previous_state	humus.circuit.previous_state
circuit.outcome	humus.circuit.outcome
retry.decision	humus.retry.decision
the counts on the breakers' log records: circuit.successes, circuit.failures, circuit.consecutive_successes, circuit.consecutive_failures and circuit.timed_out	humus.circuit.successes, humus.circuit.failures, humus.circuit.consecutive_successes, humus.circuit.consecutive_failures and humus.circuit.timed_out

The dependency's attributes are OpenTelemetry's, server.address, server.port and messaging.system, and keep their names. So do the records' bodies, such as circuit breaker open.

In Prometheus, the instruments are humus_circuit_state, humus_circuit_transitions_total, humus_circuit_calls_total and humus_retry_decisions_total, and the attributes are labels such as humus_circuit_state. The post's query for every opening, in humus.circuit.transitions, becomes increase(humus_circuit_transitions_total{humus_circuit_state="open"}[5m]).

Summary

The table becomes:

Instrument	Kind	Unit	Attributes, beside the dependency's	Recorded
humus.circuit.state	asynchronous UpDownCounter	{breaker}	humus.circuit.state: closed, open or half-open	at each collection, from Breaker.State
humus.circuit.transitions	Counter	{transition}	humus.circuit.state and humus.circuit.previous_state	at each change of the breaker's state
humus.circuit.calls	Counter	{call}	humus.circuit.outcome: success, failure, ignored or rejected	at each call the breaker counts or rejects
humus.retry.decisions	Counter	{decision}	humus.retry.decision: again, succeeded, stopped, exhausted, throttled or expired	after each attempt; not by the Kafka Producer

The first bullet and the bullet on the tokens become:

• Every name starts with humus., then circuit. for the breaker and retry. for the retries. The breakers' log records rename their attributes to match. Every attribute value is the String() of bedrock's own type: circuit.State, circuit.Outcome and retry.Decision.
• The retry budget's tokens aren't exported. A retry the budget refused is a throttled decision, counted exactly, and the tokens mean nothing without MaxTokens.

The instruments

The block becomes:

humus.circuit.state         Int64ObservableUpDownCounter  {breaker}
                            "Circuit breakers in each state. Only the current state's value is non-zero."
humus.circuit.transitions   Int64Counter                  {transition}
                            "Changes of state of a circuit breaker."
humus.circuit.calls         Int64Counter                  {call}
                            "Calls through a circuit breaker, by how each counted."
humus.retry.decisions       Int64Counter                  {decision}
                            "What retry decided after each attempt."

The sentence before the attributes' table becomes:

humus.circuit.previous_state is on the breakers' log records already, renamed from circuit.previous_state. The other three attributes are new.

What isn't measured

The bullets on the retry budget's tokens and on the probes end without "Open question 1." and "Open question 3.": both stay unmeasured.

Design decisions

The bullets "The circuit. and retry. namespaces, with bedrock's words" and "No name for a client" become:

• The humus. prefix, then bedrock's words. OpenTelemetry recommends a prefix for names it doesn't define, and libraries that define their own use their own name: gRPC, Resilience4j and Polly among them. In Prometheus, where the scope is only a label, the prefix keeps humus's series apart from another library's circuit_state. After the prefix the names follow the breakers' log records, which take the prefix too. The attribute values are bedrock's String()s, so a Decision in the code and in a query read the same.
• No name for a client. A client is told apart by its dependency's attributes, as its log records are. Clients of one dependency that keep to its contract see the same failures, so two clients of one origin add up. The rare service with two, such as two APIs behind one gateway, sees both in one series, and their transitions in one log stream. A name can come when users ask for one.

Testing

In the sketches, every instrument's name and every key takes the prefix: points(t, rm, "humus.circuit.calls") returns "humus.circuit.outcome=failure": 5, and so on.

This is added to the tests the stories add:

• The records' names. Each client's transition records carry humus.circuit.previous_state and the five counts, and no attribute whose name starts with circuit..

Changes to other files

These are added:

• The breakers' log records, in all four clients, rename circuit.previous_state and the five counts as Names shows. Their bodies don't change.
• The four clients' docs pages and package docs rename them in their lists of the records' attributes.

Migration from today

The second row becomes:

Before the reset	After wave 4	With this proposal
no breakers	a breaker's state in WARN and INFO log records only, with circuit. attributes	humus.circuit.state and humus.circuit.transitions, beside the records, whose attributes become humus.circuit.

The other rows take the names of Names. There is still nothing for the end user to migrate: no release carries the circuit. attributes.

Open questions

All four are answered above.

Out of scope

"A name for a client in its telemetry, unless open question 2 is answered yes" becomes:

• A name for a client in its telemetry, until users ask for one.

[Zaba505]

Revision 2: health.probe takes the prefix too

Revision 1 put humus. in front of every name this proposal defines, and of the breakers' log records' attributes. humus defines one other name of its own: health.probe. httphealth and grpchealth put it on the ERROR record that a monitor's error writes (httphealth/handler.go#L96, grpc/grpchealth/server.go#L113). It becomes humus.health.probe, for the same reasons as revision 1's names: OpenTelemetry defines no health. namespace, and the name is generic enough that another library could write it too.

Its values, liveness and readiness, don't change, and neither do the records' bodies and severity. No release carries it: v0.15.0 had no httphealth or grpchealth.

Answer 3 stands. The probes' answers aren't counted, so humus.health.probe is on the records only.

Names

This row is added:

In the post	Now
health.probe, on httphealth's and grpchealth's records of a monitor's error	humus.health.probe

Changes to other files

This is added:

• httphealth and grpc/grpchealth rename health.probe to humus.health.probe on their records of a monitor's error, in their package docs and on the health docs page.

health itself doesn't change: it writes no records.

[Zaba505]

Revision 3: the Kafka consumer's own attributes take the prefix too

Revision 1 put humus. in front of every name humus defines of its own. This is a standard for every attribute and instrument name. Revision 2 extended it to health.probe. Three more attributes, on the Kafka consumer's log records, break it. queue/kafka defines them inside OpenTelemetry's messaging. namespace (queue/kafka/service.go#L48-L57), which OpenTelemetry's naming rules advise against and which revision 1 found kotel doing:

• messaging.destination.partition.ids lists a topic's partitions on the records partitions assigned, partitions revoked and partitions lost. OpenTelemetry's conventions name one partition, messaging.destination.partition.id.
• messaging.kafka.offset.first and messaging.kafka.offset.last bound the offsets of the batch on the record batch skipped: its commit was refused. OpenTelemetry's conventions name one offset, messaging.kafka.offset.

They become:

Before	Now
messaging.destination.partition.ids	humus.messaging.destination.partition.ids
messaging.kafka.offset.first	humus.messaging.kafka.offset.first
messaging.kafka.offset.last	humus.messaging.kafka.offset.last

Their values, the records' bodies and their severities don't change. The records' attributes from OpenTelemetry's conventions keep their names: messaging.system, messaging.consumer.group.name, messaging.destination.name, and the rest. No release carries the three: v0.15.0 had none of them.

Changes to other files

This is added:

• queue/kafka's Service renames the three on its records, in its package doc and on the Kafka docs page.

[Zaba505]

Closing: humus's own metrics have landed

The revisions' stories are done:

• story(httpclient): measure the circuit breaker and the retries with humus's own instruments #873 measured httpclient's circuit breaker and retries with humus.circuit.state, humus.circuit.transitions, humus.circuit.calls and humus.retry.decisions, and renamed the breaker's record attributes to humus.circuit.* (story(httpclient): measure the circuit breaker and the retries with humus's own instruments #888);
• story(grpcclient): measure the circuit breaker and the retries with humus's own instruments #874 measured grpcclient's circuit breaker and retries with the same four instruments (story(grpcclient): measure the circuit breaker and the retries with humus's own instruments #900);
• story(kafka): measure the Producer's circuit breaker with humus's own instruments #875 measured the Kafka Producer's circuit breaker with the three circuit instruments, and its records gained its first broker's server.address and server.port (story(kafka): measure the Producer's circuit breaker with humus's own instruments #904);
• story(rabbitmq): measure the Publisher's circuit breaker and retries with humus's own instruments #876 measured the RabbitMQ Publisher's circuit breaker and retries with the four instruments (story(rabbitmq): measure the Publisher's circuit breaker and retries with humus's own instruments #907);
• story(docs): list humus's own instruments on the telemetry page #877 listed humus's own instruments on the telemetry page, with Views that drop and rename them (story(docs): list humus's own instruments on the telemetry page #920);
• story(health): name a monitor's error record's probe humus.health.probe #878 renamed a monitor's error record's health.probe to humus.health.probe in httphealth and grpchealth (story(health): name a monitor's error record's probe humus.health.probe #925);
• story(kafka): prefix the consumer's partition and offset attributes with humus. #880 renamed the Kafka consumer's partition and offset attributes to humus.messaging.destination.partition.ids, humus.messaging.kafka.offset.first and humus.messaging.kafka.offset.last (story(kafka): prefix the consumer's partition and offset attributes with humus. #929).

Revisions 1 to 3 left no open questions. The retry budget's tokens aren't exported, a client gets no name of its own, probe answers aren't counted, and every name humus defines starts with humus.. No Go identifier was added.

The rest is out of scope or belongs to other proposals, as "Out of scope" lists:

• Instruments for failure injection, and for authentication and authorization, are their own proposals' (Proposal: failure injection #869 and Proposal: authentication and authorization #870).
• Dashboards and alert rules are Proposal: walkthroughs and a humus MIGRATING.md (wave 6), whose services show these instruments.
• A retry's waits, attempts per call as a histogram, consumer lag, partition and connection counts, the SDK's self-observability metrics, and a name for a client stay out of scope.