Operator View: Availability - What sleds are up / down?

[smklein]

The Neuxs operator API should provide a view of which sleds on the rack are currently running, with a variety of related information (how long have they been up, health, etc).

[smklein]

FWIW, the "sled list" endpoint exists, but it not returning any information about health:

omicron/nexus/src/external_api/http_entrypoints.rs

Lines 4972 to 5000 in 75bd905

	/// List sleds
	#[endpoint {
	method = GET,
	path = "/system/hardware/sleds",
	tags = ["system"],
	}]
	async fn sled_list(
	rqctx: Arc<RequestContext<Arc<ServerContext>>>,
	query_params: Query<PaginatedById>,
	) -> Result<HttpResponseOk<ResultsPage<Sled>>, HttpError> {
	let apictx = rqctx.context();
	let nexus = &apictx.nexus;
	let query = query_params.into_inner();
	let handler = async {
	let opctx = OpContext::for_external_api(&rqctx).await?;
	let sleds = nexus
	.sleds_list(&opctx, &data_page_params_for(&rqctx, &query)?)
	.await?
	.into_iter()
	.map(|s| s.into())
	.collect();
	Ok(HttpResponseOk(ScanById::results_page(
	&query,
	sleds,
	&|_, sled: &Sled| sled.identity.id,
	)?))
	};
	apictx.external_latencies.instrument_dropshot_handler(&rqctx, handler).await
	}

Additionally, these sleds remain, even if they are powered off or unplugged

[davepacheco]

Presumably part of this includes some kind of state reporting from the Sled itself, storing that in the database, and then using that produce this information in the "list sleds" API?

[smklein]

Yeah, definitely. Currently we are storing "what sleds has Nexus ever heard of" in CRDB, through the internal Nexus API:

omicron/nexus/src/internal_api/http_entrypoints.rs

Lines 66 to 86 in b8a4617

	/// Report that the sled agent for the specified sled has come online.
	#[endpoint {
	method = POST,
	path = "/sled-agents/{sled_id}",
	}]
	async fn sled_agent_put(
	rqctx: RequestContext<Arc<ServerContext>>,
	path_params: Path<SledAgentPathParam>,
	sled_info: TypedBody<SledAgentStartupInfo>,
	) -> Result<HttpResponseUpdatedNoContent, HttpError> {
	let apictx = rqctx.context();
	let nexus = &apictx.nexus;
	let path = path_params.into_inner();
	let info = sled_info.into_inner();
	let sled_id = &path.sled_id;
	let handler = async {
	nexus.upsert_sled(*sled_id, info).await?;
	Ok(HttpResponseUpdatedNoContent())
	};
	apictx.internal_latencies.instrument_dropshot_handler(&rqctx, handler).await
	}

This... sorta works. It works in cases where the sled info is purely additive, at least? But I don't believe it ever gets updated, and it seems unlikely for a sled to report that it is no longer available (this seems like an inference that needs to be made by Nexus, either by querying MGS, or indirectly through lack-of-a-response).