Skip to content

perf(job): two-pass listing and fast queue stats #230

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
retr0h merged 3 commits into from
Mar 7, 2026
Merged

perf(job): two-pass listing and fast queue stats #230

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d1c98cb
perf(job): two-pass listing with pagination limits
retr0h Mar 7, 2026
5e5ac86
refactor(job): remove GetQueueStats and use fast path
retr0h Mar 7, 2026
bbbc3a3
docs(job): document two-pass listing and kv.Keys() limits
retr0h Mar 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 2 additions & 9 deletions cmd/client_job_list.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,16 +54,9 @@ var clientJobListCmd = &cobra.Command{
return
}

// Get queue stats for summary
statsResp, err := sdkClient.Job.QueueStats(ctx)
if err != nil {
cli.HandleError(err, logger)
return
}

jobs := jobsResp.Data.Items
totalItems := jobsResp.Data.TotalItems
statusCounts := statsResp.Data.StatusCounts
statusCounts := jobsResp.Data.StatusCounts

if jsonOutput {
displayJobListJSON(jobs, totalItems, statusCounts, statusFilter, limitFlag, offsetFlag)
Expand Down Expand Up @@ -186,6 +179,6 @@ func init() {

clientJobListCmd.Flags().
String("status", "", "Filter jobs by status (submitted, processing, completed, failed, partial_failure)")
clientJobListCmd.Flags().Int("limit", 10, "Limit number of jobs displayed (0 for no limit)")
clientJobListCmd.Flags().Int("limit", 10, "Maximum number of jobs per page (1-100, default 10)")
clientJobListCmd.Flags().Int("offset", 0, "Skip the first N jobs (for pagination)")
}
9 changes: 1 addition & 8 deletions cmd/client_job_status.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -176,13 +176,6 @@ func fetchJobsStatus() string {
statusDisplay += fmt.Sprintf(" Dead Letter Queue: %d\n", stats.DlqCount)
}

if len(stats.OperationCounts) > 0 {
statusDisplay += "\nOperation Types:\n"
for opType, count := range stats.OperationCounts {
statusDisplay += fmt.Sprintf(" %s: %d\n", opType, count)
}
}

return statusDisplay
}

Expand Down Expand Up @@ -215,7 +208,7 @@ and operation types with live refresh.`,
return
}

p := tea.NewProgram(initialJobsModel(pollIntervalSeconds))
p := tea.NewProgram(initialJobsModel(pollIntervalSeconds), tea.WithAltScreen())
_, err := p.Run()
if err != nil {
status := fetchJobsStatus()
Expand Down
52 changes: 2 additions & 50 deletions docs/docs/gen/api/get-queue-statistics.api.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ description: "Retrieve statistics about the job queue."
sidebar_label: "Get queue statistics"
hide_title: true
hide_table_of_contents: true
api: eJztVk1v1DAQ/SvWnEO75eOS24IoKhKoQBGHarVy4unG28ROx+PCEvm/o3G2S7bbQ5EAceC0SXbG7817Y3sGMBhqsj1b76CEj8hk8RZVYM02sK2D0pWPrLhBtfaVuokY8QgKYL0KUF7CW18t32mnV9ih4+X8/Gy59tXS90haVg2wKCBgHcnyBsrLAV6iJqR55Eby174qCbWBRVoUQBh67wIGKAd4OpvJzz7FD0Jgwk+41N4xOpZg3fetrTPy8TpIxgChbrDT8sSbHqEEX62xZiigJ+HJdsRjz7oV9mESax3jCgmKezwuJFi52FVIyl+JOEFZl4XaiYTfdNe3COXzp6kAIR3DsvbRcXiIjjbGyvK6Pd8ntk8l3efySlbckag2akQ6ksidEX8DdwemJDXjm/ZmRH6Epu8fVNOgNqpFZqQ7ZVMqgC2LsmNDfGLN4eO2eSAlCXg+Ozlsn89OR2482e9o1BM1Pz9T17hRhDfREprf10xI5GkSFpisWx2UPFeTd6lcKs65ihvNytd1JEKz101wqm2LRrFXNN2xW8sNsrZteAT4zne1zZns9kziQVgTUaAd8ldP14pthz5yhq69wcdsnl2RkrAH8mI2m7r7WqIOjH12aOypp8oag049UWcuxKsrW1t0rHqkzoaQj6L/7v777r546NTPgXdyWLcaD4I/cw389/YPeZsK6JAbb6CEFWbhtQwBcLz21fEoA8iwQLdIMlxMJodPYt7oz3R+2PFtmHvJzWFQQpWDoNg+nHrqNEMJb79c5NvDuiuf07ds5ys5K34OMnIzQAFCZCz85Gh2NBOheh+407mjnM5Yb5AP+vG+cMPP7vylKWusjvEbH/ettk4YRGplwVG8PEHB3XAhw1bjA8v3Yah0wM/UpiSfbyLSZhT1VpPVldR9uUgFNKgNUp7NrnEjYtQ19uLPrW6j4B9sJ5nVdl6+eX0hQ8S+IfcMyKtv/9JuM1l7GMaIC3+NLiUotiRY3iEtUko/ADwftuo=
api: eJztVk1v00AQ/SurOZs2LXDxLSBaFQlU2iIOVRSt7Um8qb3rzs4WgrX/Hc06TZ0mhyKBxIFTbGc+3ry3OzM9VOhLMh0bZyGHK2Qy+IDKs2bj2ZRe6cIFVlyjWrlC3QcMeAQZsF56yG/hoyvmn7TVS2zR8nx6eTFfuWLuOiQtUT3MMvBYBjK8hvy2h3eoCWkauBb/lStyQl3BLM4yIPSdsx495D2cTibyswvxiwAY4RMspbOMlsVYd11jypT5eOXFowdf1thqeeJ1h5CDK1ZYMmTQkeBkM+Rjx7oR9H5kayzjEgmyZzhuxFjZ0BZIyi2EHK+MTURtScIfuu0ahPzNacxAQAc/L12w7A/B0VVlJLxuLneB7UKJz7G8l4hbEMVaDZmOkmVzP2R8QU2fD1ZToa5Ug8xIj5XFmAEblsoGQa5Zs7/aiAcxisGbycm+fF+tDlw7Mj+xUq/U9PJC3eFaEd4HQ1j9OTGRyNHIzDMZu9wreapG71K5VJx8FdealSvLQITVjppwpk2DlWKnaHxjHilH1qbxL0i+1VttfEa3LYE4mLYKKKkt8ndHd4pNiy5wSl26Cl9yeLdFisNOkreTyVjdD2K1J+zrfWHPHBWmqtCqV+rC+rBYmNKgZdUhtcb71Ar+q/vvq/v2UNdNho90GLscGsHfacP/tf1L2sYMWuTaVZDDEhPxWoYwHK9ccTzQADKs6QFJhvtocl+LeIM+4/m9xVszd+KbzCCHIhlBtnk4c9Rqhhw+frtJ08PYhUvuG7TTpfSKp0VCJgNkIECGwk+OJkcTIapznludTpTVKdc58t55fE5c/3Q6f2vLGapj/MHHXaONFQSBGgk4kJc2GHgc7rLs1M6zfO/7Qnv8Sk2M8vk+IK0HUh80GV1I3bezmEGNukJKu9EdroWMssRO9HnQTZD8e9dJdqWtlucfbmR52BXkmQAp+uYvbdej2H0/WNy4O7QxQrYBwfIOcRZj/AXUzojN
sidebar_class_name: "get api-method"
info_path: gen/api/agent-management-api
custom_edit_url: null
Expand Down Expand Up @@ -138,54 +138,6 @@ Retrieve statistics about the job queue.
Count of jobs by status.


</div><SchemaItem
name={"property name*"}
required={false}
schemaName={"integer"}
qualifierMessage={undefined}
schema={{"type":"integer"}}
collapsible={false}
discriminator={false}
>

</SchemaItem>
</div>
</details>
</SchemaItem><SchemaItem
collapsible={true}
className={"schemaItem"}
>
<details
style={{}}
className={"openapi-markdown__details"}
>
<summary
style={{}}
>
<span
className={"openapi-schema__container"}
>
<strong
className={"openapi-schema__property"}
>
operation_counts
</strong><span
className={"openapi-schema__name"}
>
object
</span>
</span>
</summary><div
style={{"marginLeft":"1rem"}}
>
<div
style={{"marginTop":".5rem","marginBottom":".5rem"}}
>


Count of jobs by operation type.


</div><SchemaItem
name={"property name*"}
required={false}
Expand Down Expand Up @@ -216,7 +168,7 @@ Retrieve statistics about the job queue.
value={"Example (from schema)"}
>
<ResponseSamples
responseExample={"{\n \"total_jobs\": 42,\n \"status_counts\": {},\n \"operation_counts\": {},\n \"dlq_count\": 0\n}"}
responseExample={"{\n \"total_jobs\": 42,\n \"status_counts\": {},\n \"dlq_count\": 0\n}"}
language={"json"}
>

Expand Down
53 changes: 51 additions & 2 deletions docs/docs/gen/api/list-jobs.api.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ description: "Retrieve jobs, optionally filtered by status."
sidebar_label: "List jobs"
hide_title: true
hide_table_of_contents: true
api: eJztWN9v20YM/lcOfNoA2XW6dhgM9MHbmi1FuwZtgj0UgUFJlH2OdKfcUWk8Q//7wDtb/pk4LdqhA/pkS+LxPvIjeTwuICefOV2ztgaG8I7YabolNbOpT5QN77Es56rQJZOjXKVz5Rm58X1IgHHiYfgBXtl0/AYNTqgiw+PR+dl4ZtOxrcmhqPBwlYCnrHGa5zD8sIBfCR25UcNTWT+z6dAR5nDVXiVQo8OKmJwPogYrgiHEXSEBLUhvGnJzSMDRTaMd5TAssPSUwF3PYq17mc1pQqZHd+ywF3Eu4BZLnSOLOltppqrmeWIN2eKFb9JKM1Ouamcz8l6bicpsVZckLwvUpXxDxxrLsTw2jqBNwGdTqlC087yOQJ02E0iATFOJdZ1qSGCtHBLo1EMCcQMI1m9tcdUmOyydBi4CR5t0tEnnq1JXmr+CqyptXgwOGq0N04QcJFBpoysxfCC4C2xKhuHJYM+KN3gncso0VUpO2SIaxFY54saZvrr0pAaqsE4Zq4JJW0baovD0TVm5b+Rfe8b5a10Hm2qcaBOyow8S9Y58bY2ngOvpYCA/28pGqtSeV8r6IYAMk2ERxboudRb0PZl5kV/s47fpjDKOYViTYx13Y8tYjsV8f9DYbRgXIr1LW4WcTSVleErLYiEA6Q4lxGH47GmbwO4O6BwKZ7vv74Gp80NZVlhXIcMQmkbne2Avjb5pSOmcDOtCR8SCcWbTfuA4lpUDircV/dY4R4aX2barJXOEksfH1YigtkaxrsgzVnVY31XKQ17Ycf+UVCeucmQUR2Oe61iszzd8xq6hNoRWiM/dgBJV8VvQo3SxrngBFjln3XGjXoqYqsh7nJBoicUsqJhazzFhj2kZTcS/K3nFU+RVLaZ8y9tNLTmbj5GPa32NnlVcsOPzrYR72Ofn5HoY4K0WRYdJHqfOYp7JLqusPMzFsei+LxClpiDjgXKwArTi7yHKHmSibds2gWDgWGB8mks2zsuw+L/yyv2W5s1eMm1ZKnFQakMHStFOtk6dNba0E51hqTzdNGQyWhY9VeqCsnlWkqJbMhysfGQp6yLxeACfvX+rfvl5cLKO3lXxCbtG0m+Xp8CRPBUxJTLqh64pSRRm18Z+LCmfyJNndOF1VwqSZT4ncjI7TfmPn5XZ1ilvG5ctU3tCRorYMrXXtizLyHGtfzYVmp60jZiWpDY+HnDQpxSynBh16aWQLU/VtKR+iJ0QPCwHmvS8vwfBd8uSAFufX2vPr2zqN7+2CTw7dLCfmdCHKOldyLNaN8Bf8JB/pAtGBx0Z/BJos1kmJ2G+dbrDaWyPY/MWrxDrthSWDn3E5l2N6EjA1Da8BnFw27wh2doQf7TuOiSKbSLx0vY9pqfpjJQFW5s8Hww2eQ0hskfqyT6plwYbnlqn/6Fc9dTo/Exd01x1/el3Yv8PxP60T+ypdanOczKqp86Mb4pCZzochOQq7X24735n99tn9/mhWhyPALloSUPzhS9a38n8SmSGvoGnNochTMJQoEYZLcGTmU1BZk/uNs6TNgZR74W1SMzmOKoDOmWuYTkFkOc0CEGy/HO6uni++vsCBIE2hY3NXYQZ2571XEzOAEhAgESLT/qDfhg01NZzhSGUlsMNaR9C9O26abGOxU8f10W7mO74SV2iNuEy5UrRGv0VRnEyrJPmTh4XixQ9XbqybeV1nLLIVC7XXtqi9ZzlXpT3jKvugXNN881h3y2WjQiFwc/j9/y84dKDkFYztc9E9MhJ0IMQuonXGsOVPDgtICS4pS0nzMkFjuKqUZZRvblqr4KJli59/nh5Ibe17VTYCf2gfXVzMvMN3YtFlLiw12TaFlbQWZ6hvWrb9l9R0dQD
api: eJztWF+P2zYM/yqCnlbASXNdOwwB+pCtve2Kbju0d9hDdwhom050Z0s+iUovCwzsQ+wT7pMMlGLn/+VatEMH9CmxTVE/8kdSFBcyR5dZVZMyWg7lGySrcIbi2qQuESa8h7Kci0KVhBZzkc6FIyDv+jKRBBMnh+/kK5OOfwENE6xQ03h0fja+NunY1GiBVTh5lUiHmbeK5nL4biF/QLBoR56mvP7apEOLkMur5iqRNViokNC6IKqhQjmUcVeZSMVIbz3auUykxVuvLOZyWEDpMJF3PQO16mUmxwnqHt6RhV7EuZAzKFUOxOpMpQirmuaJ0WiK586nlSLCXNTWZOic0hORmaoukV8WoEr+BpYUlGN+9BZlk0iXTbEC1k7zOgK1Sk9kIlH7iq3rVMtErpTLRHbqZSLjBjJYv7HFVZNssXQauAgcrdPRJJ2vSlUp+gyuqpR+PthrtNKEE7QykZXSqmLDB4y7AF+SHJ4Mdqz4Be5YTmhfpWiFKaJBZIRF8lb3xaVDMRCFsUIbEUzaMNIUhcMvyspdI3/dMc7dqDrYVMNE6ZAdfclRb9HVRjsMuJ4MBvyzqWwkSuWoVdYPAaQJNbEo1HWpsqDv8bVj+cUufpNeY0YxDGu0pOJuZAjKMZvv9hq7CeOCpbdpq4CyKacMTXFZLBgg3gGHuBw+fcL+DJE6zozX5PbBgjxXseScbwLcRLTj5h9ZI2OBstzKC/FNl37JWmonq9xOlsmdbGf3o754gVbNOP2tqcQNzgXHnhO5t62tTAn/r8E58c9ff3OshhgTXNBc/w/NcLd9C9YCR+v2+wMEqXxffSmMrYDkUHqv8h2aLrW69ShUjppUoSJXjPjapH3ZsbFP8ZZzvbWoqfXnlpbMInAFO66GBZXRglSFjqCqw/rujNjnha3Am6LoxEUOBP2DMUPWYxOSKmTmdiqxqvgt6BGqWMVDgIXWGnvcqJcsJip0DibIWmIoBRVT4yiWqmNaRhP2bysvaArUhirmG972NVerfAx0XOtrcCTigi2fb5Sa+31+jrYHAV67KDqMK1hqDeQZ79LWoyP5eyC6DwUipzkQ7CmELaCWv/sou5eJpmmaRAYDxwzjw1yy1imExf+VVw5bmvudZNqwlOOgVBr3lKKtbJ1ao01pJiqDUji89agzXJZ7UaoCs3lWosAZagpWPrCUdZF4PIDP3v4mvv9ucLKK3rb4hF0j6bPl+XckT1lMsMzGeQDZjTbvS8wn/OQIbHi952iwoTPOH31UZhsrnPE2W6b2BDUXsWVqr2xZlpHjWn/2Fegeny+QlijWPu5x0IcUshwJVOm4kC37ibTEfoidEDzERzl3+y+C4JtlSZAbn18rR69M6ta/Nol8uq+lOdOhAxPctaEjsWr9P2F780AXjPY6Mvgl0GayjE/CfKOvkafxYhDb1nh5WjXkcunQB2ze1YiOBEiNpxWIvdvmHnlrjfTe2JuQKMZH4rnhfUg31xnJCzY2eTYYrPMaQmSH1JNdUi81eJoaq/7EXPTE6PwstE5dZ/6V2P8Dsd/uEntqbKryHLXoiTPtfFGoTIWDEG2lnAs3/a/sfvnsPttXi+MR0N5nPvEV8yuZn4nM0DfQ1ORyKCdhHFIDD9Xk42uTSp662VmcpK2N4N4ya5GY9UFcB3RKVMvl/IOf0yAkk+Wf0/bi+er3C8kIlC5MbO4izNj2rCaCfAbIRDKQaPFJf9API5baOKoghNJyrMPtQ4i+bTctVrH44YPKaBfhHT2uS1DhUu5tyVqjv8IQkseU3Nzx42KRgsNLWzYNv47zJZ5H5spxW7SaMB1EeWBQdwDODc7Xx5wzKD0LhZHXw/f8uLHavZDaaeJHInrgDOxeCN2sb4Xhih+sYhAc3NyWI+RoA0dx1SjLsF5ftVPBWEuXPj+9vODb2mYqbIV+0N7enPR8TfdiESUuzA3qppEtdOJn2Vw1TfMvwukvHg==
sidebar_class_name: "get api-method"
info_path: gen/api/agent-management-api
custom_edit_url: null
Expand Down Expand Up @@ -145,6 +145,55 @@ Retrieve jobs, optionally filtered by status.
schema={{"type":"integer","description":"Total number of jobs matching the filter.","example":42}}
>

</SchemaItem><SchemaItem
collapsible={true}
className={"schemaItem"}
>
<details
style={{}}
className={"openapi-markdown__details"}
>
<summary
style={{}}
>
<span
className={"openapi-schema__container"}
>
<strong
className={"openapi-schema__property"}
>
status_counts
</strong><span
className={"openapi-schema__name"}
>
object
</span>
</span>
</summary><div
style={{"marginLeft":"1rem"}}
>
<div
style={{"marginTop":".5rem","marginBottom":".5rem"}}
>


Count of all jobs by status (submitted, processing, completed, failed, partial_failure). Derived from key names during the listing pass — no extra reads.



</div><SchemaItem
name={"property name*"}
required={false}
schemaName={"integer"}
qualifierMessage={undefined}
schema={{"type":"integer"}}
collapsible={false}
discriminator={false}
>

</SchemaItem>
</div>
</details>
</SchemaItem><SchemaItem
collapsible={true}
className={"schemaItem"}
Expand Down Expand Up @@ -600,7 +649,7 @@ Retrieve jobs, optionally filtered by status.
value={"Example (from schema)"}
>
<ResponseSamples
responseExample={"{\n \"total_items\": 42,\n \"items\": [\n {\n \"id\": \"3fa85f64-5717-4562-b3fc-2c963f66afa6\",\n \"status\": \"string\",\n \"created\": \"string\",\n \"operation\": {},\n \"error\": \"string\",\n \"hostname\": \"string\",\n \"updated_at\": \"string\",\n \"responses\": {},\n \"agent_states\": {},\n \"timeline\": [\n {\n \"timestamp\": \"string\",\n \"event\": \"string\",\n \"hostname\": \"string\",\n \"message\": \"string\",\n \"error\": \"string\"\n }\n ]\n }\n ]\n}"}
responseExample={"{\n \"total_items\": 42,\n \"status_counts\": {},\n \"items\": [\n {\n \"id\": \"3fa85f64-5717-4562-b3fc-2c963f66afa6\",\n \"status\": \"string\",\n \"created\": \"string\",\n \"operation\": {},\n \"error\": \"string\",\n \"hostname\": \"string\",\n \"updated_at\": \"string\",\n \"responses\": {},\n \"agent_states\": {},\n \"timeline\": [\n {\n \"timestamp\": \"string\",\n \"event\": \"string\",\n \"hostname\": \"string\",\n \"message\": \"string\",\n \"error\": \"string\"\n }\n ]\n }\n ]\n}"}
language={"json"}
>

Expand Down
68 changes: 58 additions & 10 deletions docs/docs/sidebar/architecture/job-architecture.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ into NATS JetStream:
- **Agents** — Processes jobs, updates status, stores results

All three use the **Job Client Layer** (`internal/job/client/`), which provides
type-safe business logic operations (`CreateJob`, `GetQueueStats`,
type-safe business logic operations (`CreateJob`, `GetQueueSummary`,
`GetJobStatus`, `ListJobs`) on top of NATS JetStream.

**NATS JetStream** provides three storage backends:
Expand Down Expand Up @@ -346,17 +346,12 @@ GET /api/v1/jobs/{job-id}
{
"total_jobs": 42,
"status_counts": {
"unprocessed": 5,
"submitted": 5,
"processing": 2,
"completed": 30,
"failed": 5
},
"operation_counts": {
"node.hostname.get": 15,
"node.status.get": 19,
"network.dns.get": 8,
"network.ping.do": 23
}
"dlq_count": 0
}
```

Expand Down Expand Up @@ -663,7 +658,7 @@ The `internal/job/` package contains shared domain types and two subpackages:
| `client.go` | Publish-and-wait/collect with KV + stream |
| `query.go` | Query operations (system status, hostname, etc.) |
| `modify.go` | Modify operations (DNS updates) |
| `jobs.go` | CreateJob, RetryJob, GetJobStatus, GetQueueStats |
| `jobs.go` | CreateJob, RetryJob, GetJobStatus, ListJobs |
| `agent.go` | WriteStatusEvent, WriteJobResponse |
| `types.go` | Client-specific types and interfaces |

Expand Down Expand Up @@ -770,11 +765,64 @@ osapi agent start

## Performance Optimizations

### Two-Pass Job Listing

Job listing uses a two-pass approach to avoid reading every job payload:

**Pass 1 — Key-name scan (fast):** Calls `kv.Keys()` once to get all key names
in the bucket. Job status is derived purely from key name patterns
(`status.{id}.{event}.{hostname}.{ts}`) without any `kv.Get()` calls. This
produces ordered job IDs, per-job status, and aggregate status counts — all from
string parsing in memory.

**Pass 2 — Page fetch (bounded):** Fetches full job details (`kv.Get()`) only
for the paginated page. With `limit=10`, this is ~10 reads regardless of total
queue size.

```
Pass 1: kv.Keys() → 1 call → parse key names → status for all jobs
Pass 2: kv.Get() → N calls → full details for page only (N = limit)
```

Queue statistics (`GetQueueSummary`) and the `ListJobs` status counts both use
Pass 1 only — no `kv.Get()` calls at all.

### Pagination Limits

The API enforces a maximum page size of 100 (`MaxPageSize`). Requests with
`limit=0` or `limit > 100` return 400. The default page size is 10.

### Known Scalability Constraint: `kv.Keys()`

The two-pass approach relies on `kv.Keys()`, which returns **all key names** in
the bucket as a string slice. NATS JetStream does not support paginated key
listing (`kv.Keys(prefix, limit, offset)`) — it is all or nothing.

This is acceptable today because:

- Key names are short strings (~80 bytes each)
- The KV bucket has a 1-hour TTL, naturally bounding the key count
- Even 100K keys as strings is only a few MB of memory

However, at very large scale (millions of keys or longer/no TTL), `kv.Keys()`
would become a bottleneck in both memory and latency. If this becomes a problem,
potential approaches include:

1. **Separate status index** — a dedicated KV key (e.g., `index.status.failed`)
maintaining a list of job IDs, updated on status transitions
2. **External index** — move listing/filtering to a database (SQLite, Postgres)
while keeping NATS for job dispatch and processing
3. **NATS KV watch** — use `kv.Watch()` to maintain an in-memory index
incrementally rather than scanning all keys on each request

For now, the 1-hour TTL keeps the bucket bounded and `kv.Keys()` fast.

### Other Optimizations

1. **Batch Operations**: Agents can fetch multiple jobs per poll
2. **Connection Pooling**: Reuse NATS connections
3. **KV Caching**: Local caching of frequently accessed jobs
4. **Stream Filtering**: Agents only receive relevant job types
5. **Efficient Filtering**: Status-based key prefixes enable fast queries

## Error Handling

Expand Down
Loading