[epic] Soroban RPC Productionalization

[tsachiherman]

What ?

This epic covers the soroban-rpc productionalization efforts, grouped by areas.

Deployment Reliability

• [P1] Startup Health Check (soroban-rpc: Check DB on health endpoint stellar-cli#449)
  • Note: Make sure the http getHealth returns non-200 status code when failing.
• Optional Watchdog
  • [P3] (nice-to-have?) soroban-rpc: Add systemd sd_notify support for debian package #30
  • [P1] Get helm chart using healthcheck to alert/restart soroban-rpc (soroban-rpc: Production-ready helm-chart stellar-cli#514)
• [P2] soroban-rpc: Centralize error handling and failure stellar-cli#500
• [P1] soroban-rpc: Graceful shutdown for HTTP server stellar-cli#499

Monitoring

• [P1] soroban-rpc: Add /metrics endpoint stellar-cli#496
• [P3] soroban-rpc: Add opt-in remote metrics push so we can collect usage data #35
• [P3] soroban-rpc: Add configurable pprof support stellar-cli#498
• [P1] soroban-rpc: add Grafana dashboard stellar-cli#669

Logging

• [P1] soroban-rpc: Audit and standardize logging stellar-cli#497
  • Log clearly startup conditions
  • Define clear guidelines for each log severity and apply these.
  • Log all requests and their resolution.
• [P2] Cycle through log files
  • Stdout/stderr are automatically handled by systemd for debian install
  • Make sure logs are handled in the helm chart (soroban-rpc: Production-ready helm-chart stellar-cli#514)
• [P3] soroban-rpc: Add opt-in remote logging push so we can collect usage data #34

Tracing/Telemetry

• [P3] soroban-rpc: Support distributed tracing #33

Scalability

• [P2] Baseline Performance Testing (soroban-rpc: Establish a benchmark baseline on (a few?) different hardware profiles stellar-cli#505)
• [P2] Baseline Memory Profile (soroban-rpc: Establish a benchmark baseline on (a few?) different hardware profiles stellar-cli#505)
• [P1] soroban-rpc: Review and improve memory consumption stellar-cli#554
• [P1] Limit concurrency on high-cpu handler calls, based on number of available CPUs.
  • soroban-rpc: Limit preflight-computation concurrency through a worker pool stellar-cli#489
  • Others TBD based on outcome of performance testing
• [P2] soroban-rpc: Define and document a scaling strategy stellar-cli#506

Private Instance

• Add access control so that an operator can have a private application use
  • Planned initial infra providers have said they would rather do this themselves.

Documentation

• [P2] soroban-rpc: Runbook for infra operators stellar-cli#512
• [P3] soroban-rpc: Automate our API docs using something like swagger #32

Deployment

• [P1] soroban-rpc: Add support for configuration from a file stellar-cli#507
• Docker image + helm chart
  • [P1] Deployment Image for Soroban RPC stellar-cli#382
  • [P1] soroban-rpc: Production-ready helm-chart stellar-cli#514
• [P1] Add Soroban RPC packaging packages#183

Misc

• [P2] soroban-rpc: Audit and enforce timeouts everywhere we need them stellar-cli#509
• [P2] soroban-rpc: Audit and analyze retries on any external calls #31
• [P1] soroban-rpc: Review & fix any remaining TODOs/FIXMEs before "real" release. stellar-cli#511

[2opremio]

Related stellar/stellar-cli#435

[mollykarcher]

There are a few issues referenced here still not complete, but they are low-priority. Closing out this epic, and we'll reprioritize the remaining issues at a later time

[mollykarcher]

Don't mind me, just testing the github close action https://stellarfoundation.slack.com/archives/C02U19A2A/p1707843214915849