feat: unify grpc and plugin configs

[thlorenz]

Summary

Unify grpc-service and geyser-plugin configuration around TOML with aligned shared keys so the two configs can be translated with minimal edits.

• move grpc-service from env-driven startup config to TOML files with --config PATH support
• move geyser-plugin from JSON config to TOML while preserving raw Kafka client passthrough via [kafka.client]
• align shared Kafka and ksqlDB settings across both crates and add matching example configs and README updates

Details

This change standardizes the operator-facing config shape for the two Kafka-coupled crates in this workspace.

Shared values now use the same names and sections in both crates, centered on:

• [kafka] for bootstrap_servers and topic
• [ksql] for url and table

The plugin keeps low-level producer tuning available through [kafka.client], while grpc-service keeps a typed consumer config and adds the same optional extension point for raw client overrides if needed later.

grpc-service

• replaced environment-variable config assembly with TOML file loading
• added cargo run -- [--config PATH] [PUBKEY] support without removing the existing optional pubkey filter behavior
• updated Kafka and ksql config usage to the normalized field names
• added committed example and default config files under grpc-service/configs/
• updated the Makefile and README to use the new TOML-based workflow

geyser-plugin

• replaced JSON config parsing with TOML parsing
• moved shared Kafka settings to normalized [kafka] keys and startup restore settings to [ksql]
• moved plugin-only settings under [plugin]
• preserved direct rdkafka client passthrough through [kafka.client]
• added TOML example/runtime config files and updated the Makefile and README accordingly

Summary by CodeRabbit

• New Features

  • Services now use TOML-based configuration files instead of JSON
  • Added --config command-line flag to specify custom configuration file paths

• Documentation

  • Updated service documentation with TOML configuration examples and CLI usage patterns

• Chores

  • Refactored configuration system to support TOML file parsing
  • Added TOML library dependencies

[coderabbitai]

Warning

Rate limit exceeded

@thlorenz has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 43 minutes and 33 seconds before requesting another review.

Your organization is not enrolled in usage-based pricing. Contact your admin to enable usage-based pricing to continue reviews beyond the rate limit, or try again in 43 minutes and 33 seconds.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: d8008071-49ea-4b77-9617-78610bc6a20e

📥 Commits

Reviewing files that changed from the base of the PR and between b002b0a and 27dbd88.

📒 Files selected for processing (5)

• geyser-plugin/README.md
• geyser-plugin/src/config.rs
• geyser-plugin/src/plugin/mod.rs
• grpc-service/Makefile
• grpc-service/src/kafka.rs

Walkthrough

The PR refactors configuration handling across geyser-plugin and grpc-service from JSON and environment variables to TOML files. Both services migrate configuration structures, add TOML dependencies, reorganize config into nested sections (Kafka, ksqlDB, Plugin), and update loading logic while preserving functional equivalence.

Changes

Cohort / File(s)	Summary
Dependency Management geyser-plugin/Cargo.toml, grpc-service/Cargo.toml	Added toml crate dependency to both; added serde with derive feature to grpc-service.
Configuration Schema Refactoring geyser-plugin/src/config.rs, grpc-service/src/config.rs	Restructured flat configs into nested TOML structs (KafkaConfig, KsqlConfig/PluginConfig for geyser; KafkaConfig, KsqlConfig, ValidatorConfig, GrpcConfig for grpc). Renamed fields (broker→bootstrap_servers, server_url→url); added client: BTreeMap for extended settings. Switched parsing from serde_json (env vars) to toml::from_str (file-based). Updated validation and error reporting.
Configuration Field Access Updates geyser-plugin/src/plugin/mod.rs, grpc-service/src/kafka.rs, grpc-service/src/ksql.rs	Updated field access paths to reflect nested config structure (e.g., config.shutdown_timeout_ms→config.plugin.shutdown_timeout_ms, config.broker→config.bootstrap_servers). Improved error context and log reporting.
Build System & Initialization geyser-plugin/Makefile, grpc-service/Makefile	Updated default config variable from JSON to TOML filenames; added init-config target to create default configs from examples; changed grpc-service run target from env-var-based invocation to --config flag.
Configuration Files & Documentation geyser-plugin/plugin-config.toml, geyser-plugin/configs/plugin-config.example.toml, grpc-service/configs/config.toml, grpc-service/configs/config.example.toml, geyser-plugin/README.md, grpc-service/README.md	Added new TOML config files and examples with nested sections for Kafka, ksqlDB, and runtime settings. Updated READMEs to document TOML format, CLI usage (--config PATH), and new field organization.
Error Handling grpc-service/src/errors.rs	Replaced InvalidEnvRead/InvalidEnvValue error variants with ConfigFileRead and ConfigTomlParse to support file-based config errors. Updated CLI usage message to include --config argument.
Code Style Improvements grpc-service/examples/grpc_client.rs, grpc-service/src/grpc_service/dispatcher.rs	Consolidated nested conditional chains into single if let && condition expressions for improved readability; no behavioral change.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 47.06% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'feat: unify grpc and plugin configs' accurately and concisely summarizes the main change: standardizing configuration across both the grpc-service and geyser-plugin crates by moving to TOML-based config files with aligned shared keys.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch config

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 9

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@geyser-plugin/configs/plugin-config.example.toml`:
- Around line 13-15: The example config currently enables ksqlDB restore by
providing a non-running default url under the [ksql] block (ksql.url =
"http://localhost:8088"); change the example so restore stays disabled by
default — either comment out the entire [ksql] block or remove/blank the
ksql.url entry (or prefix it with a comment) so that the generated config from
make init-config does not contain a live ksql.url and plugin startup won't
attempt a restore.

In `@geyser-plugin/plugin-config.toml`:
- Line 1: The committed plugin-config.toml contains a platform-specific libpath
("libpath = \"target/release/libsolana_accountsdb_plugin_kafka.so\"") which
breaks non-Linux launches; remove this generated root-level file from the PR (or
replace it with a neutral example) and put the platform-specific sample under
configs/ (or update .gitignore to ignore generated plugin-config.toml). Ensure
the root plugin-config.toml either does not exist in the repo or contains a
platform-agnostic example value rather than the .so path so make init-config /
make launch behave correctly across macOS (.dylib) and Linux (.so).

In `@geyser-plugin/README.md`:
- Around line 37-39: The Quick Start README currently includes an enabled [ksql]
block (keys: url and table) which causes startup restore to require ksqlDB;
instead update the Quick Start to treat this as optional by commenting out or
removing the [ksql] block (the "[ksql]" header and its "url" and "table"
entries) or move it to an "Optional configuration" section with clear comment
that it must be omitted for minimal local runs, so plugin load won't attempt a
ksqlDB restore on first run.

In `@geyser-plugin/src/config.rs`:
- Around line 162-191: When validating ksql.url in the existing block
(referencing self.ksql.url), also validate the configured table name: if
self.restore_on_startup is true (or startup restore is enabled) and ksql.url is
present, trim self.ksql.table and return a
GeyserPluginError::ConfigFileReadError with an appropriate message (e.g.,
"invalid config field `ksql.table`: table must not be empty") when the trimmed
table is empty; use the same error construction style as the existing checks so
the validation occurs up front (refer to self.ksql.table and
restore_on_startup).

In `@geyser-plugin/src/plugin/mod.rs`:
- Around line 221-228: The code uses the raw config.ksql.url (possibly with
surrounding whitespace) when logging and calling KsqlPubkeyRestoreClient::new;
change it to use the validated/trimmed value by trimming the Option<&str> before
use (e.g., extract the Some(raw) from config.ksql.url.as_deref(), call
raw.trim() and assign to url, then use that trimmed url in the info! log and in
KsqlPubkeyRestoreClient::new). Reference symbols: config.ksql.url,
KsqlPubkeyRestoreClient::new, and the local variable url.
- Around line 95-99: The producer config currently sets "bootstrap.servers"
before applying entries from config.kafka.client, which allows a later client
override; change the order so you first apply all entries from
config.kafka.client to producer_config (using producer_config.set for each (key,
value)), and only after that call producer_config.set("bootstrap.servers",
&config.kafka.bootstrap_servers) to ensure the typed bootstrap.servers value
(from config.kafka.bootstrap_servers) wins; update the code around
ClientConfig::new(), the loop over config.kafka.client, and the subsequent set
call accordingly.

In `@grpc-service/configs/config.toml`:
- Around line 1-17: Remove the tracked local default config file (config.toml)
so the example remains the single source of truth: delete config.toml from the
repository, add its filename to .gitignore, and commit that change; ensure the
init task (make init-config) still creates config.toml from config.example.toml
when absent. Reference the two files config.toml and config.example.toml and the
repository's init flow (make init-config) when making the change so local
operator-specific edits no longer show as tracked modifications.

In `@grpc-service/Makefile`:
- Line 11: The Makefile's run target is dropping the PUBKEY argument when
invoking the service; update the run target (and any other targets that invoke
the binary) to forward PUBKEY to the executed command by adding PUBKEY=$(PUBKEY)
(or exporting PUBKEY) in the command invocation so the binary still receives the
optional [PUBKEY] argument; ensure the run target's invocation mirrors how
CONFIG is forwarded so PUBKEY is preserved.

In `@grpc-service/src/kafka.rs`:
- Around line 31-39: The client map is being applied after the typed fields so
user-supplied keys (like "bootstrap.servers", "group.id", "auto.offset.reset")
can override canonical settings; change the order so that the raw client
settings from self.config.client are applied first to client_config and then
explicitly set the canonical fields (bootstrap.servers, group.id,
auto.offset.reset, enable.auto.commit) to guarantee they are authoritative, or
alternatively filter/reject those reserved keys from self.config.client before
applying; make the same change in every place you build a ClientConfig in this
file (the blocks around client_config creation and the later similar blocks).

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: c07caaa8-a72b-4d02-a1ad-96cf1cf2757e

📥 Commits

Reviewing files that changed from the base of the PR and between e312821 and b002b0a.

⛔ Files ignored due to path filters (1)

• Cargo.lock is excluded by !**/*.lock

📒 Files selected for processing (18)

• geyser-plugin/Cargo.toml
• geyser-plugin/Makefile
• geyser-plugin/README.md
• geyser-plugin/configs/plugin-config.example.toml
• geyser-plugin/plugin-config.toml
• geyser-plugin/src/config.rs
• geyser-plugin/src/plugin/mod.rs
• grpc-service/Cargo.toml
• grpc-service/Makefile
• grpc-service/README.md
• grpc-service/configs/config.example.toml
• grpc-service/configs/config.toml
• grpc-service/examples/grpc_client.rs
• grpc-service/src/config.rs
• grpc-service/src/errors.rs
• grpc-service/src/grpc_service/dispatcher.rs
• grpc-service/src/kafka.rs
• grpc-service/src/ksql.rs