# [IMPROVEMENT] Implement Structured Logging and Improve Observability Using `solana-logger` and `tui-logger`
---
### Priority: Normal
### Labels: improvement, technical-debt
### Estimated Size: Medium
---
## 🚀 Problem Statement
Currently, the `osvm-cli` project relies on basic logging mechanisms that do not leverage structured logging paradigms or rich observability features. This limits our ability to effectively diagnose runtime issues, trace complex workflows, and provide meaningful feedback to users and developers. Improving the logging system to use structured logs and enhanced observability will enable better runtime diagnostics, improve troubleshooting, and facilitate future analytics or monitoring integrations.
---
## 🧠 Technical Context
- The project is a Rust-based CLI tool managing Solana Virtual Machines (SVMs).
- Existing logs are mostly unstructured and scattered across modules.
- We already have two crates in use for logging:
- [`solana-logger`](https://crates.io/crates/solana-logger): Provides structured log formatting aligned with Solana ecosystem standards.
- [`tui-logger`](https://crates.io/crates/tui-logger): Offers terminal UI-based log viewing and filtering.
- The goal is to unify and enhance logging by adopting these crates fully, ensuring consistent structured logs with contextual metadata and real-time observability in the terminal.
- Backward compatibility and minimal disruption of current CLI workflows are key.
---
## 🎯 Detailed Implementation Steps
1. **Audit Current Logging Implementation**
- Identify all places where `println!` or basic `log` crate macros are used.
- Document current logging levels, formats, and information content.
- Understand how logs are currently consumed (e.g., CLI output, files).
2. **Design Structured Logging Format**
- Define log schema fields: timestamps, log level, module path, thread IDs, contextual data (e.g., SVM identifiers, command names).
- Ensure compatibility with Solana ecosystem log formats.
3. **Integrate `solana-logger`**
- Replace existing logging macros with `solana-logger` macros or APIs.
- Configure global logger initialization in `main.rs` or an appropriate central place.
- Support dynamic log level configuration via CLI flags or environment variables.
4. **Integrate `tui-logger` for Interactive Observability**
- Embed `tui-logger` in CLI interactive modes or optionally enable it.
- Allow users to filter logs by level and module in real-time.
- Ensure smooth fallback for non-interactive or scripted usage.
5. **Backward Compatibility and Incremental Rollout**
- Incrementally refactor logging calls to avoid a large monolithic PR.
- Provide a compatibility shim if needed for legacy log consumers.
6. **Testing and Validation**
- Add unit tests for new logging helpers or wrappers.
- Add integration tests simulating typical CLI commands and verifying structured log output correctness.
- Validate performance impact on startup and runtime.
7. **Documentation Updates**
- Update CLI usage docs to mention new logging flags and features.
- Add a developer guide section on how to use structured logging and `tui-logger`.
- Document troubleshooting steps leveraging improved observability.
---
## 🛠️ Technical Specifications
- **Logging API:** Adopt `solana-logger` macros such as `solana_logger::log!`, `solana_logger::info!`, etc.
- **Log Levels:** Support `ERROR`, `WARN`, `INFO`, `DEBUG`, and `TRACE`.
- **Configuration:**
- Use CLI flags like `--log-level` for runtime log verbosity control.
- Support `RUST_LOG` environment variable compatibility.
- **Structured Log Fields:**
- `timestamp` in ISO 8601 UTC
- `level` (string)
- `module_path` (Rust module path)
- `thread_id` (numeric or hex)
- `message` (string)
- Optional contextual tags (e.g., `svm_id`, `command_name`)
- **Observability UI:**
- Integrate `tui-logger` in a non-blocking mode, invoked by a flag like `--log-ui`.
- Support filtering and searching logs by level and text.
- **Compatibility:** No breaking API changes to CLI commands or outputs.
---
## ✅ Acceptance Criteria
- [ ] Comprehensive audit report of current logging state is created and added to docs.
- [ ] All logging calls refactored to use structured logging via `solana-logger`.
- [ ] `tui-logger` integrated and usable interactively without breaking existing CLI workflows.
- [ ] CLI flags or environment variables allow dynamic log level configuration.
- [ ] Unit and integration tests cover new logging behavior and output formats.
- [ ] Performance benchmarks show no significant degradation (e.g., <5% overhead).
- [ ] Documentation updated with usage and developer guidelines.
---
## 🧪 Testing Requirements
- **Unit Tests:**
- Verify logging helpers produce correctly structured logs with all required fields.
- **Integration Tests:**
- Execute CLI commands with various log levels and verify logs contain expected structured data.
- Test log UI mode (`--log-ui`) for responsiveness and filtering.
- **Manual Testing:**
- Run CLI commands under normal and verbose logging modes and inspect logs.
- Validate fallback behavior when `tui-logger` UI is not enabled.
- **Performance Testing:**
- Measure startup and command execution latency before and after logging changes.
---
## 📚 Documentation Needs
- Update **User Guide** with:
- New logging CLI flags (`--log-level`, `--log-ui`).
- How to interpret structured log outputs.
- Add a **Developer Guide** section:
- How to add new structured logging calls.
- How to use `tui-logger` for debugging.
- Examples of contextual logging.
- Update **CHANGELOG.md** with improvement details.
---
## ⚠️ Potential Challenges & Risks
- **Complexity in Refactoring:**
Incremental rollout is necessary to avoid breaking CLI commands or scripts relying on existing log formats.
- **Performance Overhead:**
Structured logging and UI rendering could impact performance; must benchmark and optimize.
- **User Adoption:**
CLI users need clear documentation to leverage new logging features effectively.
- **Terminal Compatibility:**
`tui-logger` depends on terminal features; fallback modes must be tested on diverse environments.
---
## 🔗 Resources & References
- [`solana-logger` crate documentation](https://docs.rs/solana-logger)
- [`tui-logger` crate documentation](https://docs.rs/tui-logger)
- [Rust `log` crate API](https://docs.rs/log) for baseline logging concepts
- [OSVM CLI repository](https://github.com/openSVM/osvm-cli)
- [Rust CLI best practices](https://rust-cli.github.io/book/)
- Example structured logging patterns in Rust projects:
- [log4rs structured logging example](https://docs.rs/log4rs/latest/log4rs/)
- [Tracing crate examples](https://docs.rs/tracing/latest/tracing/)
---
💡 **Let's make `osvm-cli` a beacon of observability in the Solana ecosystem!**
If you have questions or want to brainstorm on design approaches, ping me here or on Discord.
---
**Happy hacking!** 🎉🚀
