Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
182 changes: 49 additions & 133 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Konfig - Dynamic Configuration Service

A distributed configuration management system built with C++17, gRPC, Kafka, and modern observability tools. Upload, validate, distribute, and roll out configuration changes across services in real-time.
A distributed configuration management system built with C++17, gRPC, Kafka, and modern observability tools. Upload, validate, and roll out configuration changes to thousands of services in real-time.

## Quick Start

Expand All @@ -15,33 +15,37 @@ make services
make cli

# 4. Upload a config
./bin/konfig upload --service my-service --file config.json --format json
./bin/configctl upload my-service --file config.yaml --format yaml

# 5. Retrieve it
./bin/konfig get --id my-service-v1
# 5. Roll it out to all instances
./bin/configctl rollout my-service-v1 --strategy ALL_AT_ONCE
```

## Architecture

```
┌──────────────────┐
CLI (konfig)
│ CLI (configctl)
│ Go / gRPC │
└────────┬─────────┘
│ :8081
┌────────▼─────────┐
│ API Service │
│ Upload/Rollout │
└────────┬─────────┘
│ Kafka
┌──────────────┼──────────────┐
│ │ │
┌──────▼──────┐ ┌────▼─────┐ ┌─────▼──────┐
│ API Service │ │ Dist │ │ Validation │
│ :8081 │ │ Service │ │ Service │
│ (gRPC) │ │ :8082 │ │ :8083 │
└──────┬──────┘ └────┬─────┘ └─────┬──────┘
│ │ │
┌───────┬───┴───┬────────┤ │
│ │ │ │ │
┌────▼──┐ ┌─▼───┐ ┌─▼────┐ ┌─▼───┐ ┌────▼──┐
│Postgres│ │Redis│ │Kafka │ │Redis│ │Postgres│
└───────┘ └─────┘ └──────┘ └─────┘ └───────┘
│ PostgreSQL │ │ Redis │ │Distribution│
│ (config, │ │ (cache) │ │ Service │
│ rollouts) │ └──────────┘ │ :8082 │
└─────────────┘ └─────┬──────┘
│ gRPC stream
┌──────▼──────┐
│ Client SDK │
│ (C++ lib) │
└─────────────┘
```

### Services
Expand All @@ -51,16 +55,14 @@ make cli
| **API Service** | 8081 | Config upload, retrieval, deletion, rollout management |
| **Distribution Service** | 8082 | Real-time config push to clients via gRPC streaming |
| **Validation Service** | 8083 | Config syntax/schema/rule validation (JSON & YAML) |
| **CLI (`konfig`)** | - | Command-line tool for interacting with the API |
| **Client SDK** | - | C++ library for services to receive config updates |

### Infrastructure

| Component | Port | Purpose |
|-----------|------|---------|
| PostgreSQL | 5432 | Config metadata, data, audit logs, validation rules |
| Redis | 6379 | Caching, validation result cache |
| Kafka | 9092/9093 | Event streaming, config update notifications |
| PostgreSQL | 5432 | Config metadata, data, audit logs, rollout state |
| Redis | 6379 | Caching layer |
| Kafka | 9092/9093 | Event streaming (rollout triggers, config notifications) |
| Prometheus | 9090 | Metrics collection |
| Grafana | 3000 | Metrics dashboards (admin/admin) |
| Kafka UI | 8080 | Topic and message inspection |
Expand All @@ -72,43 +74,21 @@ make cli
```
Konfig/
├── cmd/configctl/ # CLI tool (Go)
│ └── main.go
├── config/ # Service configuration files
│ ├── api-service.yml # Docker config
│ ├── api-service-local.yml # Local dev config
│ ├── distribution-service.yml
│ ├── distribution-service-local.yml
│ └── validation-service.yml
├── db/migrations/ # PostgreSQL schema migrations (000-008)
├── docker/
│ ├── postgres/init.sql # DB initialization (runs migrations)
│ ├── grafana/ # Grafana provisioning
│ └── services/ # Per-service Dockerfiles
│ ├── api-service.Dockerfile
│ ├── distribution-service.Dockerfile
│ └── validation-service.Dockerfile
├── db/migrations/ # PostgreSQL schema migrations
├── docker/ # Dockerfiles and init scripts
├── examples/ # Example configs and client
├── include/ # C++ headers
│ ├── api_service/
│ ├── distribution_service/
│ ├── validation_service/
│ ├── configclient/ # Client SDK headers
│ └── statsdclient/ # StatsD client
├── internal/commands/ # CLI command implementations
├── proto/ # Protocol Buffer definitions
│ ├── api.proto # ConfigAPIService RPCs
│ ├── distribution.proto # DistributionService RPCs
│ ├── validation.proto # ValidationService RPCs
│ └── config.proto # Shared message types
├── prometheus/ # Prometheus & StatsD config
├── scripts/ # Build helper scripts
├── src/
│ ├── api-service/ # API Service implementation
│ ├── distribution-service/ # Distribution Service implementation
│ ├── validation-service/ # Validation Service implementation
│ ├── client-sdk/ # Client SDK library
│ └── common/ # Shared utilities (StatsD client)
├── docker-compose.yml # All containers
├── Dockerfile.dev # Development build container
└── Makefile # Build automation
│ ├── api-service/
│ ├── distribution-service/
│ ├── validation-service/
│ ├── client-sdk/
│ └── common/
├── docker-compose.yml
└── Makefile
```

## Makefile Commands
Expand All @@ -127,20 +107,16 @@ make services-local # Build binaries locally (no Docker)
make dev # Complete setup (dirs + infrastructure + verify)
make infra-up # Start infrastructure containers
make infra-down # Stop infrastructure containers
make infra-restart # Restart infrastructure
make verify # Health check all services
```

### Build

```bash
make proto # Generate protobuf/gRPC code
make api-service # Build API service locally
make distribution-service # Build distribution service locally
make validation-service # Build validation service locally
make sdk # Build client SDK (shared + static)
make cache-test # Build disk cache test binary (bin/cache_test)
make cli # Build CLI tool (bin/konfig)
make cli # Build CLI tool (bin/configctl)
make all # Build everything locally
make clean # Remove build artifacts
make rebuild # Clean + build all
Expand All @@ -154,55 +130,18 @@ make redis-shell # Redis CLI
make kafka-topics # List Kafka topics
make kafka-ui # Open Kafka UI in browser
make grafana # Open Grafana in browser
make pgadmin # Open pgAdmin in browser
```

See [COMMANDS.md](COMMANDS.md) for the complete command reference.

## CLI Usage
### Dev Container

```bash
# Upload configuration
./bin/konfig upload --service payment-service --file config.json --format json

# Get config by ID
./bin/konfig get --id payment-service-v1

# List configs for a service
./bin/konfig list --service payment-service

# Validate without uploading
./bin/konfig validate --service payment-service --file config.json --format json

# Delete a config
./bin/konfig delete --id payment-service-v1

# Check rollout status
./bin/konfig status --id payment-service-v1

# Rollback to previous version
./bin/konfig rollback --service payment-service --version 1
make dev-up # Start dev container
make dev-shell # Enter interactive shell
make dev-build # Build inside container
make dev-sdk # Build client SDK inside container
```

## Database Schema

Managed via migrations in `db/migrations/` (000-008):

| Table | Purpose |
|-------|---------|
| `config_metadata` | Service name, version, format, timestamps |
| `config_data` | Actual config content and hashes (FK to metadata) |
| `rollout_state` | Gradual rollout tracking |
| `service_instances` | Connected client instances |
| `audit_log` | All config actions with JSONB details |
| `health_checks` | Service health status |
| `validation_schemas` | Registered validation schemas |
| `validation_rules` | Custom validation rules per service |
| `validation_history` | Validation result audit trail |

## Development

### Local Development
## Local Development

```bash
# Start infrastructure
Expand All @@ -217,33 +156,9 @@ make services-local
./bin/distribution-service config/distribution-service-local.yml
```

### Docker Development

```bash
# Start everything
make infra-up && make services

# View logs
docker compose logs -f api-service
docker compose logs -f validation-service

# Rebuild after code changes
make services
```

### Dev Container (for building inside Linux)

```bash
make dev-up # Start dev container
make dev-shell # Enter interactive shell
make dev-build # Build inside container
make dev-sdk # Build client SDK inside container
make dev-cache-test # Build disk cache test binary inside container
```

## Client SDK

The C++ client SDK (`libconfigclient`) lets services subscribe to real-time config updates with automatic reconnection and disk caching.
The C++ client SDK (`libconfigclient`) lets services subscribe to real-time config updates with automatic reconnection, configurable heartbeating, and disk caching. See [Client SDK](src/client-sdk/README.md) for the full reference including heartbeat configuration and advanced options.

```cpp
#include "configclient/config_client.h"
Expand Down Expand Up @@ -281,7 +196,7 @@ make dev-sdk && make dev-cache-test
./bin/cache_test distribution-service:8082 payment-service

# Step 2 — upload a config (separate terminal)
./bin/konfig upload --service payment-service --file examples/configs/valid-config.json --format json
./bin/configctl upload payment-service --file examples/configs/valid-config.json --format json
# cache_test prints: >>> CONFIG UPDATE <<< and [DiskCache] Saved config v1 ...

# Step 3 — restart: cache loads before gRPC connects
Expand Down Expand Up @@ -318,18 +233,19 @@ Headers are in `include/configclient/`. Libraries are in `lib/` after `make sdk`
| PostgreSQL | `localhost:5432` | `postgres:5432` |
| Redis | `localhost:6379` | `redis:6379` |
| Kafka | `localhost:9093` | `kafka:9092` |
| StatsD | `localhost:9125` | `statsd-exporter:9125` |
| API Service | `localhost:8081` | `api-service:8081` |
| Validation Service | `localhost:8083` | `validation-service:8083` |
| Distribution Service | `localhost:8082` | `distribution-service:8082` |

**Database credentials:** `configuser` / `configpass` / `configservice`

## Service Documentation
## Documentation

- [API Service](src/api-service/README.md)
- [Distribution Service](src/distribution-service/README.md)
- [Validation Service](src/validation-service/README.md)
- [CLI Reference](cmd/configctl/README.md) — all commands, flags, rollout strategies
- [Client SDK](src/client-sdk/README.md) — C++ SDK usage, heartbeat config, disk cache
- [API Service](src/api-service/README.md) — gRPC API, upload flow, components
- [Distribution Service](src/distribution-service/README.md) — streaming, rollout execution, heartbeat monitor
- [Validation Service](src/validation-service/README.md) — schema validation, rules

## License

Expand Down
Loading
Loading