Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
e3504f4
Test: verify branching workflow
codec404 Feb 15, 2026
2dde83c
workflow changes
codec404 Feb 15, 2026
5f242ee
Merge branch 'master' into dev
codec404 Feb 15, 2026
f24f0c5
removed TEST
codec404 Feb 15, 2026
bc62dd5
Merge branch 'master' into dev
codec404 Feb 15, 2026
1fd5419
newer commits only
codec404 Feb 15, 2026
d21f620
Merge branch 'master' into dev
codec404 Feb 15, 2026
d8abe9f
Merge branch 'master' into dev
codec404 Feb 15, 2026
3209db8
Feature/cli
codec404 Feb 15, 2026
d19c8ff
added dist-service
codec404 Feb 16, 2026
f797835
Merge branch 'master' into dev
codec404 Feb 16, 2026
aaa329b
added the api service
codec404 Feb 17, 2026
ecd7012
fixed linters
codec404 Feb 17, 2026
dcd7995
Merge pull request #21 from codec404/feature/api-service
codec404 Feb 17, 2026
40b2fc7
Merge branch 'master' into dev
codec404 Feb 17, 2026
fb16141
added connector t b/w sdk and services
codec404 Feb 17, 2026
b77de24
tested version
codec404 Feb 18, 2026
c9615ae
Merge pull request #23 from codec404/feature/connector
codec404 Feb 18, 2026
c9f7a68
ignoring the protos
codec404 Feb 18, 2026
5b7a76d
ignoring the protos
codec404 Feb 18, 2026
a3adf65
Merge pull request #25 from codec404/fix/hide-proto
codec404 Feb 18, 2026
e66c553
added db migration
codec404 Feb 18, 2026
930e7d2
Merge pull request #26 from codec404/feat/migration
codec404 Feb 18, 2026
e87cc31
added validation service
codec404 Feb 18, 2026
ecef7a1
Merge pull request #27 from codec404/feat/validation-service
codec404 Feb 18, 2026
2820eeb
Merge branch 'master' into dev
codec404 Feb 18, 2026
b62039d
e2e handled
codec404 Feb 18, 2026
c9cf2e1
Merge pull request #28 from codec404/feat/e2e
codec404 Feb 18, 2026
d2c91a8
security fix
codec404 Feb 18, 2026
6a02c7c
Merge pull request #30 from codec404/fix/security
codec404 Feb 18, 2026
5df0f05
Merge branch 'master' into dev
codec404 Feb 18, 2026
3c081f6
cleanup
codec404 Feb 18, 2026
44b0bc1
Merge pull request #31 from codec404/emt/cleanup
codec404 Feb 18, 2026
e4069aa
Merge branch 'master' into dev
codec404 Feb 18, 2026
aa62d69
updated README
codec404 Feb 18, 2026
05b8229
Merge branch 'master' into dev
codec404 Feb 18, 2026
df160f1
reordered example config
codec404 Feb 22, 2026
aa730d9
Merge pull request #35 from codec404/reorder
codec404 Feb 22, 2026
9fc962d
Merge branch 'master' into dev
codec404 Feb 22, 2026
cfe632c
added persistent caching to disk
codec404 Feb 22, 2026
2131213
added persistent caching to disk
codec404 Feb 22, 2026
f5879d8
Merge pull request #37 from codec404/feat/cache-to-disk
codec404 Feb 22, 2026
06015dc
Merge branch 'master' into dev
codec404 Feb 22, 2026
f2c505e
Feat/rollout (#39)
codec404 Mar 12, 2026
b79cace
Merge branch 'master' into dev
codec404 Mar 12, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
182 changes: 49 additions & 133 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Konfig - Dynamic Configuration Service

A distributed configuration management system built with C++17, gRPC, Kafka, and modern observability tools. Upload, validate, distribute, and roll out configuration changes across services in real-time.
A distributed configuration management system built with C++17, gRPC, Kafka, and modern observability tools. Upload, validate, and roll out configuration changes to thousands of services in real-time.

## Quick Start

Expand All @@ -15,33 +15,37 @@ make services
make cli

# 4. Upload a config
./bin/konfig upload --service my-service --file config.json --format json
./bin/configctl upload my-service --file config.yaml --format yaml

# 5. Retrieve it
./bin/konfig get --id my-service-v1
# 5. Roll it out to all instances
./bin/configctl rollout my-service-v1 --strategy ALL_AT_ONCE
```

## Architecture

```
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ CLI (konfig) β”‚
β”‚ CLI (configctl) β”‚
β”‚ Go / gRPC β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
β”‚
β”‚ :8081
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ API Service β”‚
β”‚ Upload/Rollout β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
β”‚ Kafka
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ β”‚ β”‚
β”Œβ”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”
β”‚ API Service β”‚ β”‚ Dist β”‚ β”‚ Validation β”‚
β”‚ :8081 β”‚ β”‚ Service β”‚ β”‚ Service β”‚
β”‚ (gRPC) β”‚ β”‚ :8082 β”‚ β”‚ :8083 β”‚
β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”˜
β”‚ β”‚ β”‚
β”Œβ”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”΄β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€ β”‚
β”‚ β”‚ β”‚ β”‚ β”‚
β”Œβ”€β”€β”€β”€β–Όβ”€β”€β” β”Œβ”€β–Όβ”€β”€β”€β” β”Œβ”€β–Όβ”€β”€β”€β”€β” β”Œβ”€β–Όβ”€β”€β”€β” β”Œβ”€β”€β”€β”€β–Όβ”€β”€β”
β”‚Postgresβ”‚ β”‚Redisβ”‚ β”‚Kafka β”‚ β”‚Redisβ”‚ β”‚Postgresβ”‚
β””β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”˜
β”‚ PostgreSQL β”‚ β”‚ Redis β”‚ β”‚Distributionβ”‚
β”‚ (config, β”‚ β”‚ (cache) β”‚ β”‚ Service β”‚
β”‚ rollouts) β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ :8082 β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”˜
β”‚ gRPC stream
β”Œβ”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”
β”‚ Client SDK β”‚
β”‚ (C++ lib) β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
```

### Services
Expand All @@ -51,16 +55,14 @@ make cli
| **API Service** | 8081 | Config upload, retrieval, deletion, rollout management |
| **Distribution Service** | 8082 | Real-time config push to clients via gRPC streaming |
| **Validation Service** | 8083 | Config syntax/schema/rule validation (JSON & YAML) |
| **CLI (`konfig`)** | - | Command-line tool for interacting with the API |
| **Client SDK** | - | C++ library for services to receive config updates |

### Infrastructure

| Component | Port | Purpose |
|-----------|------|---------|
| PostgreSQL | 5432 | Config metadata, data, audit logs, validation rules |
| Redis | 6379 | Caching, validation result cache |
| Kafka | 9092/9093 | Event streaming, config update notifications |
| PostgreSQL | 5432 | Config metadata, data, audit logs, rollout state |
| Redis | 6379 | Caching layer |
| Kafka | 9092/9093 | Event streaming (rollout triggers, config notifications) |
| Prometheus | 9090 | Metrics collection |
| Grafana | 3000 | Metrics dashboards (admin/admin) |
| Kafka UI | 8080 | Topic and message inspection |
Expand All @@ -72,43 +74,21 @@ make cli
```
Konfig/
β”œβ”€β”€ cmd/configctl/ # CLI tool (Go)
β”‚ └── main.go
β”œβ”€β”€ config/ # Service configuration files
β”‚ β”œβ”€β”€ api-service.yml # Docker config
β”‚ β”œβ”€β”€ api-service-local.yml # Local dev config
β”‚ β”œβ”€β”€ distribution-service.yml
β”‚ β”œβ”€β”€ distribution-service-local.yml
β”‚ └── validation-service.yml
β”œβ”€β”€ db/migrations/ # PostgreSQL schema migrations (000-008)
β”œβ”€β”€ docker/
β”‚ β”œβ”€β”€ postgres/init.sql # DB initialization (runs migrations)
β”‚ β”œβ”€β”€ grafana/ # Grafana provisioning
β”‚ └── services/ # Per-service Dockerfiles
β”‚ β”œβ”€β”€ api-service.Dockerfile
β”‚ β”œβ”€β”€ distribution-service.Dockerfile
β”‚ └── validation-service.Dockerfile
β”œβ”€β”€ db/migrations/ # PostgreSQL schema migrations
β”œβ”€β”€ docker/ # Dockerfiles and init scripts
β”œβ”€β”€ examples/ # Example configs and client
β”œβ”€β”€ include/ # C++ headers
β”‚ β”œβ”€β”€ api_service/
β”‚ β”œβ”€β”€ distribution_service/
β”‚ β”œβ”€β”€ validation_service/
β”‚ β”œβ”€β”€ configclient/ # Client SDK headers
β”‚ └── statsdclient/ # StatsD client
β”œβ”€β”€ internal/commands/ # CLI command implementations
β”œβ”€β”€ proto/ # Protocol Buffer definitions
β”‚ β”œβ”€β”€ api.proto # ConfigAPIService RPCs
β”‚ β”œβ”€β”€ distribution.proto # DistributionService RPCs
β”‚ β”œβ”€β”€ validation.proto # ValidationService RPCs
β”‚ └── config.proto # Shared message types
β”œβ”€β”€ prometheus/ # Prometheus & StatsD config
β”œβ”€β”€ scripts/ # Build helper scripts
β”œβ”€β”€ src/
β”‚ β”œβ”€β”€ api-service/ # API Service implementation
β”‚ β”œβ”€β”€ distribution-service/ # Distribution Service implementation
β”‚ β”œβ”€β”€ validation-service/ # Validation Service implementation
β”‚ β”œβ”€β”€ client-sdk/ # Client SDK library
β”‚ └── common/ # Shared utilities (StatsD client)
β”œβ”€β”€ docker-compose.yml # All containers
β”œβ”€β”€ Dockerfile.dev # Development build container
└── Makefile # Build automation
β”‚ β”œβ”€β”€ api-service/
β”‚ β”œβ”€β”€ distribution-service/
β”‚ β”œβ”€β”€ validation-service/
β”‚ β”œβ”€β”€ client-sdk/
β”‚ └── common/
β”œβ”€β”€ docker-compose.yml
└── Makefile
```

## Makefile Commands
Expand All @@ -127,20 +107,16 @@ make services-local # Build binaries locally (no Docker)
make dev # Complete setup (dirs + infrastructure + verify)
make infra-up # Start infrastructure containers
make infra-down # Stop infrastructure containers
make infra-restart # Restart infrastructure
make verify # Health check all services
```

### Build

```bash
make proto # Generate protobuf/gRPC code
make api-service # Build API service locally
make distribution-service # Build distribution service locally
make validation-service # Build validation service locally
make sdk # Build client SDK (shared + static)
make cache-test # Build disk cache test binary (bin/cache_test)
make cli # Build CLI tool (bin/konfig)
make cli # Build CLI tool (bin/configctl)
make all # Build everything locally
make clean # Remove build artifacts
make rebuild # Clean + build all
Expand All @@ -154,55 +130,18 @@ make redis-shell # Redis CLI
make kafka-topics # List Kafka topics
make kafka-ui # Open Kafka UI in browser
make grafana # Open Grafana in browser
make pgadmin # Open pgAdmin in browser
```

See [COMMANDS.md](COMMANDS.md) for the complete command reference.

## CLI Usage
### Dev Container

```bash
# Upload configuration
./bin/konfig upload --service payment-service --file config.json --format json

# Get config by ID
./bin/konfig get --id payment-service-v1

# List configs for a service
./bin/konfig list --service payment-service

# Validate without uploading
./bin/konfig validate --service payment-service --file config.json --format json

# Delete a config
./bin/konfig delete --id payment-service-v1

# Check rollout status
./bin/konfig status --id payment-service-v1

# Rollback to previous version
./bin/konfig rollback --service payment-service --version 1
make dev-up # Start dev container
make dev-shell # Enter interactive shell
make dev-build # Build inside container
make dev-sdk # Build client SDK inside container
```

## Database Schema

Managed via migrations in `db/migrations/` (000-008):

| Table | Purpose |
|-------|---------|
| `config_metadata` | Service name, version, format, timestamps |
| `config_data` | Actual config content and hashes (FK to metadata) |
| `rollout_state` | Gradual rollout tracking |
| `service_instances` | Connected client instances |
| `audit_log` | All config actions with JSONB details |
| `health_checks` | Service health status |
| `validation_schemas` | Registered validation schemas |
| `validation_rules` | Custom validation rules per service |
| `validation_history` | Validation result audit trail |

## Development

### Local Development
## Local Development

```bash
# Start infrastructure
Expand All @@ -217,33 +156,9 @@ make services-local
./bin/distribution-service config/distribution-service-local.yml
```

### Docker Development

```bash
# Start everything
make infra-up && make services

# View logs
docker compose logs -f api-service
docker compose logs -f validation-service

# Rebuild after code changes
make services
```

### Dev Container (for building inside Linux)

```bash
make dev-up # Start dev container
make dev-shell # Enter interactive shell
make dev-build # Build inside container
make dev-sdk # Build client SDK inside container
make dev-cache-test # Build disk cache test binary inside container
```

## Client SDK

The C++ client SDK (`libconfigclient`) lets services subscribe to real-time config updates with automatic reconnection and disk caching.
The C++ client SDK (`libconfigclient`) lets services subscribe to real-time config updates with automatic reconnection, configurable heartbeating, and disk caching. See [Client SDK](src/client-sdk/README.md) for the full reference including heartbeat configuration and advanced options.

```cpp
#include "configclient/config_client.h"
Expand Down Expand Up @@ -281,7 +196,7 @@ make dev-sdk && make dev-cache-test
./bin/cache_test distribution-service:8082 payment-service

# Step 2 β€” upload a config (separate terminal)
./bin/konfig upload --service payment-service --file examples/configs/valid-config.json --format json
./bin/configctl upload payment-service --file examples/configs/valid-config.json --format json
# cache_test prints: >>> CONFIG UPDATE <<< and [DiskCache] Saved config v1 ...

# Step 3 β€” restart: cache loads before gRPC connects
Expand Down Expand Up @@ -318,18 +233,19 @@ Headers are in `include/configclient/`. Libraries are in `lib/` after `make sdk`
| PostgreSQL | `localhost:5432` | `postgres:5432` |
| Redis | `localhost:6379` | `redis:6379` |
| Kafka | `localhost:9093` | `kafka:9092` |
| StatsD | `localhost:9125` | `statsd-exporter:9125` |
| API Service | `localhost:8081` | `api-service:8081` |
| Validation Service | `localhost:8083` | `validation-service:8083` |
| Distribution Service | `localhost:8082` | `distribution-service:8082` |

**Database credentials:** `configuser` / `configpass` / `configservice`

## Service Documentation
## Documentation

- [API Service](src/api-service/README.md)
- [Distribution Service](src/distribution-service/README.md)
- [Validation Service](src/validation-service/README.md)
- [CLI Reference](cmd/configctl/README.md) β€” all commands, flags, rollout strategies
- [Client SDK](src/client-sdk/README.md) β€” C++ SDK usage, heartbeat config, disk cache
- [API Service](src/api-service/README.md) β€” gRPC API, upload flow, components
- [Distribution Service](src/distribution-service/README.md) β€” streaming, rollout execution, heartbeat monitor
- [Validation Service](src/validation-service/README.md) β€” schema validation, rules

## License

Expand Down
Loading
Loading