first commit

.gitignore

@@ -0,0 +1,4 @@
+.idea/
+backup/
+build/
+etc/

README.md

@@ -0,0 +1,135 @@
+# CCM - Consul Config Manager
+
+CCM is a tool, which can help you automate configuration delivery for your applications.  
+It uses Consul as a backend service, to store configuration, and watches for any updates which need to be delivered.
+
+# Features
+
+## Service Registration
+Application automatically registeres itself as a service in Consul, so you can monitor its health through Consul UI.  
+![Application Service](https://raw.githubusercontent.com/leads-su/consul-config-manager/main/docs/images/service.png)
+
+Application will also provide information on its startup with all information, which will be available in the "Meta" section of Consul UI.  
+![Application Meta](https://raw.githubusercontent.com/leads-su/consul-config-manager/main/docs/images/meta.png)
+
+## KeyValue Watcher
+
+This watcher is able to detect changes made to your Consul installation, and then act accordingly.  
+Whenever change is detected, CCM will pull this information to local environment files and new configuration value will be available in a matter of seconds for you to use.
+
+### Example of supported data structures
+CCM requires you to provide KeyValue values in the following format.  
+This is required, so the CCM itself, as well as GUI could know what they are working with.
+
+**1. Number**  
+Simply tells CCM to convert this value to a number (instead of string)
+```json
+{"type":"number","value":123456}
+```
+**2. String**  
+Most basic type, all of the values by default are treated as strings
+```json
+{"type":"string","value":"database.localhost"}
+```
+**3. Array**  
+Allows CCM to build an array of values
+```json
+{"type":"array","value":["123", 456, "789"]}
+```
+**4. Reference**  
+This is a special type, it allows you to reference existing value, which will then be converted by the CCM to the real value upon receiving changed key
+```json
+{"type":"reference","value":"shared/database/mysql/username"}
+```
+
+## Task Runner
+CCM could also act as a task runner on the host it is installed on.  
+
+**For example:**
+1. You have 10 API servers
+2. You need to change the database address on all of them
+3. You need to write an Ansible role, or a pipeline to do so
+4. You also need to modify configuration values (in some cases)
+5. You need to keep 10 terminals open at a time (or just SSH to 10 servers one after another)
+
+With use of CCM and the GUI provided by us, you are able to create a single pipeline, which will watch for key changes and apply any necessary commands whenever you decide to change the database address.
+
+CCM can execute commands as itself (ccm), root (root), or any other user you specify.
+
+## Event Streaming Server
+In order to provide realtime output for the Task Runner, CCM utilizes SSE (Server Sent Events).   
+This allows to avoid hustle with WebSockets, as well as provides ability to store logs locally (and access them later through HTTP), as well as stream them to any other service.
+
+![Event Streaming Server](https://raw.githubusercontent.com/leads-su/consul-config-manager/main/docs/images/realtime_log.png)
+
+## Automatic Consul Server switching
+CCM is able to be configured with multiple servers in mind.  
+That means that in case there is a problem with one of the servers, CCM will switch to another one.  
+Also, upon initial connection, all servers will be pinged and server with lowest latency will be used.
+
+# Initila Setup
+
+By default, CCM will use `/etc/ccm.d` as its configuration folder.  
+It will also try to load `config.yml` from this directory as its default configuration source.  
+
+**IMPORTANT** - file extension must be exactly `yml` and not `yaml`, otherwise it will start with the default configuration (which you might not want).
+
+## Starting application
+
+If you wish to supply different configuration folder or configuration file name, you can do so with usage of flags.
+
+**Change configuration directory:**
+```bash
+--config-path=/etc/ccm.d
+```
+
+**Change configuration file name:**
+```bash
+--config-file=config.yml
+```
+
+**Start application:**
+```bash
+ccm start --config-path=/etc/ccm.d --config-file=config.yml
+```
+
+
+# Example Configuration
+```yaml
+agent:                                 # Agent Configuration
+  network:                             # Agent Network Configuration
+    interface: ""                      # Interfaces which will be used to obtain IP address (if "address" is empty)
+    address: "127.0.0.1"               # Manually set address which will be visible in Consul for this CCM instance
+    port: 32175                        # Port which will be used to serve metrics + SSE events
+  health_check:                        # Agent Health Checks configuration
+    ttl: true                          # Enable TTL healthcheck
+    http: true                         # Enable HTTP healthcheck
+consul:                                # Consul Configuration
+  enabled: true                        # Enable / Disable Consul service
+  datacenter: "dc0"                    # Datacenter Name
+  addresses:                           # List of Consul Servers (can be many)
+    - scheme: "http"                   # Scheme to be used to access Consul API
+      host: "consul1.local"            # Hostname of the Consul server
+      port: 8500                       # Port of the Consul server
+  token: "consul-acl-access-token"     # Access Token used to access Consul server
+  write_to: "/etc/ccm.d"               # Path, where configuration files will be written
+environment: "production"              # Application Environment
+log:                                   # Log Configuration
+  level: INFO                          # Default log level
+  write_to: "/var/log/ccm"             # Where to write logs to
+sse:                                   # Server Sent Events configuration
+  write_to: "/var/log/ccm/events"      # Where to write execution logs
+notifier:                              # Notifier configuration
+  enabled: True                        # Enable / Disable notifier
+  notify_on:                           # Enable / Disable notifications by type
+    success: false                     # Enable / Disable "success" notifications
+    error: true                        # Enable / Disable "error" notifications
+  notifiers:                           # Notifiers configuration
+    telegram:                          # Telegram notifier configuration
+      enabled: True                    # Enable / Disable Telegram notifier
+      token: "telegram-token"          # Telegram access token
+      recipients:                      # Telegram recipients (who will receive notifications)
+        - 123456789
+        - 987654321
+        - 123459876
+```

cmd/start.go

@@ -0,0 +1,124 @@
+package cmd
+
+import (
+	"fmt"
+	"time"
+
+	"github.com/leads-su/broker"
+	"github.com/leads-su/consul-config-manager/pkg/config"
+	"github.com/leads-su/consul-config-manager/pkg/http"
+	"github.com/leads-su/consul-config-manager/pkg/providers/consul"
+	"github.com/leads-su/consul-config-manager/pkg/providers/vault"
+	"github.com/leads-su/consul-config-manager/pkg/state"
+	"github.com/leads-su/logger"
+	"github.com/leads-su/updater"
+	"github.com/spf13/cobra"
+)
+
+var StartCommand = &cobra.Command{
+	Use:   "start",
+	Short: "Start CCM",
+	Long:  "Start Consul Config Manager",
+	Run: func(cmd *cobra.Command, args []string) {
+		logger.Info("cmd:start", "starting application")
+		brokerInstance, channel := initializeBroker()
+		applicationConfiguration := initializeApplicationConfiguration(brokerInstance)
+
+		logServer := http.NewLogServer(applicationConfiguration)
+		logServer.RegisterRoutes()
+
+		eventsServer := http.NewEventServer(applicationConfiguration)
+		eventsServer.RegisterRoutes()
+
+		if applicationConfiguration.Updater.Enabled {
+			updaterTicker, err := registerUpdateTicker(applicationConfiguration)
+			if err != nil {
+				logger.Warnf("cmd:start", "failed to register update checker - %s", err.Error())
+			}
+			defer updaterTicker.Stop()
+		}
+
+		if applicationConfiguration.Consul.Enabled {
+			go consul.NewConsul(applicationConfiguration)
+		}
+
+		if applicationConfiguration.Vault.Enabled {
+			go vault.NewVault(applicationConfiguration)
+		}
+
+		for {
+			switch <-channel {
+			case state.ApplicationShutdownRequested:
+				fmt.Println("Application shutdown requested")
+			case state.ApplicationRestartRequested:
+				fmt.Println("Application restart requested")
+			case state.ApplicationUpdateRequested:
+				fmt.Println("Application update requested")
+			case state.ApplicationConfigurationChanged:
+				fmt.Println("Application configuration has changed")
+			}
+		}
+	},
+}
+
+// initializeBroker initialize broker and return channel
+func initializeBroker() (*broker.Broker, chan interface{}) {
+	brokerInstance := broker.NewBroker()
+	go brokerInstance.Start()
+	channel := brokerInstance.Subscribe()
+	return brokerInstance, channel
+}
+
+// initializeApplicationConfiguration initializes application configuration (default or from file)
+func initializeApplicationConfiguration(brokerInstance *broker.Broker) *config.Config {
+	appConfig, err := config.Initialize(brokerInstance)
+	if err != nil {
+		logger.Fatal("cmd:start", "failed to initialize application configuration")
+	}
+	return appConfig
+}
+
+// registerUpdateTicker registers update ticker, so we can now
+// check if there is a new version while application is running
+func registerUpdateTicker(cfg *config.Config) (*time.Ticker, error) {
+	var service updater.UpdaterInterface
+	var err error
+	updaterTicker := time.NewTicker(60 * time.Minute)
+
+	switch cfg.Updater.Type {
+	case "gitlab":
+		service, err = updater.InitializeGitlab(updater.GitlabOptions{
+			Scheme:      cfg.Updater.Scheme,
+			Host:        cfg.Updater.Host,
+			Port:        cfg.Updater.Port,
+			ApiVersion:  4,
+			ProjectID:   cfg.Updater.ProjectID,
+			AccessToken: cfg.Updater.AccessToken,
+		})
+	case "gitea":
+		service, err = updater.InitializeGitea(updater.GiteaOptions{
+			Scheme:      cfg.Updater.Scheme,
+			Host:        cfg.Updater.Host,
+			Port:        cfg.Updater.Port,
+			Owner:       cfg.Updater.Owner,
+			Repository:  cfg.Updater.Repository,
+			AccessToken: cfg.Updater.AccessToken,
+		})
+	default:
+		return nil, fmt.Errorf("invalid updater specified - `%s`, only `gitlab` and `gitea` are supported", cfg.Updater.Type)
+	}
+
+	if err != nil {
+		return nil, err
+	}
+	service.CheckLatest()
+	go func() {
+		for {
+			select {
+			case <-updaterTicker.C:
+				service.CheckLatest()
+			}
+		}
+	}()
+	return updaterTicker, nil
+}

docs/example/config.yml

@@ -0,0 +1,45 @@
+agent:
+  network:
+    interface: ""
+    address: "127.0.0.1"
+    port: 32175
+  health_check:
+    ttl: true
+    http: true
+consul:
+  enabled: true
+  datacenter: "dc0"
+  addresses:
+    - scheme: "http"
+      host: "consul1.local"
+      port: 8500
+    - scheme: "http"
+      host: "consul2.local"
+      port: 8500
+    - scheme: "http"
+      host: "consul3.local"
+      port: 8500
+    - scheme: "http"
+      host: "consul4.local"
+      port: 8500
+    - scheme: "http"
+      host: "consul5.local"
+      port: 8500
+  token: "consul-acl-access-token"
+  write_to: "/etc/ccm.d"
+environment: "production"
+log:
+  level: DEBUG
+  write_to: "/var/log/ccm"
+sse:
+  write_to: "/var/log/ccm/events"
+notifier:
+  enabled: True
+  notify_on:
+    success: false
+    error: true
+  notifiers:
+    telegram:
+      enabled: True
+      token: "telegram-token"
+      recipients: []

go.mod

@@ -0,0 +1,59 @@
+module github.com/leads-su/consul-config-manager
+
+go 1.17
+
+require (
+	github.com/davecgh/go-spew v1.1.1
+	github.com/fsnotify/fsnotify v1.5.4
+	github.com/hashicorp/consul/api v1.12.0
+	github.com/leads-su/application v1.0.0
+	github.com/leads-su/broker v1.0.0
+	github.com/leads-su/consul v1.0.0
+	github.com/leads-su/logger v1.0.0
+	github.com/leads-su/network v1.0.0
+	github.com/leads-su/notifier v1.0.0
+	github.com/leads-su/runner v1.0.0
+	github.com/leads-su/storage v1.0.0
+	github.com/leads-su/updater v1.0.0
+	github.com/leads-su/version v1.0.0
+	github.com/r3labs/sse/v2 v2.7.7
+	github.com/sirupsen/logrus v1.8.1
+	github.com/spf13/cobra v1.4.0
+	github.com/spf13/viper v1.11.0
+)
+
+require (
+	github.com/armon/go-metrics v0.3.11 // indirect
+	github.com/cenkalti/backoff v2.2.1+incompatible // indirect
+	github.com/fatih/color v1.13.0 // indirect
+	github.com/hashicorp/go-cleanhttp v0.5.2 // indirect
+	github.com/hashicorp/go-hclog v1.2.0 // indirect
+	github.com/hashicorp/go-immutable-radix v1.3.1 // indirect
+	github.com/hashicorp/go-rootcerts v1.0.2 // indirect
+	github.com/hashicorp/go-version v1.4.0 // indirect
+	github.com/hashicorp/golang-lru v0.5.4 // indirect
+	github.com/hashicorp/hcl v1.0.0 // indirect
+	github.com/hashicorp/serf v0.9.7 // indirect
+	github.com/inconshreveable/mousetrap v1.0.0 // indirect
+	github.com/magiconair/properties v1.8.6 // indirect
+	github.com/mattn/go-colorable v0.1.12 // indirect
+	github.com/mattn/go-isatty v0.0.14 // indirect
+	github.com/mitchellh/go-homedir v1.1.0 // indirect
+	github.com/mitchellh/mapstructure v1.5.0 // indirect
+	github.com/pelletier/go-toml v1.9.5 // indirect
+	github.com/pelletier/go-toml/v2 v2.0.0 // indirect
+	github.com/pkg/errors v0.9.1 // indirect
+	github.com/spf13/afero v1.8.2 // indirect
+	github.com/spf13/cast v1.4.1 // indirect
+	github.com/spf13/jwalterweatherman v1.1.0 // indirect
+	github.com/spf13/pflag v1.0.5 // indirect
+	github.com/subosito/gotenv v1.2.0 // indirect
+	golang.org/x/net v0.0.0-20220425223048-2871e0cb64e4 // indirect
+	golang.org/x/sys v0.0.0-20220503163025-988cb79eb6c6 // indirect
+	golang.org/x/text v0.3.7 // indirect
+	gopkg.in/cenkalti/backoff.v1 v1.1.0 // indirect
+	gopkg.in/ini.v1 v1.66.4 // indirect
+	gopkg.in/natefinch/lumberjack.v2 v2.0.0 // indirect
+	gopkg.in/yaml.v2 v2.4.0 // indirect
+	gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b // indirect
+)