Golang Development Utilities

Utilities for golang developments on Conflux blockchain, especially for backend service.

Module	Description
Alert	Send notification messages to DingTalk, Telegram, SMTP email or PagerDuty.
API	REST API utilities based on gin.
Config	Initialize all modules.
DLock	Utilities for distributed lock.
Health	Utilities for health management.
HTTP	Provides common used middlewares.
Log	Based on logrus and integrated with Alert.
Metrics	To monitor system runtime.
Rate Limit	Utilities to limit request rate.
Store	Provides utilities to initialize database.
Viper	To fix some issues of original viper.

Alert

Before sending any message to notification channels, the client should create a channel robot. To construct a channel robot programmatically:

// Construct a notification channel imperatively
var notifyCh alert.Channel

// DingTalk Channel
notifyCh = alert.NewDingTalkChannel(...)
// or PagerDuty Channel
notifyCh = alert.NewPagerDutyChannel(...)
// or Smtp email Channel
notifyCh = alert.NewSmtpChannel(...)
// or Telegram Channel
notifyCh = alert.NewTelegramChannel(...)

Alternatively, you can initialize the alert channels from configuration file or environment variables, which is recommended.

// Initialize the alert channels from configurations loaded by viper
alert.MustInitFromViper()
// After initialization, you can retrieve the notification channel using a unique channel ID
notifyCh := alert.DefaultManager().Channel(chID)

Once the channel is initialized, you can send a notification message through the channel:

notifyCh.Send(context.Background(), &alert.Notification{
    Title:    "Alert testing",
    Severity: alert.SeverityLow,
    Content: `This is a test notification message`,
})

Moreover, alert can be integrated with log module, so as to send alerting message when warning or error logs occurred.

API

This module provides common HTTP responses along with standard errors in JSON format.

Uniform JSON message in HTTP response body:

type BusinessError struct {
	Code    int         `json:"code"`
	Message string      `json:"message"`
	Data    interface{} `json:"data"`
}

Code 0 indicates success, and Data is an object indicates the return value of REST API. Code with non-zero value indicates any error occurred, please refer to the Message and Data fields for more details. There are some pre-defined errors as below:

• 1: Invalid parameter, see Data for detailed error.
• 2: Internal server error, see Data for detailed error.
• 3: Too many requests, see Data for detailed error.

To distinguish backend service error and gateway error, we only use 200 and 600 as HTTP response status code:

• 200: success, or known business error, e.g. entity not found.
• 600: unexpected system error, e.g. database error, io error.

We recommend to initialize REST API from configuration file. Client only requires to provide a factory to setup controllers.

// Setup controllers.
factory := func(router *gin.Engine) {
    router.GET("url", api.Wrap(controller))
}

// Start REST API server in a separate goroutine.
go api.MustServeFromViper(factory)

Config

Initialize all modules at the entry point of program, including viper, log, metrics and alert.

config.MustInit(viperEnvPrefix string)

The viperEnvPrefix is used to overwrite configurations from environment. E.g. if the viperEnvPrefix is FOO, then client could set environment as below to overwrite config alert.dingTalk.secret:

FOO_ALERT_DINGTALK_SECRET='dsafsadf'

You could follow the example config.yaml under config folder to setup your own configuration file. Generally, you could only overwrite configurations if the default value not suitable.

Distributed Lock

The distributed lock ensures atomicity in a distributed environment, such as leader election for achieving high availability.

To create a distributed lock, you need to specify a storage backend. We provide the MySQLBackend which handles reading and writing lock information in a MySQL table. Alternatively, you can implement your own storage backend using Redis, etcd, ZooKeeper, or other options.

// Construct a lock manager with customized storage backend.
lockMan := dlock.NewLockManager(backend)

Alternatively, you can construct a lock manager with a convenient MySQL backend by using configuration files or environment variables.

// Construct a lock manager with a MySQL backend from configurations loaded by viper
lockMan = dlock.NewLockManagerFromViper()

To acquire and release a lock, you can use:

intent := NewLockIntent("dlock_key", 15 * time.Second)
// Acquire a lock with key name "dlock_key" for 15 seconds
lockMan.Acquire(context.Background(), intent)
// Release the lock immediately
lockMan.Release(context.Background(), intent)

Health

Provides utilities for error tolerant health monitoring to avoid massive duplicated alerts.

Generally, system shall not report failure if auto resolved in a short time. However, system should report failure if not solved in a short time, and periodically remind failure if unrecovered for a long time.

• Counter: manage health status based on the number of continous failures.
• TimedCounter: manage health status based on duration since the first failure.

HTTP

Provides utilities to hook middlewares to HTTP handler, e.g. remote address, API key and rate limit.

Log

We recommend initializing the log module from a configuration file or environment variables. Additionally, you can configure the alert hook to set up default notification channels for sending alert messages when warning or error logs occur.

// Initialize logging by specifying configurations
log.MustInit(conf)
// or Initialize logging from configurations loaded by viper
log.MustInitFromViper()

ErrorTolerantLogger

ErrorTolerantLogger is a thread-safe logger that incorporates error tolerance behavior based on the continuous error count.

// Construct an error tolerant logger imperatively
etlogger := log.NewErrorTolerantLogger(conf)

Alternatively, you can construct an error tolerant logger from configuration files or environment variables.

// Construct an error tolerant logger from configurations loaded by viper
etlogger := log.MustNewErrorTolerantLoggerFromViper()

Then, you can log the error message using the logger, which will mute the error message unless continuous errors happen.

// Logging without context fields
etLogger.Log(logrus.StandardLogger(), err, "Some error")
// Logging with context fields
etLogger.Log(logrus.WithField("field", val), err, "Some error")

Please note that it is important to always call the logging function, even if the error is nil, in order to reset the continuous timer.

Metrics

We recommend to initialize metrics module from configuration file. Client could also configure influxdb to report metrics periodically. See MetricsConfig for more details.

Rate Limit

Provides basic rate limit algorithms, including fixed window, token bucket, along with utilities for HTTP middleware.

Note, rate limit middleware depends on the HTTP middleware RealIP.

Store

We recommend to initialize store module from configuration file.

config := mysql.MustNewConfigFromViper()
db := config.MustOpenOrCreate(tables ...interface{})

Viper

Fixes issues when unmarshal configurations from environment. A simple way to load configuration from file is as below:

viper.MustUnmarshalKey(key string, valPtr interface{}, resolver ...ValueResolver)

// E.g. load `foo` config from file.
var config FooConfig
viper.MustUnmarshalKey("foo", &config)