Skip to content

kubemq-io/kubemq-targets

master
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
api
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
pkg
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
go
 
 
 
 
 
 
 
 
 
 

KubeMQ Targets

KubeMQ Targets connects KubeMQ Message Broker with external systems and cloud services.

KubeMQ Targets allows us to build a message-based microservices architecture on Kubernetes with minimal efforts and without developing connectivity interfaces between KubeMQ Message Broker and external systems such as databases, cache, messaging, and REST-base APIs.

Key Features:

  • Runs anywhere - Kubernetes, Cloud, on-prem, anywhere
  • Stand-alone - small docker container / binary
  • Single Interface - One interface all the services
  • Any Service - Support all major services types (databases, cache, messaging, serverless, HTTP, etc.)
  • Plug-in Architecture Easy to extend, easy to connect
  • Middleware Supports - Logs, Metrics, Retries, and Rate Limiters
  • Easy Configuration - simple yaml file builds your topology

Concepts

KubeMQ Targets building blocks are:

  • Binding
  • Source
  • Target
  • Request/Response

Binding

Binding is a 1:1 connection between Source and Target. Every Binding runs independently.

binding

Target

Target is an external service that exposes an API allowing to interact and serve his functionalists with other services.

Targets can be Cache systems such as Redis and Memcached, SQL Databases such as Postgres and MySql, and even an HTTP generic Rest interface.

KubeMQ Targets integrate each one of the supported targets and service requests based on the request data.

A list of supported targets is below.

Standalone Services

Category Target Kind Configuration Example
Cache
Redis cache.redis Usage Example
Memcached cache.memcached Usage Example
Hazelcast cache.hazelcast Usage Example
Stores/db
Postgres stores.postgres Usage Example
Mysql stores.mysql Usage Example
MSSql stores.mssql Usage Example
MongoDB stores.mongodb Usage Example
Elastic Search stores.elastic-search Usage Example
Cassandra stores.cassandra Usage Example
Couchbase stores.couchbase Usage Example
Percona stores.percona Usage Example
Cockroachdb stores.cockroachdb Usage Example
Aerospike stores.aerospike Usage Example
RethinkDB stores.rethinkdb Usage Example
SingleStore stores.singlestore Usage Example
Crate.io stores.crate Usage Similar to postgres
Messaging
Kafka messaging.kafka Usage Example
Nats messaging.nats Usage Example
RabbitMQ messaging.rabbitmq Usage Example
MQTT messaging.mqtt Usage Example
ActiveMQ messaging.activemq Usage Example
IBM-MQ messaging.ibmmq Usage Example
Storage
Minio/S3 storage.minio Usage Example
hadoop/hdfs storage.hdfs Usage Example
Filesystem storage.filesystem Usage
Serverless
OpenFaas serverless.openfaas Usage Example
Http
Http http Usage Example
Testing
Echo echo Usage

Google Cloud Platform (GCP)

Category Target Kind Configuration Example
Cache
Redis gcp.cache.redis Usage Example
Memcached gcp.cache.memcached Usage Example
Stores/db
Postgres gcp.stores.postgres Usage Example
Mysql gcp.stores.mysql Usage Example
BigQuery gcp.bigquery Usage Example
BigTable gcp.bigtable Usage Example
Firestore gcp.firestore Usage Example
Spanner gcp.spanner Usage Example
Firebase gcp.firebase Usage Example
Messaging
Pub/Sub gcp.pubsub Usage Example
Storage
Storage gcp.storage Usage Example
Serverless
Functions gcp.cloudfunctions Usage Example

Amazon Web Service (AWS)

Category Target Kind Configuration Example
Stores/db
Athena aws.athena Usage Example
DynamoDB aws.dynamodb Usage Example
Elasticsearch aws.elasticsearch Usage Example
KeySpaces aws.keyspaces Usage Example
MariaDB aws.rds.mariadb Usage Example
MSSql aws.rds.mssql Usage Example
MySQL aws.rds.mysql Usage Example
Postgres aws.rds.postgres Usage Example
RedShift aws.rds.redshift Usage Example
RedShiftSVC aws.rds.redshift.service Usage Example
Messaging
AmazonMQ aws.amazonmq Usage Example
msk aws.msk Usage Example
Kinesis aws.kinesis Usage Example
SQS aws.sqs Usage Example
SNS aws.sns Usage Example
Storage
s3 aws.s3 Usage Example
Serverless
lambda aws.lambda Usage Example
Other
Cloud Watch Logs aws.cloudwatch.logs Usage Example
Cloud Watch Events aws.cloudwatch.events Usage Example
Cloud Watch Metrics aws.cloudwatch.metrics Usage Example

Microsoft Azure

Category Target Kind Configuration Example
Stores/db
Azuresql azure.stores.azuresql Usage Example
Mysql azure.stores.mysql Usage Example
Postgres azure.stores.postgres Usage Example
Storage
Blob azure.storage.blob Usage Example
Files azure.storage.files Usage Example
Queue azure.storage.queue Usage Example
EventHubs
EventHubs azure.eventhubs Usage Example
ServiceBus
ServiceBus azure.servicebus Usage Example

Source

The source is a KubeMQ connection (in subscription mode), which listens to requests from services and route them to the appropriate target for action, and return back a response if needed.

KubeMQ Targets supports all of KubeMQ's messaging patterns: Queue, Events, Events-Store, Command, and Query.

Type Kind Configuration
Queue kubemq.queue Usage
Queue-Stream kubemq.queue-stream Usage
Events kubemq.events Usage
Events Store kubemq.events-store Usage
Command kubemq.command Usage
Query kubemq.query Usage

Request / Response

concept

Request

A request is an object that sends to a designated target with metadata and data fields, which contains the needed information to perform the requested data.

Request Object Structure
Field Type Description
metadata string, string object contains metadata information for action
data bytes array contains raw data for action
Example

Request to get a data from Redis cache for the key "log"

{
  "metadata": {
    "method": "get",
    "key": "log"
  },
  "data": null
}

Response

The response is an object that sends back as a result of executing an action in the target

Response Object Structure
Field Type Description
metadata string, string object contains metadata information result for action
data bytes array contains raw data result
is_error bool indicate if the action ended with an error
error string contains error information if any
Example

Response received on request to get the data stored in Redis for key "log"

{
  "metadata": {
    "result": "ok",
    "key": "log"
  },
  "data": "SU5TRVJUIElOVE8gcG9zdChJRCxUSVRMRSxDT05URU5UKSBWQUxVRVMKCSAgICAgICAgICAgICAgICAgICAgICA"
}

Installation

Kubernetes

  1. Install KubeMQ Cluster
kubectl apply -f https://get.kubemq.io/deploy
  1. Run Redis Cluster deployment yaml
kubectl apply -f https://raw.githubusercontent.com/kubemq-hub/kubemq-targets/master/redis-example.yaml
  1. Run KubeMQ Targets deployment yaml
kubectl apply -f https://raw.githubusercontent.com/kubemq-hub/kubemq-targets/master/deploy-example.yaml

Binary (Cross-platform)

Download the appropriate version for your platform from KubeMQ Targets Releases. Once downloaded, the binary can be run from anywhere.

Ideally, you should install it somewhere in your PATH for easy use. /usr/local/bin is the most probable location.

Running KubeMQ Targets

./kubemq-targets --config config.yaml

Windows Service

  1. Download the Windows version from KubeMQ Targets Releases. Once downloaded, the binary can be installed from anywhere.
  2. Create config.yaml configuration file and save it to the same location of the Windows binary.

Service Installation

Run:

kubemq-targets.exe --service install

Service Installation With Username and Password

Run:

kubemq-targets.exe --service install --username {your-username} --password {your-password}

Service UnInstall

Run:

kubemq-sources.exe --service uninstall

Service Start

Run:

kubemq-targets.exe --service start

Service Stop

Run:

kubemq-targets.exe --service stop

Service Restart

Run:

kubemq-targets.exe --service restart

NOTE: When running under Windows service, all logs will be emitted to Windows Events Logs.

Configuration

Build Wizard

KubeMQ Targets configuration can be build with Build and Deploy tool https://build.kubemq.io/#/sources

Structure

Config file structure:

apiPort: 8080 # kubemq targets api and health end-point port
bindings:
  - name: clusters-sources # unique binding name
    properties: # Bindings properties such middleware configurations
      log_level: error
      retry_attempts: 3
      retry_delay_milliseconds: 1000
      retry_max_jitter_milliseconds: 100
      retry_delay_type: "back-off"
      rate_per_second: 100
    source:
      kind: kubemq.query # source kind
      name: name-of-sources # source name 
      properties: # a set of key/value settings per each source kind
        .....
    target:
      kind:cache.redis # target kind
      name: name-of-target # targets name
      properties: # a set of key/value settings per each target kind
        - .....

Build Wizard

KubeMQ Targets configuration can be build with --build flag

./kubemq-targets --build

Properties

In bindings configuration, KubeMQ targets support properties setting for each pair of source and target bindings.

These properties contain middleware information settings as follows:

Logs Middleware

KubeMQ targets support level based logging to console according to as follows:

Property Description Possible Values
log_level log level setting "debug","info","error"
"" - indicate no logging on this bindings

An example for only error level log to console:

bindings:
  - name: sample-binding 
    properties: 
      log_level: error
    source:
    ......  

Retry Middleware

KubeMQ targets support Retries' target execution before reporting of error back to the source on failed execution.

Retry middleware settings values:

Property Description Possible Values
retry_attempts how many retries before giving up on target execution default - 1, or any int number
retry_delay_milliseconds how long to wait between retries in milliseconds default - 100ms or any int number
retry_max_jitter_milliseconds max delay jitter between retries default - 100ms or any int number
retry_delay_type type of retry delay "back-off" - delay increase on each attempt
"fixed" - fixed time delay
"random" - random time delay

An example for 3 retries with back-off strategy:

bindings:
  - name: sample-binding 
    properties: 
      retry_attempts: 3
      retry_delay_milliseconds: 1000
      retry_max_jitter_milliseconds: 100
      retry_delay_type: "back-off"
    source:
    ......  

Rate Limiter Middleware

KubeMQ targets support a Rate Limiting of target executions.

Rate Limiter middleware settings values:

Property Description Possible Values
rate_per_second how many executions per second will be allowed 0 - no limitation
1 - n integer times per second

An example for 100 executions per second:

bindings:
  - name: sample-binding 
    properties: 
      rate_per_second: 100
    source:
    ......  

Source

Source section contains source configuration for Binding as follows:

Property Description Possible Values
name sources name (will show up in logs) string without white spaces
kind source kind type kubemq.queue
kubemq.query
kubemq.query-stream
kubemq.command
kubemq.events
kubemq.events-store
properties an array of key/value setting for source connection see above

Target

Target section contains the target configuration for Binding as follows:

Property Description Possible Values
name targets name (will show up in logs) string without white spaces
kind source kind type type-of-target
properties an array of key/value set for target connection see above