Shipper

Http service which provides Clickhouse data via GraphQL interface.

It supports conditions, limits and offsets. All requests can be cached and authentificated based on OpenID Connect.

Features

• Support all selectable object: tables, views, materialized views, etc.
• Provide sorting (asc, desc), record limits and offsets
• Support conditions like: Match, EQ, NE, LT, LE, GT, GE
• Support groups and sums
• Include GraphQL Playground, OIDC, Prometheus metrics
• Refresh schemas automatically
• Provide response cache using size or time

Build

git clone https://github.com/click-stream/shipper.git
cd shipper/
go build

Example

./shipper --http-listen ":8081" --graphql-mode Playground \
          --clickhouse-host=HOST --clickhouse-password=PASSWORD --clickhouse-debug \
          --clickhouse-database-pattern "system" --clickhouse-table-pattern "(settings|tables)" \
          --log-template "{{.msg}}" --log-format stdout

GraphQL to Clickhouse

GraphQL request:

{
  settings(limit:1,
           valueNE: "0",
           order:[nameASC]) {
    name
    value
  }
  
  tables(limit: 2,
         nameMatch: ".*settings.*",
         order:[nameDESC]) {
    database
    engine
    name
  }
}

GraphQL response:

{
  "data": {
    "settings": [
      {
        "name": "allow_ddl",
        "value": "1"
      }
    ],
    "tables": [
      {
        "database": "system",
        "engine": "SystemSettings",
        "name": "settings"
      },
      {
        "database": "system",
        "engine": "SystemMergeTreeSettings",
        "name": "merge_tree_settings"
      }
    ]
  }
}

Clickhouse queries based on GraphQL request:

SELECT `name`, `value` FROM `system`.`settings` WHERE `value` != '0' ORDER BY `name` ASC LIMIT 1 OFFSET 0;
SELECT `database`, `engine`, `name` FROM `system`.`tables` WHERE match(`name`,'.*settings.*') = 1 ORDER BY `name` DESC LIMIT 2 OFFSET 0;

Usage

Shipper command

Usage:
  shipper [flags]
  shipper [command]

Available Commands:
  help        Help about any command
  version     Print the version number

Flags:
      --clickhouse-action-refresh string     Clickhouse action refresh (default "refresh")
      --clickhouse-cache-clean-seconds int   Clickhouse cache clean seconds
      --clickhouse-cache-life-seconds int    Clickhouse cache life seconds
      --clickhouse-cache-max-size int        Clickhouse cache max size in MB
      --clickhouse-database-pattern string   Clickhouse database pattern (default ".*")
      --clickhouse-debug                     Clickhouse debug
      --clickhouse-host string               Clickhouse host
      --clickhouse-ident-format string       Clickhouse ident format (default "`%s`")
      --clickhouse-password string           Clickhouse password
      --clickhouse-port int                  Clickhouse port (default 9000)
      --clickhouse-query-limit int           Clickhouse query limit (default 1000)
      --clickhouse-read-timeout int          Clickhouse read timeout (default 10)
      --clickhouse-refresh-interval int      Clickhouse refresh interval in seconds (default 60)
      --clickhouse-table-pattern string      Clickhouse table pattern (default ".*")
      --clickhouse-url-pattern string        Clickhouse url pattern (default "/{database}")
      --clickhouse-user string               Clickhouse user (default "default")
      --graphql-mode string                  Graphql mode: GraphiQL, Playground
      --graphql-pretty                       Graphql pretty response format
  -h, --help                                 help for shipper
      --http-cert string                     Http cert file or content
      --http-chain string                    Http CA chain file or content
      --http-external-host string            Http external host
      --http-key string                      Http key file or content
      --http-listen string                   Http listen (default ":80")
      --http-cors                            Http CORS enabled true/false (default false)
      --http-oidc-callback-url string        Http oidc callback url (default "/callback")
      --http-oidc-client-id string           Http oidc client id
      --http-oidc-client-secret string       Http oidc client secret
      --http-oidc-config-url string          Http oidc config url
      --http-oidc-default-url string         Http oidc default url
      --http-oidc-enabled                    Http oidc enabled
      --http-oidc-login-url string           Http oidc login url (default "/login")
      --http-oidc-logout-url string          Http oidc logout url (default "/logout")
      --http-oidc-scopes-url string          Http oidc scopes (default "profile, email, roles, groups")
      --http-tls                             Http TLS
      --http-url string                      Http url (default "/shipper")
      --log-format string                    Log format: json, text, stdout (default "text")
      --log-level string                     Log level: info, warn, error, debug, panic (default "info")
      --log-template string                  Log template (default "{{.func}} [{{.line}}]: {{.msg}}")
      --prometheus-listen string             Prometheus listen (default "127.0.0.1:8080")
      --prometheus-url string                Prometheus endpoint url (default "/metrics")

Environment variables

For containerization purpose all command switches have environment variables analogs.

• SHIPPER_LOG_FORMAT

• SHIPPER_LOG_LEVEL

• SHIPPER_LOG_TEMPLATE

• SHIPPER_PROMETHEUS_URL

• SHIPPER_PROMETHEUS_LISTEN

• SHIPPER_HTTP_URL

• SHIPPER_HTTP_LISTEN

• SHIPPER_HTTP_CORS

• SHIPPER_HTTP_TLS

• SHIPPER_HTTP_CERT

• SHIPPER_HTTP_KEY

• SHIPPER_HTTP_CHAIN

• SHIPPER_HTTP_EXTERNAL_HOST

• SHIPPER_HTTP_OIDC_ENABLED

• SHIPPER_HTTP_OIDC_CLIENT_ID

• SHIPPER_HTTP_OIDC_CLIENT_SECRET

• SHIPPER_HTTP_OIDC_CONFIG_URL

• SHIPPER_HTTP_OIDC_LOGIN_URL

• SHIPPER_HTTP_OIDC_LOGOUT_URL

• SHIPPER_HTTP_OIDC_CALLBACK_URL

• SHIPPER_HTTP_OIDC_DEFAULT_URL

• SHIPPER_HTTP_OIDC_SCOPES

• SHIPPER_GRAPHQL_PRETTY

• SHIPPER_GRAPHQL_MODE

• SHIPPER_CLICKHOUSE_HOST

• SHIPPER_CLICKHOUSE_PORT

• SHIPPER_CLICKHOUSE_USER

• SHIPPER_CLICKHOUSE_PASSWORD

• SHIPPER_CLICKHOUSE_DEBUG

• SHIPPER_CLICKHOUSE_URL_PATTERN

• SHIPPER_CLICKHOUSE_READ_TIMEOUT

• SHIPPER_CLICKHOUSE_DATABASE_PATTERN

• SHIPPER_CLICKHOUSE_TABLE_PATTERN

• SHIPPER_CLICKHOUSE_QUERY_LIMIT

• SHIPPER_CLICKHOUSE_IDENT_FORMAT

• SHIPPER_CLICKHOUSE_CACHE_LIFE_SECONDS

• SHIPPER_CLICKHOUSE_CACHE_CLEAN_SECONDS

• SHIPPER_CLICKHOUSE_CACHE_MAX_SIZE

• SHIPPER_CLICKHOUSE_REFRESH_INTERVAL

• SHIPPER_CLICKHOUSE_ACTION_REFRESH