Axibase Collector is a schedule-based ETL tool and and a highly customizable agent for collecting statistics into Axibase Time Series Database
Vue Python JavaScript CSS Shell
Clone or download
Latest commit 09aadfa Jul 12, 2018
Permalink
Failed to load latest commit information.
.vuepress Add ext Jun 18, 2018
files move docs back Jun 4, 2018
images Add details to http pool description Jul 2, 2018
job-templates Fix ip whitelist Jun 30, 2018
jobs selector-replace Jul 12, 2018
.dictionary Squashed commit of the following: Jul 3, 2018
.gitignore docs site files May 31, 2018
.travis.yml validate_anchors May 31, 2018
README.md Link to historize tutorial Jul 11, 2018
atsd-server-connection.md Use specify Jul 2, 2018
collections.md Replace custom ip addresses Jun 28, 2018
configure-administrator-account.md conflict-resolution Jun 5, 2018
docker-bundle.md spelling-refactor Jun 14, 2018
docker_metrics.md lint Jun 25, 2018
functions-freemarker.md fixes for no-space-in-fenced-code.js Jul 11, 2018
installation-firefox-webdriver.md fix evident lint errors Jun 26, 2018
installation-on-docker.md fixes for case-sensitive.js Jul 10, 2018
installation-on-kubernetes.md lint Jul 12, 2018
installation.md replaced insecure links with secure ones Jul 3, 2018
job-autostart.md lint Jun 20, 2018
job-generic.md Elements rename Jul 9, 2018
logging.md spelling-refactor Jun 14, 2018
monitoring.md lint Jun 20, 2018
package.json build from repo's root Jun 4, 2018
pre-configured-jobs.md spelling-refactor Jun 14, 2018
requirements.md JDK version Jun 28, 2018
scheduling.md lint Jun 20, 2018
test.json move docs back Jun 4, 2018
to_check.txt replaced insecure links with secure ones Jul 3, 2018
updating-collector-on-docker.md replaced insecure links with secure ones Jul 3, 2018
updating-collector.md replaced insecure links with secure ones Jul 3, 2018
yarn.lock fix unrecognized extensions and broken links Jun 6, 2018

README.md

Axibase Collector

Overview

Axibase Collector is a Java-based ETL application that queries external data sources on a defined schedule and uploads the data as series, properties, messages, and files into the Axibase Time Series Database (ATSD).

The Collector supports data markers to minimize the load on the source systems as well as list-based automation to efficiently copy data from multiple sources with re-usable configurations.

Use Cases

  • Download a CSV/TSV file from a URL or an FTP server and upload the file into ATSD.
  • Offload data from a relational database for long-term storage in ATSD.
  • Historize metrics by querying current values and retaining their history in ATSD.
  • Query a web service, convert JSON document into CSV format, and upload the file into ATSD.
  • Collect performance metrics from Java applications (JMX), network devices (SNMP), etc.
  • Copy data from operational support systems and plant historians for advanced analytics and visualization.

Supported Databases

  • Oracle
  • IBM DB2
  • Microsoft SQL Server
  • MySQL
  • PostgreSQL
  • Sybase
  • Apache Derby
  • Vertica
  • ATSD
  • SAP HANA
  • OSISoft PI Data Archive
  • OSISoft PIOleDBEnt
  • Custom

Supported Data and Network Protocols

  • JDBC
  • SNMP
  • JMX
  • ICMP
  • TCP
  • HTTP

Supported File Formats

  • CSV
  • TSV
  • Text
  • JSON

Supported Schemas

Supported APIs

  • Kafka
  • Amazon Web Services CloudWatch
  • Docker Engine
  • HP OVPM (Performance Manager)

Installation

Jobs

Automation

Job Types

Type Description
AWS Collects AWS statistics using CloudWatch API.
Docker Collects container, image, and volume statistics using Docker Engine API.
File Downloads CSV/TSV files from remote servers or local file system and uploads them into ATSD for parsing.
Supports HTTP/s, FTP, SFTP, SCP, and FILE protocols.
HTTP Executes HTTP requests or Web Driver scripts and stores response status, response times, and SSL certificate status as metrics.
ICMP Pings hostnames/IP addresses and stores response status.
JDBC Executes SQL queries against relational databases.
Converts rows into series, property, or message commands.
JMX Collects MBean attribute values from Java applications.
JSON Downloads JSON files, parses the documents and converts JSON fields into series, property, and message commands using JSONPath.
Kafka Reads JSON messages from Kafka brokers, parses the messages and converts them into series, property, and message commands.
OVPM Offloads statistics from HP OpenView Performance Manager.
PI Extracts PI points archive data via JDBC driver.
SNMP Queries SNMP devices using built-in and custom MIB files.
Socrata Downloads JSON documents published in Socrata schema and converts them into series, property, and message commands.
TCP Connects to hostnames/IP addresses and stores connection status.

Administration

Examples

Name Job Type Description
ActiveMQ JMX Collect metrics about brokers, queues, pub/sub topics
Derby Database JMX Collect uptime metrics for the database
File File Download CSV files
HP OpenView OVPM Offload CODA metrics from OVPM
Jetty JMX Collect sessions, requests, status from the Jetty server
JSON JSON Download JSON file and convert the file to CSV format
JVM JMX Collect key JVM performance metrics
MySQL JDBC Collect database performance metrics
nginx File Collect key web server metrics for nginx
nginx-plus File Collect extended web server metrics for nginx-plus
Oracle EM JDBC Offload incremental database and application metrics collected by Oracle EM
PI PI Copy incremental tag values from PI Data Archive
PostgreSQL JDBC Collect database performance metrics
SCOM JDBC Offload incremental server metrics collected by SCOM
Socrata Socrata Download and parse datasets published in Socrata format
SolarWinds JDBC Offload incremental network and server metrics collected by SolarWinds
Tomcat JMX Collect key container metrics exposed by Tomcat
VMware JDBC Offload incremental cluster, host, VM metrics collected by VMware vCenter