Axibase Time Series Database API Client for Python
Python
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.

README.md

ATSD Python Client

Table of Contents

Overview

The ATSD API Client for Python simplifies the process of interacting with Axibase Time Series Database through REST API and SQL endpoints.

References

Requirements

Check Python version.

python -V

The client supports the following versions of Python:

  • Python 2: v2.7.9 and later
  • Python 3: all versions

Installation

Installing Module with pip

Install atsd_client module with pip.

If necessary, install pip with apt-get install python-pip on Ubuntu.

pip install atsd_client

Upgrade setup tools with pip install --upgrade setuptools.

To install a specific version, specify that version in the pip install command.

pip install atsd_client==2.3.0

Use this command to downgrade the module as well.

Check the currently installed pip module.

pip show atsd-client
Name: atsd-client
Version: 2.3.0
Summary: Axibase Time Series Database API Client for Python
Home-page: https://github.com/axibase/atsd-api-python
Author: Axibase Corporation
Author-email: support-atsd@axibase.com
License: Apache 2.0
Location: /usr/local/lib/python2.7/dist-packages
Requires: tzlocal, requests, python-dateutil
Required-by:

To install the client on a system without Internet access, follow the Offline Installation Instructions.

Installing from Source

Clone the repository and run the installation manually.

git clone https://github.com/axibase/atsd-api-python.git
cd atsd-api-python
python setup.py install

Verify Installation

Check that modules are installed.

python -c "import tzlocal, requests, dateutil, atsd_client"

An empty output corresponds to a successful installation. Otherwise, the output displays an error which enumerates missing modules.

Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named atsd_client

Upgrade

Execute pip install command to upgrade the client to the latest version.

pip install atsd_client --upgrade --upgrade-strategy only-if-needed

Run pip list to view the currently installed modules.

pip list
Package             Version
------------------- ------------------
asn1crypto          0.24.0
atsd-client         2.2.2
certifi             2018.4.16
cffi                1.11.5
...

Hello World

Create a connect_url_check.py file with a basic connection test.

from atsd_client import connect_url

# Update connection properties and user credentials
connection = connect_url('https://atsd_hostname:8443', 'john.doe', 'password')

# Retrieve JSON from '/api/v1/version' endpoint
# https://axibase.com/docs/atsd/api/meta/misc/version.html
response = connection.get('v1/version')
build_info = response['buildInfo']
print('Revision: %s ' % build_info['revisionNumber'])
python connect_url_check.py
INFO:root:Connecting to ATSD at https://atsd_hostname:8443 as john.doe user.
Revision: 19020

Connecting to ATSD

To connect to an ATSD instance, gather hostname and port details. By default ATSD listens for connection requests on port 8443. Create a user account on the Settings > Users page if needed.

Establish a connection with the connect_url method.

from atsd_client import connect_url
connection = connect_url('https://atsd_hostname:8443', 'username', 'password')

Alternatively, create a connection.properties file.

base_url=https://atsd_hostname:8443
username=username
password=password
ssl_verify=False

Specify path to connection.properties file in the connect method.

from atsd_client import connect
connection = connect('/path/to/connection.properties')

Logging

Logging to stdout is enabled by default. To disable logging, append the logger.disabled parameter to the script.

import logging
logger = logging.getLogger()
logger.disabled = True

Services

The client supports services to insert and query particular types of records in the database. These include Series, Property, and Message records as well as metadata records such as Entity, Metric, and EntityGroup.

from atsd_client.services import *
svc = SeriesService(conn)

Available services:

Models

Use this service to insert and query particular types of records in the database, which are implemented as Python classes for convenience.

Inserting Data

Inserting Series

Initialize a Series object and populate object with timestamped values.

from atsd_client.models import Series

series = Series(entity='sensor123', metric='temperature')
series.add_samples(
    Sample(value=1, time="2018-05-18T17:14:30Z"),
    Sample(value=2, time="2018-05-18T17:16:30Z")
)
svc.insert(series)

Inserting Properties

Initialize a Property object.

from atsd_client.models import Property

property = Property(type='disk', entity='nurswgvml007',
                    key={"mount_point": "sda1"},
                    tags={"fs_type": "ext4"})

svc = PropertiesService(conn)
svc.insert(property)

Inserting Messages

Initialize a Message object.

from atsd_client.models import Message

message = Message(entity='nurswgvml007', type="application", source="atsd", severity="MAJOR",
                  tags={"path": "/", "name": "sda"},
                  message="connect_to localhost port 8881 failed.")

svc = MessageService(conn)
svc.insert(message)

Querying Data

Querying Series

To query series in the database, pass the following filters to the SeriesService:

SeriesFilter is required. Optionally use other filters to further specify series more precisely.

  • SeriesFilter: Defines the metric name. Alternatively, include data type, series tags, and other parameters.
  • EntityFilter: Accepts a single entity name, an array of multiple entity name, an entity group name, or an expression to filter entities.
  • DateFilter: Specifies startDate, endDate, and interval fields. Provide either startDate and endDate fields or either startDate or endDate and interval or only interval to define period. If only interval is defined, current time is set as endtime. Provide startDate and endDate fields as calendar syntax keywords, an ISO 8601 formatted string, Unix milliseconds, or a Python datetime object.
from atsd_client.models import *

sf = SeriesFilter(metric="temperature")
ef = EntityFilter(entity="sensor123")
df = DateFilter(start_date="2018-02-22T13:37:00Z", end_date=datetime.now())
query_data = SeriesQuery(series_filter=sf, entity_filter=ef, date_filter=df)
svc = SeriesService(conn)
result = svc.query(query_data)

# Print first Series object
print(result[0])
2018-07-18T17:14:30+00:00             1
2018-07-18T17:16:30+00:00             2
metric: temperature
entity: sensor123
tags: tz=local

Optional filters:

Refer to API Documentation for additional details.

Querying Data with SQL

To perform SQL queries, use the query method implemented in SQLService. The returned table is an instance of the DataFrame class.

from atsd_client import connect_url
from atsd_client.services import SQLService

conn = connect_url('https://atsd_hostname:8443', 'user', 'passwd')

# Single-line SQL query
# query = 'SELECT datetime, time, entity, value FROM jvm_memory_free LIMIT 3';

# Multi-line SQL query, enclose in triple quotes (single or double)
query = """
SELECT datetime, time, entity, value
  FROM "jvm_memory_free"
ORDER BY datetime DESC
  LIMIT 3
"""

svc = SQLService(conn)
df = svc.query(query)

print(df)
                   datetime           time entity      value
0  2018-05-17T12:36:36.971Z  1526560596971   atsd  795763936
1  2018-05-17T12:36:21.970Z  1526560581970   atsd  833124808
2  2018-05-17T12:36:06.973Z  1526560566973   atsd  785932984

Pandas options used by atsd_client:

'display.expand_frame_repr' = False

Querying Properties

To retrieve property records from the database, specify the property type name and pass the following filters to the PropertiesService:

  • EntityFilter: Accepts a single entity name, an array of multiple entity name, an entity group name, or an expression to filter entities.
  • DateFilter: Specifies startDate, endDate, and interval fields. Provide either startDate and endDate fields or either startDate or endDate and interval or only interval to define period. If only interval is defined, current time is set as endtime. Provide startDate and endDate fields as calendar syntax keywords, an ISO 8601 formatted string, Unix milliseconds, or a Python datetime object.
from atsd_client.models import *

ef = EntityFilter(entity="nurswgvml007")
df = DateFilter(start_date="today", end_date="now")
query = PropertiesQuery(type="disk", entity_filter=ef, date_filter=df)
svc = PropertiesService(conn)
result = svc.query(query)

# Print first Property object
print(result[0])
type: disk
entity: nurswgvml007
key: command=com.axibase.tsd.Server
tags: fs_type=ext4
date: 2018-05-21 14:46:42.728000+00:00

Optionally use additional property filter fields in PropertiesQuery, for example, key and key_tag_expression.

Refer to API Documentation for additional details.

Querying Messages

To query messages, initialize a MessageQuery object and pass it to the MessageService with the following filters:

  • EntityFilter: Accepts a single entity name, an array of multiple entity name, an entity group name, or an expression to filter entities.
  • DateFilter: Specifies startDate, endDate, and interval fields. Provide either startDate and endDate fields or either startDate or endDate and interval or only interval to define period. If only interval is defined, current time is set as endtime. Provide startDate and endDate fields as calendar syntax keywords, an ISO 8601 formatted string, Unix milliseconds, or a Python datetime object.
  • Additional filter fields: type, source, severity, and tags. To select records with a non-empty value for the given tag, set the filter value to * wildcard.
from atsd_client.models import *

ef = EntityFilter(entity="nurswgvml007")
df = DateFilter(start_date="today", end_date="now")
query = MessageQuery(entity_filter=ef, date_filter=df, type="application", tags={"syslog": "*"}, limit=1000)
svc = MessageService(conn)
messages = svc.query(query)

print("received messages: ", len(messages))

for msg in messages:
  print(msg)
entity: nurswgvml007
type: application
source: atsd
date: 2018-05-21 15:42:04.452000+00:00
severity: MAJOR
tags: syslog=ssh
message: connect_to localhost port 8881 failed.
persist: True

Refer to API Documentation for additional details.

Analyzing Data

Convert to pandas

Install the pandas module for advanced data manipulation and analysis.

pip install pandas

Use Pandas set_option to format output:

import pandas as pd

pd.set_option('display.max_columns', None)
pd.set_option('display.width', 2000)
pd.set_option('max_rows', None)
pd.set_option('max_columns', None)
pd.set_option('max_colwidth', -1)
pd.set_option('display.expand_frame_repr', False)

Series

Access the Series object in Pandas with the built-in to_pandas_series() and from_pandas_series() methods.

ts = series.to_pandas_series()

# 'pandas.tseries.index.DatetimeIndex'
print(ts)
2018-04-10 17:22:24.048000    11
2018-04-10 17:23:14.893000    31
2018-04-10 17:24:49.058000     7
2018-04-10 17:25:15.567000    22
2018-04-13 14:00:49.285000     9
2018-04-13 15:00:38            3

Entities

To retrieve Entity list as Pandas DataFrame use query_dataframe method:

entities = svc.query_dataframe(expression="createdDate > '2018-05-16T00:00:00Z'")

print(entities)
                createdDate  enabled            lastInsertDate          name
0  2018-07-12T14:52:21.599Z     True  2018-07-23T15:39:51.542Z  nurswgvml007
1  2018-07-17T20:08:02.213Z     True  2018-07-17T20:08:04.813Z  nurswghbs001
2  2018-07-12T14:52:21.310Z     True  2018-07-23T15:39:49.164Z          atsd

Pandas options used by atsd_client:

'display.expand_frame_repr' = False
'max_colwidth' = -1

Messages

To retrieve Message records as Pandas DataFrame use query_dataframe method:

messages = svc.query_dataframe(query, columns=['entity', 'date', 'message'])

print(messages)
         entity                             date                                            message
0  nurswgvml007 2018-07-17 18:49:24.749000+03:00  Scanned 0 directive(s) and 0 block(s) in 0 mil...
1  nurswgvml007 2018-07-17 18:48:24.790000+03:00  Scanned 0 directive(s) and 0 block(s) in 0 mil...
2  nurswgvml007 2018-07-17 18:48:16.129000+03:00                Indexing started, type: incremental

Pandas options used by atsd_client:

'display.expand_frame_repr' = False
'max_colwidth' = -1

Properties

To retrieve Property records as Pandas DataFrame use query_dataframe method:

properties = svc.query_dataframe(query)

print(properties)
                       date        entity    id  type  fs_type
0  2018-07-23T15:31:03.000Z  nurswgvml007   fd0  disk     ext3
1  2018-07-23T15:31:03.000Z  nurswgvml007   sda  disk     ext4
2  2018-07-23T15:31:03.000Z  nurswgvml007  sda1  disk     ext4

Pandas options used by atsd_client:

'display.expand_frame_repr' = False
'max_colwidth' = -1

Graph Results

To plot a series with matplotlib, use the plot() function:

>>> import matplotlib.pyplot as plt
>>> series.plot()
>>> plt.show()

Working with Versioned Data

Versioning enables keeping track of value changes. See Versioning Documentation for more information.

Enable versioning for specific metrics and add optional versioning fields to samples which contain the version argument.

from datetime import datetime
other_series = Series('sensor123', 'power')
other_series.add_samples(
    Sample(3, datetime.now(), version={"source":"TEST_SOURCE", "status":"TEST_STATUS"})
)

To retrieve series values with versioning fields, add the VersionedFilter to the query and enable the versioned field.

import time
from atsd_client.models import *

cur_unix_milliseconds = int(time.time() * 1000)
sf = SeriesFilter(metric="power")
ef = EntityFilter(entity="sensor123")
df = DateFilter(startDate="2018-02-22T13:37:00Z", endDate=cur_unix_milliseconds)
vf = VersioningFilter(versioned=True)

query_data = SeriesQuery(series_filter=sf, entity_filter=ef, date_filter=df, versioning_filter=vf)
result = svc.query(query_data)

print(result[0])
           time         value   version_source   version_status
1468868125000.0           3.0      TEST_SOURCE      TEST_STATUS
1468868140000.0           4.0      TEST_SOURCE      TEST_STATUS
1468868189000.0           2.0      TEST_SOURCE      TEST_STATUS
1468868308000.0           1.0      TEST_SOURCE      TEST_STATUS
1468868364000.0          15.0      TEST_SOURCE      TEST_STATUS
1468868462000.0          99.0      TEST_SOURCE      TEST_STATUS
1468868483000.0          54.0      TEST_SOURCE      TEST_STATUS

Examples

Preparation

Name Description
version_check.py Print Python and module version information.

Connection

Name Description
connect_url_check.py Connect to the target ATSD instance, retrieve the database version, timezone and current time with the connect_url('https://atsd_hostname:8443', 'user', 'password') function.
connect_path_check.py Connect to the target ATSD instance, retrieve the database version, timezone and current time with the connect(/home/axibase/connection.properties) function.
connect_check.py Connect to the target ATSD instance, retrieve the database version, timezone and current time with the connect() function.

Inserting Records

Name Description
nginx_access_log_tail.py Continuously read nginx access.log via tail -F, parse request logs as CSV rows, discard bot requests, insert records as messages.

Data Availability

Name Description
find_broken_retention.py Find series that ignore metric retention days.
metrics_without_last_insert.py Find metrics without a last insert date.
entities_without_last_insert.py Find entities without a last insert date.
find_series_by_value_filter.py Retrieve versioned series with value filter to discard the deleted values.
find_lagging_series_for_entity_expression.py Find series for matching entities that have not been updated for more than one day.
find_lagging_series_for_entity.py Find series for the specified entity that have not been updated for more than one day.
find_lagging_series_for_metric.py Find series for the specified metric that have not been updated for more than one day.
find_lagging_series.py Find series with last insert date which lag behind the maximum last insert date by more than specified grace interval.
high_cardinality_series.py Find series with excessive tag combinations.
high_cardinality_metrics.py Find metrics with series that have too many tag combinations.
find_lagging_entities.py Find entities that match the specified expression filter which no longer collect data.
find_stale_agents.py Find entities which no longer collect data for a subset metrics.
metrics_created_later_than.py Find metrics created after the specified date.
entities_created_later_than.py Find entities created after the specified date.
find_delayed_entities.py Find entities more than N hours behind the metric last_insert_date.
series_statistics.py Retrieve series for a given metric, for each series fetch first and last value.
frequency_violate.py Print values that violate metric frequency.
migration.py Compare series query responses before and after ATSD migration.
data-availability.py Monitor availability of data for parameters defined in data-availability.csv.

Data Manipulation

Name Description
copy_data.py Copy data to a different period.
copy_data_for_the_metric.py Copy data to a new metric.

Data Removal and Cleanup

Name Description
find_non-positive_values.py Find series with non-positive values for the specified metric and delete if required.
delete_series.py Delete samples for the given metric, entity, and any tags within the specified date interval.
delete_series_data_interval.py Delete data for a given series with tags within the specified date interval.
delete_series_for_all_entity_metrics.py Delete series for all metrics for the specified entity with names beginning with the specified prefix.
delete_series_for_entity_metric_tags.py Delete all series for the specified entity, metric and series tags.
docker_delete.py Delete docker host entities and related container/image/network/volume entities without data insertion during the previous seven days.
entities_expression_delete.py Delete entities that match the specified expression filter.
delete_entity_tags.py Delete specific entity tags from entities that match an expression.
delete_entity_tags_starting_with_expr.py Delete entity tags beginning with the defined expression.
update_entity_tags_from_property.py Update entity tags from the corresponding property tags.

Reports

Name Description
sql_query.py Execute SQL query and convert results into a DataFrame.
entity_print_metrics_html.py Print metrics for entity into HTML or ASCII table.
export_messages.py Export messages into CSV.
message_dataframe.py Execute Message query and convert results into a DataFrame.
message_dataframe_filtered.py Execute Message query, convert results into a DataFrame, group by tag and filter.
message_dataframe_filtered_and_ordered.py Execute Message query, convert results into a DataFrame, group by tag, filter, and sort by date.
message_referrer_report.py Query messages convert result into a HTML table.

Some of the examples above use the prettytable module to format displayed records.

pip install prettytable
# pip install https://pypi.python.org/packages/source/P/PrettyTable/prettytable-0.7.2.tar.gz