forked from OpenTSDB/opentsdb.net
/
cli.html
254 lines (238 loc) · 11.8 KB
/
cli.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
<!DOCTYPE html>
<html lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
<title>Command-line tool - OpenTSDB - A Distributed, Scalable Monitoring System</title>
<link rel="stylesheet" href="css/style.css" type="text/css" media="screen"/>
<!--[if lte IE 8]>
<link rel="stylesheet" href="css/ie.css" type="text/css" media="screen"/>
<![endif]-->
<script type="text/javascript">
var _gaq = _gaq || [];
_gaq.push(['_setAccount', 'UA-18339382-1']);
_gaq.push(['_setDomainName', 'none']);
_gaq.push(['_setAllowLinker', true]);
_gaq.push(['_trackPageview']);
(function() {
var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
})();
</script>
</head>
<body>
<header><div id="headerbar">
<h1><a href="index.html">OpenTSDB</a></h1>
<nav><ul id="navbar">
<li><a href="overview.html">Overview</a></li>
<li><a href="getting-started.html">Getting Started</a></li>
<li><a href="manual.html">Manual</a></li>
<li><a href="faq.html">FAQ</a></li>
</ul></nav>
</div></header>
<!--[if lte IE 8]>
<div class="iesucks">Warning: You're using an unsupported, archaic browser.
Get a better, modern browsing experience with
<a href="http://www.google.com/chrome">Chrome</a> or
<a href="http://www.mozilla.com/firefox">Firefox</a>.</div>
<![endif]-->
<section id="content">
<section id="cli">
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.3.2/jquery.min.js" type="text/javascript"></script>
<script src="misc/toc.js" type="text/javascript"></script>
<h2>Command-Line tool</h2>
OpenTSDB comes with a simple command-line tool, <code>./tsdb</code>.
This page documents the various subcommands of the tool.
<div id="toc"></div>
<h3>Common flags</h3>
All the subcommands of the tool accept the following flags:
<div class="code"> --table=TABLE Name of the HBase table where to store the time series (default: tsdb).
--uidtable=TABLE Name of the HBase table to use for Unique IDs (default: tsdb-uid).
--zkbasedir=PATH Path under which is the znode for the -ROOT- region (default: /hbase).
--zkquorum=SPEC Specification of the ZooKeeper quorum to use (default: localhost).
</div>
<h3>Site-specific configuration</h3>
The common flags are required by all the subcommands. It can be tedious to
manually type them over and over again. You can instead override the default
values in a configuration file, <code>./tsdb.local</code>. This file is
expected to be a shell script and will be sourced by <code>./tsdb</code>
if it exists.
<h4>Setting default values for common flags</h4>
If, for example, your ZooKeeper quorum is behind the DNS name
"zookeeper.example.com" (a name with 5 A records), instead of always passing
<code>--zkquorum=zookeeper.example.com</code> to the command-line tool each
time you use it, you can create <code>./tsdb.local</code> with the
following contents:
<div class="code">#!/bin/bash
MY_ARGS='--zkquorum=zookeeper'
set x $MY_ARGS "$@"
shift
</div>
<h4>Overriding the timezone of the TSD</h4>
Servers are frequently using UTC as their timezone. By default, the TSD
renders graphs using the local timezone of the server. You can override
this to have graphs in your local time by specifying a timezone in
<code>./tsdb.local</code>. For example, if you're in California,
this will force the TSD to use your timezone:
<div class="code">echo export TZ=PST8PDT >>tsdb.local</div>
On most Linux and BSD systems, you can look under
<code>/usr/share/zoneinfo</code> for names of timezones supported on your
system.
<h4>Changing JVM parameters</h4>
You might want to adjust JVM parameters, for instance to turn on GC activity
logging or to set the size of various memory regions. In order to do so,
simply set the variable <code>JVMARGS</code> in <code>./tsdb.local</code>.
<p>
Here is an example that is recommended for production use:
<div class="code">GCARGS="-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps\
-XX:+PrintTenuringDistribution -Xloggc:/tmp/tsd-gc-`date +%s`.log"
if test -t 0; then # if stdin is a tty, don't turn on GC logging.
GCARGS=
fi
# The Sun JDK caches all name resolution results forever, which is stupid.
# This forces you to restart your application if any of the backends change
# IP. Instead tell it to cache names for only 10 minutes at most.
UNFUCK_DNS='-Dsun.net.inetaddr.ttl=600'
JVMARGS="$JVMARGS $GCARGS $UNFUCK_DNS"
</div>
<h3>uid</h3>
OpenTSDB assigns various UIDs (Unique IDs) to metric names, tag names and tag
values. Read more about the <a href="schema.html">schema</a>. This tool
allows you to query, create and rename UIDs. Beware that TSDs aren't
expecting UIDs to change, so after renaming a UID you should do a rolling
restart of your TSDs.
<div class="code">$ ./tsdb uid grep metrics 'disk.*write'
metrics iostat.disk.msec_write: [0, 3, -67]
metrics iostat.disk.write_merged: [0, 3, -69]
metrics iostat.disk.write_requests: [0, 3, -70]
metrics iostat.disk.write_sectors: [0, 3, -68]
</div>
You can also lookup the UID of an individual entity, for instance a tag name:
<div class="code">$ ./tsdb uid tagk host
tagk host: [0, 0, 1]
</div>
<p>
If you ever decide to rename something, for instance a metric, you can do so
using the tool, but once again, be advised that TSDs don't expect UIDs to
change names, so you should do a rolling restart of your TSDs after changing a
name:
<div class="code">$ ./tsdb uid rename metrics my.metric new.metric.name
metrics my.metric -> new.metric.name
</div>
<p>
Because UIDs are stored in a separate table (named <code>tsdb-uid</code> by
default), there's a sub-command called <code>fsck</code> to check that the
table is consistent.
<div class="code">$ ./tsdb uid fsck
8668 KVs analyzed in 1709ms (~5071 KV/s)
No errors found.
</div>
<h3>mkmetric</h3>
This tool is used to register new metrics in OpenTSDB. Although you can also
register new metrics with the <a href="#uid"><code>uid</code></a> tool, using
the <code>mkmetric</code> tool is easier. Just give it a bunch of metric
names and it'll register them:
<div class="code">$ ./tsdb mkmetric my.first.metric my.other.metric
metrics my.first.metric: [0, 0, -58]
metrics my.other.metric: [0, 0, -57]
</div>
<h3>import</h3>
The import command takes one or more paths in argument and performs a batch
import of all the data points. If an input path ends with <code>.gz</code>,
the import tool will decompress the file on the fly. This is a lot more
efficient and is recommended. If you generate large amounts of data points,
gzip them as you generate them as the import will be a lot faster with a
gzipped input.
<p>
The format of the data accepted by the tool is the same as throughout the rest
of OpenTSDB: metric name, timestamp, value, tags, all separated by a single
space. Here's a well formed example:
<div class="code">proc.stat.cpu 1297574486 54.2 host=foo type=user
proc.stat.cpu 1297574501 39.6 host=foo type=user
proc.stat.cpu 1297574486 2.66 host=foo type=nice
proc.stat.cpu 1297574501 0.06 host=foo type=nice
proc.stat.cpu 1297574486 5.0 host=foo type=system
proc.stat.cpu 1297574501 1.6 host=foo type=system
</div>
<p>
Important note: batch imports do not go through a TSD, they go directly to
HBase and bypass its WAL (Write Ahead Log). This results in a 10x speed up
but if an HBase node crashes during the import, the data you imported may
not be stored safely on disk. In this case, it is recommended to run the
import again to ensure that all the data is imported properly. Importing
the same data twice is harmless.
<h3>query</h3>
This command allows you to make queries from the command-line. The query will
not go through any TSD as the command-line tool will read directly from HBase.
The syntax of the query is very similar to that of the
<a href="http-api.html#/q">HTTP API</a> except it uses spaces instead of
colons to separate the different components of a query.
<div class="code">$ ./tsdb query 2011/02/12-22:05 sum rate proc.stat.cpu host=foo type=idle
proc.stat.cpu 1297576227 352.333333 {host=foo, type=idle}
proc.stat.cpu 1297576242 342.800000 {host=foo, type=idle}
<i>[...]</i>
</div>
Note that if you want to integrate OpenTSDB with third-party tools (e.g.
alerting systems, graphing systems), it is recommended to extract data out of
OpenTSDB using the <a href="http-api.html#/q">HTTP interface</a> of a TSD, not
with the command-line tool. The TSD has much better query performance thanks
to various caches and existing open connections to HBase.
<h3>fsck</h3>
This lets command lets you scan your data and find anything that might be
wrong with it. It can also fix some errors automatically.
<p>
Its usage is very similar to the <a href="#query"><code>query</code></a>
in the sense that you have to specify a query and whatever data is covered
by this query will be checked.
<div class="code">$ ./tsdb fsck 2011/06/24 2011/06/25 sum proc.meminfo.memfree
<i>[...]</i>
3592067 KVs (in 15288 rows) analyzed in 6212ms (~578246 KV/s)
No error found.
</div>
<h3>scan</h3>
This command lets you scan the HBase table and helps you make sense of all the
binary byte arrays stored in it. It is useful to try to debug the raw data
stored in the <code>tsdb</code> table. It can also be used to delete data
from the table, in case you mistakenly imported bad data.
<p>
Its usage is very similar to the <a href="#query"><code>query</code></a>
command, only the output is different and reflects more exactly how the data
is stored in HBase. It is also more resilient to invalid or corrupted data,
to help you debug such situations should they arise.
<div class="code">$ ./tsdb scan 2011/02/12-22:05 sum rate proc.stat.cpu host=foo type=idle
<i>[...]</i>
[0, 4, 2, 77, 87, 113, 8, 0, 0, 1, 0, 0, -33, 0, 0, 3, 0, 2, 76] proc.stat.cpu 1297576200 (Sat Feb 12 21:50:00 PST 2011) {type=idle, host=foo}
[0, -57] [0, 0, 0, 0, 60, 102, -59, -8] 12 l 1013368312 1297576212 (Sat Feb 12 21:50:12 PST 2011)
[1, -73] [0, 0, 0, 0, 60, 102, -38, -99] 27 l 1013373597 1297576227 (Sat Feb 12 21:50:27 PST 2011)
[2, -89] [0, 0, 0, 0, 60, 102, -18, -77] 42 l 1013378739 1297576242 (Sat Feb 12 21:50:42 PST 2011)
<i>[...]</i>
</div>
The first line shows the raw byte array of the row key, followed by its
interpretation. We can infer that <code>[0, 4, 2]</code> is the metric ID for
<code>proc.stat.cpu</code>. <code>[77, 87, 113, 8]</code> is simply the 32
bit integer for the timestamp <code>1297576200</code>. Then the tags:
<code>[0, 0, 1, 0, 0, -33]</code> is for <code>host=foo</code> and
<code>[0, 0, 3, 0, 2, 76]</code> is for <code>type=idle</code>.
The next 3 lines show the first 3 cells stored in this row. The first 2 byte
array is the column qualifier, the second byte array is the value. The 3rd
field is the interpreted timestamp delta from the column qualifier, then
there's a one character flag to tell us the type of value: <code>l</code> for
an integer value or <code>f</code> for a floating point value, followed by the
value itself. The last numerical value is the timestamp of that data point
(the sum of the base time stored in the row key and the delta stored in the
column qualifier) as well as a human readable version of it.
<p>
<a name="--delete"></a>
The <code>scan</code> command accepts a <code>--delete</code> flag. If this
flag is present, it will delete rows as it scans them, and print them to the
standard output in a format that is suitable for the
<a href="#import"><code>import</code></a> command. So make sure you save the
output to a file in case you mistakenly delete too much data and you want to
re-import it back later! This is also useful to quickly fix up data. You can
<code>scan --delete</code> it to a file, do some text manipulations on the
file using your favorite editor, and then import it back into OpenTSDB.
</section>
<footer>
© 2010–2012 The OpenTSDB Authors.
</footer>
</section></body></html>