Allow "tsdb import" to read from STDIN #580

Merged
merged 1 commit into from Mar 23, 2016

Projects

None yet

3 participants

@hynd
Contributor
hynd commented Oct 6, 2015

To facilitate streaming data from one OpenTSDB instance to another, avoiding intermediate files. If the import "path" is "-", it'll read from STDIN instead.

For example, something like this for migrating between two configs (with different salting or byte width configs etc):

tsdb scan --config=old.conf 1y-ago sum my.metric | tsdb import --skip-errors -- -

There may already be another way to achieve this?

A very quick check of the above got ~70k/s on an anaemic VM backed by a standalone Hbase (~85k unique metrics out of the 999981):

2015-10-06 21:23:42,263 INFO  [main] [TextImporter.importFile]: Processed - in 14855 ms, 999981 data points (67316.1 points/s)
2015-10-06 21:23:42,264 INFO  [main] [TextImporter.main]: Total: imported 999981 data points in 14.861s (67287.2 points/s)

(i wondered if TextImporter could also leverage aec67bf to improve its speed?)

@hynd hynd referenced this pull request Nov 5, 2015
Closed

tsdb import from stdin #596

@hynd
Contributor
hynd commented Feb 9, 2016

Hey @manolama - what are the chances of this getting wangled into 2.2? Might help people with migrating data after upgrades...

@johann8384
Member

Hey @manolama - perhaps this can go into 2.3?

@johann8384 johann8384 added this to the v2.3.0 milestone Mar 19, 2016
@johann8384 johann8384 merged commit c2d26cb into OpenTSDB:next Mar 23, 2016

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment