Skip to content

Releases: TimechoLab/influxdb2tsfile

Release list

influxdb2tsfile 1.0.0

influxdb2tsfile 1.0.0 Latest
Latest

Choose a tag to compare

View all tags
@github-actions github-actions released this 16 Sep 02:12
v1.0.0
eaf30ff

influxdb2tsfile 1.0.0

First public release — a command-line tool that migrates InfluxDB data to Apache TsFile.

Downloads

File What it is
influxdb2tsfile-1.0.0-bin.tar.gz Runnable package: launcher bin/influxdb2tsfile + fat-jar in lib/ + README (EN/中文) + LICENSE
influxdb2tsfile-1.0.0.jar The shaded fat-jar alone (all dependencies included)

Requires JDK 17 or newer (Apache TsFile 2.4.0 is compiled to Java 17 bytecode). The launcher checks the JDK version and prints a clear message if it is too old.

Run it

tar xzf influxdb2tsfile-1.0.0-bin.tar.gz
cd influxdb2tsfile-1.0.0
export JAVA_HOME=/path/to/jdk-17          # 17 / 21 / 25 all work

# 1) inspect the schema and the TsFile mapping
bin/influxdb2tsfile discover --url http://127.0.0.1:8086 --database telegraf \
  --username admin --password secret --counts

# 2) migrate (time-sliced, parallel, resumable, verified)
bin/influxdb2tsfile migrate --url http://127.0.0.1:8086 --database telegraf \
  --username admin --password secret --start -30d --end now \
  --time-slice 1d --parallel 4 -o ./tsfile-out --resume --verify

# 3) read the result back and compare it with the manifest
bin/influxdb2tsfile inspect ./tsfile-out --manifest ./tsfile-out/manifest.json

Or run the jar directly:

java -jar influxdb2tsfile-1.0.0.jar migrate --url http://127.0.0.1:8086 --database telegraf --token <api-token> \
  --time-slice 1d --parallel 4 -o ./tsfile-out --verify

Highlights

  • InfluxDB 1.x and 2.x: InfluxQL with username/password, or the InfluxQL compatibility API with an API token — same commands for both
  • Offline path: lp2tsfile converts line protocol files or stdin, for instances that are already gone
  • Schema mapping: measurement → table, tag → TAG column (device id = series), field → FIELD column (float→DOUBLE, integer/unsigned→INT64, boolean→BOOLEAN, string→STRING), timestamp → nanosecond time column, missing fields → null
  • Sharded, streaming writes: --time-slice, --max-file-size, --max-rows-per-file, --batch-size, LZ4 by default (ZSTD/GZIP/SNAPPY selectable)
  • Parallel and resumable: one work unit (measurement × time slice) owns its files; --resume re-validates the recorded output files and only redoes what is missing or stale; re-running the same command is idempotent
  • Verifiable: --verify and inspect --manifest read every written file back and compare row counts per table; a mismatch exits with code 1
  • Auditable: manifest.json records per-unit files, rows, devices, time ranges and the original→TsFile column mapping

Verified with

  • InfluxDB 1.8.10 (authentication enabled) and InfluxDB 2.7.11 (API token): 23,520 points across 4 measurements → 16 TsFile parts (210 KB), row counts matched the source exactly
  • Interruption/resume and injected failures: no duplicated rows, --resume completed the run and passed verification
  • CI on GitHub Actions: JDK 17 / 21 / 25 all green (mvn test, 28 unit + integration tests including a mock InfluxDB and real TsFile read/write round-trips)

Documentation

Known limitations

  • Time-series data only; InfluxDB metadata (users, dashboards, continuous queries, subscriptions, tasks) is out of scope
  • One measurement maps to one table; multiple databases/buckets need --table-prefix
  • Only the default retention policy is read by default (--rp for others)
  • Fields with mixed data types need --type-conflict text or source-side cleanup
  • Command line only — no web UI, by design

中文摘要

首个正式版本：把 InfluxDB 数据迁移到 Apache TsFile 的命令行工具。

下载：influxdb2tsfile-1.0.0-bin.tar.gz（解压即用的发行包：启动脚本 + fat-jar + 中英文 README + LICENSE）或 influxdb2tsfile-1.0.0.jar（单独 fat-jar）。

要求：JDK 17 及以上（TsFile 2.4.0 的字节码是 Java 17）。

tar xzf influxdb2tsfile-1.0.0-bin.tar.gz && cd influxdb2tsfile-1.0.0
export JAVA_HOME=/path/to/jdk-17
bin/influxdb2tsfile discover --url http://127.0.0.1:8086 --database telegraf --username admin --password secret --counts
bin/influxdb2tsfile migrate  --url http://127.0.0.1:8086 --database telegraf --username admin --password secret \
  --time-slice 1d --parallel 4 -o ./tsfile-out --resume --verify
bin/influxdb2tsfile inspect  ./tsfile-out --manifest ./tsfile-out/manifest.json

映射规则：measurement→表、tag→TAG 列（设备 ID 即 series）、field→FIELD 列（float→DOUBLE、integer/unsigned→INT64、boolean→BOOLEAN、string→STRING）、时间戳→纳秒时间列、缺失字段→null。

可靠性：工作单元（measurement × 时间切片）独立成文件，可并行；同参数重跑幂等；--resume 会校验产物文件，只补做缺失/失效的单元；--verify / inspect --manifest 回读比对行数，不一致退出码 1。

实测：InfluxDB 1.8.10（开启认证）与 2.7.11（Token），23,520 点 → 16 个 tsfile 分片，行数与源端逐表一致；CI 在 JDK 17/21/25 上全绿。

Apache License 2.0

Assets 4
Loading