feat: row protocol support for opentsdb #2468

Lilit0x · 2023-09-21T20:05:58Z

I hereby agree to the terms of the GreptimeDB CLA

What's changed and what's your intention?

Support for row inserts in opentsdb

Please explain IN DETAIL what the changes are in this PR and why they are needed:
OpenTSDB now uses the row insert protocol as against column inserts

Checklist

I have written the necessary rustdoc comments.
I have added the necessary unit tests and integration tests.

close #2444

MichaelScofield · 2023-09-26T05:09:28Z

@Lilit0x can you fix the failed tests?

codecov · 2023-09-26T08:59:35Z

Codecov Report

Merging #2468 (fcf65a1) into develop (201acd1) will decrease coverage by 0.57%.
The diff coverage is 100.00%.

❗ Current head fcf65a1 differs from pull request most recent head ff300d2. Consider uploading reports for the commit ff300d2 to get more accurate results

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #2468      +/-   ##
===========================================
- Coverage    85.28%   84.72%   -0.57%     
===========================================
  Files          730      724       -6     
  Lines       116093   115218     -875     
===========================================
- Hits         99009    97615    -1394     
- Misses       17084    17603     +519

fengjiachun · 2023-09-27T03:41:54Z

Hi, @Lilit0x , can we make OpentsdbProtocolHandler accept a collection of DataPoints? In this way, we can convert the collection of DataPoints into a RowInsertRequests at once. I think this will improve efficiency. cc @MichaelScofield

Lilit0x · 2023-09-27T06:23:59Z

Hi, @Lilit0x , can we make OpentsdbProtocolHandler accept a collection of DataPoints? In this way, we can convert the collection of DataPoints into a RowInsertRequests at once. I think this will improve efficiency. cc @MichaelScofield

Hi, that makes sense. I will work on it.

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* feat: add two grpc config options Those options are for: * Limit receiving(decoding) message size * Limit sending(enoding) message size * test: add integration tests for message size limit

chore: remove unused region_status method form table

* feat: added show tables command * fix(tests): fixed parser and statement unit tests * chore: implemeted display trait for table type * fix: handled no tabletype and error for usopprted command in show databse * chore: removed full as a show kind, instead as a show option * chore(tests): fixed failing test and added more tests for show full * chore: refactored table types to use filters * fix: changed table_type to tables

…2466) * test: test on_compaction_finished * fix: avoid submit same region to compact * feat: persist and recover compaction time window * test: fix test * test: sort like result

* refactor: remove is_timestamp_compatible. * chore: fmt * refactor: remove int64 to timestamp match * chore * chore: apply suggestions from code review Co-authored-by: dennis zhuang <killme2008@gmail.com> * chore: fmt --------- Co-authored-by: dennis zhuang <killme2008@gmail.com>

* feat: implement new histogram data model * feat: use prometheus table format for histogram * refactor: remove duplicated code * fix: histogram tag column * fix: use accumulated count in buckets * refactor: using row based protocol for otlp WIP * refactor: use row based writer for otlp. Also updated row writer for owned keys * refactor: use row writers for otlp * test: add integration tests for histogram * refactor: change le column name

…ies (#2474) fix: reset is_sorted flag to true after the merger finishing one series

fix: normalize region dir in RegionOpener

* refactor: datetime time unit * Update src/common/time/src/datetime.rs Co-authored-by: Yingwen <realevenyag@gmail.com> * chore: cr. --------- Co-authored-by: Yingwen <realevenyag@gmail.com>

) * refactor: set meta leader lease secs to 3s * fix: correct default heartbeat interval * refactor: ask meta leader in parallel * feat: configure heartbeat client timeout to 500ms * fix: trigger to send heartbeat immediately after fail * fix: fix clippy

* feat: re-support query engine execute dml Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * chore: remove region_number in InsertRequest Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * chore: add doc comments Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> --------- Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: convert datetime to chrono datetime Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * chore: typo Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fix the bad fix Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> --------- Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* feat: distribute truncate table * chore: add metrics for truncate table * test: add sqlness test * chore: cr * test: add multi truncate * chore: add trace id to the header

* impl macro stack_trace_debug Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * manually mark external error Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * ignore warnings Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix clippy warnings Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * use debug print Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * simplify the error and warn macro Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix ut Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * add docs Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * replace snafu backtrace with location Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* feat: add cache manager * feat: add cache to reader builder * feat: add AsyncFileReaderCache * feat: Impl AsyncFileReaderCache * chore: move moka dep to workspace * feat: add moka cache to the manager * feat: implement parquet meta cache * test: test cache manager * feat: consider vec size * style: fix clippy * test: fix config api test * feat: divide cache * test: test disabling meta cache * test: fix config api test * feat: remove meta cache if file is purged

* fix: check for table scan before expanding Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * change assert_ok to unwrap Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix clippy warning Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * update sqlness result Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * don't skip dml Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * uncomment ignored tests Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

…he dir (#2504) fix: fix region drop task runs multiple times but never clean the directory

* refactor: simplify the error! and warn! macros Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * support display format Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * err.msg to err Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* chore: set default connect_timeout_millis to 1000 * fix: re-create heartbeat stream ASAP * chore: apply suggestions

* chore: test return msg * fix: test_child_error Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * chore: fix test * chore: minor fix grpc return value * chore: format return msg * chore: use root error as return value * chore: fix empty err display * chore: iter through external error * chore: remove err msg * chore: remove unused field --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> Co-authored-by: Ruihang Xia <waynestxia@gmail.com>

* fix: allow `.`(dot) literal in table name * fix: resolve PR comments

feat: pushdown filters to region engine Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

feat: open_tsdb support for row insert requests

2df6350

Lilit0x marked this pull request as draft September 21, 2023 20:06

Lilit0x marked this pull request as ready for review September 25, 2023 21:19

MichaelScofield requested a review from fengjiachun September 26, 2023 05:07

waynexia and others added 22 commits October 2, 2023 11:39

refactor: remove SqlStatementExecutor (#2464)

67651e2

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

feat: limit grpc message size (#2459)

2292c41

* feat: add two grpc config options Those options are for: * Limit receiving(decoding) message size * Limit sending(enoding) message size * test: add integration tests for message size limit

chore: remove unused region_stats method form table (#2458)

f9315bd

chore: remove unused region_status method form table

feat: update proto and remove create_if_not_exists (#2467)

8a12762

fix(mito): compaction scheduler schedules more tasks than expected (#…

e2a8c0c

…2466) * test: test on_compaction_finished * fix: avoid submit same region to compact * feat: persist and recover compaction time window * test: fix test * test: sort like result

chore: typo (#2470)

e7c1d91

fix(mito): reset is_sorted to true after the merger finishing one ser…

2d22d60

…ies (#2474) fix: reset is_sorted flag to true after the merger finishing one series

fix(mito): normalize region dir in RegionOpener (#2475)

a3ff532

fix: normalize region dir in RegionOpener

chore: refactor options (#2476)

e4cbe5f

chore: fix typo (#2477)

e051841

chore: add integration test for issue2437 (#2481)

198fcba

fix: changed tags to string type

c6c8c2a

fix: print root cause error message to user facing interface (#2486)

87589e7

refactor(mito): remove #[allow(dead_code)] (#2479)

958adf7

refactor: datetime time unit (#2469)

6efc972

* refactor: datetime time unit * Update src/common/time/src/datetime.rs Co-authored-by: Yingwen <realevenyag@gmail.com> * chore: cr. --------- Co-authored-by: Yingwen <realevenyag@gmail.com>

fix(tests): fixed open tsdb integration test

212c490

fix(tests): aligned top row in table for failing test

6ede1be

fix(tests): aligned top row in table for failing test

662e9bb

zhongzc and others added 13 commits October 2, 2023 11:39

feat: distribute truncate table in region server (#2414)

d80977a

* feat: distribute truncate table * chore: add metrics for truncate table * test: add sqlness test * chore: cr * test: add multi truncate * chore: add trace id to the header

fix(mito): fix region drop task runs multiple times but never clean t…

987eefb

…he dir (#2504) fix: fix region drop task runs multiple times but never clean the directory

fix: re-create heartbeat stream ASAP (#2499)

5d8f46f

* chore: set default connect_timeout_millis to 1000 * fix: re-create heartbeat stream ASAP * chore: apply suggestions

fix: allow .(dot) literal in table name (#2483)

a71bf83

* fix: allow `.`(dot) literal in table name * fix: resolve PR comments

feat: push down filters to region engine (#2513)

6eb49b5

feat: pushdown filters to region engine Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

chore: changed opentsd handler to accept collection of datapoints

9d72117

Lilit0x closed this Oct 2, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: row protocol support for opentsdb #2468

feat: row protocol support for opentsdb #2468

Lilit0x commented Sep 21, 2023 •

edited by MichaelScofield

MichaelScofield commented Sep 26, 2023

codecov bot commented Sep 26, 2023 •

edited

fengjiachun commented Sep 27, 2023

Lilit0x commented Sep 27, 2023

feat: row protocol support for opentsdb #2468

feat: row protocol support for opentsdb #2468

Conversation

Lilit0x commented Sep 21, 2023 • edited by MichaelScofield

What's changed and what's your intention?

Checklist

MichaelScofield commented Sep 26, 2023

codecov bot commented Sep 26, 2023 • edited

Codecov Report

fengjiachun commented Sep 27, 2023

Lilit0x commented Sep 27, 2023

Lilit0x commented Sep 21, 2023 •

edited by MichaelScofield

codecov bot commented Sep 26, 2023 •

edited