Verify connection && Refactor async connection framework #12667
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
liyuheng55555
changed the title
liyuheng55555
commented
Jun 12, 2024
...fignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/schema/SetTTLProcedure.java
Outdated
Show resolved
Hide resolved
...e/src/main/java/org/apache/iotdb/db/protocol/client/cn/AsyncConfigNodeRequestRPCHandler.java
Outdated
Show resolved
Hide resolved
.../src/main/java/org/apache/iotdb/db/protocol/client/cn/AsyncConfigNodeTSStatusRPCHandler.java
Outdated
Show resolved
Hide resolved
...e/datanode/src/main/java/org/apache/iotdb/db/protocol/client/dn/AsyncDataNodeRPCHandler.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
maybe extract some code
.../iotdb/confignode/client/async/ConfigNodeToConfigNodeInternalServiceAsyncRequestManager.java
Outdated
Show resolved
Hide resolved
...he/iotdb/confignode/client/async/ConfigNodeToDataNodeInternalServiceAsyncRequestManager.java
Outdated
Show resolved
Hide resolved
.../node-commons/src/main/java/org/apache/iotdb/commons/client/request/AsyncRequestContext.java
Outdated
Show resolved
Hide resolved
...va/org/apache/iotdb/commons/client/request/ConfigNodeInternalServiceAsyncRequestManager.java
Show resolved
Hide resolved
...main/java/org/apache/iotdb/commons/client/request/DataNodeInternalServiceRequestManager.java
Show resolved
Hide resolved
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/client/request/Utils.java
Show resolved
Hide resolved
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/ClusterManager.java
Show resolved
Hide resolved
...tdb/confignode/client/async/handlers/rpc/SubmitTestConnectionTaskToConfigNodeRPCHandler.java
Outdated
Show resolved
Hide resolved
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/ClusterManager.java
Show resolved
Hide resolved
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/ClusterManager.java
Outdated
Show resolved
Hide resolved
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/protocol/client/cn/DnToCnRequestType.java
Show resolved
Hide resolved
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/protocol/client/dn/DnToDnRequestType.java
Show resolved
Hide resolved
.../main/java/org/apache/iotdb/db/queryengine/plan/execution/config/sys/TestConnectionTask.java
Outdated
Show resolved
Hide resolved
JackieTien97
added a commit
that referenced
this pull request
Jun 21, 2024
* [IOTDB-6336] Add max retry time duration and whether to retry for unknown errors configurations * Optimize error message when creating unsupported data type (#12650) Optimize error message when creating unsupported data type * Pipe: fix uninitialized config when validate PipeHistoricalDataRegionTsFileExtractor parameters & Subscription: avoid poll messages from unsubscribed topics (#12648) * Pipe: fix uninitialized config when validate PipeHistoricalDataRegionTsFileExtractor parameters * Subscription: avoid poll messages from unsubscribed topics * Fix region migration's timeout judgement #12639 * fix compaction recover skip negative time partition (#12637) * Fix Concurrency Instability When IoTConsensus LogDispatcher Exits #12655 * JDBC: Fix IndexOutOfBoundsException when calling getBytes (#12662) * JDBC: Fix ArrayOutOfBoundException when calling getBytes * Update JDBCCharsetExample.java * [IOTDB-6335] Redundant rows when using GROUP BY TIME with LIMIT * Pipe: Enhance error handling logic in pipe async connector's handlers (#12669) * Print log when read resource error (#12666) * Load: Make load SQL option VERIFY can only be set to true (#12670) * Pipe: Improved hybrid algorithm and avoid the caculations of tsfiles in connector pending queue tsfiles to block the forwarding of realtime requests (#12668) * Migrate to Develocity Maven Extension (#12658) * Migrate to Develocity Maven Extension * Update to Common Custom User Data Maven Extension 2.0 * Update import schema logic implementation and optimize -help display information (#12643) * add import-schema.sh/bat and export-schema.sh/bat in cli/tools * fix import schema * fix import schema --------- Co-authored-by: 2b3c511 <rong.li@timecho.com> * Subscription: fix topic now timestamp precision (#12663) * [IOTDB-6337] Refine the count calculation in RegionScan framework * Bump version of master branch to 1.3.3-SNAPSHOT (#12672) * Pipe: Reduce warn log when PipeConfigRegionExtractorMetrics does not exist (#12673) * [IOTDB-6061] Fix the instability failure caused by initServer in IoTConsensus UT not binding to the corresponding port (#12674) * fix concurrent * fix concurrent * Load: Make load SQL option VERIFY can only be set to true (A follow-up fix for #12670) (#12676) * Pipe: add compression level config for connector ZSTD compressor (#12630) Co-authored-by: Steve Yurong Su <rong@apache.org> * [PY-client] Unify setup.py and requirement.txt (#12601) * Pipe IT: Ignore failed cases caused by cluster restart failure (#12678) * update import and export data -help description (#12677) Co-authored-by: 2b3c511 <rong.li@timecho.com> * Pipe: Fix potential NPE when lastEvent == null in PipeConnectorSubtask (#12680) * Fix wrong calculation for maxBytesCanReserve for IdentitySinkOperator * Change floating point encoding method in IT * Throw `IndexOutOfBound` exception in row implementation. * Thread safely SeriesPartitionTable (#12679) * finish * use concurrentskiplist * bug fix --------- Co-authored-by: OneSizeFitQuorum <tanxinyu@apache.org> * Pipe: Refactor some features for user access (#12686) * Set timestamp precision same as time column (#12681) * done * done * rollback change * fix IT * fix IT * Set configuration on node (#12626) Co-authored-by: Haonan <hhaonan@outlook.com> * Fix cli report error when ignoreTimestamp is true (#12691) * done * done * rollback change * fix IT * fix IT * done * invoke ci * done * invoke ci * Load: Add check for BufferUnderflowException (#12690) Co-authored-by: xz m <xiangzhi.meng@timecho.com> * remove excess stack * Fix new UDTF execution framework dead loop error. * Perfect methods of IAnalysis to adapt the write process of table model * Load: Support storing piecenode in multiple folders during 2nd phase (#12675) Co-authored-by: Steve Yurong Su <rong@apache.org> * Pipe: add decompressed length in RPC compression payload to avoid potential OOM on receiver (#12701) * Load: Update loadTsFileDirs after all newLoadTsFileDirs are generated to avoid undefined behavior (#12712) * Pipe: Fixed NPE in schema pattern parsing & Improved connector subtask retry logic and logs & Fixed the executor used by consensus pipe (#12704) * Procedure: Fix ProcedureInfo on CN leader may decrease its procId (#12711) * Fix missing of linux memory metrics (#12713) * fix missing memory metric * remove the print of type * update import and export tsfile description (#12684) Co-authored-by: 2b3c511 <rong.li@timecho.com> * Subscription: skip on setup and cluster failure when running subscription restart IT & fix some bugs in SubscriptionExecutorServiceManager (#12710) * Update pom.xml to fix security issue (#12705) Co-authored-by: Christofer Dutz <cdutz@apache.org> * Rename IoTV2Consensus to IoTConsensusV2 (#12715) Signed-off-by: OneSizeFitQuorum <tanxinyu@apache.org> * Fix NumberFormatException when upgrade from old version #12719 * Replenish effective mode in iotdb-system.properties (#12706) * load-tsfile script: disable redirection & load: handle exceptions using the Analysis objects instead of throwing exceptions & pipe: handle SYSTEM_READ_ONLY correctly on receiver side (#12716) * Subscription: improve parsing logic when using JAVA SDK client & refactor subscription IT & intro `getSubscribedTopicNames` API (#12721) * Pipe: Fix schema events can not report & Fix delete data events in data regions may fail to mark at schema metrics (#12722) * Pipe: Support `"source.history.loose-range" = "path"` in iotdb-source (#12651) Co-authored-by: Steve Yurong Su <rong@apache.org> * [IOTDB-6338] Fix wrong query result while using some value filter with LIMIT/OFFSET * Pipe: fix threads of IoTDB-Pipe-Processor-Executor-Pool stucked by PipeTsFileInsertionEvent#waitForTsFileClose (#12727) * [ISSUE 12499] Reject query with massive time span in limited memroy * Fix/include thrift lib and headers in cpp client (#12734) * fix: Hopefully fixed the build of the cpp-example on windows systems * fix: Increase the timeout even more to finally make the build succeed. * fix: Fix the build. * chore: Fixed the assembly descriptor of the cpp-client to include the thrift libs and header files. * Pipe / Load / Subscription: Support new TsDataTypes (STRING / BLOB / TIMESTAMP / DATE) (#12665) * Optimize the log printing when multiple error occur (#12732) * Update packege hierachy in Client-cpp readme (#12736) * Support agg(*) query in templated align by device situation * fix config node use IoTDBDescriptor (#12730) * PipeConsensus: Delete WAL after dropping database & fix receiver directory recovery (#12738) * fix wal deletion & create receiver dir * update consensus protocol class * merge judge * modify directory path * modify directory path * Add instructions of compiling client-cpp with VS2019. (#12739) * Subscription: adapt pipe completion signal for automatically drop subscription (#12724) * Verify connection && Refactor async connection framework (#12667) * PipeConsensus: add metrics and fix some bugs for pipeConsensus (#12723) * Decrease TTL Deletion in compaction modification cache (#12687) * decrease TTL Deletion in compaction modification cache * Update iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/execute/utils/MultiTsFileDeviceIterator.java Co-authored-by: Jiang Tian <jt2594838@163.com> * modify MultiTsFileDeviceIterator * fix spotless --------- Co-authored-by: Jiang Tian <jt2594838@163.com> * Stabilize verify connection IT (#12745) * done * more ! * Fix compaction npe when set modified status (#12729) * fix compaction npe when set modified status * check empty page * Subscription IT: execute jstack for potential stuck test cases (#12728) * Pipe: Smoothed the rate in pipe's remaining time calculations (#12699) * Pipe: Fix PipeDataRegionEventCounter not work in PipeRealtimeDataRegionExtractor (#12747) * Fix python CI (#12754) * Support show device ttl on path pattern (#12709) * support show ttl on pathPattern * fix UT * modify details * Support WAL Compression (#12476) * enable wal compression remove metrics in mem table flush task, cache hash code in partial path, use gzip to compress wal batch update metrics * fix bug * fix compilation problem * remove useless code * recover some code * support compression type in WAL Compress Header * support multi version WAL * edit configuration item * add log for WAL size * temp for debug * fix bug * remove useless log * remove one configuration * use compression rate to update wal disk usage * fix ut * fix test * set default to uncompress * fix wal ut * optimize calculating of wal size * close wal file when the origin size of wal buffer is larger than threshold * add the size of magic string * may be fix the bug * fix with comment * edit with review * fix test * add test for wal compression * add hot reload * clean the code to make it more readable * reuse the byte buffer if possible * Indicate the encoding of String * Edit according to comment * spotless * Optimize the devicePath generation efficiency of PlanNode when deserializing (#12749) * Pipe: Support "source.realtime.loose-range" = "path" in iotdb-source (#12751) * Support active schema query (#12718) * Check duplicate in start-node and stop-node scripts (#12756) * check duplicate key in start node script * modify stop node scripts * use \n as line separator * fix config node send showConfiguration to config node classCastException * [IOTDB-6339] Optimize the time slice control of SeriesScanOperator and AlignedSeriesScanOperator * Optimize node startup process & Normalize system.properties management (#12752) * [IOTDB-6340] Clear Coordinator.queryExecutionMap while encountering MemoryNotEnough Exception * [IOTDB-6342] Clear FragmentInstanceManager.dataNodeQueryContextMap while encountering MemoryNotEnough Exception (#12762) * Refactor node registration check (#12766) * done * done * Fix overlap between tsfile is not correctly marked (#12748) * file fileOverlap is not correctly marked * fix repair compaction may generate too large chunk * Subscription: support topic loose range for path and time (#12760) * Add clean logic for FragmentInstance in case that callback is not added. (#12768) * SystemPropertiesHandler use Files::move instead of File::renameTo (#12770) * Files.move * ? * ? * ? * Pipe: Filter devices by pattern before reading device metadata from TsFile (#12765) Currently we read the metadata of all devices and measurements when constructing a TsFileInsertionDataContainer. This is both time-consuming and memory-wasting if only a few devices match the pattern. This PR filters the devices by pattern before reading the metadata of devices and measurements, saving memory and I/O cost when there are many unmatched devices. Note: this only works when TsFile metadata are not cached (due to high memory usage of TsFile metadata cache), because cached metadata can not be filtered so that pipes with arbitrary patterns can use it. --------- Co-authored-by: Steve Yurong Su <rong@apache.org> * Pipe: Adjusted some loggers of metrics to avoid unnecessary warns & Include "lastEvent" into pipe's event count metrics & Removed the "userConflict" judgment to data sync failure caused by METADATA_ERROR (#12758) * Synchronize takeSnapshotAsync on RaftGroupID (#12767) * SystemPropertiesHandler close reader and writer #12772 * Pipe Consensus: Fix events in connector are not in order because of parameter `'realtime-first' = 'true'` (#12773) * Pipe: Avoid event being continuously rate-limited even its pipe is dropped (#12753) Co-authored-by: Steve Yurong Su <rong@apache.org> * Load: Add some load metrics of time cost, write point and disk throughput (#12735) Co-authored-by: Steve Yurong Su <rong@apache.org> --------- Signed-off-by: OneSizeFitQuorum <tanxinyu@apache.org> Co-authored-by: Chen YZ <43774645+Cpaulyz@users.noreply.github.com> Co-authored-by: V_Galaxy <vgalaxies@apache.org> Co-authored-by: Li Yu Heng <liyuheng55555@126.com> Co-authored-by: shuwenwei <55970239+shuwenwei@users.noreply.github.com> Co-authored-by: Xiangpeng Hu <65238551+BUAAserein@users.noreply.github.com> Co-authored-by: Steve Yurong Su <rong@apache.org> Co-authored-by: YangCaiyin <ycycse@gmail.com> Co-authored-by: Haonan <hhaonan@outlook.com> Co-authored-by: Caideyipi <87789683+Caideyipi@users.noreply.github.com> Co-authored-by: Clay Johnson <cjohnson@gradle.com> Co-authored-by: Summer <43237967+2b3c511@users.noreply.github.com> Co-authored-by: 2b3c511 <rong.li@timecho.com> Co-authored-by: Zikun Ma <55695098+DanielWang2035@users.noreply.github.com> Co-authored-by: Liao Lanyu <1435078631@qq.com> Co-authored-by: Zhihao Shen <zhihaoshen7@qq.com> Co-authored-by: Yongzao <yongzao@apache.org> Co-authored-by: OneSizeFitQuorum <tanxinyu@apache.org> Co-authored-by: ppppoooo <1214343862@qq.com> Co-authored-by: xz m <xiangzhi.meng@timecho.com> Co-authored-by: CritasWang <critas@outlook.com> Co-authored-by: Beyyes <cgf1993@foxmail.com> Co-authored-by: Itami Sho <42286868+MiniSho@users.noreply.github.com> Co-authored-by: ZhangHongYin <46039728+SpriCoder@users.noreply.github.com> Co-authored-by: Christofer Dutz <cdutz@apache.org> Co-authored-by: liuminghui233 <36565497+liuminghui233@users.noreply.github.com> Co-authored-by: Jiang Tian <jt2594838@163.com> Co-authored-by: yschengzi <87161145+yschengzi@users.noreply.github.com> Co-authored-by: Peng Junzhi <78788603+Pengzna@users.noreply.github.com> Co-authored-by: 周沛辰 <45144903+choubenson@users.noreply.github.com> Co-authored-by: Liu Xuxin <37140360+THUMarkLau@users.noreply.github.com> Co-authored-by: William Song <48054931+SzyWilliam@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.