Apache IoTDB 2.0.3

Features & Improvements

• Data Query: Added new aggregate function count_if and scalar functions greatest / least to the table model.
• Data Query: Significantly improved the performance of full-table count(*) queries in the table model.
• AI Management: Added timestamps to the results returned by AINode.
• System Management: Optimized the performance of the table model's metadata module.
• System Management: Enabled the table model to actively listens and loads TsFile.
• System Management: Added support for TsBlock deserialization in the Python and Go client query interfaces.
• Ecosystem Integration: Expanded the table model's ecosystem to integrate with Spark.
• Scripts and Tools: The import-schema and export-schema scripts now support importing and exporting metadata for the table model.
• ...

Bugs

• Fixed the issue where a single write request exceeding the total size of the WAL queue caused write queries to hang.
• Fixed the issue where the receiver experienced OOM (Out of Memory) after resuming synchronization following a long period of inactivity.
• Fixed the issue where repeatedly setting TTL for DB and Table led to inserted data being unqueryable and returning an empty list.
• Fixed the issue where a regular user with create+insert permissions on a table encountered exceptions when loading tsfile.
• Fixed the issue where passwords were logged when SessionPool getSession timed out.
• Fixed the issue in the Go client tree model query interface where the absence of a check for the Time column led to an "index out of range [-1]" error when retrieving Time column data.
• Fixed the issue where distinct hits aggregate pushdown optimization and is used with group by date_bin, causing execution exceptions in aggregate queries.
• Fixed the issue of whitespace characters at the beginning and end of port and directory address parameters in the configuration file.
• Fixed the issue where setting the maximum number of concurrent RPC clients less than the number of CPU threads caused DN startup failure.
• Fixed the issue where using a template, after activation, writing to extended columns, and then creating a pipe, caused the series under the device to double.
• Fixed the issue where metadata synchronization, creating a pipe after a template, caused the series to double when using show timeseries.
• Fixed the issue where a regular user with INSERT permissions encountered exceptions when exporting metadata using export-schema.sh.
• ...

Full Changelog: v2.0.2-1...v2.0.3

Apache IoTDB 2.0.2-1

This is a bug-fix version of 2.0.2

• Fix the bug that will remove the data partition table by mistake in case of us/ns time precision and using ttl

Apache IoTDB 1.3.4-1

This is a bug-fix version of 1.3.4

• Fix the bug that will remove the data partition table by mistake in case of us/ns time precision and using ttl

Apache IoTDB 2.0.2

Features & Improvements

• Data Query: Added management of table model UDFs, including user-defined scalar functions (UDSF) and user-defined aggregate functions (UDAF).
• Data Query: Table model now supports permission management, user management, and authorization for related operations.
• Data Query: Introduced new system tables and various O&M statements to optimize system management.
• System Management: The tree model and table model are now fully isolated at the database level.
• System Management: The built-in MQTT Service is now compatible with the table model.
• System Management: The CSharp client now supports the table model.
• System Management: The Go client now supports the table model.
• System Management: Added a C++ Session write interface for the table model.
• Data Synchronization: The table model now supports metadata synchronization and synchronization delete operations.
• Scripts and Tools: The import-data/export-data scripts now support the table model and local TsFile Load.
• ...

Bugs

• Fixed the memory leak issue when writing data using SQL.
• Fixed the issue of duplicate timestamps appearing in table model queries.
• Fixed the issue of duplicate removal anomaly in table model aggregate queries with GROUP BY.
• Fixed the handling of Long.MIN_VALUE or Long.MAX_VALUE during write and merge processes.
• Fixed the issue of Long.MIN_VALUE timestamp causing time partition overflow and subsequent load failure.
• Fixed the issue of out-of-order data within a single TSFile on the destination data node during region migration in load operations.
• Fixed the issue where Explain Analyze caused the execution plan to fail to properly perform column pruning.
• Fixed the issue where the C# Session could not correctly fetch result sets when querying large amounts of data (exceeding fetch_size) on a cluster with more than one node.
• Fixed the inconsistency in reading JDK environment variables by ConfigNode and DataNode on Windows.
• Fixed the issue where the query distribution time statistics in Explain Analyze were larger than actual values, and changed the query distribution time monitoring from FI level to Query level.
  ...

Full Changelog: v2.0.1-beta...v2.0.2

Apache IoTDB 1.3.4

Features & Improvements

• Data Query: Users can now control the loading of JAR packages via URI for UDF, PipePlugin, Trigger, and AINode through configuration items.
• Data Query: Added monitoring for TimeIndex cached during the merge process.
• System Management: Expanded UDF functions with the addition of the pattern_match function for pattern matching.
• System Management:The Python session SDK now includes a parameter for connection timeout.
• System Management:Introduced authorization for cluster management-related operations.
• System Management:ConfigNode/DataNode now supports scaling down using SQL.
• System Management:ConfigNode automatically cleans up partition information exceeding the TTL (cleans up every 2 hours).
• Data Synchronization: Supports specifying authorization information for the receiver on the sender's end.
• Ecosystem Integration: Supports Kubernetes Operator.
• Scripts and Tools: The import-data/export-data scripts have been expanded to support new data types (strings, large binary objects, dates, timestamps).
• Scripts and Tools:The import-data/export-data scripts have been iterated to support importing and exporting data in three formats: TsFile, CSV, and SQL.
  ...

Bugs

• Fixed the issue where an ArrayIndexOutOfBoundsException occurred when a column name did not exist in the HAVING clause of the tree model.
• Fixed the issue where the target sequence in SELECT INTO contained backticks, resulting in incorrect sequences being written.
• Fixed the issue where an empty iot-consensus file was generated after an abnormal power outage, causing the DataNode (dn) to fail to start.
• Fixed the issue where the storage engine reported an error during asynchronous recovery after manually deleting the resource file, leading to Pipe startup failure.
• Fixed the issue where data forwarded by external Pipe could not be synchronized between dual-lives.
• Fixed the issue where the C# Session could not correctly fetch result sets when querying large amounts of data (exceeding fetch_size) on a cluster with more than one node.
• Fixed the issue where the order of query results was incorrect in the C# client.
• Fixed the issue where duplicate timestamps were included in query result sets.
• Fixed the issue where query results were incorrect for single-device queries with sort+offset+limit+align by device.
• Fixed the issue where data synchronization failed when a sequence S1 of data type A was deleted and then a sequence S1 of data type B was written, and a TTL existed.
• Fixed the issue where MergeReader needed to consider memory allocation to avoid negative available memory during out-of-order and reverse queries.
• Fixed the inconsistency in how ConfigNode and DataNode read the JDK environment variables on Windows.
• ...

Full Changelog: v1.3.3...v1.3.4

Apache IoTDB 2.0.1-beta

Features & Improvements

• Table Model: IoTDB has introduced a new model named table model, and supports standard SQL query syntax, including SELECT, WHERE, JOIN, GROUP BY, ORDER BY, LIMIT clause and subQuery.
• Data Query: The table model supports a variety of functions and operators, including logical operators, mathematical functions, and the time-series specific function DIFF, etc.
• Data Query: The databases of the table model and tree model are invisible to each other, and users can choose the appropriate model based on their needs.
• Data Query: Users can control the loading of UDF, PipePlugin, Trigger, and AINode via URI with configuration items to load JAR packages.
• Storage Engine: The table model supports data ingestion through the Session interface, and the Session interface supports automatic metadata creation.
• Storage Engine: The Python client now supports four new data types: String, Blob, Date, and Timestamp.
• Storage Engine: The comparison rules for the priority of same-type merge tasks have been optimized.
• Data Synchronization: Support for specifying authentication information of the receiving end at the sending end.
• Stream Processing Module: TsFile Load now supports the table model.
• Stream Processing Module: Pipe now supports the table model.
• System Management: The Benchmark tool has been adapted to support the table model.
• System Management: The Benchmark tool now supports four new data types: String, Blob, Date, and Timestamp.
• System Management: The stability of DataNode scaling down has been enhanced.
• System Management: Users are now allowed to perform drop database operations in readonly mode.
• Scripts and Tools: The import-data/export-data scripts have been extended to support new data types (string, binary large objects, date, timestamp).
• Scripts and Tools: The import-data/export-data scripts have been iterated to support the import and export of three types of data: TsFile, CSV, and SQL.
• Ecosystem Integration: Support for Kubernetes Operator.
  ...

Bugs

• Fixed the issue where the query result set contained duplicate timestamps.
• Fixed the issue where deleted data could be queried again when triggered to merge after deletion.
• Fixed the issue where the target sequence in SELECT INTO containing backticks would result in writing the wrong sequence.
• Fixed the issue where an array out-of-bounds exception was thrown in the HAVING clause of the tree model due to a non-existent column name.
• Fixed the issue where MergeReader needed to consider memory allocation to avoid negative available memory during out-of-order and reverse queries.
• Fixed the issue where the CN in the cluster could not register large pipe plugins (greater than 100MB) and the parameters were not configurable.
• Fixed the issue of controlling the memory size of TimeIndex referenced by Pipe for TsFileResource.
• Fixed the issue where the Storage Engine - File Count - mods displayed negative values on the monitoring dashboard.
• Fixed the issue where the query result order was incorrect in the C# client.

...

Apache IoTDB 1.3.3

Features & Improvements

• Storage Engine: Added new data types String, Blob, Date, and Timestamp.
• Storage Engine: Multi-level storage has added a rate-limiting mechanism.
• Storage Engine: New merge target file splitting feature, with additional configuration file parameters, and improved memory control performance of the merge module.
• Data Query: Filter performance optimization, enhancing the speed of aggregate queries and where condition queries.
• Data Query: New client query requests load balancing optimization.
• Data Query: New active metadata statistics query added.
• Data Query: Optimized memory control strategy during the query planning phase.
• Data Synchronization: The sender supports transferring files to a specified directory, and the receiver automatically loads them into IoTDB.
• Data Synchronization: The receiver has a new automatic conversion mechanism for data type requests.
• Data Synchronization: Enhanced observability on the receiver side, supporting ops/latency statistics for multiple internal interfaces, consolidated into a single pipeTransfer display.
• Data Loading: DataNode actively listens and loads TsFiles, with additional observability metrics.
• Stream Processing Module: New data subscription capability, supporting subscription to database data in the form of data points or tsfile files.
• Stream Processing Module: Alter Pipe supports the ability to alter the source.
• System Management: Optimized configuration files, with the original three configuration files merged into one, reducing user operational costs.
• System Management: Optimized restart recovery performance, reducing startup time.
• System Management: Internal addition of monitoring items such as device count, estimated remaining time for data synchronization, size of data to be synchronized, and synchronization speed.
• Scripts and Tools: The import-tsfile script is expanded to support running the script on a different server from the IoTDB server.
• Scripts and Tools: New metadata import and export scripts added.
• Scripts and Tools: New support for Kubernetes Helm added.
• AINode: AINode module added.
  ...

Bugs

• Fixed the issue of NullPointerException (NPE) when merging chunks with modifications and empty pages in the sequential space.
• Fixed the issue where the wrong parent file was used when reassigning the file position for skipped files during merge, leading to failure in creating hard links.
• Fixed the issue where the newly added four data types had null values written, and the TsFile handling of the STRING type was incorrect, causing a BufferUnderflowException: null.
• Fixed the issue in the high availability scenario where stopping the DataNode resulted in a PipeException: Failed to start consensus pipe.
• Fixed the issue in Stream mode where the first batch of written data points might require a flush to be synchronized.
• Fixed the compatibility issue with pipe plugin upgrades.
• Fixed the issue where the ORDER BY clause became ineffective when used in combination with LIMIT in the last query.
  ...

New Contributors

• @alxkhi made their first contribution in #12296
• @SihanLiu2024 made their first contribution in #12323
• @supersshhhh made their first contribution in #12512
• @bdemers made their first contribution in #12971

Full Changelog: v1.3.2...v1.3.3

Apache IoTDB 1.3.2

Features & Improvements

• Storage Module: Performance improvement in the insertRecords interface for writing
• Query Module: New Explain Analyze statement added (monitoring the time spent on each stage of a single SQL execution)
• Query Module: New UDAF (User-Defined Aggregate Function) framework added
• Query Module: New MaxBy/MinBy functions added, supporting the retrieval of maximum/minimum values along with the corresponding timestamps
• Query Module: Performance improvement in value filtering queries
• Data Synchronization: Path matching supports path pattern
• Data Synchronization: Supports metadata synchronization (including time series and related attributes, permissions, etc.)
• Stream Processing: Added Alter Pipe statement, supporting hot updates of plugins for Pipe tasks
• System Module: System data point count statistics now include statistics for data imported by loading TsFile
• Scripts and Tools: New local upgrade backup tool added (backing up original data through hard links)
• Scripts and Tools: New export-data/import-data scripts added, supporting data export in CSV, TsFile formats or SQL statements
• Scripts and Tools: Windows environment now supports distinguishing ConfigNode, DataNode, and Cli by window name
  ...

Bugs

• Optimize the error message when a NullPointerException (NPE) occurs due to a timeout when dropping a database.
• Add logs for notifyLeaderReady, notifyLeaderChanged, and procedure worker.
• Add compatibility handling for existing erroneous data during file merging.
• Fix the deadlock issue caused by flushing empty files during querying.
• Fix the issue where Ratis becomes unresponsive during read, write, and delete operations.
• Fix the concurrent bug in load and merge operations.
• Fix the issue where the system's compression ratio is recorded as a negative number in the file for certain scenarios.
• Fix the ConcurrentModificationException issue during memory estimation for merge tasks.
• Fix potential deadlocks that may occur when writing, automatically creating, and deleting databases concurrently.
  ...

What's Changed

• Delete write mem control parameters by @l2280212 in #12007
• Pipe: Ensure at-least-once semantic by forcibly starting PipeHistoricalDataRegionTsFileExtractor after pipe restart by @Caideyipi in #12010
• Separate the synchronized block of AbstractCompactionEstimator file info cache by @shuwenwei in #12019
• Pipe: using clusterId to judge whether the target cluster is source cluster by @xuanronaldo in #11994
• Storage Engine: fix wal delete thread exception because multi thread safety question by @caozj1011 in #12020
• PBTree: Fix Dead Lock and Refactor write/update inteface by @bigreybear in #11985
• Remove sql merge by @shuwenwei in #12021
• Load: Trigger last flush time map & tsfile resource degrading after successful load by @SteveYurongSu in #12024
• feat(jdbc):getDouble add type check by @CritasWang in #12023
• Remove infinite retry logic for replica inconsistency to avoid potential problems by @OneSizeFitsQuorum in #12028
• Pipe: Support "sink.host" and "connector.host" key in receiver IP specification by @Caideyipi in #12017
• Fixed Ratis query not retrying when DataNode restarts by @OneSizeFitsQuorum in #12029
• Pipe: support replace and modify mode for alter pipe sql by @VGalaxies in #12018
• Storage Engine: put DeviceCache into the total memory of the storage engine by @caozj1011 in #12016
• Bump jackson.version from 2.15.2 to 2.15.3 by @dependabot in #12014
• IT framework supports restarting cluster by @l2280212 in #12022
• Fix typo for parameter dn_multi_dir_strategy by @OneSizeFitsQuorum in #12033
• Upgrade github action version to fix the warning by @HTHou in #12032
• Introduce UDAF(User-Defined Aggregation Functions) by @Sh-Zh-7 in #12005
• Pipe: Move parsing event logic into PipeProcessorSubtask to separate CPU intensive and IO intensive tasks by @DanielWang2035 in #11965
• DataNodeDevicePathCache should use free memory of memtable by @HTHou in #12034
• Pipe: Revert "Pipe: stop pipe using restarting strategy to unpin the wal's reference count to avoid WAL stacking (#11971)" to avoid unnecessary pipe drop during subtask exception handling by @Caideyipi in #12031
• Pipe: Removed mayLinkedTsFileCountReachDangerousThreshold judgement from stuck restart logic by @Caideyipi in #12041
• Pipe: fixed the bug that processor dies when encountered memory shortage for tablets in a period of time by @Caideyipi in #12042
• Pipe: Reduced pipe logs and controlled how often the log is printed below a certain frequency by @Caideyipi in #11973
• [IOTDB-6296] Fix memory leak in MQTTService by @JackieTien97 in #12046
• Pipe: Pattern parsing pruning: When pattern is at the level below database, the parsing logic can be skipped if tsfiles / tablets completely match with the pattern by @Caideyipi in #12049
• Bump org.apache.commons:commons-compress from 1.21 to 1.26.0 by @dependabot in #12058
• Bump ip from 2.0.0 to 2.0.1 in /iotdb-connector/grafana-plugin by @dependabot in #12057
• Pipe: fixed potential lose point bug caused by cancelled flush of historical extractor by @Caideyipi in #12056
• Fix it which use 'merge' command by @shuwenwei in #12060
• Pipe: Fixed the bug that CN cannot sense drop pipe failure in meta sync and may lead to constantly skip of drop pipe by @Caideyipi in #12059
• [IOTDB-6297] Optimize the distribute plan in the situation of aggregation with align by device by @Beyyes in #12043
• [IOTDB-6298] Fix number overflow in group by time interval by @JackieTien97 in #12065
• [IOTDB-6299] Fix bug in merging overlapped data process caused by filter & offset push down by @liuminghui233 in #12068
• Introduce Aggregation Function MinBy by @lancelly in #12071
• Add recover IT for Procedure, and delete ProcedureStore by @liyuheng55555 in #12045
• PBTree: Implement dual-buffer container for MNode management by @linxt20 in #12048
• Optimize flushing memtable check by @HTHou in #12081
• [IOTDB-6302] Enhance the support of ISO_LOCAL_DATE_TIME timestamp format by @HTHou in #12086
• IT: revert some changes in restart IT to make the error log clear when failed to restart by @l2280212 in #12066
• change ci to github ci by @CritasWang in #12073
• feat(bat): add title and set format by @CritasWang in #12069
• Fix dead lock in PBTree.SchemaFile by @bigreybear in #12079
• fix.root privileges lost and grant role. by @ColinLeeo in #12091
• [IOTDB-6300] Support place time column at any column index in insert statement by @JackieTien97 in #12074
• Make Drop database timeout return message more detailed by @Cpaulyz in #12082
• Fix VolatileSubtreeIterator missing update buffer if all nodes in new buffer are not qualified by @Cpaulyz in #12093
• Pbtree: MNode iterating with merge sort upon disk and buffer by @linxt20 in #12077
• Add FileUtils.moveFileSafe function by @liyuheng55555 in #12094
• Add logs for data partition allocation by @CRZbulabula in #12090
• fix: IoTDBPreparedStatement implement addBatch function by @CritasWang in #12095
• [IOTDB-6303] Correct the default path of user, role and procedure folder by @CRZbulabula in #12089
• Cache clusterId in IoTDBConfig by @liyuheng55555 in #12053
• Implement new plan optimizer: PredicatePushDown by @liuminghui233 in #11859
• Optimize ConfigNode ConsensusManager init logic by @OneSizeFitsQuorum in #12098
• Do not wait to retry when configLeader exists by @Pengzna in #12075
• Faster repair data by @shuwenwei in #12039
• Bump org.scala-lang:scala-library from 2.12.18 to 2.12.19 by @dependabot in #12104
• Bump flink.version from 1.17.0 to 1.17.2 by @dependabot in #12103
• [IOTDB-6305] Add show current_timestamp statement by @yx-zhang in #12106
• [IOTDB-6306] Fix t...

Apache IoTDB 1.3.1

Features & Improvements

• Add cluster script for one-click start-stop (start-all/stop-all.sh & start-all/stop-all.bat)
• Add script for one-click instance information collection (collect-info.sh & collect-info.bat)
• Add new statistical aggregators include stddev and variance
• Add repair tsfile data command
• Support setting timeout threshold for Fill clause. When time beyond the threshold, do not fill value.
• Simplify the time range specification for data synchronization, directly set start and end times
• Improved system observability (adding dispersion monitoring of cluster nodes, observability of distributed task scheduling framework)
• Optimized default log output strategy
• Enhance memory control for Load TsFile, covering the entire process
• Rest API (Version 2) adds column type return.
• Improve the process of query execution
• Session automatically fetch all available DataNodes
  ...

Bugs

• Fix issue with abnormal behavior when time precision is not in milliseconds during grouping by month.
• Fix issue with abnormal behavior when duration contains multiple units during grouping by month.
• Fix bug where limit and offset cannot be pushed down when there is an order by clause.
• Fix abnormal behavior in combination scenario of grouping by month + align by device + limit.
• Fix deserialization errors during IoT protocol synchronization.
• Fix concurrent exceptions in deleting timeseries.
• Fix issue where group by level in view sequences does not execute correctly.
• Fix potential issue of metadata creation failure when increasing election timeout

New Contributors

• @Jgj1006 made their first contribution in #11730

Full Changelog: v1.3.0...v1.3.1

Apache IoTDB 1.3.0

Features & Improvements

• Optimize the permission module and support timeseries permission control
• Optimize heap and off-heap memory configuration in startup script
• Computed-type view timeseries support LAST queries
• Add pipe-related monitoring metrics
• Pipe rename 'Extractor' to 'Source' and 'Connector' to 'Sink'
• [IOTDB-6138] Support Negative timestamp
• [IOTDB-6193] Reject Node startup when loading configuration file failed
• [IOTDB-6194] Rename target_config_node_list to seed_config_node
• [IOTDB-6200] Change schema template to device template
• [IOTDB-6207] Add Write Point Metrics for load function
• [IOTDB-6208] Node error detection through broken thrift pipe
• [IOTDB-6217] When the number of time series reaches the upper limit, the prompt message should be changed to prioritize using device templates
• [IOTDB-6218] Rename storage_query_schema_consensus_free_memory_proportion to datanode_memory_proportion
• [IOTDB-6219] Fix the display problem of explain that the print result is not aligned
• [IOTDB-6220] Pipe: Add check logic to avoid self-transmission
• [IOTDB-6222] Optimize the performance of Python client
• [IOTDB-6230] Add HEAPDUMP configuration in datanode-env.sh
• [IOTDB-6231]SchemaCache supports precise eviction
• [IOTDB-6232] Adding SSL function to dn_rpc_port

Bugs

• [IOTDB-6160] while using ` in target path, select into will throw error
• [IOTDB-6167] DataNode can't register to cluster when fetch system configuration throws NPE
• [IOTDB-6168] ConfigNode register retry logic does not worked
• [IOTDB-6171] NPE will be thrown while printing FI with debug on
• [IOTDB-6184] Merge Sort finishes one iterator too long
• [IOTDB-6191] Fix group by year not considering leap years
• [IOTDB-6226] Fix the problem of inaccurate GC monitor detection at the beginning and adjusting the alert threshold
• [IOTDB-6239] Show regions display error create time

New Contributors

• @clayburn made their first contribution in #11243
• @chunshao90 made their first contribution in #11385

Full Changelog: v1.2.2...v1.3.0