4.2.0

SUMMARY

We have implemented SIMD-based vector execution of queries, such as

select sum(value) from table

This is around 100x faster than non-vector based execution. This is just the beginning. We are vectorising more operations.

Try our first implementation in this release and stay tuned for more features in the upcoming releases!

IMPORTANT

Metadata file format has been changed to include a new flag for columns of type symbol. It is necessary to convert existing tables to new format. Running the following sql: repair table myTable, will update the table metadata.

What is new?

• Java: vectorized sum(), avg(), min(), max() for DOUBLE, LONG, INT
• Java: select distinct symbol optimisation
• FreeBSD support
• Automatically restore data consistency and recover from partial data loss.

What we fixed

• SQL: NPE when parsing SQL text with malformed table name expression , for example ')', or ', blah'
• SQL: parsing 'fill' clause in sub-query context was causing unexpected syntax error (#115)
• SQL: possible internal error when ordering result of group-by or sample-by
• Data Import: Ignore byte order marks (BOM) in table names created from an imported CSV (#114)
• SQL: 'timestamp' propagation thru group-by code had issues. sum() was tripping over null values. Added last() aggregate function. (#113)
• LOG: make service log names consistent on windows (#106)
• SQL: deal with the following syntax 'select * from select ( select a from ....)'
• SQL: allow the following syntax 'case cast(x as int) when 1 then ...'
  fix(griffin): syntax check for "case"-')' overlap, e.g. "a + (case when .. ) end"

4.1.5

IMPORTANT

Metadata file format has been changed to include version and more column flags. It is necessary to convert existing tables to new format. For more information please follow this guide.

What is new?

• Java: TableWriter.addIndex() implementation - adds index to existing table
• SQL: CASE statement implementation
• SQL: CAST implementation (documentation)
• SQL: RENAME TABLE implementation
• SQL: round() multiple implementations (documentation)
• SQL: ALTER TABLE X ALTER COLUMN SYM ADD INDEX - SQL to add index to SYMBOL columns of existing tables (documentation)
• SQL: symbol in (list) function implementation
• SQL: optimiser improvements
• UI: Web Console has simple Charts implementation

select * from trades where ts>'2015-01-02T00:00:00.000Z' and ts<'2015-01-02T10:30:00.000Z' and sym in ('EEM', 'VXX', 'BCLI', ...) order by sym

What we fixed

• SQL: double-to-string implementation parses IEEE number, performance is up 80% on simpler values and 30% on complex ones

• SQL: aggregation without key, e.g. no 'group by' clause are 6x faster than before.

• SQL: 'SAMPLE BY' queries are faster than before

• SQL: 'COPY' is limited to picking up files only from configured location (documentation)

• SQL: improved error reporting

• UI: LONG256 0 is printed as '0x00'

• UI: Double "inifinity" values (division by zero) used to corrupt JSON

4.0.4

Release highlights

• production ready version of Influx Line Protocol Receiver. It works well with existing line protocol publishers and is much faster than Influx's own implementation. Refer to this how-to

• choice of transaction durability

What's new ?

• GRIFFIN: 'asof' join performs much faster
• GRIFFIN: systimestamp() SQL function
• line protocol receiver supports abbreviated Boolean values, such as 'T', 'F' and capitalised TRUE and FALSE
• line protocol receiver 30% performance boost
• line protocol receiver supports all of InfluxDB's timestamp formats
• CAIRO: durable commits, support for NOSYNC, ASYNC and SYNC as documented [here]
• HTTP: improved performance of JSON HTTP end point by 100% - SQL results are serialised and delivered to browser faster.
• Web Console: grid headers can be clicked to help construction of column list in 'select' clause

What's fixed ?

• line protocol receiver could fail on field values that have space
• GRIFFIN: INSERT statement could have incorrectly referenced column names of the target table and fail uncontrollably
• GRIFFIN: column name in timestamp() clause wasn't validate leading to uncontrollable failure
• CAIRO: UTF8 encoding on Windows. Although QuestDB was unaffected, directory names containing UTF8 characters were garbled.
• Web Console: highlighted more SQL keywords
• Web Console: removed unfinished "visualisation" tab
• Web Console: trim SQL statement before sending to server. Cleaner logging on server.
• HTTP: headers are now case insensitive
• HTTP: NPE when 'content-disposition' header is missing
• improved hash function distribution leading to overall performance boost by 10%
• windows docker image is smaller. Reduced the size of JRE package

Download

Via Docker or cross-platform binary below:

4.0.3

What we fixed?

• removed accidental garbage generation via lambdas in PG wire implementation
• reduced size of binaries
• error handling from INSERT statement execution over PG wire
• OS error reporting out of network interface
• optimised ASOF join for non-keyed joins
• cached SQL plan re-evaluation when table structure changes

What's new?

• Influx UDP receiver can either use shared threads or its own busy thread via configuration

4.0.2

What we fixed?

• import single column text file #91
• 'insert' bugfixes
• switched to parallel GC to improve performance across the board. CMS we used to use, although not running explicitly, was slowing down our code.
• NPE in TableReader under some conditions

What is new?

• Influx line-protocol receiver is started by default when server starts
• 'alter table drop partition' implementation
• to_long(TIMESTAMP) implementation (thanks to clickingbuttons)
• 'insert' implementation over PostgreSQL wire

4.0.1

What's new?

• docker files for Windows, Linux and ARM64 Linux

What did we fix?

• compatibility with Java 11
• #90
• print architecture ID as recognised by application
• #84
• ANSI SQL compatible count(*) support
• #88

4.0.0

Welcome to QuestDB 4.0.0 Release Notes

Before you continue, some keywords we used and their meaning:

CAIRO - storage engine
GRIFFIN - SQL engine
STD - QuestDB standard libraries

Whats new?

• PostgresSQL wire server implementation (BETA)

• GRIFFIN: new algo for sample by .. fill none without key. It executes a little faster than generic algo.

• GRIFFIN: order by implementation for cursors without random access to rows.

• GRIFFIN: group by implementation

• GRIFFIN: support multiple columns in latest by clause

• GRIFFIN: renamed to_char for date and timestamp to to_str to make them easier to find

• GRIFFIN: optimised and and or operators to recognise constant conditions better and avoid unnecessary data processing.

• GRIFFIN: = implementation for SYMBOL = CHAR and STRING = CHAR

• GRIFFIN: 'sum' performance optimisation

• Network: ability to set multicast TTL and bind port for unicast

• GRIFFIN: "insert into ... values" implementation

• GRIFFIN: implementation of 'select count() from table' that would use result of base.size() when its available. This speed up exploratory 'count()' queries dramatically

• GRIFFIN: UNION and UNION ALL implementations and tests

• GRIFFIN: "splice join" implementation

• GRIFFIN: alter table drop column SQL implementation

• GRIFFIN: SQL limit implementation

• GRIFFIN: INNER, OUTER, AS OF join implementation

• CAIRO: "remove partition" API on TableWriter

• CAIRO: new LONG256 data type

• STD: ARM64 Linux support and binary

• HTTP server implementation. The server is able to send and receive unlimited amount of data from and to data store.

• STD: Os.currentTimeNanos() system clock in nanosecond resolution (not to be confused with System.nanoTime())

• STD: (network) set/get TCP_NODELAY

• STD: (network) implementation of interrupt for blocking accept()

• STD: (network) bridge to native IP_MULTICAST_LOOP and IP_MULTICAST_IF

• STD: (network) connect() and configureNoLinger() bridges

• STD: (network) SO_REUSEADDR and SO_REUSEPORT bridges

• STD: (log) implemented log writer that can roll files on size and/or date.

• LineProtoSender can now be bound to a specific interface. It also receives IP addresses as inti n order to offload dealing with address parsing errors somewhere else.

• STD: (threading) available() method is made available only on SP- and SC- sequences. available() is new to SPSequence and allows bulk publishing and enables higher queue throughput STD: Worker can have CPU affinity now

Whats changed?

• new top level package name! This is a breaking change. We wanted to make package name consistent with our product web site.

What did we fix?

• logging OS error code whenever close(fd) fails.
• null for symbol value now corresponds to null for int. This removes unnecessary complexity around implementing getInt()
• fixed transactional behaviour of TableWriter.truncate(). TableReader will now process this event correctly.
• CAIRO: TableWriter maintains minTimestamp so that TableReader doesn't have to count directories to determine data interval
• CAIRO: TableReader no longer relies on partition directories to be removed in order to remove logical data chunk. This is necessary for Windows where open files cannot be removed.
• CAIRO: ByteSequence could allow reading beyond the size of the blob.
• get/set lastModified on Windows could set incorrect time
• fixed TextImportProcessor resume function
• Files.exists() does not use Files.getLastModified() anymore
• network send() and recv() on Linux did not handle errors correctly, which sometimes results in broken wire protocols.
• Numbers.parseInt() could overflow without throwing exception
• Removed garbage-generating lambdas.
• CAIRO: SymbolTable reload could produce an exception

QuestDB 3.0.0

Beta release of new storage system dubbed CAIRO and new query system GRIFFIN.

CAIRO improves on now legacy system in several key areas:

1. Performance. It is at least 30% faster in every test.
2. Resilience. Cairo has been tested to withstand file system errors, such as corruption, space issue as well deal hardware stability issues, such as power loss or sudden reboots. Cairo will perform automatic data recovery in all of these circumstances
3. Data management. Cairo implements functions of adding and removing table columns and partition management.
4. Stability. Cairo is production ready. Data structures have been extensively tested to resolve all known defects.

Downsides:

1. Object store and retrieval is not implemented against CAIRO. The system focus is on zero-GC embedded storage.

GRIFFIN - SQL implementation that uses CAIRO as the store.

This system is under active development. For now lacks features of legacy SQL, such as joins, group-by and analytic functions - these are not yet developed. However GRIFFIN provides sample-by implementation with various missing data fill options, including linear interpolation and more extensive language basics, such as select from function. Eventually GRIFFIN will implement all features of legacy SQL and well beyond that.

GRIFFIN also is not yet linked to HTTP server or any network service.

QuestDB 2.0.1

Release notes

• BUGFIX: microsecond timer also works on pre-Windows 8 systems
• BUGFIX: race condition in CAIRO WriterPool
• BUGFIX: microsecond time returned incorrect time
• OS-specific binary is loaded implicitly when required
• CAIRO engine performance improvement

QuestDB 2.0.0

This release introduces new storage system, CAIRO. I am writing it to run in parallel with legacy storage and keep existing functionality completely unaffected. CAIRO is available to try if you look for any of the following:

• easy DDL: add/remove column, create/rename/drop table etc. These operations are either possible or much faster than before.
• dedicated writer and reader implementations. Writer reuses resources very aggressively, caches very little and is double the speed of legacy implementation
• CAIRO is tested for every possible outcome of file system operation, it does not corrupt data under any circumstances, attempts to recover automatically and concisely reports errors
• commits are much faster, there is very little performance difference between committing batches and every record
• i18n support out of box, table names and column names accept UTF8 encoded characters

CAIRO is not yet integrated with neither HTTP server nor SQL. These are the things to come.

Also included in this release CUTLASS: UDP receiver for Influx line protocol. Everything broadcast for Influx can be easily stored in QuestDB. CUTLASS supports on-the-fly table definition and column addition and it stores data in CAIRO.

Stay tuned for more exciting releases ;-)