Skip to content

21.05-1.17.0

Choose a tag to compare

View all tags
@jarohen jarohen released this 28 May 16:52
21.05-1.17.0
This tag was signed with the committer鈥檚 verified signature.
jarohen James Henderson
GPG key ID: 956F20CA2105F582
Verified
Learn about vigilant mode.
a20acc7
This commit was signed with the committer鈥檚 verified signature.
jarohen James Henderson
GPG key ID: 956F20CA2105F582
Verified
Learn about vigilant mode.

Hey folks, here's 1.17.0! 馃殌 Highlights include:

  • We now keep more index statistics to help with choosing the right query plan - in certain cases with many filters this can change the query plan (and hence the query time) significantly. This change requires an 'index version bump' - see below for more details.

  • JDBC users are advised to migrate the tx_events index to remove the compacted column - see below.

  • There's one breaking change: the removal of the previously deprecated :full-results? flag:

    {:find [e]
 :where [...]
 :full-results? true}

;; =>

{:find [(pull e [*])]
 :where [...]}

  • The usual array of bugfixes and performance improvements.

Query plan improvements (requires index version bump)

When you query Crux, you don't have to worry about the order of your :where clauses - Crux checks its index statistics and picks a query plan for you. In this release, we've added more of these statistics so that we can better understand the shape of your data, and hence pick better plans.

In this release, we've added HyperLogLog statistics, which can efficiently approximate the amount of distinct values in a large set. We then use these statistics to make better choices when your queries have filters on multiple attributes. Given a choice, we choose to filter first by attributes with a higher proportion of distinct elements (or higher 'selectivity') so that we can reduce the size of intermediate query results as much as possible as early as possible.

As a result, this release contains an 'index version bump'. This means that you'll need to clear your Crux query indices, and re-index from the transaction log.

In green/blue production settings, we recommend doing this by starting a new cluster of Crux nodes, waiting for them to catch up with the tx-log, switching over, and decommissioning the old nodes. If you've got any questions/concerns about this, please do get in touch via crux@juxt.pro - we're happy to help out.

JDBC - Required migration (#1463, PR #1504)

Prior to 1.17.0, Crux JDBC created an index on its tx_events table based on (compacted, event_key). The queries that previously accessed this no longer filter on the compacted key, meaning that the index wasn't being used, and ingestion times were negatively affected.

Crux 1.17.0 nodes will attempt to remove this index (tx_events_event_key_idx) on startup if it is present, and idempotently create a more suitable index (tx_events_event_key_idx_2), based on event_key alone. For most databases, this is a table-locking DDL migration, which may mean that the Crux cluster is unavailable for transactions while the index is being created.

If high availability is required, it's advisable for Crux users to run a non-locking variant of this migration offline, at which point Crux will not need to perform its own migration. For example:

  • In PostgreSQL, DROP INDEX tx_events_event_key_idx; CREATE INDEX CONCURRENTLY tx_events_event_key_idx_2 ON tx_events(event_key);
  • In MySQL 5.6+, the index DDL statements do not take table locks.
  • In MSSQL, DROP INDEX tx_events.tx_events_event_key_idx; CREATE INDEX tx_events_event_key_idx_2 ON tx_events(event_key) WITH (ONLINE = ON);

Thanks to Mateusz Probachta and Tijs Mallaerts for reporting this one!

Elsewhere in this release:

  • (#1519) Fix NodeOutOfSyncException when submitting an empty tx - thanks @jacobobryant!
  • (#1398) Can now use vanilla java.util.Maps in documents - thanks @AlistairONeill
  • (#1462) Can now start Crux on JDK 16 - we recommend adding --add-opens java.base/java.util.concurrent=ALL-UNNAMED to preserve JDK <16 cache performance. Thanks @nivekuil!
  • (#1513) Kafka transit now round-tripping #crux/ids correctly.
  • (#1506) Remote API clients now support :in parameters.
  • (#1479) Kafka doc-store now fails fast if the requested document isn't present.
  • (#960) Transaction function argument docs in aborted transactions are fully evictable.
  • (#1469) Compact the second-chance cache to avoid scanning through a large empty map.
  • (#1503) Fix metrics NPE on startup - thanks again @nivekuil!

As always, a big thanks to everyone who's contributed to this release by submitting PRs, raising issues and helping with repros!

Cheers,

Crux Team

Assets 5
Loading