Skip to content
This repository has been archived by the owner on Jul 31, 2023. It is now read-only.

Improve query engine state management #19

Open
tylerwilding-wk opened this issue Jul 4, 2019 · 0 comments
Open

Improve query engine state management #19

tylerwilding-wk opened this issue Jul 4, 2019 · 0 comments
Labels
enhancement New feature or request query engine

Comments

@tylerwilding-wk
Copy link
Contributor

There are some notable inefficiencies in the internal representations for query state. Several naive representations were experimented with, all of which had the potential to dominate execution time far beyond IO costs. However, there exist well-defined solutions that could be implemented to drastically reduce overall runtime of the query engine under many or most circumstances.

In particular, there are two elements of state that need improvement:

  1. The manner in which variable bindings are stored eagerly evaluates all cross products, requiring repeated linear traversals over potentially tens or hundreds of thousands of immutable maps.
  2. The manner in which the query engine tracks whether or not it has already evaluated a particular predicate.

To address (1), we intend to use or build a lazy relational algebra system - https://github.com/Workiva/lazy-tables

To address (2), we intend to explore feasible alternatives and choose one/more to implement.

The expectation is that we could drastically improve the query engine performance and possibly simplify several code paths.

@tylerwilding-wk tylerwilding-wk added enhancement New feature or request query engine labels Jul 4, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement New feature or request query engine
Projects
None yet
Development

No branches or pull requests

1 participant