Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add refresh_sql to dataset acceleration #1073

Merged
merged 57 commits into from Apr 12, 2024
Merged

Add refresh_sql to dataset acceleration #1073

merged 57 commits into from Apr 12, 2024

Conversation

lukekim
Copy link
Contributor

@lukekim lukekim commented Apr 10, 2024 •
edited by phillipleblanc

Part of #1056

Add a refresh_sql that will be evaluated when refreshing the accelerated table to filter the data that is accelerated.

datasets:
  - from: spice.ai/eth.recent_blocks
    name: eth_recent_blocks
    acceleration:
       enabled: true
       refresh_mode: full
       refresh_interval: 1m
       refresh_sql: |
          SELECT * FROM eth_recent_blocks WHERE number > 15000000

phillipleblanc and others added 30 commits March 21, 2024 12:47
@phillipleblanc
Reorganizes the DataConnector registry
94e8dde
@phillipleblanc
rename
f9e680a
@phillipleblanc
concurrent
0808c75
@phillipleblanc
more explicit
a0c1e82
@phillipleblanc
Merge branch 'trunk' into phillip/240320-rearch
c03f69a
@phillipleblanc
wip
412a91e
@phillipleblanc
implement Read, Write for Databricks
03bfc2e
@phillipleblanc
wip
c239e4d
@phillipleblanc
wip
f7e09cd
@phillipleblanc
Merge branch 'trunk' into phillip/240320-rearch-2
6bd6fc9
@phillipleblanc
wip
f42e603
@phillipleblanc
Merge branch 'trunk' into phillip/240320-rearch-2
edb34f9
@phillipleblanc
wip
f869d02
@phillipleblanc
Merge branch 'trunk' into phillip/240320-rearch-2
454cac0
@phillipleblanc
wip
f370202
@phillipleblanc
wip
2282a1b
@phillipleblanc
wip
927e90a
@phillipleblanc
fix
b9828a1
@phillipleblanc
wip
2031b03
@phillipleblanc
wip
5ea2803
@phillipleblanc
wip
2977fbb
@phillipleblanc
Implement insert_into for the AcceleratedTable
cc62d88
@phillipleblanc
wip
12e5814
@phillipleblanc
Rework building the data accelerator
24be099
@lukekim
Merge branch 'trunk' into phillip/240320-rearch-2
21320cc
@phillipleblanc
wip
f321f06
@phillipleblanc
mut FlightClient
b64c198
@phillipleblanc
Merge branch 'trunk' into phillip/240320-rearch-2
9fc3449
@phillipleblanc
DuckDB Accelerator
41f1396
@phillipleblanc
Fix feature flag
efccbc1
@lukekim lukekim requested a review from a team as a code owner April 10, 2024 22:49
@lukekim lukekim self-assigned this Apr 10, 2024
@lukekim lukekim marked this pull request as draft April 10, 2024 22:50
@lukekim lukekim added the kind/enhancement New feature or request label Apr 10, 2024
@lukekim lukekim added this to the v0.11-alpha milestone Apr 10, 2024
@lukekim lukekim linked an issue Apr 10, 2024 that may be closed by this pull request
Enhancement: Custom refresh SQL #1056
Closed
1 task
Base automatically changed from phillip/240320-rearch-2 to trunk April 11, 2024 03:31
@phillipleblanc phillipleblanc marked this pull request as ready for review April 11, 2024 15:17
phillipleblanc
phillipleblanc previously approved these changes Apr 11, 2024
View reviewed changes
@lukekim lukekim enabled auto-merge (squash) April 12, 2024 02:28
@lukekim lukekim merged commit 9438e29 into trunk Apr 12, 2024
16 checks passed
@lukekim lukekim deleted the lukim/refresh-sql-2 branch April 12, 2024 02:32
@phillipleblanc phillipleblanc mentioned this pull request Apr 12, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@phillipleblanc phillipleblanc phillipleblanc approved these changes

@digadeesh digadeesh Awaiting requested review from digadeesh

Assignees

@lukekim lukekim

Labels
kind/enhancement New feature or request
Projects
None yet
Milestone
v0.11-alpha
Development

Successfully merging this pull request may close these issues.

Enhancement: Custom refresh SQL
3 participants
@lukekim @phillipleblanc @digadeesh