Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Convert temporal column and refresh period to datafusion expr #1187

Merged
merged 19 commits into from
Apr 25, 2024

Conversation

y-f-u
Copy link
Contributor

@y-f-u y-f-u commented Apr 24, 2024

Add a timestamp converter to simplify the logic.
Convert the refresh period logic to expr.

There will be following PR to enable the filter when fetching the data.

Fixes #1175

Edit:
With the merge of #1188 . This allows the period to work properly

  - from: s3://spiceai-demo-datasets/taxi_trips/2024/
    name: taxi_trips
    time_column: tpep_pickup_datetime
    acceleration:
      enabled: true
      refresh_interval: 10m
      refresh_period: 1000w
      refresh_sql: |
        SELECT * FROM taxi_trips WHERE passenger_count = 1;

@y-f-u y-f-u requested a review from a team as a code owner April 24, 2024 00:04
@y-f-u y-f-u self-assigned this Apr 24, 2024
sgrebnov
sgrebnov previously approved these changes Apr 24, 2024
@phillipleblanc phillipleblanc merged commit 3036285 into trunk Apr 25, 2024
16 checks passed
@phillipleblanc phillipleblanc deleted the transalate-refresh-period-to-sql branch April 25, 2024 01:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Generate sql filter based on dataset schema and temporal column and format for refresh period
4 participants