New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Clickhouse offline store support #1224
Merged
ahmadnazeri
merged 64 commits into
featureform:feature/clickhouse_provider
from
gingerwizard:clickhouse_store
Jan 9, 2024
Merged
Clickhouse offline store support #1224
ahmadnazeri
merged 64 commits into
featureform:feature/clickhouse_provider
from
gingerwizard:clickhouse_store
Jan 9, 2024
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Co-authored-by: sdreyer <sterling@featureform.com>
Co-authored-by: sdreyer <sterling@featureform.com>
…1193) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: sdreyer <sterling@featureform.com>
… cli. (featureform#962) Co-authored-by: sdreyer <sterling@featureform.com>
ahmadnazeri
reviewed
Dec 20, 2023
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good for the most part! Left some comments on the PR. There are two parts missing:
- End to End Behave Tests
- Having instant of Clickhouse available for CICD like we do for Postgres
@ahmadnazeri i believe ive resolved all issues and added ci/cd tests + support for batch features. This should be ready to test. |
epps
reviewed
Jan 8, 2024
epps
reviewed
Jan 8, 2024
@epps addressed comments |
ahmadnazeri
approved these changes
Jan 9, 2024
ahmadnazeri
added a commit
that referenced
this pull request
Jan 16, 2024
Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Dale McDiarmid <dale@clickhouse.com> Co-authored-by: Ali Olfat <ali@featureform.com> Co-authored-by: sdreyer <sterling@featureform.com> Co-authored-by: Erik Eppel <erik@featureform.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: anthonylasso <34227093+anthonylasso@users.noreply.github.com>
github-actions bot
pushed a commit
that referenced
this pull request
Jan 16, 2024
Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Dale McDiarmid <dale@clickhouse.com> Co-authored-by: Ali Olfat <ali@featureform.com> Co-authored-by: sdreyer <sterling@featureform.com> Co-authored-by: Erik Eppel <erik@featureform.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: anthonylasso <34227093+anthonylasso@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Support for ClickHouse as offline store. Closes #1160
Type of change
This adds support for ClickHouse as an offline store. While CH could also be used as online store (and vector db) this is the main functional usecase.
I've made a few key design decisions/questions:
writeUpdate
, which according to sql.go base provider is always done byentity
andts
. For this table we therefore use a ReplacingMergeTree - inserts with the same entity and ts will be de-duplicated (latest preserved) at insert and query time (final is set on all queries).entity, ts
as the ordering key on any offline tables. This key combination is also required to uniquely identify rows in our ReplacingMergeTree. Thets
column can be null though - this means i need to useallow_nullable_key
for the table - this is generally not recommended but probably ok for this usecase.Its not clear to me what a materialization is vs an offline table i.e.
OfflineTable
. The former i assume is when a feature is represented and "materialized" into the store. OfflineTable looks the same structurally -entity
,ts
,value
columns.The above should be fine provided materializations cant recieve upserts like offline tables. If they are, we can make these tables a ReplacingMergeTree also.
A few other things:
materializationCreate
andtransformationCreate
have been modified to return []string vs string. We can't do these in a single statement and don't support multiple statements in a single query. I've updated other offline stores and discussed with @ahmadnazeriDoes this correspond to an open issue?
#1160
Select type(s) of change
Checklist: