-
Notifications
You must be signed in to change notification settings - Fork 948
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Contrib azure provider with synapse/mssql offline store and Azure registry store #3072
Merged
Merged
Changes from 46 commits
Commits
Show all changes
51 commits
Select commit
Hold shift + click to select a range
e1e210d
Broken state
kevjumba 011d1e0
working state
kevjumba a6a2fce
Fix the lint issues
kevjumba 57b63bb
Semi working state
kevjumba ae7ed8a
Fix
kevjumba 421645b
Fremove print
kevjumba 07fece5
Fix lint
kevjumba 4062031
Run build-sphinx
kevjumba cb39329
Add tutorials
kevjumba 554ca1a
Fix
kevjumba 4a969e7
Fix?
kevjumba 116320a
Fix lint
kevjumba c0b16ef
Fix
kevjumba 44d09d0
Fix lint
kevjumba b6f0a79
Begin configuring tests
adchia 2b2ff40
Fix
kevjumba 4616366
Working version
kevjumba c7d9852
Fix
kevjumba d2e290b
Fix
kevjumba a726a9a
Fix
kevjumba 32992e3
Fix lint
kevjumba ebb934b
Fix lint
kevjumba e456acb
Fix
kevjumba 45f479f
Fix lint
kevjumba 4b8c4a2
Fix
kevjumba b1bf602
Fix
kevjumba 4586f00
Fix azure
kevjumba 3b88c0b
Fix
kevjumba 9ae8ee3
Fix
kevjumba 1b12e4a
Fix lint and address issues
kevjumba 0ca5048
Fix integration tests
kevjumba 883f314
Fix
kevjumba ccf8716
Fix lint and address issues
kevjumba f05288e
Fix
kevjumba ee30e73
Fix
kevjumba ab17db9
Fix
kevjumba be162f5
Revert
kevjumba f5aa476
Fix
kevjumba 4423dfa
Fix
kevjumba 5806507
Fix
kevjumba 7a4d055
Fix lint
kevjumba 78b74b1
Fix
kevjumba a9e8119
Fix lint
kevjumba 1341e3e
Fix pyarrow
kevjumba 3d42093
Fix lint
kevjumba 1c591f0
add requirements files
adchia b4da607
fix name of docs
adchia c3a0423
fix offline store readme
adchia 576b57e
fix offline store readme
adchia 69940ac
fix
adchia 516ff76
fix
adchia File filter
Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,29 @@ | ||
# MsSQL source (contrib) | ||
|
||
## Description | ||
|
||
MsSQL data sources are Microsoft sql table sources. | ||
These can be specified either by a table reference or a SQL query. | ||
|
||
## Disclaimer | ||
|
||
The MsSQL data source does not achieve full test coverage. | ||
Please do not assume complete stability. | ||
|
||
## Examples | ||
|
||
Defining a MsSQL source: | ||
|
||
```python | ||
from feast.infra.offline_stores.contrib.mssql_offline_store.mssqlserver_source import ( | ||
MsSqlServerSource, | ||
) | ||
|
||
driver_hourly_table = "driver_hourly" | ||
|
||
driver_source = MsSqlServerSource( | ||
table_ref=driver_hourly_table, | ||
event_timestamp_column="datetime", | ||
created_timestamp_column="created", | ||
) | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,59 @@ | ||
# MsSQL/Synapse offline store (contrib) | ||
|
||
## Description | ||
|
||
The MsSQL offline store provides support for reading [MsSQL Sources](../data-sources/mssql.md). Specifically, it is developed to read from [Synapse SQL](https://docs.microsoft.com/en-us/azure/synapse-analytics/sql/overview-features) on Microsoft Azure | ||
|
||
* Entity dataframes can be provided as a SQL query or can be provided as a Pandas dataframe. | ||
|
||
## Disclaimer | ||
|
||
The MsSQL offline store does not achieve full test coverage. | ||
Please do not assume complete stability. | ||
|
||
## Example | ||
|
||
{% code title="feature_store.yaml" %} | ||
```yaml | ||
registry: | ||
registry_store_type: AzureRegistryStore | ||
path: ${REGISTRY_PATH} # Environment Variable | ||
project: production | ||
provider: azure | ||
online_store: | ||
type: redis | ||
connection_string: ${REDIS_CONN} # Environment Variable | ||
offline_store: | ||
type: mssql | ||
connection_string: ${SQL_CONN} # Environment Variable | ||
``` | ||
{% endcode %} | ||
|
||
## Functionality Matrix | ||
|
||
The set of functionality supported by offline stores is described in detail [here](overview.md#functionality). | ||
Below is a matrix indicating which functionality is supported by the Spark offline store. | ||
|
||
| | MsSql | | ||
| :-------------------------------- | :-- | | ||
| `get_historical_features` (point-in-time correct join) | yes | | ||
| `pull_latest_from_table_or_query` (retrieve latest feature values) | yes | | ||
| `pull_all_from_table_or_query` (retrieve a saved dataset) | yes | | ||
| `offline_write_batch` (persist dataframes to offline store) | no | | ||
| `write_logged_features` (persist logged features to offline store) | no | | ||
|
||
Below is a matrix indicating which functionality is supported by `MsSqlServerRetrievalJob`. | ||
|
||
| | MsSql | | ||
| --------------------------------- | --- | | ||
| export to dataframe | yes | | ||
| export to arrow table | yes | | ||
| export to arrow batches | no | | ||
| export to SQL | no | | ||
| export to data lake (S3, GCS, etc.) | no | | ||
| export to data warehouse | no | | ||
| local execution of Python-based on-demand transforms | no | | ||
| remote execution of Python-based on-demand transforms | no | | ||
| persist results in the offline store | yes | | ||
|
||
To compare this set of functionality against other offline stores, please see the full [functionality matrix](overview.md#functionality-matrix). |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,26 @@ | ||
# Azure | ||
|
||
## Description | ||
|
||
* Offline Store: Uses the **MsSql** offline store by default. Also supports File as the offline store. | ||
* Online Store: Uses the **Redis** online store by default. Also supports Sqlite as an online store. | ||
|
||
## Disclaimer | ||
|
||
The Azure provider does not achieve full test coverage. | ||
Please do not assume complete stability. | ||
|
||
## Example | ||
|
||
{% code title="feature_store.yaml" %} | ||
```yaml | ||
registry: | ||
registry_store_type: AzureRegistryStore | ||
path: ${REGISTRY_PATH} # Environment Variable | ||
project: production | ||
provider: azure | ||
online_store: | ||
type: redis | ||
connection_string: ${REDIS_CONN} # Environment Variable | ||
``` | ||
{% endcode %} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not spark?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed