Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Fix push API to respect feature view's already inferred entity types #3172

Merged
merged 3 commits into from
Sep 7, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
6 changes: 1 addition & 5 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -356,15 +356,11 @@ lint-go: compile-protos-go compile-go-lib

# Docker

build-docker: build-ci-docker build-feature-server-python-docker build-feature-server-python-aws-docker build-feature-transformation-server-docker build-feature-server-java-docker
build-docker: build-feature-server-python-docker build-feature-server-python-aws-docker build-feature-transformation-server-docker build-feature-server-java-docker

push-ci-docker:
docker push $(REGISTRY)/feast-ci:$(VERSION)

# TODO(adchia): consider removing. This doesn't run successfully right now
build-ci-docker:
docker buildx build -t $(REGISTRY)/feast-ci:$(VERSION) -f infra/docker/ci/Dockerfile --load .

push-feature-server-python-docker:
docker push $(REGISTRY)/feature-server:$$VERSION

Expand Down
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -185,8 +185,8 @@ The list below contains the functionality that contributors are planning to deve
* [ ] Batch transformation (In progress. See [RFC](https://docs.google.com/document/d/1964OkzuBljifDvkV-0fakp2uaijnVzdwWNGdz7Vz50A/edit))
* **Streaming**
* [x] [Custom streaming ingestion job support](https://docs.feast.dev/how-to-guides/creating-a-custom-provider)
* [x] [Push based streaming data ingestion to online store (Alpha)](https://docs.feast.dev/reference/data-sources/push)
* [x] [Push based streaming data ingestion to offline store (Alpha)](https://docs.feast.dev/reference/data-sources/push)
* [x] [Push based streaming data ingestion to online store](https://docs.feast.dev/reference/data-sources/push)
* [x] [Push based streaming data ingestion to offline store](https://docs.feast.dev/reference/data-sources/push)
* **Deployments**
* [x] AWS Lambda (Alpha release. See [RFC](https://docs.google.com/document/d/1eZWKWzfBif66LDN32IajpaG-j82LSHCCOzY6R7Ax7MI/edit))
* [x] Kubernetes (See [guide](https://docs.feast.dev/how-to-guides/running-feast-in-production#4.3.-java-based-feature-server-deployed-on-kubernetes))
Expand All @@ -202,7 +202,7 @@ The list below contains the functionality that contributors are planning to deve
* [x] Model-centric feature tracking (feature services)
* [x] Amundsen integration (see [Feast extractor](https://github.com/amundsen-io/amundsen/blob/main/databuilder/databuilder/extractor/feast_extractor.py))
* [x] DataHub integration (see [DataHub Feast docs](https://datahubproject.io/docs/generated/ingestion/sources/feast/))
* [x] Feast Web UI (Alpha release. See [docs](https://docs.feast.dev/reference/alpha-web-ui))
* [x] Feast Web UI (Beta release. See [docs](https://docs.feast.dev/reference/alpha-web-ui))


## 🎓 Important Resources
Expand Down
2 changes: 0 additions & 2 deletions docs/reference/data-sources/push.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
# Push source

**Warning**: This is an _experimental_ feature. It's intended for early testing and feedback, and could change without warnings in future releases.

## Description

Push sources allow feature values to be pushed to the online store and offline store in real time. This allows fresh feature values to be made available to applications. Push sources supercede the
Expand Down
9 changes: 5 additions & 4 deletions docs/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,15 +43,16 @@ The list below contains the functionality that contributors are planning to deve
* [ ] Batch transformation (In progress. See [RFC](https://docs.google.com/document/d/1964OkzuBljifDvkV-0fakp2uaijnVzdwWNGdz7Vz50A/edit))
* **Streaming**
* [x] [Custom streaming ingestion job support](https://docs.feast.dev/how-to-guides/creating-a-custom-provider)
* [x] [Push based streaming data ingestion to online store (Alpha)](https://docs.feast.dev/reference/data-sources/push)
* [x] [Push based streaming data ingestion to offline store (Alpha)](https://docs.feast.dev/reference/data-sources/push)
* [x] [Push based streaming data ingestion to online store](https://docs.feast.dev/reference/data-sources/push)
* [x] [Push based streaming data ingestion to offline store](https://docs.feast.dev/reference/data-sources/push)
* **Deployments**
* [x] AWS Lambda (Alpha release. See [RFC](https://docs.google.com/document/d/1eZWKWzfBif66LDN32IajpaG-j82LSHCCOzY6R7Ax7MI/edit))
* [x] Kubernetes (See [guide](https://docs.feast.dev/how-to-guides/running-feast-in-production#4.3.-java-based-feature-server-deployed-on-kubernetes))
* **Feature Serving**
* [x] Python Client
* [x] [Python feature server](https://docs.feast.dev/reference/feature-servers/python-feature-server)
* [x] [Go feature server](https://docs.feast.dev/reference/feature-servers/go-feature-server)
* [x] [Java feature server (alpha)](https://github.com/feast-dev/feast/blob/master/infra/charts/feast/README.md)
* [x] [Go feature server (alpha)](https://docs.feast.dev/reference/feature-servers/go-feature-server)
* **Data Quality Management (See [RFC](https://docs.google.com/document/d/110F72d4NTv80p35wDSONxhhPBqWRwbZXG4f9mNEMd98/edit))**
* [x] Data profiling and validation (Great Expectations)
* **Feature Discovery and Governance**
Expand All @@ -60,4 +61,4 @@ The list below contains the functionality that contributors are planning to deve
* [x] Model-centric feature tracking (feature services)
* [x] Amundsen integration (see [Feast extractor](https://github.com/amundsen-io/amundsen/blob/main/databuilder/databuilder/extractor/feast_extractor.py))
* [x] DataHub integration (see [DataHub Feast docs](https://datahubproject.io/docs/generated/ingestion/sources/feast/))
* [x] Feast Web UI (Alpha release. See [docs](https://docs.feast.dev/reference/alpha-web-ui))
* [x] Feast Web UI (Beta release. See [docs](https://docs.feast.dev/reference/alpha-web-ui))
4 changes: 2 additions & 2 deletions infra/charts/feast/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Feast Helm Charts
# Feast Java Helm Charts (alpha)

This repo contains Helm charts for Feast components that are being installed on Kubernetes:
This repo contains Helm charts for Feast Java components that are being installed on Kubernetes:
* Feast (root chart): The complete Helm chart containing all Feast components and dependencies. Most users will use this chart, but can selectively enable/disable subcharts using the values.yaml file.
* [Feature Server](charts/feature-server): High performant JVM-based implementation of feature server.
* [Transformation Service](charts/transformation-service): Transformation server for calculating on-demand features
Expand Down
4 changes: 2 additions & 2 deletions infra/charts/feast/README.md.gotmpl
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Feast Helm Charts
# Feast Java Helm Charts (alpha)

This repo contains Helm charts for Feast components that are being installed on Kubernetes:
This repo contains Helm charts for Feast Java components that are being installed on Kubernetes:
* Feast (root chart): The complete Helm chart containing all Feast components and dependencies. Most users will use this chart, but can selectively enable/disable subcharts using the values.yaml file.
* [Feature Server](charts/feature-server): High performant JVM-based implementation of feature server.
* [Transformation Service](charts/transformation-service): Transformation server for calculating on-demand features
Expand Down
7 changes: 1 addition & 6 deletions sdk/python/feast/feature_store.py
Original file line number Diff line number Diff line change
Expand Up @@ -1478,13 +1478,8 @@ def write_to_online_store(
feature_view = self.get_feature_view(
feature_view_name, allow_registry_cache=allow_registry_cache
)
entities = []
for entity_name in feature_view.entities:
entities.append(
self.get_entity(entity_name, allow_registry_cache=allow_registry_cache)
)
provider = self._get_provider()
provider.ingest_df(feature_view, entities, df)
provider.ingest_df(feature_view, df)

@log_exceptions_and_usage
def write_to_offline_store(
Expand Down
6 changes: 4 additions & 2 deletions sdk/python/feast/infra/passthrough_provider.py
Original file line number Diff line number Diff line change
Expand Up @@ -193,7 +193,6 @@ def online_read(
def ingest_df(
self,
feature_view: FeatureView,
entities: List[Entity],
df: pd.DataFrame,
):
set_usage_attribute("provider", self.__class__.__name__)
Expand All @@ -204,7 +203,10 @@ def ingest_df(
table, feature_view.batch_source.field_mapping
)

join_keys = {entity.join_key: entity.value_type for entity in entities}
join_keys = {
entity.name: entity.dtype.to_value_type()
for entity in feature_view.entity_columns
}
rows_to_write = _convert_arrow_to_proto(table, feature_view, join_keys)

self.online_write_batch(
Expand Down
2 changes: 0 additions & 2 deletions sdk/python/feast/infra/provider.py
Original file line number Diff line number Diff line change
Expand Up @@ -123,15 +123,13 @@ def online_write_batch(
def ingest_df(
self,
feature_view: FeatureView,
entities: List[Entity],
df: pd.DataFrame,
):
"""
Persists a dataframe to the online store.

Args:
feature_view: The feature view to which the dataframe corresponds.
entities: The entities that are referenced by the dataframe.
df: The dataframe to be persisted.
"""
pass
Expand Down