feast example#53
Open
tmvfb wants to merge 18 commits into
Open
Conversation
…orkflow Reworked version of the Feast example: - Include redis-cr.yaml with namespace placeholder (no docs lookup needed) - Use feast[redis] pip dependency for Redis online store support - Replace MLflow section with a practical feature evolution workflow: define new computed features, register them, and query across views
- Remove features.py — all definitions live in the notebook - Replace CLI-based feast apply with store.apply() Python API - Add On Demand Feature View (ODFV) example for derived features - Explain why /tmp/registry.db is ephemeral and why that's OK - Single linear flow: setup → data → define → register → train → materialize → serve
…xplain ODFV warning - Add value_type=ValueType.INT64 to Entity to fix DeprecationWarning - Make entity_df construction more verbose with comments explaining the query - Add note about ODFV RuntimeWarning (experimental, irrelevant at this scale)
…n Redis The derived feature is now computed during materialization and stored in Redis, giving the same sub-ms latency as raw features at serving time.
Explains how the workflow changes in production: - Feature definitions in Git (auto-applied on pod start) - Registry Server exposed for remote clients - Notebooks use the operator-generated client ConfigMap - Materialization runs as a CronJob, not from notebooks
Replace abstract driver-stats example with a concrete retailer use case: - Entity: customer_id with order history features - FeatureView: total_orders, total_returns, avg_order_value, days_since_last_order - ODFV: return_rate and return_risk (derived, written to online store) - Preprocessing: filter new customers (< 3 orders), drop nulls, normalize - Model: LogisticRegression to predict return probability - Serving: flag high-risk orders for proactive customer service Also add prod-readiness disclaimer to README and update project name to retail_features.
At serving time we don't know whether the customer will return — that's what the model predicts. Labels come from a separate source and are only joined during training.
…st 0.63 The ODFV materialization to Redis hits a serialization bug with the entity key. Use on-the-fly computation instead, which works reliably and is fast enough for simple transformations like return_rate.
Only the SQLite registry on /tmp is not production-ready. Redis is managed by the OpsTree operator (prokube provides this), and parquet on PVC or S3/MinIO is fine for the offline store.
The notebook now auto-discovers the FeatureStore CR in the current
namespace, reads the operator-published feast-<name>-client ConfigMap,
and writes a feature_store.yaml that points at the registry gRPC server
backed by the operator's PVC. Feature definitions persist across notebook
restarts and are visible to every other client in the namespace.
- feast-cr.yaml: enable services.registry.local.server: {} and disable the
istio sidecar on the feast-server pod (sidecar.istio.io/inject: false).
The operator generates the registry Service with port name 'http' and
no appProtocol, so istio mis-classifies gRPC traffic; the registry only
carries metadata, so dropping mTLS here has minimal impact.
- feast_example.ipynb: read project + remote registry from the operator
ConfigMap; override online_store with direct Redis (read from the secret
referenced by the CR) so materialize() works from the notebook.
- Remove on-demand feature views: feast 0.63 hangs when invoking ODFV UDFs
loaded from a remote registry. Compute return_rate/return_risk in plain
pandas after get_historical_features / get_online_features.
- README rewritten to describe the new flow.
- feast-notebook-rbac.yaml: ClusterRole granting notebook SAs read access
to FeatureStore CRs (already covered by kubeflow-roles on prokube; kept
for portability to other Kubeflow installs).
Ports Henrik's commit (4d7a8ff on feature/feast-example-v2) forward to v3, adapted to the ODFV-free flow: the FeatureService bundles only customer_order_stats (no ODFV, since those hang with a remote registry on feast 0.63). Consumers pass the service object to get_historical_features() and get_online_features() instead of listing individual feature names.
- feast-cr.yaml: use excludeInboundPorts: "6570" instead of disabling sidecar injection; the three-part fix (annotation + alt-Service + DestinationRule) preserves mTLS on the feast-server pod for all other ports. - feast-istio-workaround.yaml: new file with the alt-Service (appProtocol: grpc) and DestinationRule (tls: DISABLE) templates. - feast_example.ipynb: restore on-demand feature view (customer_risk_features) and add PandasTransformation.from_proto patch that bypasses dill+typeguard hang by injecting the live UDF by name (confirmed working in 0.6 s on cluster). - README.md: update istio section to describe three-part fix; remove the "no ODFV" limitation entry.
The operator Service name:http misleads BOTH envoy sidecars, not just the client side. The server-side envoy classifies port 6570 as HTTP/1.1 and rejects gRPC (HTTP/2) with a protocol error. Restores the three-part fix with accurate comments explaining each piece: - excludeInboundPorts:6570 bypasses the misconfigured server-side envoy - alt-Service appProtocol:grpc fixes client-side protocol detection - tls:DISABLE because server-side envoy is bypassed, no mTLS termination
tmvfb
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Merges
feature/feast-example-v2(local SQLite SQL registry) andfeature/feast-example-v3(remote operator gRPC registry) into a single, self-contained example that supports both modes.Supersedes the draft PR #52 (from
feature/feast-example).What changed
feast/registry/local/— FeatureStore CR,feature_store.yaml, and README for the local SQLite SQL registry mode (from v2). No Istio workaround needed, ODFVs work out of the box, registry is ephemeral on/tmpby default.feast/registry/remote/— FeatureStore CR,feature_store.yaml,feast-istio-workaround.yaml, and README for the remote gRPC registry mode (from v3). Registry is persistent on the operator PVC and shared across clients in the namespace.feast/feast_example.ipynb— single notebook for both modes. Anipywidgetsradio button at the top lets the user pick local or remote; the setup cell branches accordingly. The dill+typeguard monkey-patch is wrapped inif REGISTRY_MODE == "remote":.feast/README.md— rewritten to explain both modes, trade-offs, and setup steps for each.feast-cr.yaml,feature_store.yaml,feast-istio-workaround.yaml.Why two modes?
/tmp(ephemeral)Remote mode requires feast-dev/feast#6367 to clean up its workarounds. Once that merges and the operator is upgraded, the monkey-patch and
feast-istio-workaround.yamlcan be dropped and local mode retired.Branches to close after merge
feature/feast-examplefeature/feast-example-v2