Feast Java Server unable to pull data from Elasticache Redis

[harmeet-singh-discovery]

Setup

I am using Elasticache Redis

• Engine version: 6.2.6
• Cluster mode: Enabled

I am using following application-override-feast.yaml

feast:
  project: project
  registry: s3://<bkt_name>/registry.db
  entityKeySerializationVersion: 2
  registryRefreshInterval: 30
  activeStore: online
  awsRegion: us-east-1
  stores:
    - name: online
      type: REDIS_CLUSTER
      config:
        connection_string: fabc.def.clustercfg.use1.cache.amazonaws.com:6379
        timeout: PT0.5S
        read_from: REPLICA

If I fetch online features using feast sdk from the pod that is running feast-java-server, as given below:

def run_demo():
    store = FeatureStore(repo_path="../feature_repo")

    print("\n--- Online features ---")
    features = store.get_online_features(
        features=[
            "feature_view:feature",
        ],
        entity_rows=[
            {
                "id": "1",
            }
        ],
    ).to_dict()
    for key, value in sorted(features.items()):
        print(key, " : ", value)


if __name__ == "__main__":
    run_demo()

I get the data from online store.

However, Java Server returns the following error:

java.lang.RuntimeException: Unexpected error when pulling data from Redis
        at feast.serving.connectors.redis.retriever.RedisOnlineRetriever.getFeaturesFromRedis(RedisOnlineRetriever.java:129)
        at feast.serving.connectors.redis.retriever.RedisOnlineRetriever.getOnlineFeatures(RedisOnlineRetriever.java:59)
        at feast.serving.service.OnlineServingServiceV2.retrieveFeatures(OnlineServingServiceV2.java:321)
        at feast.serving.service.OnlineServingServiceV2.getOnlineFeatures(OnlineServingServiceV2.java:123)
        at feast.serving.service.grpc.OnlineServingGrpcServiceV2.getOnlineFeatures(OnlineServingGrpcServiceV2.java:56)
        at feast.proto.serving.ServingServiceGrpc$MethodHandlers.invoke(ServingServiceGrpc.java:314)
        at io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:172)
        at io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35)
        at io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23)
        at io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:40)
        at io.grpc.Contexts$ContextualizedServerCallListener.onHalfClose(Contexts.java:86)
        at io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35)
        at io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23)
        at io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:40)
        at io.opentracing.contrib.grpc.TracingServerInterceptor$2.onHalfClose(TracingServerInterceptor.java:235)
        at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:331)
        at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:820)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)

[harmeet-singh-discovery]

It turns out the error was in application-override-file.yaml
read_from: REPLICA was replaced with read_from: MASTER as there were no replica nodes in our setup.