client.start_offline_to_online_ingestion job is failing in minimal_ride_hailing example

[Adedolapo-Akin]

Expected Behavior

{'driver_id': [100651,
561506,
304334,
997026,
420223,
546179,
308686,
888397,
731657,
684814],
'driver_statistics:avg_daily_trips': [614,
71,
801,
341,
158,
747,
509,
568,
844,
315]}

Current Behavior

{'driver_id': [507133,
891268,
940489,
750166,
830970,
815675,
85423,
56280,
230535,
846488],
'driver_statistics:acc_rate': [None,
None,
None,
None,
None,
None,
None,
None,
None,
None]}

kubectl logs feast-release-feast-jupyter-86444477b7-vxptg

Exception in thread "main" redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: Failed connecting to localhost:6379
Caused by: java.net.ConnectException: Connection refused (Connection refused)

Steps to reproduce

features = client.get_online_features(
feature_refs=["driver_statistics:avg_daily_trips"],
entity_rows=entities_sample).to_dict()

features

Specifications

• Version:
  helm repo add feast-charts https://feast-charts.storage.googleapis.com
  helm repo update

  app: feast-core
  app.kubernetes.io/managed-by: Helm
  chart: feast-core-0.8.2
  component: core
  heritage: Helm
  release: feast-release
  managedFields:

• Platform:

• Subsystem:

Possible Solution

[woop]

Thanks for raising this issue @Adedolapo-Akin. I'll try to reproduce this problem today.

[woop]

Quick question @Adedolapo-Akin, did you just run kubectl logs feast-release-feast-jupyter-86444477b7-vxptg to get the logs? What command did you run to get the below logs, did you perhaps get it from the serving service?

Exception in thread "main" redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: Failed connecting to localhost:6379
Caused by: java.net.ConnectException: Connection refused (Connection refused)

Are you sure that Redis doesn't have any security enabled?

[Adedolapo-Akin]

Quick question @Adedolapo-Akin, did you just run kubectl logs feast-release-feast-jupyter-86444477b7-vxptg to get the logs?
yes, this was the command I used.

Are you sure that Redis doesn't have any security enabled?
I followed the installation guide on the feast website which used helm for the installation (helm repo add feast-charts https://feast-charts.storage.googleapis.com) .
I did not change any settings for Redis as this was not specified in the installation guide.

[woop]

@Adedolapo-Akin would you mind providing a bit more information. Are you running on GCP?

[Adedolapo-Akin]

Yes...I am using GKE on GCP

[woop]

@Adedolapo-Akin what isn't clear to me is where the logs came from. kubectl logs feast-release-feast-jupyter-86444477b7-vxptg shouldn't be giving you any Redis connection logs. Did the logs come from the online serving API?

[Adedolapo-Akin]

I have attached a copy of the log from feast-release-feast-jupyter-86444477b7-vxptg so that you can verify for yourself
jupyter.log

[woop]

Ok, so you are using Spark from within the notebook.

Starting with config IngestionJobConfig(Online,FeatureTable(driver_trips,default,List(Field(driver_id,INT64)),List(Field(trips_today,INT32)),None,Map()),KafkaSource(10.52.10.236:9092,driver_trips,AvroFormat({"type": "record", "name": "DriverTrips", "fields": [{"name": "driver_id", "type": "long"}, {"name": "trips_today", "type": "int"}, {"name": "datetime", "type": {"type": "long", "logicalType": "timestamp-micros"}}]}),Map(),datetime,Some(datetime),Some()),2021-02-02T12:18:00.400Z,2021-02-02T12:18:00.465Z,RedisConfig(localhost,6379,false),None,None,None,0,None,true)

Can you try to set Client(redis_host="your redis address"), the address should be something like feast-redis-headless.namespace.svc

[Adedolapo-Akin]

@woop
Thanks .
Setting the redis host solved my problem.

[woop]

Great!

[theofpa]

I had the same problem using the helm installation method in an on-premise k8s cluster, resolved using:

client = Client(redis_host=os.environ["FEAST_RELEASE_REDIS_MASTER_SERVICE_HOST"])

[woop]

Thanks @theofpa !