Return missing feature metadata in response

[thisisandreeeee]

There is currently no simple mechanism for differentiating between missing and null features. A missing feature would be one where the feature creation pipeline for the corresponding timestamp has not been run yet, thus feature retrieval yields no results. A null feature would be one where the pipeline has been executed, but the feature value for that entity is null e.g. event counts for an entity would be 0 if the said entity did not trigger any events within the time window.

It would be useful if Feast returned metadata on features that have not been ingested yet. This could take on the form of a boolean mask that is applied to the resultant dataframe.

[woop]

Thanks for the suggestion @thisisandreeeee. We definitely want to support this for 0.3. Will reply with a proposal in this thread.

[woop]

Just so that others have more context on this. There are two types of missing feature values:

1. Missing key for a specific feature set. For example if you are looking for a specific feature for a specific customer but no data has been ingested for that customer, then there wont be any keys set.
2. Missing feature value within a feature set. In this case we do have feature data for a specific customer at a specific point in time (so we get a hit on the key), but not all of the feature values are set. Some of them could be set to null. Setting all values in a feature row (when ingesting data) is required. However, we allow you to set values to Null.

For example, let's say you are doing a query for feature_1 and feature_2 for customers 1001, 1002, and 1003 at 2019-01-01 00:00:00.

customer 1001 has all of their features.
customer 1002 has no data
customer 1003 has all of their features set, but one of them has been set to null by the feature creator.

+---------------------+-------------+-----------+-----------+
|      datetime       | customer_id | feature_1 | feature_2 |
+---------------------+-------------+-----------+-----------+
| 2019-01-01 00:00:00 |        1001 | 1         | 10.0      |
| 2019-01-01 00:00:00 |        1002 | Null      | Null      |
| 2019-01-01 00:00:00 |        1003 | 2         | Null      |
+---------------------+-------------+-----------+-----------+

I have some questions regarding your ask:

• As a thought experiment, why can't you determine the missing values from the returned response. As long as we return all the columns and rows that you requested and set the values to null.
• How will you use the differentiated features?
• How many classes of missing values do you need? I can identify at least 3. Missing key, missing value, value found but outside max age (this would only be for serving)

[thisisandreeeee]

Thanks for clearing up the terminology. I think it's fine if Feast handled missing values by returning nulls in the response. What is more problematic is the fact that Feast currently returns nulls for missing keys as well.

If a feature has a missing key for a timestamp, we may want to handle it differently from a missing value (where you may just impute or drop the feature). If keys are missing, it is possible that upstream feature ingestion pipelines have failed or have not been executed. I would prefer to raise an exception in such scenarios so that we can notify the feature creators, instead of handling this silently in our model training pipelines. This would be rather important if we were to consume features created by other creators.

I think values found outside max-age would also be a relevant concern during training. It would be useful to know when features are beyond a staleness threshold so that we can decide if we want to proceed with using them or not.

[woop]

e fact that Feast currently returns nulls for missing keys as well.

I think it would be pretty straightforward to return metadata on whether there was a missing key. That could be at the feature level.

I am not sure how we want to handle values found outside of max age. For online serving we can return these since we will always get the last written value's timestamp, but we wont find older values for historical data. @zhilingc are you currently getting the last written timestamp when running the query, or is it being filtered out by max_age?

[woop]

Bumping this.

@zhilingc do you have any more inputs?

[ches]

Since this is referenced as a roadmap item for 0.5, could we clarify the scope of what it entails at this point? Maybe this issue on its own is sufficient and only the Google Doc needs more clarity: it says "Add support for missing values"—that's rather broad, I think it's evident in this thread that e.g. imputation as some form of first-class feature is not in scope now.

[woop]

Since this is referenced as a roadmap item for 0.5, could we clarify the scope of what it entails at this point? Maybe this issue on its own is sufficient and only the Google Doc needs more clarity: it says "Add support for missing values"—that's rather broad, I think it's evident in this thread that e.g. imputation as some form of first-class feature is not in scope now.

I have updated the roadmap item to be a bit more descriptive. This card is definitely different from #454 (registration vs data availability)

The roadmap item would be focused purely on this GitHub issue. Roughly I was thinking along these lines

• Return metadata about missing feature values for both online and offline responses
• Return metadata about missing feature rows for both online and offline responses
• Type codes to explain why data is missing
  • Excluded due to max age?
  • Was it imputed from defaults?

I was hoping to pick this up again and scope it out at the start of next week.

[woop]

Started drafting out my ideas here RFC: Missing Feature Metadata

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.