fix: Parse inline data sources #3036

felixwang9817 · 2022-08-08T21:49:27Z

What this PR does / why we need it:

Which issue(s) this PR fixes:

sdk/python/feast/stream_feature_view.py

sdk/python/feast/repo_operations.py

achals · 2022-08-08T22:18:28Z

sdk/python/feast/repo_operations.py

+                # Handle stream sources defined with feature views.
+                if obj.stream_source:
+                    stream_source = obj.stream_source
+                    if not any((stream_source is ds) for ds in res.data_sources):


Suggested change

if not any((stream_source is ds) for ds in res.data_sources):

if stream_source not in res.data_sources:

is there any reason this wouldn't work?

yeah so the reason I want to filter by is instead of == is precisely to avoid deduping

the only kind of duplication we should allow is directly importing an object from another file in the feature repo

What other kind of duplication are you thinking about?

you might e.g. define a file source in a different .py file with just the path data.parquet and then re-define it inline for a feature view with the same path

this would result in two unique FileSource objects that are ==, which I think we should prevent

achals · 2022-08-08T22:19:23Z

sdk/python/feast/repo_operations.py

+                if not any((batch_source is ds) for ds in res.data_sources):
+                    res.data_sources.append(batch_source)


Suggested change

if not any((batch_source is ds) for ds in res.data_sources):

res.data_sources.append(batch_source)

if batch_source not in res.data_sources:

res.data_sources.append(batch_source)

May be cleaner? Or cleaner still to continue using a set and just add blindly and let the set dedupe.

same reason as above

felixwang9817 · 2022-08-08T23:43:22Z

also note that the test errors here will be fixed once #3037 lands

codecov-commenter · 2022-08-08T23:45:10Z

Codecov Report

Merging #3036 (e733c89) into master (0ed1a63) will increase coverage by 8.53%.
The diff coverage is 85.33%.

@@            Coverage Diff             @@
##           master    #3036      +/-   ##
==========================================
+ Coverage   67.44%   75.98%   +8.53%     
==========================================
  Files         169      202      +33     
  Lines       14936    16914    +1978     
==========================================
+ Hits        10074    12852    +2778     
+ Misses       4862     4062     -800

Flag	Coverage Δ
integrationtests	`67.05% <26.66%> (-0.40%)`	⬇️
unittests	`58.28% <85.33%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
sdk/python/feast/cli.py	`41.59% <ø> (-0.10%)`	⬇️
sdk/python/feast/repo_operations.py	`49.53% <5.00%> (+24.16%)`	⬆️
...unit/local_feast_tests/test_local_feature_store.py	`97.88% <97.36%> (+66.55%)`	⬆️
sdk/python/feast/feature_store.py	`85.26% <100.00%> (+3.20%)`	⬆️
sdk/python/tests/utils/cli_repo_creator.py	`98.03% <100.00%> (+9.15%)`	⬆️
...s/contrib/postgres_offline_store/tests/__init__.py	`100.00% <0.00%> (ø)`
...line_stores/contrib/postgres_repo_configuration.py	`100.00% <0.00%> (ø)`
sdk/python/feast/loaders/yaml.py	`18.18% <0.00%> (ø)`
...k/python/feast/infra/materialization/lambda/app.py	`26.66% <0.00%> (ø)`
..._stores/contrib/postgres_offline_store/postgres.py	`35.22% <0.00%> (ø)`
... and 102 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

adchia · 2022-08-09T23:19:38Z

nit: have more descriptive PR title since this goes into the changelog

felixwang9817 · 2022-08-09T23:26:14Z

@adchia fixed!

Signed-off-by: Felix Wang <wangfelix98@gmail.com>

adchia

/lgtm

feast-ci-bot · 2022-08-15T14:27:03Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: adchia, felixwang9817

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [adchia,felixwang9817]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

# [0.24.0](v0.23.0...v0.24.0) (2022-08-25) ### Bug Fixes * Check if on_demand_feature_views is an empty list rather than None for snowflake provider ([#3046](#3046)) ([9b05e65](9b05e65)) * FeatureStore.apply applies BatchFeatureView correctly ([#3098](#3098)) ([41be511](41be511)) * Fix Feast Java inconsistency with int64 serialization vs python ([#3031](#3031)) ([4bba787](4bba787)) * Fix feature service inference logic ([#3089](#3089)) ([4310ed7](4310ed7)) * Fix field mapping logic during feature inference ([#3067](#3067)) ([cdfa761](cdfa761)) * Fix incorrect on demand feature view diffing and improve Java tests ([#3074](#3074)) ([0702310](0702310)) * Fix Java helm charts to work with refactored logic. Fix FTS image ([#3105](#3105)) ([2b493e0](2b493e0)) * Fix on demand feature view output in feast plan + Web UI crash ([#3057](#3057)) ([bfae6ac](bfae6ac)) * Fix release workflow to release 0.24.0 ([#3138](#3138)) ([a69aaae](a69aaae)) * Fix Spark offline store type conversion to arrow ([#3071](#3071)) ([b26566d](b26566d)) * Fixing Web UI, which fails for the SQL registry ([#3028](#3028)) ([64603b6](64603b6)) * Force Snowflake Session to Timezone UTC ([#3083](#3083)) ([9f221e6](9f221e6)) * Make infer dummy entity join key idempotent ([#3115](#3115)) ([1f5b1e0](1f5b1e0)) * More explicit error messages ([#2708](#2708)) ([e4d7afd](e4d7afd)) * Parse inline data sources ([#3036](#3036)) ([c7ba370](c7ba370)) * Prevent overwriting existing file during `persist` ([#3088](#3088)) ([69af21f](69af21f)) * Register BatchFeatureView in feature repos correctly ([#3092](#3092)) ([b8e39ea](b8e39ea)) * Return an empty infra object from sql registry when it doesn't exist ([#3022](#3022)) ([8ba87d1](8ba87d1)) * Teardown tables for Snowflake Materialization testing ([#3106](#3106)) ([0a0c974](0a0c974)) * UI error when saved dataset is present in registry. ([#3124](#3124)) ([83cf753](83cf753)) * Update sql.py ([#3096](#3096)) ([2646a86](2646a86)) * Updated snowflake template ([#3130](#3130)) ([f0594e1](f0594e1)) ### Features * Add authentication option for snowflake connector ([#3039](#3039)) ([74c75f1](74c75f1)) * Add Cassandra/AstraDB online store contribution ([#2873](#2873)) ([feb6cb8](feb6cb8)) * Add Snowflake materialization engine ([#2948](#2948)) ([f3b522b](f3b522b)) * Adding saved dataset capabilities for Postgres ([#3070](#3070)) ([d3253c3](d3253c3)) * Allow passing repo config path via flag ([#3077](#3077)) ([0d2d951](0d2d951)) * Contrib azure provider with synapse/mssql offline store and Azure registry store ([#3072](#3072)) ([9f7e557](9f7e557)) * Custom Docker image for Bytewax batch materialization ([#3099](#3099)) ([cdd1b07](cdd1b07)) * Feast AWS Athena offline store (again) ([#3044](#3044)) ([989ce08](989ce08)) * Implement spark offline store `offline_write_batch` method ([#3076](#3076)) ([5b0cc87](5b0cc87)) * Initial Bytewax materialization engine ([#2974](#2974)) ([55c61f9](55c61f9)) * Refactor feature server helm charts to allow passing feature_store.yaml in environment variables ([#3113](#3113)) ([85ee789](85ee789))

feast-ci-bot added approved size/XL labels Aug 8, 2022

achals reviewed Aug 8, 2022

View reviewed changes

felixwang9817 force-pushed the fix_repo_parsing branch from 3cbf934 to 6fb1620 Compare August 8, 2022 23:21

felixwang9817 force-pushed the fix_repo_parsing branch from 6fb1620 to ed3c4de Compare August 9, 2022 20:22

felixwang9817 assigned achals Aug 9, 2022

felixwang9817 force-pushed the fix_repo_parsing branch from ed3c4de to 4f4ada1 Compare August 9, 2022 23:25

felixwang9817 changed the title ~~fix: Fix repo parsing logic~~ fix: Ensure inline data sources are correctly parsed Aug 9, 2022

felixwang9817 changed the title ~~fix: Ensure inline data sources are correctly parsed~~ fix: Correctly parse inline data sources Aug 9, 2022

felixwang9817 changed the title ~~fix: Correctly parse inline data sources~~ fix: Parse inline data sources Aug 9, 2022

felixwang9817 force-pushed the fix_repo_parsing branch from 4f4ada1 to 8786519 Compare August 10, 2022 18:24

felixwang9817 added 3 commits August 12, 2022 13:09

Remove deprecated CLI warnings

047437b

Signed-off-by: Felix Wang <wangfelix98@gmail.com>

Fix repo parsing logic

b0dc405

Signed-off-by: Felix Wang <wangfelix98@gmail.com>

Add tests

e733c89

Signed-off-by: Felix Wang <wangfelix98@gmail.com>

felixwang9817 force-pushed the fix_repo_parsing branch from 8786519 to e733c89 Compare August 12, 2022 20:09

adchia approved these changes Aug 15, 2022

View reviewed changes

feast-ci-bot assigned adchia Aug 15, 2022

feast-ci-bot added the lgtm label Aug 15, 2022

feast-ci-bot merged commit c7ba370 into feast-dev:master Aug 15, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: Parse inline data sources #3036

fix: Parse inline data sources #3036

felixwang9817 commented Aug 8, 2022

achals Aug 8, 2022

felixwang9817 Aug 8, 2022

achals Aug 9, 2022

felixwang9817 Aug 9, 2022

achals Aug 8, 2022

felixwang9817 Aug 8, 2022

felixwang9817 commented Aug 8, 2022

codecov-commenter commented Aug 8, 2022 •

edited

adchia commented Aug 9, 2022

felixwang9817 commented Aug 9, 2022

adchia left a comment

feast-ci-bot commented Aug 15, 2022

	if not any((stream_source is ds) for ds in res.data_sources):
	if stream_source not in res.data_sources:

		if not any((batch_source is ds) for ds in res.data_sources):
		res.data_sources.append(batch_source)

fix: Parse inline data sources #3036

fix: Parse inline data sources #3036

Conversation

felixwang9817 commented Aug 8, 2022

achals Aug 8, 2022

Choose a reason for hiding this comment

felixwang9817 Aug 8, 2022

Choose a reason for hiding this comment

achals Aug 9, 2022

Choose a reason for hiding this comment

felixwang9817 Aug 9, 2022

Choose a reason for hiding this comment

achals Aug 8, 2022

Choose a reason for hiding this comment

felixwang9817 Aug 8, 2022

Choose a reason for hiding this comment

felixwang9817 commented Aug 8, 2022

codecov-commenter commented Aug 8, 2022 • edited

Codecov Report

adchia commented Aug 9, 2022

felixwang9817 commented Aug 9, 2022

adchia left a comment

Choose a reason for hiding this comment

feast-ci-bot commented Aug 15, 2022

codecov-commenter commented Aug 8, 2022 •

edited