Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Added Redshift and Spark typecheck to data_source event_timestamp_col inference #2389

Merged
merged 8 commits into from
Mar 17, 2022
Merged

fix: Added Redshift and Spark typecheck to data_source event_timestamp_col inference #2389

merged 8 commits into from
Mar 17, 2022

Conversation

shaibruhis
Copy link
Contributor

What this PR does / why we need it:
RedshiftSource is a supported data_source, but the update_data_sources_with_inferred_event_timestamp_col() method omits it as a valid value in the assertion. This was causing the assertion to fail when using a RedshiftSource
Which issue(s) this PR fixes:

Fixes #

@shaibruhis shaibruhis changed the title added Redshift typecheck to data_source event_timestamp_col inference fix: Added Redshift typecheck to data_source event_timestamp_col inference Mar 9, 2022
@adchia adchia marked this pull request as ready for review March 9, 2022 05:23
@adchia adchia requested a review from a team as a code owner March 9, 2022 05:23
@adchia adchia requested review from mavysavydav and removed request for a team March 9, 2022 05:23
@adchia
Copy link
Collaborator

adchia commented Mar 9, 2022

/ok-to-test

@adchia
Copy link
Collaborator

adchia commented Mar 9, 2022

Thanks for catching and making a PR!

Mind updating to master? Also mind adding a test e.g. at https://github.com/feast-dev/feast/blob/master/sdk/python/tests/integration/registration/test_inference.py#L87-L87 ?

@codecov-commenter
Copy link

codecov-commenter commented Mar 9, 2022

Codecov Report

Merging #2389 (e72c216) into master (c2389da) will increase coverage by 0.01%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #2389      +/-   ##
==========================================
+ Coverage   85.47%   85.49%   +0.01%     
==========================================
  Files         122      122              
  Lines       10714    10727      +13     
==========================================
+ Hits         9158     9171      +13     
  Misses       1556     1556              
Flag Coverage Δ
integrationtests 75.34% <100.00%> (-0.19%) ⬇️
unittests 57.08% <42.85%> (-0.03%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
sdk/python/feast/inference.py 83.82% <ø> (ø)
...ts/integration/feature_repos/repo_configuration.py 96.10% <100.00%> (+0.05%) ⬆️
...n/tests/integration/registration/test_inference.py 100.00% <100.00%> (ø)
sdk/python/feast/go_server.py 89.18% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c2389da...e72c216. Read the comment docs.

@shaibruhis
Copy link
Contributor Author

Hey Danny! Not sure what you mean by "updating to master", I set the base to master?

Also, the test you listed already checks the RedshiftSource. I think the right test here is to add a RedshiftSource in here: https://github.com/feast-dev/feast/blob/master/sdk/python/tests/integration/registration/test_inference.py#L119-L123. I saw there is a utils class RedshiftDataSourceCreator but curious if there is a test Redshift data source?

@adchia
Copy link
Collaborator

adchia commented Mar 9, 2022

Hey Danny! Not sure what you mean by "updating to master", I set the base to master?

Also, the test you listed already checks the RedshiftSource. I think the right test here is to add a RedshiftSource in here: https://github.com/feast-dev/feast/blob/master/sdk/python/tests/integration/registration/test_inference.py#L119-L123. I saw there is a utils class RedshiftDataSourceCreator but curious if there is a test Redshift data source?

Ah sorry I was unclear. I meant pulling the latest changes from master.

With the test, I accidentally referenced a specific line there, but was referring to the overall file. There is a test Redshift data source available actually. You'll want to probably pass in the test fixtures we generate (e.g. https://github.com/feast-dev/feast/blob/master/sdk/python/tests/integration/feature_repos/repo_configuration.py#L222-L222 are test sources we generate) -> an example of where we pull this is in test_historical_features where it passes in the universal_data_sources test fixture. That gives you access to the data sources defined.

Let me know if you have any questions!

@shaibruhis shaibruhis changed the title fix: Added Redshift typecheck to data_source event_timestamp_col inference fix: Added Redshift and Spark typecheck to data_source event_timestamp_col inference Mar 9, 2022
sdk/python/feast/inference.py Outdated Show resolved Hide resolved
Signed-off-by: Shai Bruhis <shaibruhis@gmail.com>
Signed-off-by: Shai Bruhis <shaibruhis@gmail.com>
Signed-off-by: Shai Bruhis <shaibruhis@gmail.com>
Signed-off-by: Shai Bruhis <shaibruhis@gmail.com>
…allow for inference

Signed-off-by: Shai Bruhis <shaibruhis@gmail.com>
@shaibruhis
Copy link
Contributor Author

Hey @adchia, anything else i should do to make land?

Copy link
Collaborator

@adchia adchia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@adchia
Copy link
Collaborator

adchia commented Mar 14, 2022

Looks like you need to pass the linter by sorting.

You can run this linter locally with make lint-python. You also should probably install the precommit hooks (https://github.com/feast-dev/feast/blob/master/CONTRIBUTING.md#pre-commit-hooks) so this is run automatically

Signed-off-by: Shai Bruhis <shaibruhis@gmail.com>
@feast-ci-bot feast-ci-bot removed the lgtm label Mar 15, 2022
@adchia
Copy link
Collaborator

adchia commented Mar 15, 2022

The linter looks now to be failing and your commit wasn't signed?

If you give me push privileges to your branch btw happy to also help resolve quickly.

@shaibruhis
Copy link
Contributor Author

Fixed the lint issues! I realized I had the wrong linter set which was causing problems for me. Apologies for that

Copy link
Collaborator

@adchia adchia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

Signed-off-by: Shai Bruhis <shaibruhis@gmail.com>
@feast-ci-bot feast-ci-bot removed the lgtm label Mar 15, 2022
Copy link
Collaborator

@adchia adchia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@feast-ci-bot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: adchia, shaibruhis

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@feast-ci-bot feast-ci-bot merged commit 04dea73 into feast-dev:master Mar 17, 2022
@shaibruhis shaibruhis deleted the add-redshift-data_source-event_timestamp_col-typecheck branch March 17, 2022 07:29
achals pushed a commit that referenced this pull request Apr 14, 2022
# [0.20.0](v0.19.0...v0.20.0) (2022-04-14)

### Bug Fixes

* Add inlined data sources to the top level registry ([#2456](#2456)) ([356788a](356788a))
* Add new value types to types.ts for web ui ([#2463](#2463)) ([ad5694e](ad5694e))
* Add PushSource proto and Python class ([#2428](#2428)) ([9a4bd63](9a4bd63))
* Add spark to lambda dockerfile ([#2480](#2480)) ([514666f](514666f))
* Added private_key auth for Snowflake ([#2508](#2508)) ([c42c9b0](c42c9b0))
* Added Redshift and Spark typecheck to data_source event_timestamp_col inference ([#2389](#2389)) ([04dea73](04dea73))
* Building of go extension fails ([#2448](#2448)) ([7d1efd5](7d1efd5))
* Bump the number of versions bumps expected to 27 ([#2549](#2549)) ([ecc9938](ecc9938))
* Create __init__ files for the proto-generated python dirs ([#2410](#2410)) ([e17028d](e17028d))
* Don't prevent apply from running given duplicate empty names in data sources. Also fix repeated apply of Spark data source. ([#2415](#2415)) ([b95f441](b95f441))
* Dynamodb deduplicate batch write request by partition keys ([#2515](#2515)) ([70d4a13](70d4a13))
* Ensure that __init__ files exist in proto dirs ([#2433](#2433)) ([9b94f7b](9b94f7b))
* Fix DataSource constructor to unbreak custom data sources ([#2492](#2492)) ([712653e](712653e))
* Fix default feast apply path without any extras ([#2373](#2373)) ([6ba7fc7](6ba7fc7))
* Fix definitions.py with new definition ([#2541](#2541)) ([eefc34a](eefc34a))
* Fix entity row to use join key instead of name ([#2521](#2521)) ([c22fa2c](c22fa2c))
* Fix Java Master ([#2499](#2499)) ([e083458](e083458))
* Fix registry proto ([#2435](#2435)) ([ea6a9b2](ea6a9b2))
* Fix some inconsistencies in the docs and comments in the code ([#2444](#2444)) ([ad008bf](ad008bf))
* Fix spark docs ([#2382](#2382)) ([d4a606a](d4a606a))
* Fix Spark template to work correctly on feast init -t spark ([#2393](#2393)) ([ae133fd](ae133fd))
* Fix the feature repo fixture used by java tests  ([#2469](#2469)) ([32e925e](32e925e))
* Fix unhashable Snowflake and Redshift sources ([cd8f1c9](cd8f1c9))
* Fixed bug in passing config file params to snowflake python connector ([#2503](#2503)) ([34f2b59](34f2b59))
* Fixing Spark template to include source name ([#2381](#2381)) ([a985f1d](a985f1d))
* Make name a keyword arg for the Entity class ([#2467](#2467)) ([43847de](43847de))
* Making a name for data sources not a breaking change ([#2379](#2379)) ([71d7ae2](71d7ae2))
* Minor link fix in `CONTRIBUTING.md` ([#2481](#2481)) ([2917e27](2917e27))
* Preserve ordering of features in _get_column_names ([#2457](#2457)) ([495b435](495b435))
* Relax click python requirement to >=7 ([#2450](#2450)) ([f202f92](f202f92))
* Remove date partition column field from datasources that don't s… ([#2478](#2478)) ([ce35835](ce35835))
* Remove docker step from unit test workflow ([#2535](#2535)) ([6f22f22](6f22f22))
* Remove spark from the AWS Lambda dockerfile ([#2498](#2498)) ([6abae16](6abae16))
* Request data api update ([#2488](#2488)) ([0c9e5b7](0c9e5b7))
* Schema update ([#2509](#2509)) ([cf7bbc2](cf7bbc2))
* Simplify DataSource.from_proto logic ([#2424](#2424)) ([6bda4d2](6bda4d2))
* Snowflake api update ([#2487](#2487)) ([1181a9e](1181a9e))
* Support passing batch source to streaming sources for backfills ([#2523](#2523)) ([90db1d1](90db1d1))
* Timestamp update ([#2486](#2486)) ([bf23111](bf23111))
* Typos in Feast UI error message ([#2432](#2432)) ([e14369d](e14369d))
* Update feature view APIs to prefer keyword args ([#2472](#2472)) ([7c19cf7](7c19cf7))
* Update file api ([#2470](#2470)) ([83a11c6](83a11c6))
* Update Makefile to cd into python dir before running commands ([#2437](#2437)) ([ca32155](ca32155))
* Update redshift api ([#2479](#2479)) ([4fa73a9](4fa73a9))
* Update some fields optional in UI parser ([#2380](#2380)) ([cff7ac3](cff7ac3))
* Use a single version of jackson libraries and upgrade to 2.12.6.1 ([#2473](#2473)) ([5be1cc6](5be1cc6))
* Use dateutil parser to parse materialization times ([#2464](#2464)) ([6c55e49](6c55e49))
* Use the correct dockerhub image tag when building feature servers ([#2372](#2372)) ([0d62c1d](0d62c1d))

### Features

* Add `/write-to-online-store` method to the python feature server ([#2423](#2423)) ([d2fb048](d2fb048))
* Add description, tags, owner fields to all feature view classes ([#2440](#2440)) ([ed5e928](ed5e928))
* Add DQM Logging on GRPC Server with FileLogStorage for Testing ([#2403](#2403)) ([57a97d8](57a97d8))
* Add Feast types in preparation for changing type system ([#2475](#2475)) ([4864252](4864252))
* Add Field class ([#2500](#2500)) ([1279612](1279612))
* Add support for DynamoDB online_read in batches ([#2371](#2371)) ([702ec49](702ec49))
* Add Support for DynamodbOnlineStoreConfig endpoint_url parameter ([#2485](#2485)) ([7b863d1](7b863d1))
* Add templating for dynamodb table name ([#2394](#2394)) ([f591088](f591088))
* Allow local feature server to use Go feature server if enabled ([#2538](#2538)) ([a2ef375](a2ef375))
* Allow using entity's join_key in get_online_features ([#2420](#2420)) ([068c765](068c765))
* Data Source Api Update ([#2468](#2468)) ([6b96b21](6b96b21))
* Go server ([#2339](#2339)) ([d12e7ef](d12e7ef)), closes [#2354](#2354) [#2361](#2361) [#2332](#2332) [#2356](#2356) [#2363](#2363) [#2349](#2349) [#2355](#2355) [#2336](#2336) [#2361](#2361) [#2363](#2363) [#2344](#2344) [#2354](#2354) [#2347](#2347) [#2350](#2350) [#2356](#2356) [#2355](#2355) [#2349](#2349) [#2352](#2352) [#2341](#2341) [#2336](#2336) [#2373](#2373) [#2315](#2315) [#2372](#2372) [#2332](#2332) [#2349](#2349) [#2336](#2336) [#2361](#2361) [#2363](#2363) [#2344](#2344) [#2354](#2354) [#2347](#2347) [#2350](#2350) [#2356](#2356) [#2355](#2355) [#2349](#2349) [#2352](#2352) [#2341](#2341) [#2336](#2336) [#2373](#2373) [#2379](#2379) [#2380](#2380) [#2382](#2382) [#2364](#2364) [#2366](#2366) [#2386](#2386)
* Graduate write_to_online_store out of experimental status ([#2426](#2426)) ([e7dd4b7](e7dd4b7))
* Make feast PEP 561 compliant ([#2405](#2405)) ([3c41f94](3c41f94)), closes [#2420](#2420) [#2418](#2418) [#2425](#2425) [#2426](#2426) [#2427](#2427) [#2431](#2431) [#2433](#2433) [#2420](#2420) [#2418](#2418) [#2425](#2425) [#2426](#2426) [#2427](#2427) [#2431](#2431) [#2433](#2433)
* Makefile for contrib for Issue [#2364](#2364) ([#2366](#2366)) ([a02325b](a02325b))
* Support on demand feature views in go feature server ([#2494](#2494)) ([6edd274](6edd274))
* Switch from Feature to Field ([#2514](#2514)) ([6a03bed](6a03bed))
* Use a daemon thread to monitor the go feature server exclusively ([#2391](#2391)) ([0bb5e8c](0bb5e8c))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants