Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor Feast SDK test suite #1496

Closed
woop opened this issue Apr 22, 2021 · 1 comment
Closed

Refactor Feast SDK test suite #1496

woop opened this issue Apr 22, 2021 · 1 comment

Comments

@woop
Copy link
Member

woop commented Apr 22, 2021

Our current test suite has the following problems

  • Lack of generalization across different sources, stores, providers. Ideally we would have a high level set of e2e tests that use the user API to test different combination of source, stores, providers and other infra config. An example of the problem here is https://github.com/feast-dev/feast/blob/master/sdk/python/tests/test_historical_retrieval.py#L313 where we have a bigquery test and a parquet test.
  • Related to the above problem is the fact that we aren't reusing our setup/teardown for tests. Ideally the setup/teardown would be externalized and reusable across different kinds of tests (testing online retrieval, historical retrieval, materialization). Even better if we can just do the setup/teardown once, and then run a range of tests on the staged infrastructure.
  • Ideally we would have a single "kitchen sink" use case that can be applied to all tests, removing the need for test authors to think about what kind of data they need to use
  • We also need a place to add smaller tests, which are closer to unit tests. An example of this is testing whether entity dataframe inferencing works Infer entity dataframe event timestamp column #1495. Its not scalable to keep adding conditional flags and rerun the whole test suite. Eventually our integration tests will take too long to complete.
  • Lack of parallelism. We can run many of these tests async if they are well isolated.
  • Smaller point: We need to test all the types that we support. Currently we're not testing fringe types with our integration tests.
@stale
Copy link

stale bot commented Aug 27, 2021

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the wontfix This will not be worked on label Aug 27, 2021
@woop woop added the keep-open label Aug 27, 2021
@stale stale bot closed this as completed Sep 3, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant