Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Regression test for all three hello-world demos #2193

Closed
4 tasks done
hanxiao opened this issue Mar 17, 2021 · 6 comments · Fixed by #2218
Closed
4 tasks done

Regression test for all three hello-world demos #2193

hanxiao opened this issue Mar 17, 2021 · 6 comments · Fixed by #2218
Assignees
Labels
priority/critical Urgent: Security, critical bugs, blocking issues. drop everything until this issue is addressed. status/done This issue is complete. type/maintenance This issue is not a bug or a feature_request.

Comments

@hanxiao
Copy link
Member

hanxiao commented Mar 17, 2021

Describe the feature

hello-world examples are important, if it breaks for a first-time user, then we lose this user.

The current tests on hello-world is not enough, it does not check (1) if index/search is successful (2) the sanity of retrieved results.

Your proposal

Please add regression test for three hello-world.


Environment

Screenshots

Tasks:

  • helloworld
  • multimodal
  • chatbot
  • ci/cd pipeline
@hanxiao hanxiao added the priority/critical Urgent: Security, critical bugs, blocking issues. drop everything until this issue is addressed. label Mar 17, 2021
@JoanFM
Copy link
Member

JoanFM commented Mar 17, 2021

Example for multimodal can be taken from reverted change in #2178. It was reverted for other reasons, but the test would still be valid

@FionnD FionnD added status/available The issue is available for someone to begin working on. type/maintenance This issue is not a bug or a feature_request. labels Mar 17, 2021
@bwanglzu
Copy link
Member

glad to work on it with other engineers.

@JoanFM
Copy link
Member

JoanFM commented Mar 17, 2021

Multimodal helloworld has already been added, please add the other integration tests in the same subfolders

@bwanglzu bwanglzu self-assigned this Mar 18, 2021
@JoanFM
Copy link
Member

JoanFM commented Mar 18, 2021

One thing to be aware of.

Adding these tests, make the tests dependant on the hub submodule, which will bring to a situation where the tests in core do not pass because the hub needs an update and the hub will not be updatable because the new core version is not released

@JoanFM
Copy link
Member

JoanFM commented Mar 18, 2021

@bwanglzu can u make sure the helloworld examples tests are moved to their own folder independent of integration tests? Like this we can easily skip it when necessary much easier

@bwanglzu
Copy link
Member

@JoanFM I will

@bwanglzu bwanglzu linked a pull request Mar 18, 2021 that will close this issue
@FionnD FionnD added status/in-progess This issue is activity being worked on. and removed status/available The issue is available for someone to begin working on. labels Mar 19, 2021
@bwanglzu bwanglzu linked a pull request Mar 23, 2021 that will close this issue
@FionnD FionnD added status/done This issue is complete. and removed status/in-progess This issue is activity being worked on. labels Mar 29, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
priority/critical Urgent: Security, critical bugs, blocking issues. drop everything until this issue is addressed. status/done This issue is complete. type/maintenance This issue is not a bug or a feature_request.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants