Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize cargo flaky #3273

Open
irevoire opened this issue Dec 22, 2022 · 3 comments
Open

Optimize cargo flaky #3273

irevoire opened this issue Dec 22, 2022 · 3 comments
Labels
help wanted Extra attention is needed maintenance Issue about maintenance (CI, tests, refacto...)

Comments

@irevoire
Copy link
Member

irevoire commented Dec 22, 2022

Currently, the tests appear to be flaky in our CI (tldr: they're not).

That's because cargo-flaky doesn't handle workspaces correctly.
A simple fix is to simply cd in each sub-directory to run flaky test instead of running it at the root of the project.
But the real fix should be implemented in cargo-flaky; since our use-case is pretty simple and we're our only users, it should be fairly simple to extract all the existing modules and their associated paths and see if it matches the binary we're trying to run; if it's the case, we execute the process from this path.

In the future, we should definitely look into a better way to handle this process, I'm sure we're not the first one stumbling on this kind of issue.
Also, I took a quick glance at cargo-nextest and I must admit I did not really understand when they're changing their path before running their test, so maybe there is another way that I don't know.

@irevoire irevoire added maintenance Issue about maintenance (CI, tests, refacto...) help wanted Extra attention is needed labels Dec 22, 2022
@curquiza curquiza added the bug Something isn't working as expected label Jan 2, 2023
@curquiza
Copy link
Member

curquiza commented Jan 2, 2023

More additional details about this issue in the old created one: #3196 (comment)

@curquiza curquiza added this to the v1.1.0 milestone Jan 4, 2023
@curquiza
Copy link
Member

https://github.com/meilisearch/meilisearch/pull/3348/files partially fixes the issue. However, we will have to add every new sub crate to the CI, which is not convenient. Ideally, we should revert this commit and fix cargo-flaky directly.

bors bot added a commit that referenced this issue Jan 17, 2023
3348: fix cargo flaky r=curquiza a=irevoire

Partially fix #3273

Ideally, we should revert this commit and fix cargo-flaky directly to ensure we never forget to add a sub-crate to the CI.

-----

Here is an example of the CI running (and thus working); https://github.com/meilisearch/meilisearch/actions/runs/3932783699/jobs/6725755801

Co-authored-by: Tamo <tamo@meilisearch.com>
@bors bors bot closed this as completed in 1af3089 Jan 18, 2023
@curquiza
Copy link
Member

reopening, this PR does not fix the issue totally

@curquiza curquiza reopened this Jan 18, 2023
@curquiza curquiza changed the title Fix the flaky tests Optimize cargo flaky Jan 18, 2023
@curquiza curquiza removed this from the v1.1.0 milestone Jan 18, 2023
@ManyTheFish ManyTheFish removed the bug Something isn't working as expected label Jul 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Extra attention is needed maintenance Issue about maintenance (CI, tests, refacto...)
Projects
None yet
Development

No branches or pull requests

3 participants