Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Non deterministic test output leads to flaky CI Jobs #98

Closed
1 task done
Munsio opened this issue May 8, 2024 · 0 comments
Closed
1 task done

Non deterministic test output leads to flaky CI Jobs #98

Munsio opened this issue May 8, 2024 · 0 comments
Assignees
@Munsio
Labels
bug Something isn't working
Milestone
v0.5.0

Comments

@Munsio
Copy link
Contributor

Munsio commented May 8, 2024 •
edited
Loading

It seems that with #94 we introduced some non-determinism into the tests for CSV output.
https://github.com/symflower/eval-dev-quality/actions/runs/8987716044/job/24686885243#logs

Tasks:

  • Sort the models before writing the CSV output
@Munsio Munsio added the bug Something isn't working label May 8, 2024
@Munsio Munsio added this to the v0.5.0 milestone May 8, 2024
@Munsio Munsio self-assigned this May 8, 2024
Munsio added a commit that referenced this issue May 8, 2024
@Munsio
fix, Sort map by model before creating the CSV row data to create an …
7041452 
…deterministic output

Closes #98
@zimmski zimmski mentioned this issue May 16, 2024
Open
45 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Munsio Munsio

Labels
bug Something isn't working
Projects
None yet
Milestone
v0.5.0
Development

No branches or pull requests
1 participant
@Munsio