Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

run fuzz in CI #165

Merged
merged 1 commit into from
Mar 23, 2023
Merged

run fuzz in CI #165

merged 1 commit into from
Mar 23, 2023

Conversation

jakkdl
Copy link
Member

@jakkdl jakkdl commented Mar 22, 2023

let's see how long they take to run in CI, I usually don't run them in my workflow locally since they're slow - but it's probably nice if they're run regularly somewhere

@jakkdl
Copy link
Member Author

jakkdl commented Mar 22, 2023

The check is once again fixed in #163, I kind of expect the fuzz tests to fail currently - so make sure to rerun after merging the others ... if the runtime looks the least bit reasonable.

@Zac-HD
Copy link
Member

Zac-HD commented Mar 22, 2023

~20 minutes to run the fuzztests seems fine to me, we could make that a non-required CI check. pytest-xdist would probably make that ~2x faster too.

If we're getting up above 20-30 minutes, I'd probably make this a scheduled job and look at the results occasionally. Fancier schemes can actually share a cache between them and use that to feed discovered failures back into the main job, but that's still pretty early for Hypothesis+HypoFuzz so I don't think I'd use it here yet.

@jakkdl
Copy link
Member Author

jakkdl commented Mar 23, 2023

Added a flag so slow_tests only runs the slow tests, added xdist, and rebased on top of main.
with "non-required" do you simple mean not listing it in needs for the release test?

Amusingly enough it actually runs faster in CI than on my personal computer now, due to me having way more installed packages ... _iter_python_files() gives 3900 files 😅 (inside tox environment it's a more reasonable 1561 ... which itself takes 11 mins on my laptop 😔 )

@Zac-HD Zac-HD merged commit e66e8a1 into python-trio:main Mar 23, 2023
@Zac-HD
Copy link
Member

Zac-HD commented Mar 23, 2023

with "non-required" do you simple mean not listing it in needs for the release test?

I mean "oops, let me go set up some branch protection rules so you can't merge a PR until the tests pass" 😆

@jakkdl jakkdl deleted the CI_tests branch March 24, 2023 10:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants