-
Notifications
You must be signed in to change notification settings - Fork 636
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conditional testing #27
Comments
Is there a graceful way to handle failed link checks, without failing the whole test suite? |
I'd rather keep running the whole test suite and just remove link checking, but it raises the question of when will we run "occasional" tests? |
@MCGallaspy why do you wanna remove it? It's extremely useful to avoid dead links in the docs, that's why Sphinx has that command built in ;)
Yup, the conditional thing is conditional :D My suggestion is to base it off git changesets, mean if you change something in |
It's useful, but if it occasionally times out and fails the whole suite, its utility goes down in my eyes. I just want to ensure that all the unit tests/integration tests/etc are run regardless of the changeset, as opposed to just a specific app label. However if you're proposing that we only run the link checker when the docs are in the changeset, then I think that's a fine idea. |
But that's the point of this issue?
If you have good heuristics, you don't need to bloat the CI :)
That's the first proposal. But I hope we expand on it so we can make tests faster. I'd personally like a |
Just a few points:
Remember This is not KA Lite. |
Comments from the meeting:
|
I really agree with this, it's a matter of adjusting it with very explicit exceptions. And if we cannot identify an exception to the rule (test everything), then we test everything :) Reality has been quite cruel.. for KA Lite. Even though we tested stuff locally to avoid waiting 20-40 minutes for the CI, a lot of tears were spilled over shaky test frameworks.. because they caused communication work regarding false negatives.. and some traumatizing debugging :) First case of this same phenomena in Kolibri is the sphinx link checker.. which IMO is really useful but should only be executed when explicitly asking for a full set of tests or when changing .rst files. Btw. I maintain an open source project in which only some of the tests were running but all lights were green... when we finally fixed the test discoverer, we had at least a couple severe issues :) -> https://github.com/django-money/django-money/pull/162/files ...with that said, I'm alert to putting too much complexity into the testing frameworks in case should be unknowing that our software has become Real Media Player because our test framework says otherwise :) |
It's done! |
…-logly Add extra logging when we raise exceptions
Some additional i18n explanation
Sphinx link checking fails because of network timeouts (no docs changes and the whole test may still break). I think we know the story of BDD / Saucelabs random failures. I'm vouching we should consider conditional testing based on git commit changesets. I've mentioned it before.
This can speed up testing.
Of course we should still do full testing once in a while.
Something based on this command:
The text was updated successfully, but these errors were encountered: