Improvements for integration tests

[brianjohnson5972]

Changed label for tests that we expect to fail and marked some longer tests with medium_running_tests to reduce the runtime for this group

Other

Added nonparallelizable_failing_tests and long_running_failing_tests labels to allow avoiding running tests that have not been fixed to work with changes made to the Wire network resources allocations.

[valthon]

I appreciate switching the label for long running tests that are (currently) failing, as that's consistent with the other changes we've made to narrow down the "must pass" suite. However, I'm not in favor of adding medium_running_tests as it really feels redundant with the long_running_tests label. Too much nuance here is just asking for missed tests and confused future developers. Either it's a long-running test or it's fast enough to run in our normal set.

[bearcubsvet]

I appreciate switching the label for long running tests that are (currently) failing, as that's consistent with the other changes we've made to narrow down the "must pass" suite. However, I'm not in favor of adding medium_running_tests as it really feels redundant with the long_running_tests label. Too much nuance here is just asking for missed tests and confused future developers. Either it's a long-running test or it's fast enough to run in our normal set.

Do you have any flag name in mind @valthon ?

[brianjohnson5972]

I think he wants medium and long to combine under long_running_tests. I'm fine with that, but my reason for creating the new one is that both take about 15 minutes to run, so it was to prevent having to lock up your machine, or a cloud resource, for 30 minutes

…

On Mon, Jun 23, 2025, 8:00 AM Svetla Syrimis ***@***.***> wrote: *bearcubsvet* left a comment (Wire-Network/wire-sysio#24) <#24 (comment)> I appreciate switching the label for long running tests that are (currently) failing, as that's consistent with the other changes we've made to narrow down the "must pass" suite. However, I'm not in favor of adding medium_running_tests as it really feels redundant with the long_running_tests label. Too much nuance here is just asking for missed tests and confused future developers. Either it's a long-running test or it's fast enough to run in our normal set. Do you have any flag name in mind @valthon <https://github.com/valthon> ? — Reply to this email directly, view it on GitHub <#24 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACOLKW6N6LDHLJ7NJU7PTFL3E73AXAVCNFSM6AAAAAB7WT7WG6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDSOJWGQYDONJYGU> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[valthon]

yeah, i think we just stick with one flag. if we split "medium" and "long" then what's to stop us from having "sort of long," "really long," "no more than 2 minutes," "less than 18 seconds," etc. It's better practice to stick to a binary for that sort of differentiation. The basic assumption is that developers will usually only run the fastest option unless they need full coverage in which case the longest option needs to be run. Any "middle ground" turns out to have extremely minimal benefits in exchange for unnecessary complexity. In scenarios where you're not just running the fastest or the most coverage you're usually only needing to run one or a small number of explicitly-relevant tests -- extremely rare that a full "long but not really long" suite adds value

[brianjohnson5972]

You sold me on it

…

On Mon, Jun 23, 2025, 8:35 AM David J Parrott ***@***.***> wrote: *valthon* left a comment (Wire-Network/wire-sysio#24) <#24 (comment)> yeah, i think we just stick with one flag. if we split "medium" and "long" then what's to stop us from having "sort of long," "really long," "no more than 2 minutes," "less than 18 seconds," etc. It's better practice to stick to a binary for that sort of differentiation. The basic assumption is that developers will usually only run the fastest option unless they need full coverage in which case the longest option needs to be run. Any "middle ground" turns out to have extremely minimal benefits in exchange for unnecessary complexity. In scenarios where you're not just running the fastest or the most coverage you're usually only needing to run one or a small number of explicitly-relevant tests -- extremely rare that a full "long but not really long" suite adds value — Reply to this email directly, view it on GitHub <#24 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACOLKWYWMMUACTVAY23RPHT3E77B3AVCNFSM6AAAAAB7WT7WG6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDSOJWGUZDSNZSGU> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[bearcubsvet]

@brianjohnson5972

[dtaghavi]

yeah, i think we just stick with one flag. if we split "medium" and "long" then what's to stop us from having "sort of long," "really long," "no more than 2 minutes," "less than 18 seconds," etc. It's better practice to stick to a binary for that sort of differentiation. The basic assumption is that developers will usually only run the fastest option unless they need full coverage in which case the longest option needs to be run. Any "middle ground" turns out to have extremely minimal benefits in exchange for unnecessary complexity. In scenarios where you're not just running the fastest or the most coverage you're usually only needing to run one or a small number of explicitly-relevant tests -- extremely rare that a full "long but not really long" suite adds value

I agree with this, that's probably what I'd do run the fastest and once I am "finished" run the full suite. Don't want it to get to convoluted.

[dtaghavi]

Just remove the medium-running from README to match your latest commit, otherwise looks good to me.