Fix compare mode to not check node bucket files#179
Conversation
Compare mode now always filters to buckets the current process actually ran, regardless of whether node_index is set. This fixes parallel test support (e.g. parallel_tests gem) where each process only sees a subset of specs but compares against the full shitlist. Previously, this filtering was gated behind the parallel? check which required node_index. Now node_index is only needed for save mode (to write shard files).
The filter was a no-op: buckets not run by this process have identical values in both normalized and stored (both read from the same shitlist file), so they always compare equal.
When node_index is set during compare, target_path would return the shard path (which does not exist after merge). This caused normalized_deprecation_messages to read an empty file instead of the full shitlist, breaking the comparison for non-run buckets.
|
I'm curious, does it even make sense to use compare for a shard? what I mean is that if you add a new test file, maybe your files tested in each shard change, rendering the compare kinda useless cause you won't be comparing the same files, or if using the timing function in CircleCI to split tests, if tests are added/removed that will change the times and maybe also change the files executed in each shard I feel like compare should be done at the end of the process after calling the merge command and not for a single shard? |
@arielj correct, this is the fix to follow that, compare should be run at the end (we cannot enforce that, can we?), what we can is ignore the node files even if the node_index is added, the compare command is only going to use the “general” shitlist file. Let me double check the changes 🤔 |
JuanVqz
changed the title
2 participants
Summary
While replying to #98, I realized that compare mode in parallel setups should not check for node-specific shard files since those nodes do not have full test suite data, which would result in false positives.
node_indexis now only needed for save mode (to write shard files)node_indexis only for save modenode_indexis safely ignored in compare mode