Stop sharding the gameplay tests twice on CI - #1034
Merged
Merged
Conversation
The runner defaulted its shard to CIRCLE_NODE_INDEX/CIRCLE_NODE_TOTAL, which CircleCI sets on every container as soon as parallelism is above 1. The all-games job therefore cut the run down twice: `--list` printed only this container's quarter of the games, `circleci tests split` split that quarter again, and the run sharded what was left once more. With 4 containers each one ended up testing a single game, so 4 of the 45 games with gameplay tests were tested on main instead of all of them. Sharding now only happens when --shard-index/--shard-total are passed explicitly. The CI splits with `circleci tests split`, which balances by the timings recorded by store_test_results; this script runs what it is given. That also explains the "No timing found" lines: with only four games ever run, there were no timings for any of the others. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01M2jc7PVAvmMmirAQude2v1
Contributor
One test per game, for the ones whose result is worth looking at: the tank's target after the shell exploded on it, the target that was shot at in the first person shooter, the car past the finish line, the RTS selection walking to where it was sent, the two 3D boards after a piece was dropped or a tile placed, the RPG scene after the NPC was answered, the first person farming inventory holding the harvested seed, and both platformers once their coins are collected. They are 9 to 17 KB each and the CI already stores them next to the results of the game they belong to. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01M2jc7PVAvmMmirAQude2v1
The message only said "the gameplay tests runner or the CI configuration changed", leaving the reader to work out which of the six files it was, and why a Pull Request that touches no game was testing all 45 of them. It now names the file and says what it implies. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01M2jc7PVAvmMmirAQude2v1
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On
main, the all-games job was testing 4 of the 45 games instead of all of them.What was happening
run-gameplay-tests.jsdefaulted its shard toCIRCLE_NODE_INDEX/CIRCLE_NODE_TOTAL, which CircleCI sets on every container as soon asparallelismis above 1. The games were therefore cut down three times over:--list(sharded by the env vars)circleci tests splitsplits that againReproduced locally by setting the same variables:
The
changedjob was unaffected because its parallelism is 1, so the variables never kicked in — which is why only the all-games job looked wrong.This also explains the
No timing found for ...lines:circleci tests split --split-by=timingslooks up each game in the timings recorded bystore_test_results, and with only four games ever run there were no timings for the rest. It should settle on its own once every game runs again.The fix
Sharding now only happens when
--shard-index/--shard-totalare passed explicitly. The CI splits withcircleci tests split, which balances by recorded timings; the script runs what it is given.After the change, with the same environment variables set:
--liston a containercircleci tests splitto divide)--projects-file--shard-total=4 --shard-index=0Not changed: the screenshots
Checked separately — the artifacts are correct, there is simply nothing to store. No test calls
harness.takeScreenshot(...)(0 across all 45 games), and the harness only captures when asked, so a failing test produces no screenshot either. The storage path itself works: with a test that does take one, the artifacts come out asMaking failures self-documenting would mean capturing a screenshot automatically when a test fails, which belongs in the GDevelop harness rather than here.
🤖 Generated with Claude Code
https://claude.ai/code/session_01M2jc7PVAvmMmirAQude2v1
Generated by Claude Code