Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speculative fix for flaky test in user_test.rb #45824

Merged
merged 1 commit into from
Apr 14, 2022

Conversation

bethanyaconnor
Copy link
Contributor

The test test_index_user_levels_by_level_id_returns_first_created_user_level_if_updated_at_is_identical failed in the DTT today with the output:

[0m[1000D[K FAIL["test_index_user_levels_by_level_id_returns_first_created_user_level_if_updated_at_is_identical", "UserTest", 597.216460313648]
 test_index_user_levels_by_level_id_returns_first_created_user_level_if_updated_at_is_identical#UserTest (597.22s)
        --- expected
        +++ actual
        @@ -1 +1 @@
        -{752=>#<UserLevel id: 34, user_id: 274, level_id: 752, attempts: 0, created_at: "2022-04-13 18:23:55", updated_at: "2022-04-12 18:23:55", best_result: nil, script_id: 458, level_source_id: nil, submitted: nil, readonly_answers: nil, unlocked_at: nil, time_spent: nil, deleted_at: nil, properties: nil>}
        +{752=>#<UserLevel id: 35, user_id: 274, level_id: 752, attempts: 0, created_at: "2022-04-13 18:23:56", updated_at: "2022-04-12 18:23:56", best_result: nil, script_id: 458, level_source_id: nil, submitted: nil, readonly_answers: nil, unlocked_at: nil, time_spent: nil, deleted_at: nil, properties: nil>}
        test/models/user_test.rb:4213:in `block in <class:UserTest>'
        test/testing/setup_all_and_teardown_all.rb:36:in `run'

Notably, the updated_at timestamps are not identical, so my guess that the clock ticked between the creation of the user levels in this test. I believe freezing time should prevent this from happening, but obviously it's very difficult to know 馃槄

Links

Testing story

Deployment strategy

Follow-up work

Privacy

Security

Caching

PR Checklist:

  • Tests provide adequate coverage
  • Privacy and Security impacts have been assessed
  • Code is well-commented
  • New features are translatable or updates will not break translations
  • Relevant documentation has been added or updated
  • User impact is well-understood and desirable
  • Pull Request is labeled appropriately
  • Follow-up work items (including potential tech debt) are tracked and linked

@bethanyaconnor bethanyaconnor changed the title Freeze time in flaky unit test Speculative fix for flaky test in user_test.rb Apr 13, 2022
@bethanyaconnor bethanyaconnor requested a review from a team April 13, 2022 23:31
Copy link
Contributor

@jamescodeorg jamescodeorg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, good catch. Another fix might be to just calculate the updated_at time once, store it in a variable, and passing it to all three create calls. The current fix is fine too.

@bethanyaconnor
Copy link
Contributor Author

I thought of that but slightly preferred this solution. No strong reason why though so if there's a benefit to calculating the time once, I'm happy to update in the future.

@bethanyaconnor bethanyaconnor merged commit fbfa051 into staging Apr 14, 2022
@bethanyaconnor bethanyaconnor deleted the bethany/fix-flaky-test branch April 14, 2022 15:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants