Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speculative deflake for assessments_controller_test #45638

Merged

Conversation

bethanyaconnor
Copy link
Contributor

@bethanyaconnor bethanyaconnor commented Apr 4, 2022

This has failed a few times recently in the DTT.

Example log:

==========�[0m�[1000D�[K FAIL["test_verified_teacher_should_get_assessments_responses", "Api::V1::AssessmentsControllerTest", 1207.9027286842465]
 test_verified_teacher_should_get_assessments_responses#Api::V1::AssessmentsControllerTest (1207.90s)
        --- expected
        +++ actual
        @@ -1 +1 @@
        -{"914"=>{"student_name"=>"student_0", "responses_by_assessment"=>{"2118"=>{"lesson"=>"bogus-script-16", "puzzle"=>1, "question"=>"Long assessment 1", "url"=>"//test-studio.code.org/s/bogus-script-16/lessons/1/levels/1?section_id=1&user_id=914", "multi_correct"=>1, "multi_count"=>4, "match_correct"=>1, "match_count"=>3, "submitted"=>true, "timestamp"=>Mon, 04 Apr 2022 21:38:23 UTC +00:00, "level_results"=>[{"student_result"=>"This is a free response", "status"=>"", "type"=>"FreeResponse"}, {"type"=>"Multi", "student_result"=>[0], "status"=>"correct"}, {"type"=>"Multi", "student_result"=>[1], "status"=>"incorrect"}, {"type"=>"Multi", "student_result"=>[], "status"=>"unsubmitted"}, {"type"=>"Match", "student_result"=>[nil, nil], "status"=>["unsubmitted", "unsubmitted"]}, {"type"=>"Match", "student_result"=>[0, 1], "status"=>["submitted", "submitted"]}, {"type"=>"Match", "student_result"=>[1, 0], "status"=>["submitted", "submitted"]}, {"type"=>"Multi", "status"=>"unsubmitted"}]}}}}
        +{"914"=>{"student_name"=>"student_0", "responses_by_assessment"=>{"2118"=>{"lesson"=>"bogus-script-16", "puzzle"=>1, "question"=>"Long assessment 1", "url"=>"//test-studio.code.org/s/bogus-script-16/lessons/1/levels/1?section_id=1&user_id=914", "multi_correct"=>1, "multi_count"=>4, "match_correct"=>1, "match_count"=>3, "submitted"=>true, "timestamp"=>"2022-04-04T21:38:22.000+00:00", "level_results"=>[{"type"=>"FreeResponse", "student_result"=>"This is a free response", "status"=>""}, {"type"=>"Multi", "student_result"=>[0], "status"=>"correct"}, {"type"=>"Multi", "student_result"=>[1], "status"=>"incorrect"}, {"type"=>"Multi", "student_result"=>[], "status"=>"unsubmitted"}, {"type"=>"Match", "student_result"=>[nil, nil], "status"=>["unsubmitted", "unsubmitted"]}, {"type"=>"Match", "student_result"=>[0, 1], "status"=>["submitted", "submitted"]}, {"type"=>"Match", "student_result"=>[1, 0], "status"=>["submitted", "submitted"]}, {"type"=>"Multi", "status"=>"unsubmitted"}]}}}}
        test/controllers/api/v1/assessments_controller_test.rb:282:in `block in <class:AssessmentsControllerTest>'
        test/testing/setup_all_and_teardown_all.rb:36:in `run'

It's a bit difficult to read but the only difference is the timestamp. The expected timestamp is one second later than the actual timestamp. I can repro this locally by sleeping for half a second before creating the sublevel user levels (line 237 in this PR).

Looking at the code, timestamp in the response is supposed to be the level group's timestamp, not the sublevel's. Likely, assigning the sublevels to user_level was a mistake that just went unnoticed until recently. I renamed that variable to be more clear as well.

This is a new area of the codebase for me so let me know if I misunderstood anything!

PR Checklist:

  • Tests provide adequate coverage
  • Privacy and Security impacts have been assessed
  • Code is well-commented
  • New features are translatable or updates will not break translations
  • Relevant documentation has been added or updated
  • User impact is well-understood and desirable
  • Pull Request is labeled appropriately
  • Follow-up work items (including potential tech debt) are tracked and linked

@bethanyaconnor bethanyaconnor marked this pull request as ready for review April 5, 2022 03:00
@bethanyaconnor bethanyaconnor requested a review from a team April 5, 2022 03:01
Copy link
Contributor

@maureensturgeon maureensturgeon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! I didn't notice it was using the wrong timestamp, good catch!

@@ -233,13 +233,13 @@ class Api::V1::AssessmentsControllerTest < ActionController::TestCase
]

# create user_level for level_group
user_level = create :user_level, user: @student_1, best_result: 100, script: script, level: level1, submitted: true
level_group_user_level = create :user_level, user: @student_1, best_result: 100, script: script, level: level1, submitted: true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice. Great find!

@bethanyaconnor bethanyaconnor merged commit 5bbe0a8 into staging Apr 5, 2022
@bethanyaconnor bethanyaconnor deleted the bethany/deflake-assessments-controller-test branch April 5, 2022 16:29
@bethanyaconnor bethanyaconnor restored the bethany/deflake-assessments-controller-test branch June 22, 2022 21:47
@fisher-alice fisher-alice deleted the bethany/deflake-assessments-controller-test branch July 13, 2022 22:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants