Adding script to update time_spent to seconds from milliseconds

[jmkulwik]

The time_spent field in the user_levels table has been recording time in milliseconds. This will cause time_spent to max out at ~25 days due to mysql's integer size. Instead we want to record time in seconds. This will cause time_spent to max out at ~68 years.

Recording time_spent was temporarily disabled by this pr so we can make this change on all existing UserLevel records.

Links

• jira

Testing story

Reviewer Checklist:

• Tests provide adequate coverage
• Code is well-commented
• New features are translatable or updates will not break translations
• Relevant documentation has been added or updated
• User impact is well-understood and desirable
• Pull Request is labeled appropriately
• Follow-up work items (including potential tech debt) are tracked and linked

[clareconstantine]

One way to mitigate the impact of running this query on such a big table would be to use a method like find_each (info) that can be chained to a where, that gets the records in batches rather than trying to instantiate them all at once.

[clareconstantine]

Additionally, it might make sense to do this in a transaction so that if something goes wrong, you won't end up in a state where some records are in seconds and other are in milliseconds. A good example is probably this script - although you would want all of your updates to happen in one transaction. That might negate the benefits of using find_each, though, depending on how transactions work.

[clareconstantine]

I like your approach! this seems like a great way to combine the transaction + batches :)

[clareconstantine]

if UserLevels could have been deleted, do we also need the ending id of each slice? And is there anything you want to capture if it fails, like which row it failed on? (maybe this is captured in whatever error message you get by default)

[jmkulwik]

Ending id: I don't think we need it. It'll just be staring_id + 5000
Row it failed on: Hmm. I thought about this for a bit. I'm super curious if you have additional thoughts on the matter. This is my first migration, so I don't have much experience to go on. My gut feeling: It might be useful as a learning experience for me, but for the migration itself, I don't think it would end up saving any time. Since the migration logic is so short and the slices are relatively small, the time spent writing error logging would outweigh the benefits of having that logging.

[sureshc]

I don't think we need to wrap each slice of 5000 rows in a single transaction. Can we log any row that we fail to update (Rescue any error and log the id and the reason) and then move on the next row?

[sureshc]

Could we use find_each.with_index to track where we are and occasionally (every 5000 rows?) output where we are, so we can monitor this long running process similar to the way we were previously logging the start of each new slice?

[jmkulwik]

👍

[sureshc]

The ActiveRecord update_column method might be appropriate to use here. We have callbacks on this model, and performance might improve and we might avoid unintended side effects by skipping callbacks.
https://apidock.com/rails/ActiveRecord/Persistence/update_column

[jmkulwik]

From running this on the adhoc, I'm seeing only the column we care about being updated

Before:
+----------+----------+---------------------+---------------------+-------------+-----------+-----------------+-----------+------------------+-------------+------------+
| level_id | attempts | created_at | updated_at | best_result | script_id | level_source_id | submitted | readonly_answers | unlocked_at | time_spent |
+----------+----------+---------------------+---------------------+-------------+-----------+-----------------+-----------+------------------+-------------+------------+
| 16073 | 1 | 2019-10-14 15:30:12 | 2020-08-20 15:31:20 | 100 | 375 | 852310263 | 0 | NULL | NULL | 0 |
| 11878 | 5 | 2019-10-14 16:37:44 | 2020-08-20 13:49:50 | 100 | 302 | 613711192 | 0 | NULL | NULL | 6618 |
| 11879 | 38 | 2019-10-14 16:40:23 | 2020-08-20 13:50:07 | 20 | 302 | 951400461 | 0 | NULL | NULL | 10165 |
| 11986 | 1 | 2019-10-14 16:50:10 | 2020-08-20 13:50:10 | 100 | 302 | 613341647 | 0 | NULL | NULL | 0 |

After:
+----------+----------+---------------------+---------------------+-------------+-----------+-----------------+-----------+------------------+-------------+------------+
| level_id | attempts | created_at | updated_at | best_result | script_id | level_source_id | submitted | readonly_answers | unlocked_at | time_spent |
+----------+----------+---------------------+---------------------+-------------+-----------+-----------------+-----------+------------------+-------------+------------+
| 16073 | 1 | 2019-10-14 15:30:12 | 2020-08-20 15:31:20 | 100 | 375 | 852310263 | 0 | NULL | NULL | 0 |
| 11878 | 5 | 2019-10-14 16:37:44 | 2020-08-20 13:49:50 | 100 | 302 | 613711192 | 0 | NULL | NULL | 7 |
| 11879 | 38 | 2019-10-14 16:40:23 | 2020-08-20 13:50:07 | 20 | 302 | 951400461 | 0 | NULL | NULL | 11 |
| 11986 | 1 | 2019-10-14 16:50:10 | 2020-08-20 13:50:10 | 100 | 302 | 613341647 | 0 | NULL | NULL | 0 |

Similarly, entries that should not be updated have not been changed.

[sureshc]

I know we're pretty committed to this solution, but wanted to note here a couple of other possible solutions ...

A single SQL statement that updates all 600K rows?

UPDATE user_levels
SET time_spent = time_spent / 1000
WHERE time_spent IS NOT NULL;

Or SELECT & export the id and time_spent column for the 600K rows to a CSV and then implement a script that issues a raw SQL UPDATE for each row, one at a time, as it reads through the CSV. This would eliminate the overhead of instantiating a UserLevel model for each row.