node_ip changes while sorting the node ips based on last used

[AmeerHajAli]

I started an internal use case with 50 workers and got the following error:

2021-02-21 07:10:57,533 ERROR autoscaler.py:139 -- StandardAutoscaler: Error during autoscaling.
Traceback (most recent call last):
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/autoscaler.py", line 137, in update
    self._update()
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/autoscaler.py", line 180, in _update
    sorted_node_ids = self._sort_based_on_last_used(nodes, last_used)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/autoscaler.py", line 327, in _sort_based_on_last_used
    return sorted(nodes, key=last_time_used, reverse=True)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/autoscaler.py", line 325, in last_time_used
    return updated_last_used[node_ip]
KeyError: '172.31.24.212'

I am assuming that the call to self.provider.internal_ip(node_id) changes between when we set the default to -1 and when we sort.

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(