race condition: Add lock around catchpointsMu to avoid race condition

[Eric-Warehime]

Summary

While upgrading to golang 1.17.9 I've run into a couple of race conditions which have been detected during E2E tests. This fixes one of them during which a write happens at https://github.com/algorand/go-algorand/blob/master/ledger/catchpointtracker.go#L417 and a read happens at https://github.com/algorand/go-algorand/blob/master/ledger/catchpointtracker.go#L173

Test Plan

e2e tests

[codecov-commenter]

Codecov Report

Merging #3944 (8052cb4) into master (cfe6e57) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master    #3944   +/-   ##
=======================================
  Coverage   49.97%   49.98%           
=======================================
  Files         400      400           
  Lines       68633    68635    +2     
=======================================
+ Hits        34301    34305    +4     
- Misses      30630    30631    +1     
+ Partials     3702     3699    -3     

Impacted Files	Coverage Δ
ledger/catchpointtracker.go	57.06% <100.00%> (+0.16%)	⬆️
ledger/blockqueue.go	82.18% <0.00%> (-2.88%)	⬇️
catchup/service.go	68.14% <0.00%> (-1.24%)	⬇️
data/transactions/verify/txn.go	44.15% <0.00%> (-0.87%)	⬇️
cmd/tealdbg/debugger.go	72.69% <0.00%> (-0.81%)	⬇️
ledger/acctupdates.go	69.30% <0.00%> (+0.79%)	⬆️
network/wsPeer.go	71.66% <0.00%> (+2.77%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cfe6e57...8052cb4. Read the comment docs.

[cce]

This same lock is taken to update ct.roundDigest below, @algorandskiy do you think both ct.roundDigest and ct.lastCatchpointLabel need to be updated in the same locked section?

[algorandskiy]

Technically this is not necessary since we generate it every 10k rounds, and not every round.
But for consistency I agree, it is better to move this if statement under the lock below.
@Eric-Warehime could you please do that?

[cce]

@algorandskiy for historical purposes I looked it up in an old release before the catchpointtracker was introduced, like 3.0.1, and au.lastCatchpointLabel is updated just before au.accountsMu.Lock() too

https://github.com/algorand/go-algorand/blob/v3.0.1-stable/ledger/acctupdates.go#L2265
https://github.com/algorand/go-algorand/blob/v3.0.1-stable/ledger/acctupdates.go#L644

[tolikzinovyev]

FYI, catchpointTracker.accountDataResourceSeparationRound is not protected my a mutex in all places -- another data race.