[Dynatrace v2] Ensure synchronization when taking snapshots

[pirgeo]

We found a timing bug that can appear when using the Dynatrace v2 exporter, usually only when recording data at high volumes. In previous versions, recording a snapshot was not a synchronized operation. That means that it was possible to write data to a DynatraceSummary, while the snapshot was being created. In very rare cases this can lead to invalid data. This PR moves the synchronization while taking a snapshot to the innermost level, and ensures that no writes can happen while a Snapshot is produced.

Edit: As requested below, here is some more context:

This error is really hard to reproduce, since it only happens in a very limited number of cases. One case would be: A counter recording only ones, but a lot of them. The resulting export would have min=1, max=1, sum=999 and count=1000. It means that between reading the sum (999) and the count (1000), another 1 was recorded. This is inconsistent data, and is therefore rejected with an error message like:

inconsistent gauge fields: min <= avg <= max doesn't hold: min: 1.000000, max: 1.000000, sum: 500467.000000, count: 500468, avg: 0.999998, tolerance: 0.000001

With this change, it is possible that there is a very small delay when recording a new value at the exact time a snapshot is taken.

[pirgeo]

@shakuzen or @jonatan-ivanov, could you take a look? I would love to get this into the next patch release, if possible 😄

[jonatan-ivanov]

Since I applied the parameter change suggestion, the build is broken and I can't fix it on the branch but I think I can fix it with applying these suggestions. :)

[jonatan-ivanov]

@pirgeo Thank you for fixing this!
Could you please do two more things?

1. Could you please add a note in the description that with this change recordings can be blocked during publishing for a very small amount of time while we get and reset the values for the snapshot?
2. Could you please add some example error message that Dynatrace will produce if this happens so users who get it and search on it might discover this issue?

[pirgeo]

@jonatan-ivanov done! Should I send another PR for adding the missing test that you pointed out?

Edit: Created a follow up, see here: #3626