retry more operations when accessing storage + enforcer in tests

[isoos]

• Retry ClientException from package:http #8337
• The RetryEnforcerStorage wrapper (used only in tests) and related classes parse the current stacktrace when a storage-method is called, and throw an assertion error if retry is not in the stack. The detection is rather rudimentary (e.g. needs reentry detection for list + arbitrary processing), but it provides some value already. Further improvements can be done with better parsing (TBD later).
• Most of the methods are currently ignoring the lack of retry, as it seems a lot to fix everything all at once. The current updates focus on the API exporter admin test (it seems to be covered 100%), enabling the assertion error while I was investigating the test case.
• I think for some operation it could have been a simple withStorage<R>(Future<R> Function(Storage storage)) {...} helper method that retries internally, but since these operation are not in a transaction, it seems wasted to retry more than one at once.

[sigurdm]

Should we also detect that we only have a single retry call on the stack?

[isoos]

I think we should allow multiple re-entry in the retry stack, e.g. one for a high-level iteration of items, while the second one may be narrower operations on it.

We could ignore re-entry detection in the retry stack if we could check if the current call is sufficiently close to the latest retry invocation. But I would consider that check to come later, after the low-hanging fixes (e.g. no retry at all) has been done.

[sigurdm]

I think we should allow multiple re-entry in the retry stack, e.g. one for a high-level iteration of items, while the second one may be narrower operations on it.

Yeah - we would need to have some kind of design here. But nested retries easily leads to n^2 attempts...

But I would consider that check to come later,

SG.