Beef up and harden import/export tests

[ribasushi]

Fixes ( or at least makes cleaner ) #7123

At this point I am convinced that this comment I made way back is in fact real: https://github.com/ipfs/go-ipfs/pull/7011/files/aa5019413741f2d34c9723d376591cf8c4b6b465#diff-15172926c4147422f472139313467179R330-R341
We did switch away from gclock a while back, and likely other changes I made cause it to trigger less frequently. Reproducing this locally is nigh impossible however :(

So instead the timeout+QUIT contraptions should tell us more next time it happens on the slower circleci boxes.

[Stebalien]

Are we talking about two separate issues here? There's:

1. The sharness issue.
2. The GC issue.

Your tests prove that this issue can't be related to GC locks by successfully verifying that none of the pinned blocks in the CARs are garbage collected. If this were a GC race, we'd be deleting pinned data.

We know what at least part of the GC issue is. The issue is that we're garbage collecting parts of the pin set. We could be generating and garbage collecting additional data, but we'd have to look into that by looking at the blocks GC is deleting.

[ribasushi]

@Stebalien not a GC race - a GC deadlock. I am 99% sure this is what you observed in 7123. What I am modifying is that if we trigger a generous timeout: we will dump a full stacktrace where I "expect nothing" otherwise.

The other thing ( ephemeral pinsets showing up ) is tracked in a different ticket.

[Stebalien]

Ah, got it.

The bug we were running into there was that we were invoking the CoreAPI while already holding the PinLock. However, we could definitely have a different deadlock somewhere.

[Stebalien]

Doing some spelunking: I don't think it's a starvation issue. That is, repeated GC should eventually allow a dag import to proceed.

[Stebalien]

I agree that the behavior of sharness is surprising, but I'm concerned that this will cause more problems than it fixes. It already has a bunch of special cases to avoid rejecting valid code.

I'd rather document this in the sharness README and move on unless we can find a more reliable way to do this.

[ribasushi]

Nobody will ever discover the sharness README :)
No worries though - I will pull this commit out, and use it locally instead.
Unless you are open to have it gated on an environment variable...?

[ribasushi]

Force-pushed a removal ( it was the first commit )

[Stebalien]

Ideally, we'd trigger it when we get a failure. However, in this case, that wouldn't have helped regardless because we weren't checking for errors.

You're right that nobody will read the README... Honestly, it's not terrible given that we can catch and deal with any false positives. What if we include it but put in a comment explaining exactly what it does, where it comes from (it's non-obvious that we're patching it in), and that it's a heuristic and that we should remove it if it causes problems?

[ribasushi]

I'll open a separate PR once I had a chance to write up better prose. The the error will link to that.

[ribasushi]

For now let's merge the rest to catch the lock if it happens again ( all the "$( stuff, while annoying was a red herring )