db: split up TestCompaction

We split off the main part of `TestCompaction`. Now each test file
initializes new state (which is less fragile and allows reproducing a
specific test file failure with a given seed).

Author: RaduBerinde