Feature/list objects eqc addition (squashed)

[shino]

This PR adds two EQC properties for listing all objects.

• Two EQC properties for listing objects (GET Bucket),
  one without delimiter and one with delimiter "/".
  Those properties assert following two are equal:

  1. keys from listing objects by repetitive usage of
     riak_cs_list_objects_fsm_v2
  2. active keys from generated [{State, Key},...]

• When counterexample is found, states and keys are
  written to file $PWD/.riak_cs_list_objects_fsm_v2_eqc.txt.
  Each line consists of line number, status and key.
  Example output:

  1,active,0123456/10
  2,scheduled_delete,0123456/1000
  

• To generate manifests, generator of whole manifests is avoided
  and generate only states and keys of manifests.
  The property list-all-objects generates several thousands of nested tuples,
  #lfs_manifest_v3 and #acl_v2 in this case.
  But generators of nested tuples seem not so fast.

TODOs remaining (not this PR's TODOs but ones for future)

• prefix of request parameter
• non-default max-keys

[reiddraper]

• non-default max-keys

Added in 0cdb828.

[reiddraper]

This has uncovered a new Riak CS bug. When a delimiter is included, S3 returns an XML node called NextMarker, which is to be used by the client for pagination. Riak CS currently does not support this. Instead, we mistakenly assume the client will just use the last key. Further, when a request is made with both a marker and delimiter, we should be treating the marker as a common delimiter and 'skipping past' it, like we do internally. Once the NextMarker XML node is returned, we should be able to update the EQC test to take advantage of it, and continue running the test. We also may need to update erlcloud, if it doesn't already support this node.

[reiddraper]

Update, erlcloud does not support the NextMarker XML node.

[reiddraper]

I've fixed the lists:last error in 0d2caed, by supporting the NextMarker element. The test is still failing, but it's now failing for a different reason. Progress.

[reiddraper]

Smallest failing case for with_prefix is now something like:

1,writing,ZZZZZZZZ/1
2,writing,ZZZZZZZZ/2
3,active,ZZZZZZZZ/3

[reiddraper]

The bug appears to be related to what @shino initially suspected: we incorrectly skip past a prefix, even if it's deleted. When it's deleted, it won't be included in the result set, and we might be skipping past other keys that fall under that prefix but are not deleted.

[reiddraper]

Removing the skip_past_prefix_and_delimiter optimization makes the test pass. We can't to keep the optimization, but need some way of not using it if the only object that fits under the delimiter is not in the active state.

[reiddraper]

Need to clean things up and squash, but I think I've addressed all of the bugs here. In addition to supporting NextMarker, I've fixed the skip_past_prefix_and_delimiter edge-case here: 3424892.

[shino]

supporting NextMarker

I cherry-picked eqc part of NextMarker support and pushed some prefix generator modification.

[shino]

At the current HEAD (ed88f35), I confirmed

• eqc tests passes with test number 10,000 each (with/without delimiter) with fixes and
• without those fixes (partly), eqc tests generates conterexamples we faced so far (original 1003 keys, common prefixes but no keys in result, false prefix skipping).

Everything seems nice. I will +1 to that branch after squash! 😂

[reiddraper]

Some of the NextMarker support is missing in the cherry-picking. I'll add it and then re-order the commits.

[shino]

Close in favor of #788