Offline delete execution after orphan block scan

[shino]

• Refactor select_gc_bucket and offline_delete scripts to calculate vnode IDs
  in offline_delete side
• Use offline_delete for output files from block_audit and ensure_orphan_blocks
• Add test for select_gc_bucket
• Misc refactoring

[kuenishi]

I'd recommend using CSBucket and CSKey as this will be confusing (originally I wrote this, sorry!)

[kuenishi]

If you can fix nitpicks above today, I'll wait .

[kuenishi]

... wait +1ing.

[shino]

I pushed the update, master.

[kuenishi]

Output of manual test said:

$ riak_cs/rel/riak-cs/bin/riak-cs escript ~/cs-2.1/riak_cs/priv/tools/internal/block_audit.erl -h localhost -o /tmp/select_gc.txt         
15:09:56.477 [info] Application lager started on node nonode@nohost
15:09:56.512 [info] Retrieved bucket list. There are 1 buckets, including tombstones.
15:09:56.512 [info] Searching for orphaned blocks. This may take a while...
15:09:56.512 [info] Finding Orphaned blocks for Bucket "test"
15:09:56.537 [info] Logged 2465 block keys to test
15:09:56.736 [info] handle_manifest_fold done for bucket: "test"

I thought the code

    info("Total number of UUIDs that has any orphaned blocks: ~p [count]", [UUIDs]),
    info("Total number of orphaned blocks: ~p [count]", [Blocks]),
    info("Orphaned Blocks written to ~p", [filename:absname(Filename)]).

shows the file to use next, but it didn't appear. Any idea what was wrong?

[kuenishi]

I just removed whole data in LevelDB.

[kuenishi]

Ah, that's because I deleted bucket record. mmmm.....

[kuenishi]

fmm, failing again

$ riak_ee/rel/riak/bin/riak escript ~/cs-2.1/riak_cs/priv/tools/internal/offline_delete.erl  -r 8 --dry-run ~/cs-2.1/riak_ee/rel/riak/data/bitcask /tmp/select2_gc.txt/test
[Warning]
Make sure any Riak process using '/Users/kuenishi/cs-2.1/riak_ee/rel/riak/data/bitcask' is not running or your data may corrupt.
Accept the terms of conditions? [y/N] y
64 bitcask directories at /Users/kuenishi/cs-2.1/riak_ee/rel/riak/data/bitcask opened.
Using bitcask key version: 1.
escript: exception error: no function clause matching mochihex:dehex(116) (src/mochihex.erl, line 32)
  in function  mochihex:to_bin/2 (src/mochihex.erl, line 61)
  in call from offline_delete:for_each_line/6 (/Users/kuenishi/cs-2.1/riak_cs/priv/tools/internal/offline_delete.erl, line 133)
  in call from offline_delete:offline_delete/3 (/Users/kuenishi/cs-2.1/riak_cs/priv/tools/internal/offline_delete.erl, line 108)
  in call from escript:run/2 (escript.erl, line 747)
  in call from escript:start/1 (escript.erl, line 277)
  in call from init:start_it/1 
  in call from init:start_em/1 

(??)

[kuenishi]

oh, I need to run ensure script before.

[kuenishi]

Hey @borshop

[kuenishi]

Merging as @borshop is temporarily unavailable, manually tested at laptop.