DELETE object does not delete writing state manifests [JIRA: RCS-316]

[shino]

At tag 1.2.2 (expected result, {leeway_seconds, 2} in app.config):

# Try PUT 1GB object and disconnect in uploading (by Ctr-C)
$ s3cmd --disable-multipart -c .s3cfg.8071.alice put /data/static/rand.1GB s3://test
/data/static/rand.1GB -> s3://test/rand.1GB  [1 of 1]
   26419200 of 1073741824     2% in    2s    11.39 MB/s^CSee ya!

# Then inspect the bucket, there is one 'writing' manifest
$ scripts.local/riak_cs_inspector.sh 127.0.0.1 dev1 test
Key ============================: Sibl. == State ========== UUID =========================== Content-Length== First Block ====
rand.1GB                        : 1        writing          8334e5fc480c4ea1836226701a643451       1073741824 Found

# Wait 2 sec (= leeway_seconds) and delete it
(py27_base)shino@shino-xub-vbox$ s3cmd -v -c .s3cfg.8071.alice del s3://test/rand.1GB
INFO: Applying --exclude/--include
INFO: Summary: 1 remote files to delete
File s3://test/rand.1GB deleted

# The manifest turns into 'scheduled_delete', nice
$ scripts.local/riak_cs_inspector.sh 127.0.0.1 dev1 test
Key ============================: Sibl. == State ========== UUID =========================== Content-Length== First Block ====
rand.1GB                        : 1        scheduled_delete 8334e5fc480c4ea1836226701a643451       1073741824 **Not Found**

# After GC, the manifest disappears
$ scripts.local/riak_cs_inspector.sh 127.0.0.1 dev1 test
Key ============================: Sibl. == State ========== UUID =========================== Content-Length== First Block ====

At 1.4.3 (unexpected, also {leeway_seconds, 2} in app.config)

# Try PUT 1GB object and disconnect in uploading (by Ctr-C)
$ s3cmd --disable-multipart -c .s3cfg.8071.alice put /data/static/rand.1GB s3://test
/data/static/rand.1GB -> s3://test/rand.1GB  [1 of 1]
   19075072 of 1073741824     1% in    1s    17.64 MB/s^CSee ya!

# Then inspect the bucket, there is one 'writing' manifest
$ scripts.local/riak_cs_inspector.sh 127.0.0.1 dev1 test
Key ============================: Sibl. == State ========== UUID =========================== Content-Length== First Block ====
rand.1GB                        : 1        writing          4e1c3f19082048e496c44c7a05eeccc9       1073741824 Found

# Wait 2 sec (= leeway_seconds) and delete it
$ s3cmd -v -c .s3cfg.8071.alice del s3://test/rand.1GB
INFO: Applying --exclude/--include
INFO: Summary: 1 remote files to delete
File s3://test/rand.1GB deleted

# The manifest still remains in 'writing' state.
$ scripts.local/riak_cs_inspector.sh 127.0.0.1 dev1 test
Key ============================: Sibl. == State ========== UUID =========================== Content-Length== First Block ====
rand.1GB                        : 1        writing          4e1c3f19082048e496c44c7a05eeccc9       1073741824 Found

1.4.0 and 1.3.2 show the same result as 1.4.3.

[kuenishi]

Partial solution is #1280 that collects manifest and blocks when a socket was closed by client or any other errors. Do we add another work on "existing" stale write objects?

[shino]

For existing manifests (writing and pending_delete) should be treated in another way, I think. #937 is the issue for that. So, I agree that #1280 is good solution for this issue.

[kuenishi]

For release note:

The issue was, if there was a canceled upload, manifest was left as writing and no overwrite happened then those partially uploaded blocks and manifest were to be left as it was and occupies certain disk space. This is fixed by adding additional error handling routine to move the manifest to GC bucket by catching socket error.

[kuenishi]

Although, there might still be chance to leave such manifest in case of CS node failure, which are to be collected by #937 fixes.

[kuenishi]

So far I close this issue and let us track #937.