rgw: can't remove shadow file when copy obj from other placement

[joke-lee]

hi,
i create 2 placement as flow

./bin/radosgw-admin realm  create --rgw-realm=default
./bin/radosgw-admin zonegroup modify --rgw-zonegroup=default  --rgw-realm=default  --master
./bin/radosgw-admin zone modify --rgw-zonegroup=default --rgw-zone=default  --rgw-realm=default  --master
./bin/ceph osd pool create default.rgw.buckets.test  8 8
./bin/radosgw-admin zone placement add --rgw-zone=default --placement-id=placement-1  --index-pool default.rgw.buckets.index  --data-pool default.rgw.buckets.data  --data-extra-pool=default.rgw.buckets.non-ec
./bin/radosgw-admin zone placement add --rgw-zone=default --placement-id=placement-2  --index-pool default.rgw.buckets.index  --data-pool default.rgw.buckets.test  --data-extra-pool=default.rgw.buckets.non-ec
./bin/radosgw-admin zonegroup placement add --placement-id=placement-1 --rgw-zonegroup=default
./bin/radosgw-admin zonegroup placement add --placement-id=placement-2 --rgw-zonegroup=default
./bin/radosgw-admin period update --commit

{
        "key": "placement-1",
        "val": {
            "index_pool": "default.rgw.buckets.index",
            "data_pool": "default.rgw.buckets.data",
            "data_extra_pool": "default.rgw.buckets.non-ec",
            "index_type": 0,
            "compression": "" 
        }
    },
  {
        "key": "placement-2",
        "val": {
            "index_pool": "default.rgw.buckets.index",
            "data_pool": "default.rgw.buckets.test",
            "data_extra_pool": "default.rgw.buckets.non-ec",
            "index_type": 0,
            "compression": "" 
        }
    }

and create 2 bucket

s3cmd mb s3://test1 --bucket-location=default:placement-1
s3cmd mb s3://test2 --bucket-location=default:placement-2

upload 10M file to test1 bucket
and copy test1/10M to test2/10M_copy

from boto3.session import Session
import boto3
access_key = "yly"
secret_key = "yly"
session = Session(access_key, secret_key)
url = "http://127.0.0.1"
config_dict = { 'signature_version' : 's3', 'connect_timeout': 30000, 'read_timeout': 30000}
config = boto3.session.Config(**config_dict)
s3_client = session.client('s3', endpoint_url=url, config=config)
print s3_client.copy_object(Bucket="test2", Key="10M_copy", CopySource=str('test1'+'/'+"10M"))

and remove objs

s3cmd rm s3://test1/*
s3cmd rm s3://test2/*

and process gc

./bin/radosgw-admin gc process --include-all

and two shadow file will never be deleted

./bin/ceph df
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
2018-07-30 22:44:02.310 7f2324cf1700 -1 WARNING: all dangerous and experimental features are enabled.
2018-07-30 22:44:02.412 7f2324cf1700 -1 WARNING: all dangerous and experimental features are enabled.
GLOBAL:
    SIZE       AVAIL      RAW USED     %RAW USED
    59 GiB     25 GiB       34 GiB         57.78
POOLS:
    NAME                          ID     USED        %USED     MAX AVAIL     OBJECTS
    .rgw.root                     1      3.9 KiB         0        24 GiB          15
    default.rgw.control           2          0 B         0        24 GiB           8
    default.rgw.meta              3      3.6 KiB         0        24 GiB          21
    default.rgw.log               4         87 B         0        24 GiB         210
    default.rgw.buckets.test      5          0 B         0        24 GiB           0
    default.rgw.buckets.index     6          0 B         0        24 GiB           2
    default.rgw.buckets.data      7        6 MiB      0.02        24 GiB           2

./bin/rados -p default.rgw.buckets.data ls
2018-07-30 22:43:50.162 7fef49004b00 -1 WARNING: all dangerous and experimental features are enabled.
2018-07-30 22:43:50.218 7fef49004b00 -1 WARNING: all dangerous and experimental features are enabled.
2018-07-30 22:43:50.246 7fef49004b00 -1 WARNING: all dangerous and experimental features are enabled.
04dd1d34-d049-4725-8a5a-0088c5853b7f.4168.2__shadow_.wrmeRy1HXHhOvwXX55qPhMW-w6rXVYV_2
04dd1d34-d049-4725-8a5a-0088c5853b7f.4168.2__shadow_.wrmeRy1HXHhOvwXX55qPhMW-w6rXVYV_1

this will cause garbage data which nerver be delete, and only can delete by rados command

we need dose the same way like https://github.com/ceph/ceph/blob/master/src/rgw/rgw_rados.cc#L7787

fix https://tracker.ceph.com/issues/25168

Signed-off-by: root root@ylycephdev.novalocal

[joke-lee]

hi, @cbodley do you mind to help review this ? thanks

[joke-lee]

hi @mattbenjamin can you please help review this ?

[joke-lee]

jenkins test this please

[joke-lee]

jenkins test this please

[mattbenjamin]

@joke-lee sure

[mattbenjamin]

can you explain how this works?

[ZVampirEM77]

@joke-lee I think attrs.erase(RGW_ATTR_ID_TAG) has been done in https://github.com/ceph/ceph/blob/master/src/rgw/rgw_rados.cc#L8413

[ZVampirEM77]

So as @chuang-he says, maybe this is a repeated commit ~

[chuang-he]

the same as the #22613

[mattbenjamin]

@joke-lee so, because deleting the attr will cause a new tail tag to be generated? commit should indicate this

[joke-lee]

same pr close now