Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[quincy] RGW backports #51806

Open
wants to merge 14 commits into
base: quincy
Choose a base branch
from
Open

[quincy] RGW backports #51806

wants to merge 14 commits into from

Conversation

soumyakoduri
Copy link
Contributor

@soumyakoduri soumyakoduri commented May 29, 2023 •
edited

This PR contains backports of below fixes -

#50098
#50172
#51192
#50676
#51511
#49803
#52160

Fixes: https://tracker.ceph.com/issues/59674
Fixes: https://tracker.ceph.com/issues/59644
Fixes: https://tracker.ceph.com/issues/61391
Fixes: https://tracker.ceph.com/issues/61389
Fixes: https://tracker.ceph.com/issues/58897
Fixes: https://tracker.ceph.com/issues/61481
Fixes: https://tracker.ceph.com/issues/61877

Signed-off-by: Soumya Koduri skoduri@redhat.com

Checklist

  • Tracker (select at least one)
    • References tracker ticket
    • Very recent bug; references commit where it was introduced
    • New feature (ticket optional)
    • Doc update (no ticket needed)
    • Code cleanup (no ticket needed)
  • Component impact
    • Affects Dashboard, opened tracker ticket
    • Affects Orchestrator, opened tracker ticket
    • No impact that needs to be tracked
  • Documentation (select at least one)
    • Updates relevant documentation
    • No doc update is appropriate
  • Tests (select at least one)
Show available Jenkins commands
  • jenkins retest this please
  • jenkins test classic perf
  • jenkins test crimson perf
  • jenkins test signed
  • jenkins test make check
  • jenkins test make check arm64
  • jenkins test submodules
  • jenkins test dashboard
  • jenkins test dashboard cephadm
  • jenkins test api
  • jenkins test docs
  • jenkins render docs
  • jenkins test ceph-volume all
  • jenkins test ceph-volume tox
  • jenkins test windows

soumyakoduri and others added 5 commits May 29, 2023 13:26
@soumyakoduri
rgw/cloud: Add custom headers for objects transitioned to cloud
9dc8d0e 
Some of the cloud services, (like MCG Noobaa/Azure Namespace store),
may not be able to map and store the objects which contain s3 style
metadata keys.

To help such services determine if the objects being transitioned
are from RGW and whether or not ignore such attrs, added below two
headers for the objects being copied/transitioned from RGW -

1) x-rgw-cloud : true/false
(set to "true" if the object is being transitioned/synced from RGW)

2) x-rgw-cloud-keep-attrs : true/false
- if set to default "true" , the cloud service should store all the
x-amz-meta-* attrs. If cannot be mapped/stored, the operation
should fail
- if set to "false", the destination cloud can ignore such attrs and
just store the object data being sent.

Also fixed a bug in the cloudtier module wherein the user-defined attrs
were not being copied to the cloud endpoint as part of transition

Fixes: https://tracker.ceph.com/issues/57980
Fixes: https://tracker.ceph.com/issues/58796

Signed-off-by: Soumya Koduri <skoduri@redhat.com>
(cherry picked from commit 9c3264f)
@cbodley @soumyakoduri
rgw: catch manifest decode errors in GetObj/PutObj/CopyObj
e83dc36 
if we read an object that doesn't have a manifest attribute,
RGWRados::get_obj_state_impl() adds an empty one and causes decode to
throw an end_of_buffer exception

Fixes: https://tracker.ceph.com/issues/58794

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 77e0b98)
(cherry picked from commit 9e476ee)
@cbodley @soumyakoduri
rgw: add s3 error message to cloud-tiered objects
e328fba 
better to tell the s3 client why their request failed than to log it as
an ERROR; this is normal/expected when cloud tiering is enabled

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 5713a04)
(cherry picked from commit b70d27f)
@cbodley @soumyakoduri
rgw/rados: don't use attrset[] to create entries that don't exist
a0b121e 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 3327347)
(cherry picked from commit 3e22cb4)
@cbodley @soumyakoduri
rgw/rados: cosmetic cleanup of attr handling
629ccb0 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 8ee9b3f)
(cherry picked from commit cb9c387)
@soumyakoduri soumyakoduri requested a review from a team as a code owner May 29, 2023 08:04
@github-actions github-actions bot added this to the quincy milestone May 29, 2023
@soumyakoduri
rgw/cloudtier: Fix bug with decoding tier_targets
c55e6b5 
Signed-off-by: Soumya Koduri <skoduri@redhat.com>
(cherry picked from commit e33946b)
(cherry picked from commit 5071cb6)
@soumyakoduri
rgw/cloudtransition: Allow multisite zones to sync cloudtiered objects
ff63d3e 
In a multisite configuration, zones should be able to fetch & sync
cloud-transitioned objects as well. To allow this, a new header
'x-rgwx-sync-cloudtiered' is added to be used by sync client to GET
such objects.

Signed-off-by: Soumya Koduri <skoduri@redhat.com>
(cherry picked from commit cec19b0)
(cherry picked from commit f7e3e94)
@soumyakoduri
rgw: Dump tier_* config in RGWObjManifest
bb22717 
This was missed as part of refactoring done to encoder
and decoder methods (ceph@2414c75)

Signed-off-by: Soumya Koduri <skoduri@redhat.com>
(cherry picked from commit 4db6325)
(cherry picked from commit dacf068)
@soumyakoduri
rgw/cloud-transition: New attrs to detect cloudtiered objects
34b0925 
Add new attrs "RGW_ATTR_CLOUD_TIER_TYPE" and "RGW_ATTR_CLOUD_TIER_CONFIG"
to store details about cloud-tiered objects so that they get synced accordingly
in a multisite environment.

Signed-off-by: Soumya Koduri <skoduri@redhat.com>
(cherry picked from commit 0f69011)
(cherry picked from commit 201f430)
@soumyakoduri
rgw: Disable logging for datalog and bilog if zone doesn't export data
3d34685 
Zones (such as archive zone) which do not export data should have
sync logging disabled.

Fixes# https://tracker.ceph.com/issues/59106

Signed-off-by: Soumya Koduri <skoduri@redhat.com>
(cherry picked from commit 85f0cb5)
(cherry picked from commit 2e411d5)
@soumyakoduri
Copy link
Contributor Author

jenkins test submodules

@soumyakoduri
rgw/archive: Correct the bucket sync status cmd output
f9e1db1 
Signed-off-by: Soumya Koduri <skoduri@redhat.com>
(cherry picked from commit d7af6c2)
(cherry picked from commit ede6e61)
@soumyakoduri
rgw/multisite: Fix issue with parsing replication rules
0e34e64 
Signed-off-by: Soumya Koduri <skoduri@redhat.com>
(cherry picked from commit 570de79)
(cherry picked from commit 25ca69b)
@soumyakoduri
rgw/sync-policy: Support disabling per-bucket replication
5178ffd 
When the zones replicate, allow disabling replication
for specific buckets using sync policy.

These are the semantics to be followed while resolving the policy
conflicts -

==================================================
zonegroup		bucket		Result
==================================================
enabled			enabled		enabled
			allowed		enabled
			forbidden	disabled
allowed			enabled		enabled
			allowed		disabled
			forbidden	disabled
forbidden		enabled		disabled
			allowed		disabled
			forbidden	disabled

In case multiple group policies are set to reflect for any sync pair
(<source-zone,source-bucket>, <dest-zone,dest-bucket>), the following
rules are applied in the order -
1) Even if one policy status is FORBIDDEN, the sync will be disabled
2) Atleast one policy should be ENABLED for the sync to be allowed.

Various cases tested are outlined here -
https://docs.google.com/document/d/19oBQA-bYxLBR4BnekA2DTwJJaTFvjAfrqAk9G3RGU0I/edit#heading=h.4qac9dpc76m

Signed-off-by: Soumya Koduri <skoduri@redhat.com>
(cherry picked from commit aeb3a74)
(cherry picked from commit 587910c)
@soumyakoduri soumyakoduri requested a review from cbodley May 29, 2023 12:54
@soumyakoduri
Copy link
Contributor Author

jenkins test submodules

@cbodley
Copy link
Contributor

cbodley commented Jul 7, 2023

jenkins test api

@cbodley
Copy link
Contributor

cbodley commented Jul 7, 2023

jenkins test make check

@cbodley cbodley added needs-qa and removed needs-qa labels Jul 7, 2023
@cbodley
Copy link
Contributor

cbodley commented Jul 7, 2023 

/home/jenkins-build/build/workspace/ceph-pull-requests/src/rgw/rgw_rados.cc:4717:81: error: 'key' is a protected member of 'rgw::sal::Object'
    ldpp_dout(dpp, 0) << __func__ << " ERROR: failed to transition obj(" << obj.key << ") read_mtime = " << read_mtime << " doesn't match mtime = " << mtime << dendl;
                                                                                ^
/home/jenkins-build/build/workspace/ceph-pull-requests/src/rgw/rgw_sal.h:861:17: note: declared protected here
    rgw_obj_key key;
                ^

@soumyakoduri
rgw: Do not update mtime in set_attrs
4a60ab3 
As per https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html,
the only way for users to modify object metadata is to make a copy of the object and
set the metadata. Hence retain old mtime for any other attr changes

Signed-off-by: Soumya Koduri <skoduri@redhat.com>
(cherry picked from commit 31666d7)
@soumyakoduri
Copy link
Contributor Author 

/home/jenkins-build/build/workspace/ceph-pull-requests/src/rgw/rgw_rados.cc:4717:81: error: 'key' is a protected member of 'rgw::sal::Object'
    ldpp_dout(dpp, 0) << __func__ << " ERROR: failed to transition obj(" << obj.key << ") read_mtime = " << read_mtime << " doesn't match mtime = " << mtime << dendl;
                                                                                ^
/home/jenkins-build/build/workspace/ceph-pull-requests/src/rgw/rgw_sal.h:861:17: note: declared protected here
    rgw_obj_key key;
                ^

thanks. Fixed it.

@github-actions
Copy link

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@cbodley cbodley Awaiting requested review from cbodley

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
documentation needs-rebase rgw
Projects
None yet
Milestone
quincy
2 participants
@soumyakoduri @cbodley