Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Missing Seaweed-* headers for active-active replication and async backup #3532

Closed
lewisl9029 opened this issue Aug 27, 2022 · 2 comments
Closed

Comments

@lewisl9029
Copy link

lewisl9029 commented Aug 27, 2022

Describe the bug
Both active-active replication and async backup seem to fail to replicate Seaweed-* headers. (not sure if related, happy to split out into separate issues if not)

System Setup
I've set up a repo with docker-compose and detailed steps to repro both of these issues:
https://github.com/lewisl9029/seaweed-async-replication-repro

Happy to provide details if anything is still unclear.

Expected behavior
Files replicated through active-active replication and async backup should include Seaweed-* headers.

Screenshots
See screenshots in https://github.com/lewisl9029/seaweed-async-replication-repro/blob/main/README.md

Additional context
Original discussion: #3529

@chrislusf
Copy link
Collaborator

Thanks for the detailed bug report! I added a fix. Please help to confirm.

@lewisl9029
Copy link
Author

lewisl9029 commented Sep 15, 2022

@chrislusf thank you so much for the fix! Was just able to confirm that the headers are showing up in async replication!

Though I just ran into a different issue for backup. A whole bunch of errors that look like this:

reflame-storage-primary  | backup  | E0915 00:24:00.211679 filer_pb_tail.go:79 process directory:"/resources" event_notification:{old_entry:{name:"4chiq_97qWIqjkE9FI656HUZZwHEDRBb96j0RRA06uU" chunks:{file_id:"1,0110a4dfdd50" size:15705 mtime:1663198121169588555 e_tag:"mWap5CpVrTPRSGTelR5DBw==" source_file_id:"5,0118375d8618" fid:{volume_id:1 file_key:272 cookie:2766134608} source_fid:{volume_id:5 file_key:280 cookie:928876056}} attributes:{file_size:15705 mtime:1663198121 file_mode:432 crtime:1663198121 mime:"application/json" md5:"\x99f\xa9\xe4*U\xad3\xd1Hdޕ\x1eC\x07"} extended:{key:"Seaweed-Meta" value:"{\"headers\":{\"cache-control\":\"public, max-age=31536000, immutable\",\"content-type\":\"application/json\"}}"}} new_entry:{name:"4chiq_97qWIqjkE9FI656HUZZwHEDRBb96j0RRA06uU" chunks:{file_id:"7,0209a4947b05" size:15705 mtime:1663200833339500005 e_tag:"mWap5CpVrTPRSGTelR5DBw==" fid:{volume_id:7 file_key:521 cookie:2761194245}} attributes:{file_size:15705 mtime:1663200833 file_mode:432 crtime:1663198121 mime:"application/json" md5:"\x99f\xa9\xe4*U\xad3\xd1Hdޕ\x1eC\x07"} extended:{key:"Seaweed-Meta" value:"{\"headers\":{\"cache-control\":\"public, max-age=31536000, immutable\",\"content-type\":\"application/json\"}}"}} delete_chunks:true new_parent_path:"/resources" signatures:1384205955} ts_ns:1663200833345144630: InvalidArgument: The header 'x-amz-tagging' shall be encoded as UTF-8 then URLEncoded URL query parameters without tag name duplicates.
reflame-storage-primary  | backup  |    status code: 400, request id: PWZ95KK5PPDP93V1, host id: LkxVG2FYizeCyxrgChJB1gZb1dgTiyDOD8c2MYok8h0rZxtbRBoABv1wXqSUYicwjN9lzfNn2BU=

Relevant part seems to be this message near the end: InvalidArgument: The header 'x-amz-tagging' shall be encoded as UTF-8 then URLEncoded URL query parameters without tag name duplicates.

Maybe we're missing a URL encoding step somewhere?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants