Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rptest: Fix s3.copy_object when running on GCP #16428

Merged
merged 1 commit into from
Feb 1, 2024

Conversation

savex
Copy link
Contributor

@savex savex commented Feb 1, 2024

When running boto3 client versus AWS it uses own header to point to object that needs to be copied.
The same function is usable for GCP's storage, but with custom header.

This PR adds custom_headers handling to CopyObject function

Backport needed only to v23.3.x due to GCP runs only supported for that version
Also v23.1.x will reach end of support soon.

Fixes: #15211

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v23.3.x
  • v23.2.x
  • v23.1.x

Release Notes

  • none

@savex savex requested a review from abhijat February 1, 2024 15:29
@savex savex self-assigned this Feb 1, 2024
@savex savex marked this pull request as ready for review February 1, 2024 15:34
@savex
Copy link
Contributor Author

savex commented Feb 1, 2024

Running on GCP env

ubuntu@rp-client-0-firstly-exciting-macaque:~/tests$  cd /home/ubuntu/tests ; /usr/bin/env /bin/python3 /home/ubuntu/.vscode-server/extensions/ms-python.debugpy-2024.0.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher 39305 -- -m ducktape --cluster=ducktape.cluster.json.JsonCluster --cluster-file=cluster.json --globals=globals.json --max-parallel=1 --repeat=1 --test-runner-timeout=86400000 rptest/tests/services_self_test.py::BucketScrubSelfTest.test_missing_segment 
[INFO:2024-02-01 15:23:09,186]: starting test run with session id 2024-02-01--004...
[INFO:2024-02-01 15:23:09,186]: running 1 tests...
[INFO:2024-02-01 15:23:09,187]: Triggering test 1 of 1...
[INFO:2024-02-01 15:23:10,128]: RunnerClient: Loading test {'directory': '/home/ubuntu/redpanda/tests/rptest/tests', 'file_name': 'services_self_test.py', 'cls_name': 'BucketScrubSelfTest', 'method_name': 'test_missing_segment', 'injected_args': {'cloud_storage_type': <CloudStorageType.S3: 1>}}
[INFO:2024-02-01 15:23:10,138]: RunnerClient: rptest.tests.services_self_test.BucketScrubSelfTest.test_missing_segment.cloud_storage_type=CloudStorageType.S3: on run 1/1
[INFO:2024-02-01 15:23:10,154]: RunnerClient: rptest.tests.services_self_test.BucketScrubSelfTest.test_missing_segment.cloud_storage_type=CloudStorageType.S3: Setting up...
[INFO:2024-02-01 15:23:16,427]: RunnerClient: rptest.tests.services_self_test.BucketScrubSelfTest.test_missing_segment.cloud_storage_type=CloudStorageType.S3: Running...
[WARNING - 2024-02-01 15:24:31,233 - redpanda - stop_and_scrub_object_storage - lineno:4067]: Non-fatal anomalies in remote storage: {
  "anomalies": {
.....
[INFO:2024-02-01 15:25:07,872]: RunnerClient: rptest.tests.services_self_test.BucketScrubSelfTest.test_missing_segment.cloud_storage_type=CloudStorageType.S3: PASS
[INFO:2024-02-01 15:25:07,873]: RunnerClient: rptest.tests.services_self_test.BucketScrubSelfTest.test_missing_segment.cloud_storage_type=CloudStorageType.S3: Data: None
test_id:    rptest.tests.services_self_test.BucketScrubSelfTest.test_missing_segment.cloud_storage_type=CloudStorageType.S3
status:     PASS
run time:   1 minute 57.734 seconds
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
===============================================================================================================================================================================================================================================================================================================================
SESSION REPORT (ALL TESTS)
ducktape version: 0.8.18
session_id:       2024-02-01--004
run time:         1 minute 59.324 seconds
tests run:        1
passed:           1
flaky:            0
failed:           0
ignored:          0
opassed:          0
ofailed:          0
===============================================================================================================================================================================================================================================================================================================================

Copy link
Contributor

@abhijat abhijat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@piyushredpanda piyushredpanda merged commit 16ef8e0 into dev Feb 1, 2024
20 checks passed
@piyushredpanda piyushredpanda deleted the rp-15211-fix-gcp-copy-object branch February 1, 2024 18:20
@vbotbuildovich
Copy link
Collaborator

/backport v23.3.x

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

CI Failure (InvalidArgument calling CopyObject) in BucketScrubSelfTest.test_missing_segment
4 participants