Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(core): service add OpenStack Swift support #3461

Merged
merged 10 commits into from
Nov 3, 2023

Conversation

morristai
Copy link
Member

@morristai morristai commented Nov 1, 2023

Description

Additional Notes

  • There are some tricky parts in Swift that require workarounds. I've attempted to solve them all, but I still need some help from reviewers.
  • Swift doesn't support creating directories directly, as there's no such concept in Swift. Instead, I use pseudo folders (Doc), but this creates some complications when listing.
  • Swift doesn't support renaming files. My current implementation involves using the copy and delete API to achieve this.
  • Swift doesn't support appending to the same file. It uses DLO (Dynamic Large Object) and SLO (Static Large Object) for large object storage. For SLO, it would be challenging to support at the moment because it requires manipulating the manifest contents. So the current append operation will use DLO for that.
  • The behavior test results are shown below. I suspect that some of them are related to swift.
failures:
    # some bug when append size is larger >≈ 15MiB
    behavior::test_appender_futures_copy
    behavior::test_fuzz_pr_3395_case_1
    behavior::test_blocking_appender_std_copy
    behavior::test_blocking_fuzz_appender
    behavior::test_fuzz_appender

TODO

  • Add new GitHub Action e2e test workflow, using openstackswift/saio:py3. (As openstackswift/saio is broken)

@morristai
Copy link
Member Author

Seems like there're some unrelated problems in CI

@Xuanwo
Copy link
Member

Xuanwo commented Nov 2, 2023

  • Swift doesn't support creating directories directly, as there's no such concept in Swift. Instead, I use pseudo folders (Doc), but this creates some complications when listing.

You can use S3 as a reference, it's the same.

  • Swift doesn't support renaming files. My current implementation involves using the copy and delete API to achieve this.

Service should only support native operations. Please don't support rename if the service doesn't provide it.

  • Swift doesn't support appending to the same file.

The same. Don't support append if services not support.

@morristai
Copy link
Member Author

  • Swift doesn't support creating directories directly, as there's no such concept in Swift. Instead, I use pseudo folders (Doc), but this creates some complications when listing.

You can use S3 as a reference, it's the same.

  • Swift doesn't support renaming files. My current implementation involves using the copy and delete API to achieve this.

Service should only support native operations. Please don't support rename if the service doesn't provide it.

  • Swift doesn't support appending to the same file.

The same. Don't support append if services not support.

  • Drop append and rename as native API doesn't support
  • Merge func create_dir with create_object to align native API

(I should ask for implementation details first, spent some time flighting with append 😅)

core/src/services/swift/error.rs Outdated Show resolved Hide resolved
core/src/services/swift/backend.rs Outdated Show resolved Hide resolved
core/src/services/swift/backend.rs Outdated Show resolved Hide resolved
core/src/services/swift/backend.rs Outdated Show resolved Hide resolved
core/src/services/swift/backend.rs Show resolved Hide resolved
core/src/services/swift/core.rs Outdated Show resolved Hide resolved
core/src/services/swift/core.rs Outdated Show resolved Hide resolved
core/src/services/swift/core.rs Outdated Show resolved Hide resolved
core/src/services/swift/core.rs Outdated Show resolved Hide resolved
core/src/services/swift/pager.rs Outdated Show resolved Hide resolved
@morristai morristai force-pushed the feat/service_add_openstack_swift branch from a23b574 to 25313fd Compare November 3, 2023 01:49
Copy link
Member

@Xuanwo Xuanwo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, let's rock!

@Xuanwo Xuanwo merged commit 68335f3 into apache:main Nov 3, 2023
138 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add Openstack Swift API Support
2 participants