Conversation
Add non-empty Add S3 directory marker modes
|
Failed to generate code suggestions for PR |
chuanbao666
approved these changes
Mar 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景
当前 AGFS 在 S3 上会通过目录 marker 来模拟目录语义,但不同的 S3 兼容后端对目录 marker 的要求并不一致:
变更内容
1. 引入新的目录 marker 配置项
在 S3 配置中新增
directory_marker_mode,支持三种模式:empty:写入 0 字节目录 marker,保留空目录语义nonempty:写入非空目录 marker,适配拒绝 0 字节目录 marker 的后端none:不创建目录 marker,使用更接近原生 S3 prefix 的目录语义其中默认值保持为
empty,保证现有常见场景的行为不变。2. 调整 Python -> AGFS 插件的配置下发
AGFSManager现在只向s3fs插件下发directory_marker_mode。3. 更新 Go 侧 s3fs 插件实现
在
s3fs插件中补充了:directory_marker_mode的配置解析与合法性校验具体行为如下:
empty:创建 0 字节目录 marker,并继续要求父目录存在nonempty:创建非空目录 marker,并继续要求父目录存在none:跳过目录 marker 创建,同时不再强依赖父目录 marker 存在,更接近原生 S3 prefix 行为4. 补充测试和文档
补充了 Python 和 Go 两侧测试,覆盖:
directory_marker_mode默认值同时更新了中英文文档,补充三种模式的说明、适用场景和配置示例。
行为变化说明
directory_marker_mode默认仍为emptynonemptynonenonempty_directory_marker不再支持兼容性说明
这是一次配置模型收敛:
对新配置
directory_marker_mode提供完整支持;