[s3] put fakedir/ key object #3086

kmlebedev · 2022-05-21T08:02:34Z

required for integration with NextCloud
cmd put key

aws --debug --endpoint-url http://127.0.0.1:8333 s3api put-object --bucket test --key fakedir/
2022-05-21 13:01:18,267 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=PUT, url=http://127.0.0.1:8333/test/fakedir/, headers={'User-Agent': b'aws-cli/2.5.1 Python/3.9.12 Darwin/20.6.0 source/x86_64 prompt/off command/s3api.put-object', 'Content-MD5': b'1B2M2Y8AsgTpgAmY7PhCfg==', 'X-Amz-Date': b'20220521T080118Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=some_access_key1/20220521/us-west-2/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-date, Signature=0e8e0ad443c9fad591a885de961a475f2fb6979bd22500e3e40940b7dedf5dfc', 'Content-Length': '0'}>
2022-05-21 13:01:18,268 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTP connection (1): 127.0.0.1:8333
2022-05-21 13:01:18,274 - MainThread - urllib3.connectionpool - DEBUG - http://127.0.0.1:8333 "PUT /test/fakedir/ HTTP/1.1" 200 0
2022-05-21 13:01:18,274 - MainThread - botocore.parsers - DEBUG - Response headers: {'Accept-Ranges': 'bytes', 'Content-Length': '0', 'Server': 'SeaweedFS S3', 'X-Amz-Request-Id': '1653120078293682028', 'Date': 'Sat, 21 May 2022 08:01:18 GMT'}
2022-05-21 13:01:18,274 - MainThread - botocore.parsers - DEBUG - Response body:
b''
2022-05-21 13:01:18,275 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.PutObject: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x10816ba00>>
2022-05-21 13:01:18,275 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
2022-05-21 13:01:18,275 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.PutObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x10816ba90>>
2022-05-21 13:01:18,275 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.PutObject: calling handler <function enhance_error_msg at 0x103971280>
2022-05-21 13:01:18,275 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.PutObject: calling handler <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at 0x10816b5b0>>
2022-05-21 13:01:18,275 - MainThread - awscli.formatter - DEBUG - RequestId: 1653120078293682028

logs put key

s3_1      | I0521 07:58:13     1 s3api_object_handlers.go:49] PutObjectHandler test /fakedir/
s3_1      | I0521 07:58:13     1 filer_client.go:234] mkdir: directory:"/buckets" entry:{name:"test/fakedir/" is_directory:true attributes:{mtime:1653119893 file_mode:2147484159 crtime:1653119893}}
filer_1   | I0521 07:58:13     1 filer_grpc_server.go:138] CreateEntry /buckets/test/fakedir/
filer_1   | I0521 07:58:13     1 filer.go:190] UpdateEntry /buckets/test/fakedir/: old entry: 
filer_1   | I0521 07:58:13     1 filer.go:202] CreateEntry /buckets/test/fakedir/: created

cmd get key

aws --debug --endpoint-url http://127.0.0.1:8333 s3api get-object --bucket test --key fakedir/ /tmp/fackedir

logs get key with /

s3_1      | I0521 08:05:50     1 s3api_object_handlers.go:136] GetObjectHandler test /fakedir/
s3_1      | I0521 08:05:50     1 error_handler.go:85] status 501 application/xml: <?xml version="1.0" encoding="UTF-8"?>
s3_1      | <Error><Code>NotImplemented</Code><Message>A header you provided implies functionality that is not implemented</Message><Resource>/test/fakedir/</Resource><RequestId>1653120350340582368</RequestId><Key>fakedir/</Key><BucketName>test</BucketName></Error>

fs.meta.cat

> fs.meta.cat buckets/test/fakedir
{
  "name": "fakedir",
  "isDirectory": true,
  "chunks": [
  ],
  "attributes": {
    "fileSize": "0",
    "mtime": "1653119827",
    "fileMode": 2147484159,
    "uid": 0,
    "gid": 0,
    "crtime": "1653119827",
    "mime": "",
    "replication": "",
    "collection": "",
    "ttlSec": 0,
    "userName": "",
    "groupName": [
    ],
    "symlinkTarget": "",
    "md5": null,
    "diskType": "",
    "rdev": 0,
    "inode": "0"
  },
  "extended": {
  },
  "hardLinkId": null,
  "hardLinkCounter": 0,
  "content": null,
  "remoteEntry": null,
  "quota": "0"
}
chunks 0 meta size: 31 gzip:59

aws --debug --endpoint-url http://127.0.0.1:8333 s3api put-object --bucket test --key "fakedir/."
s3_1      | I0521 08:40:03     1 s3api_object_handlers.go:49] PutObjectHandler test /fakedir/.
s3_1      | I0521 08:40:03     1 s3api_object_handlers.go:100] putToFiler uploadUrl: http://filer:8888/buckets/test/fakedir/.
filer_1   | I0521 08:40:03     1 filer_server_handlers_read_dir.go:55] listDirectory /buckets/test/fakedir, last file fakedir, limit 100: 1 items
s3_1      | E0521 08:40:03     1 s3api_object_handlers.go:414] failing to read upload to http://filer:8888/buckets/test/fakedir/. : <!DOCTYPE html>
s3_1      | I0521 08:40:03     1 error_handler.go:85] status 500 application/xml: <?xml version="1.0" encoding="UTF-8"?>
s3_1      | <Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message><Resource>/test/fakedir/.</Resource><RequestId>1653122403355812077</RequestId><Key>fakedir/.</Key><BucketName>test</BucketName></Error>

https://docs.nextcloud.com/server/latest/admin_manual/configuration_files/external_storage_configuration_gui.html

s3_1         | I0525 07:54:09     1 s3api_object_handlers.go:49] PutObjectHandler test /nextcloud_external_dir/
s3_1         | I0525 07:54:09     1 filer_server_handlers_read.go:98] Not found /buckets/test/nextcloud_external_dir: filer: no entry is found in filer store
s3_1         | E0525 07:54:09     1 s3api_object_handlers.go:416] failing to read upload to http://172.18.0.4:8888/buckets/test/nextcloud_external_dir/. : 
s3_1         | I0525 07:54:09     1 error_handler.go:85] status 500 application/xml: <?xml version="1.0" encoding="UTF-8"?>
s3_1         | <Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.</Message><Resource>/test/nextcloud_external_dir/</Resource><RequestId>1653465249823314600</RequestId><Key>nextcloud_external_dir/</Key><BucketName>test</BucketName></Error>

minio work:

minio:9000 [REQUEST s3.ListObjectsV2] [2022-05-25T16:36:32:000] [Client IP: 172.18.0.6]
minio:9000 GET /nextcloud?list-type=2&prefix=folder%2F&max-keys=1
minio:9000 Proto: HTTP/1.1
minio:9000 Host: minio:9000
minio:9000 Aws-Sdk-Retry: 0/0
minio:9000 Content-Length: 0
minio:9000 X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
minio:9000 Authorization: AWS4-HMAC-SHA256 Credential=some_access_key1/20220525/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-user-agent, Signature=b9b57b5dbee317c10d38e3d6fbed47a5e50e8de58a0722d1aefce620f60e5043
minio:9000 Aws-Sdk-Invocation-Id: 553ad64750d8847d39090f67e71231a5
minio:9000 User-Agent: aws-sdk-php/3.212.2 OS/Linux/5.10.25-linuxkit lang/php/8.0.18 GuzzleHttp/7
minio:9000 X-Amz-Date: 20220525T113632Z
minio:9000 X-Amz-User-Agent: aws-sdk-php/3.212.2
minio:9000
minio:9000 [RESPONSE] [2022-05-25T16:36:32:000] [ Duration 684µs  ↑ 130 B  ↓ 901 B ]
minio:9000 200 OK
minio:9000 Accept-Ranges: bytes
minio:9000 X-Amz-Request-Id: 16F25604EA79315C
minio:9000 X-Xss-Protection: 1; mode=block
minio:9000 Content-Length: 585
minio:9000 Content-Security-Policy: block-all-mixed-content
minio:9000 Content-Type: application/xml
minio:9000 Server: MinIO
minio:9000 Strict-Transport-Security: max-age=31536000; includeSubDomains
minio:9000 Vary: Origin,Accept-Encoding
minio:9000 X-Content-Type-Options: nosniff
minio:9000 <?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>nextcloud</Name><Prefix>folder/</Prefix><KeyCount>1</KeyCount><MaxKeys>1</MaxKeys><Delimiter></Delimiter><IsTruncated>false</IsTruncated><Contents><Key>folder/</Key><LastModified>2022-05-25T11:36:32.178Z</LastModified><ETag>&#34;d41d8cd98f00b204e9800998ecf8427e&#34;</ETag><Size>0</Size><Owner><ID>02d6176db174dc93cb1b899f7c6078f08654445fe8cf1b6ce98d8855f66bdbf4</ID><DisplayName>minio</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents></ListBucketResult>
minio:9000

The text was updated successfully, but these errors were encountered:

seaweedfs#3086

kmlebedev · 2022-05-25T13:58:26Z

cmds for testing

aws --endpoint-url http://127.0.0.1:8333 s3api put-object --bucket test --key "fakedir/"
aws --endpoint-url http://127.0.0.1:8333 s3api list-objects-v2 --bucket test --prefix "fakedir/"
{
    "Contents": [
        {
            "Key": "fakedir/",
            "LastModified": "2022-05-25T13:57:23+00:00",
            "ETag": "\"d41d8cd98f00b204e9800998ecf8427e-0\"",
            "Size": 0,
            "StorageClass": "STANDARD",
            "Owner": {
                "ID": "0"
            }
        }
    ]
}

chrislusf · 2022-05-30T00:51:06Z

I could not figure out what is the expected behavior.

kmlebedev · 2022-05-30T09:46:42Z

I could not figure out what is the expected behavior.

An example of what I want to achieve:

aws --endpoint-url http://127.0.0.1:8333 s3api list-objects-v2 --bucket test --prefix "lol/fakedir14/"
aws --endpoint-url http://127.0.0.1:8333 s3api put-object --content-type "httpd/unix-directory" --bucket test --key "lol/fakedir141"
{
    "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\""
}
aws --endpoint-url http://127.0.0.1:8333 s3api put-object --content-type "httpd/unix-directory" --bucket test --key "lol/fakedir14/"
 aws --endpoint-url http://127.0.0.1:8333 s3api list-objects-v2 --bucket test --prefix "lol/fakedir14/"
{
    "Contents": [
        {
            "Key": "lol/fakedir14/",
            "LastModified": "2022-05-30T09:44:42+00:00",
            "ETag": "\"d41d8cd98f00b204e9800998ecf8427e-0\"",
            "Size": 0,
            "StorageClass": "STANDARD",
            "Owner": {
                "ID": "0"
            }
        },
        {
            "Key": "lol/fakedir141",
            "LastModified": "2022-05-30T09:44:39+00:00",
            "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
            "Size": 0,
            "StorageClass": "STANDARD",
            "Owner": {
                "ID": "0"
            }
        }
    ]
}

seaweedfs#3086

kmlebedev · 2022-05-30T14:56:42Z

I could not figure out what is the expected behavior.

Created empty folder not showing in NextCloud

kmlebedev · 2022-05-30T16:49:31Z

seaweedfs#3086

chrislusf · 2022-08-03T08:24:12Z

Is this covered in the s3 tests? I want to make some changes, but not sure whether the changes will cause regression.

kmlebedev · 2022-08-03T08:31:39Z

Is this covered in the s3 tests? I want to make some changes, but not sure whether the changes will cause regression.

this is covered in make s3tests tests

bucket_listv2_delimiter_prefix_ends_with_delimiter
bucket_listv2_delimiter_prefix

kmlebedev · 2023-04-11T09:04:28Z

Neet test after https://github.com/seaweedfs/seaweedfs/pull/4240/files

kmlebedev · 2023-04-11T09:28:18Z

seaweedfs-s3-1         | <ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>nextcloudext</Name><Prefix>rootFolder/</Prefix><MaxKeys>1</MaxKeys><IsTruncated>false</IsTruncated><KeyCount>0</KeyCount></ListBucketResult>
seaweedfs-s3-1         | I0411 09:25:13.313582 s3api_object_handlers.go:189 HeadObjectHandler nextcloudext /rootFolder/
seaweedfs-s3-1         | I0411 09:25:13.313597 s3api_object_handlers.go:354 s3 proxying HEAD to http://172.18.0.4:8888/buckets/nextcloudext/rootFolder/
seaweedfs-s3-1         | I0411 09:25:13.313744 filer_server_handlers_read.go:101 Not found /buckets/nextcloudext/rootFolder: filer: no entry is found in filer store
seaweedfs-s3-1         | I0411 09:25:13.313823 error_handler.go:96 status 404 application/xml: <?xml version="1.0" encoding="UTF-8"?>
seaweedfs-s3-1         | <Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Resource>/nextcloudext/rootFolder/</Resource><RequestId>1681205113313806137</RequestId><Key>rootFolder/</Key><BucketName>nextcloudext</BucketName></Error>
seaweedfs-s3-1         | I0411 09:25:13.322352 s3api_object_handlers.go:50 PutObjectHandler nextcloudext /rootFolder/
seaweedfs-s3-1         | I0411 09:25:13.322380 filer_client.go:235 mkdir: directory:"/buckets" entry:{name:"nextcloudext/rootFolder" is_directory:true attributes:{mtime:1681205113 file_mode:2147484159 crtime:1681205113 mime:"httpd/unix-directory"}}
seaweedfs-s3-1         | I0411 09:25:13.323138 filer_grpc_server.go:139 CreateEntry /buckets/nextcloudext/rootFolder
seaweedfs-s3-1         | I0411 09:25:13.323188 filer.go:236 find uncached directory: /buckets/nextcloudext
seaweedfs-s3-1         | I0411 09:25:13.323205 filer.go:200 InsertEntry /buckets/nextcloudext/rootFolder: new entry: rootFolder
seaweedfs-s3-1         | I0411 09:25:13.323292 filer.go:221 CreateEntry /buckets/nextcloudext/rootFolder: created
seaweedfs-s3-1         | I0411 09:25:13.323540 error_handler.go:96 status 200 : 
seaweedfs-s3-1         | I0411 09:25:13.324458 s3api_object_handlers.go:189 HeadObjectHandler nextcloudext /rootFolder
seaweedfs-s3-1         | I0411 09:25:13.324470 s3api_object_handlers.go:354 s3 proxying HEAD to http://172.18.0.4:8888/buckets/nextcloudext/rootFolder
seaweedfs-s3-1         | I0411 09:25:13.324567 filer_server_handlers_read_dir.go:55 listDirectory /buckets/nextcloudext/rootFolder, last file , limit 100000: 0 items
seaweedfs-s3-1         | I0411 09:25:13.326645 error_handler.go:96 status 404 application/xml: <?xml version="1.0" encoding="UTF-8"?>
seaweedfs-s3-1         | <Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Resource>/nextcloudext/rootFolder</Resource><RequestId>1681205113326624012</RequestId><Key>rootFolder</Key><BucketName>nextcloudext</BucketName></Error>
se

> fs.meta.cat /buckets/nextcloudext/rootFolder
{
  "name": "rootFolder",
  "isDirectory": true,
  "chunks": [],
  "attributes": {
    "fileSize": "0",
    "mtime": "1681205113",
    "fileMode": 2147484159,
    "uid": 0,
    "gid": 0,
    "crtime": "1681205113",
    "mime": "httpd/unix-directory",
    "ttlSec": 0,
    "userName": "",
    "groupName": [],
    "symlinkTarget": "",
    "md5": "",
    "rdev": 0,
    "inode": "0"
  },
  "extended": {},
  "hardLinkId": "",
  "hardLinkCounter": 0,
  "content": "",
  "remoteEntry": null,
  "quota": "0"
}chunks 0 meta size: 56 gzip:84

seaweedfs#3086

* s3 fix get list of dir object key with slash suffix #3086 * list only entry dir eq prefix --------- Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>

kmlebedev added a commit to kmlebedev/seaweedfs that referenced this issue May 23, 2022

docker compose file for nextcloud testing

a344383

seaweedfs#3086

kmlebedev added a commit to kmlebedev/seaweedfs that referenced this issue May 30, 2022

show fake dirs

26f3646

seaweedfs#3086

kmlebedev added a commit to kmlebedev/seaweedfs that referenced this issue May 30, 2022

list self dir

62e5e38

seaweedfs#3086

kmlebedev mentioned this issue May 30, 2022

Create folders in s3 via nextcloud #3114

Merged

chrislusf closed this as completed May 31, 2022

kmlebedev added a commit to kmlebedev/seaweedfs that referenced this issue Jun 1, 2022

fix show empty dir

b86628f

seaweedfs#3086

kmlebedev mentioned this issue Jun 1, 2022

fix show empty dir #3130

Merged

kmlebedev mentioned this issue Jul 18, 2022

Deleting a directory via S3 does not delete directory keys in the Redis filer store #3324

Closed

chrislusf mentioned this issue Aug 12, 2022

Fix s3 pagination #3436

Merged

2 tasks

kmlebedev mentioned this issue Apr 11, 2023

make directory lister trigger if mime type is httpd/unix-directory #4169

Merged

kmlebedev pushed a commit to kmlebedev/seaweedfs that referenced this issue Apr 11, 2023

s3 fix get fake dir object key

80cfd2b

seaweedfs#3086

kmlebedev mentioned this issue Apr 11, 2023

s3 fix get fake dir object key #4390

Merged

2 tasks

kmlebedev pushed a commit to kmlebedev/seaweedfs that referenced this issue Apr 11, 2023

s3 fix get list of dir object key with slash suffix

146347c

seaweedfs#3086

kmlebedev mentioned this issue Apr 11, 2023

s3 fix get list of dir object key with slash suffix #4391

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[s3] put fakedir/ key object #3086

[s3] put fakedir/ key object #3086

kmlebedev commented May 21, 2022 •

edited

Loading

kmlebedev commented May 25, 2022

chrislusf commented May 30, 2022

kmlebedev commented May 30, 2022

kmlebedev commented May 30, 2022

kmlebedev commented May 30, 2022

chrislusf commented Aug 3, 2022

kmlebedev commented Aug 3, 2022 •

edited

Loading

kmlebedev commented Apr 11, 2023

kmlebedev commented Apr 11, 2023 •

edited

Loading

[s3] put fakedir/ key object #3086

[s3] put fakedir/ key object #3086

Comments

kmlebedev commented May 21, 2022 • edited Loading

kmlebedev commented May 25, 2022

chrislusf commented May 30, 2022

kmlebedev commented May 30, 2022

kmlebedev commented May 30, 2022

kmlebedev commented May 30, 2022

chrislusf commented Aug 3, 2022

kmlebedev commented Aug 3, 2022 • edited Loading

kmlebedev commented Apr 11, 2023

kmlebedev commented Apr 11, 2023 • edited Loading

kmlebedev commented May 21, 2022 •

edited

Loading

kmlebedev commented Aug 3, 2022 •

edited

Loading

kmlebedev commented Apr 11, 2023 •

edited

Loading