Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

When i using "weed filer.remote.gateway", ≈ 99% will be good, ≈ 1% will push a 0-byte file to the remote s3. #2465

Closed
imfantuan opened this issue Nov 22, 2021 · 1 comment
Closed

When i using "weed filer.remote.gateway", ≈ 99% will be good, ≈ 1% will push a 0-byte file to the remote s3. #2465

imfantuan opened this issue Nov 22, 2021 · 1 comment

Comments

@imfantuan
Copy link
Contributor

Describe the bug

Some times: The Remote Gateway function will push a 0-byte file to the backend s3.

System Setup

Local Server (192.168.100.5)

weed master / filer / volume / S3

nohup weed master -ip.bind 0.0.0.0 -ip 192.168.100.5 -mdir /home/seaweedfs/master -defaultReplication 000 -metrics.address 192.168.100.5:9091 >/dev/null 2>&1 &
nohup weed volume -port 8080 -max 200 -ip.bind 0.0.0.0 -ip 192.168.100.5 -dir /home/raid50_stor/seaweedfs/volume -mserver 192.168.100.5:9333 -dataCenter dc01 -rack r01 -idleTimeout 300 >/dev/null 2>&1 &
nohup weed filer -port 8888 -ip.bind 0.0.0.0 -ip 192.168.100.5 -master 192.168.100.5:9333 -dataCenter dc01 -rack r01 -concurrentUploadLimitMB 5120 -maxMB 32 -defaultStoreDir /home/seaweedfs/filer -saveToFilerLimit=10240 >/dev/null 2>&1 &
nohup weed -v=4 s3 -allowEmptyFolder >/dev/null 2>&1 &

weed filer.remote.gateway -createBucketAt=backends3

remote.configure

{
  "type": "s3",
  "name": "backends3",
  "s3AccessKey": "xxxyyyzzz",
  "s3SecretKey": "******************************",
  "s3Region": "us-east-2",
  "s3Endpoint": "http://192.168.100.3:8555",
  "s3ForcePathStyle": true
}
  • CentOS 8.4
  • version 8000GB 2.77 linux amd64

Content of filer.toml

[postgres2]
enabled = true
createTable = """
  CREATE TABLE IF NOT EXISTS "%s" (
    dirhash   BIGINT,
    name      VARCHAR(65535),
    directory VARCHAR(65535),
    meta      bytea,
    PRIMARY KEY (dirhash, name)
  );
"""
hostname = "localhost"
port = 5432
username = "postgres"
password = "xxxyyyzzz"
database = "weedinfo"          # create or use an existing database
schema = ""
sslmode = "disable"
connection_max_idle = 500
connection_max_open = 500
connection_max_lifetime_seconds = 0
# if insert/upsert failing, you can disable upsert or update query syntax to match your RDBMS syntax:
enableUpsert = true
upsertQuery = """INSERT INTO "%[1]s" (dirhash,name,directory,meta) VALUES($1,$2,$3,$4) ON CONFLICT (dirhash,name) DO UPDATE SET meta = EXCLUDED.meta WHERE "%[1]s".meta != EXCLUDED.meta"""

Remote Server (192.168.100.3 => backends3)

weed master / filer / volume / S3

nohup weed master -ip.bind 0.0.0.0 -ip 192.168.100.3 -mdir /home/seaweedfs/master -defaultReplication 010 -metrics.address 192.168.100.5:9091 >/dev/null 2>&1 &
nohup weed volume -port 8080 -max 200,200,200,200 -ip.bind 0.0.0.0 -ip 192.168.100.3 -dir /stor_sda/seaweedfs/volume,/stor_sdb/seaweedfs/volume,/stor_sdc/seaweedfs/volume,/stor_sdd/seaweedfs/volume -mserver 192.186.100.3:9333 -dataCenter dc01 -rack r01 -idleTimeout 300 >/dev/null 2>&1 &
nohup weed filer -port 8888 -ip.bind 0.0.0.0 -ip 192.168.100.3 -master 192.168.100.3:9333 -dataCenter dc01 -rack r01 -concurrentUploadLimitMB 5120 -maxMB 32 -defaultStoreDir /home/seaweedfs/filer >/dev/null 2>&1 &
nohup weed -v=4 s3 -allowEmptyFolder --port=8555 >/dev/null 2>&1 &
  • CentOS 8.4
  • version 8000GB 2.77 linux amd64

Content of filer.toml

[leveldb3]
enabled = true
dir = "/home/seaweedfs/filer_db/filerldb3"

Expected behavior

  1. Upload file to Local Server ✅

by "boto3" and Python 3.8

s3.meta.client.upload_file(temp_file, bucket_name, s3_key, ExtraArgs={'ContentType': self.headers.get('Content-Type', ''), 'ContentDisposition': self.headers.get('Content-Disposition', '')})
  1. Local Server (192.168.100.5) -> weed filer.remote.gateway -> Remote S3 Server (192.168.100.3 => backends3)

In Local Server：

All of them are ✅

In Remote Server (backends3)

≈ 99% of them are ✅
≈ 1% of them are 0 byte, and has 0 chunk ❌

fs.meta.cat in Local Server

> fs.meta.cat /buckets/some_bucket/~/source/xxx_file
{
  "name": "xxx_file",
  "isDirectory": false,
  "chunks": [
    {
      "fileId": "192,15b726043711df",
      "offset": "0",
      "size": "7375981",
      "mtime": "1637492030786420056",
      "eTag": "s1P7vxelLelI1Wfj1QRzyg==",
      "sourceFileId": "",
      "fid": {
        "volumeId": 192,
        "fileKey": "1423142",
        "cookie": 70717919
      },
      "sourceFid": null,
      "cipherKey": null,
      "isCompressed": false,
      "isChunkManifest": false
    }
  ],
  "attributes": {
    "fileSize": "7375981",
    "mtime": "1637492030",
    "fileMode": 432,
    "uid": 0,
    "gid": 0,
    "crtime": "1637492030",
    "mime": "image/jpeg",
    "replication": "000",
    "collection": "some_bucket",
    "ttlSec": 0,
    "userName": "",
    "groupName": [
    ],
    "symlinkTarget": "",
    "md5": "s1P7vxelLelI1Wfj1QRzyg==",
    "diskType": ""
  },
  "extended": {
    "Content-Disposition": ""
  },
  "hardLinkId": null,
  "hardLinkCounter": 0,
  "content": null,
  "remoteEntry": {
    "storageName": "backends3",
    "lastLocalSyncTsNs": "1637492031321026560",
    "remoteETag": "\"b353fbbf17a52de948d567e3d50473ca\"",
    "remoteMtime": "1637492031",
    "remoteSize": "7375981"
  }
}
chunks 1 meta size: 268 gzip:296

fs.meta.cat in Remote Server

> fs.meta.cat /buckets/some_bucket/~/source/xxx_file
{
  "name": "xxx_file",
  "isDirectory": false,
  "chunks": [
  ],
  "attributes": {
    "fileSize": "0",
    "mtime": "1637492031",
    "fileMode": 432,
    "uid": 0,
    "gid": 0,
    "crtime": "1637492030",
    "mime": "",
    "replication": "010",
    "collection": "some_bucket",
    "ttlSec": 0,
    "userName": "",
    "groupName": [
    ],
    "symlinkTarget": "",
    "md5": "1B2M2Y8AsgTpgAmY7PhCfg==",
    "diskType": ""
  },
  "extended": {
    "X-Amz-Tagging-Content-Disposition": ""
  },
  "hardLinkId": null,
  "hardLinkCounter": 0,
  "content": null,
  "remoteEntry": null
}
chunks 0 meta size: 118 gzip:146

Some logs of weed filer.remote.gateway

I1122 15:41:00 84737 filer_remote_gateway_buckets.go:224] rmdir  backends3/some_bucket/.uploads/0a2fa54a-4032-4146-8f9f-0c71a5bb2b9b
I1122 15:41:00 84737 filer_remote_gateway_buckets.go:224] rmdir  backends3/some_bucket/.uploads/0a2fa54a-4032-4146-8f9f-0c71a5bb2b9b
I1122 15:41:00 84737 filer_remote_gateway_buckets.go:224] rmdir  backends3/some_bucket/.uploads/0a2fa54a-4032-4146-8f9f-0c71a5bb2b9b
I1122 15:41:00 84737 filer_remote_gateway_buckets.go:224] rmdir  backends3/some_bucket/.uploads/0a2fa54a-4032-4146-8f9f-0c71a5bb2b9b
I1122 15:41:00 84737 filer_remote_gateway_buckets.go:224] rmdir  backends3/some_bucket/.uploads/0a2fa54a-4032-4146-8f9f-0c71a5bb2b9b
I1122 15:41:00 84737 filer_remote_gateway_buckets.go:227] delete backends3/some_bucket/.uploads/b97537ff-3711-40c1-8ad9-9619e45bec06/0002.part
I1122 15:41:00 84737 filer_remote_gateway_buckets.go:227] delete backends3/some_bucket/.uploads/b97537ff-3711-40c1-8ad9-9619e45bec06/0002.part
I1122 15:41:00 84737 filer_remote_gateway_buckets.go:227] delete backends3/some_bucket/.uploads/b97537ff-3711-40c1-8ad9-9619e45bec06/0002.part
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:227] delete backends3/some_bucket/.uploads/b97537ff-3711-40c1-8ad9-9619e45bec06/0002.part
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:227] delete backends3/some_bucket/.uploads/b97537ff-3711-40c1-8ad9-9619e45bec06/0002.part
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:227] delete backends3/some_bucket/.uploads/b97537ff-3711-40c1-8ad9-9619e45bec06/0002.part
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:227] delete backends3/some_bucket/.uploads/b97537ff-3711-40c1-8ad9-9619e45bec06/0002.part
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:227] delete backends3/some_bucket/.uploads/b97537ff-3711-40c1-8ad9-9619e45bec06/0002.part
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:227] delete backends3/some_bucket/.uploads/b97537ff-3711-40c1-8ad9-9619e45bec06/0002.part
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:227] delete backends3/some_bucket/.uploads/b97537ff-3711-40c1-8ad9-9619e45bec06/0002.part
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:224] rmdir  backends3/some_bucket/.uploads/b97537ff-3711-40c1-8ad9-9619e45bec06
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:224] rmdir  backends3/some_bucket/.uploads/b97537ff-3711-40c1-8ad9-9619e45bec06
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:227] delete backends3/some_bucket/.uploads/b97537ff-3711-40c1-8ad9-9619e45bec06/0002.part
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:224] rmdir  backends3/some_bucket/.uploads/b97537ff-3711-40c1-8ad9-9619e45bec06
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:224] rmdir  backends3/some_bucket/.uploads/b97537ff-3711-40c1-8ad9-9619e45bec06
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:227] delete backends3/some_bucket/.uploads/b97537ff-3711-40c1-8ad9-9619e45bec06/0002.part
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:224] rmdir  backends3/some_bucket/.uploads/b97537ff-3711-40c1-8ad9-9619e45bec06
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:227] delete backends3/some_bucket/.uploads/b97537ff-3711-40c1-8ad9-9619e45bec06/0002.part
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:227] delete backends3/some_bucket/.uploads/b97537ff-3711-40c1-8ad9-9619e45bec06/0002.part
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:227] delete backends3/some_bucket/.uploads/b97537ff-3711-40c1-8ad9-9619e45bec06/0002.part
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:227] delete backends3/some_bucket/.uploads/b97537ff-3711-40c1-8ad9-9619e45bec06/0002.part
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:227] delete backends3/some_bucket/.uploads/b97537ff-3711-40c1-8ad9-9619e45bec06/0002.part
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:227] delete backends3/some_bucket/.uploads/b97537ff-3711-40c1-8ad9-9619e45bec06/0002.part
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:224] rmdir  backends3/some_bucket/.uploads/b97537ff-3711-40c1-8ad9-9619e45bec06
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:224] rmdir  backends3/some_bucket/.uploads/b97537ff-3711-40c1-8ad9-9619e45bec06
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:224] rmdir  backends3/some_bucket/.uploads/b97537ff-3711-40c1-8ad9-9619e45bec06
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:224] rmdir  backends3/some_bucket/.uploads/b97537ff-3711-40c1-8ad9-9619e45bec06
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:224] rmdir  backends3/some_bucket/.uploads/b97537ff-3711-40c1-8ad9-9619e45bec06
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:224] rmdir  backends3/some_bucket/.uploads/b97537ff-3711-40c1-8ad9-9619e45bec06
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:224] rmdir  backends3/some_bucket/.uploads/b97537ff-3711-40c1-8ad9-9619e45bec06
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:224] rmdir  backends3/some_bucket/.uploads/b97537ff-3711-40c1-8ad9-9619e45bec06
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:224] rmdir  backends3/some_bucket/.uploads/b97537ff-3711-40c1-8ad9-9619e45bec06
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:224] rmdir  backends3/some_bucket/.uploads/b97537ff-3711-40c1-8ad9-9619e45bec06
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:224] rmdir  backends3/some_bucket/.uploads/b97537ff-3711-40c1-8ad9-9619e45bec06
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:224] rmdir  backends3/some_bucket/.uploads/b97537ff-3711-40c1-8ad9-9619e45bec06
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:224] rmdir  backends3/some_bucket/.uploads/b97537ff-3711-40c1-8ad9-9619e45bec06
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:35] remote sync localhost:8888 progressed to 2021-11-22 15:41:01.146460374 +0800 CST 0.00/sec
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:224] rmdir  backends3/some_bucket/.uploads/e9a53d50-5fe5-44bd-8730-d6973544976d
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:224] rmdir  backends3/some_bucket/.uploads/e9a53d50-5fe5-44bd-8730-d6973544976d
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:224] rmdir  backends3/some_bucket/.uploads/e9a53d50-5fe5-44bd-8730-d6973544976d
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:224] rmdir  backends3/some_bucket/.uploads/e9a53d50-5fe5-44bd-8730-d6973544976d
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:224] rmdir  backends3/some_bucket/.uploads/e9a53d50-5fe5-44bd-8730-d6973544976d
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:224] rmdir  backends3/some_bucket/.uploads/e9a53d50-5fe5-44bd-8730-d6973544976d
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:224] rmdir  backends3/some_bucket/.uploads/e9a53d50-5fe5-44bd-8730-d6973544976d
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:224] rmdir  backends3/some_bucket/.uploads/e9a53d50-5fe5-44bd-8730-d6973544976d
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:224] rmdir  backends3/some_bucket/.uploads/e9a53d50-5fe5-44bd-8730-d6973544976d
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:224] rmdir  backends3/some_bucket/.uploads/e9a53d50-5fe5-44bd-8730-d6973544976d
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:224] rmdir  backends3/some_bucket/.uploads/e9a53d50-5fe5-44bd-8730-d6973544976d
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:224] rmdir  backends3/some_bucket/.uploads/e9a53d50-5fe5-44bd-8730-d6973544976d
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:224] rmdir  backends3/some_bucket/.uploads/e9a53d50-5fe5-44bd-8730-d6973544976d
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:224] rmdir  backends3/some_bucket/.uploads/e9a53d50-5fe5-44bd-8730-d6973544976d
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:224] rmdir  backends3/some_bucket/.uploads/e9a53d50-5fe5-44bd-8730-d6973544976d
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:224] rmdir  backends3/some_bucket/.uploads/e9a53d50-5fe5-44bd-8730-d6973544976d
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:224] rmdir  backends3/some_bucket/.uploads/e9a53d50-5fe5-44bd-8730-d6973544976d
I1122 15:41:01 84737 filer_remote_gateway_buckets.go:224] rmdir  backends3/some_bucket/.uploads/e9a53d50-5fe5-44bd-8730-d6973544976d
I1122 15:41:04 84737 filer_remote_gateway_buckets.go:35] remote sync localhost:8888 progressed to 2021-11-22 15:41:04.170261674 +0800 CST 0.00/sec
I1122 15:41:04 84737 filer_remote_gateway_buckets.go:224] rmdir  backends3/some_bucket/.uploads/8b3f4912-92f5-41b8-953a-c23d05b09a0b
I1122 15:41:04 84737 filer_remote_gateway_buckets.go:224] rmdir  backends3/some_bucket/.uploads/8b3f4912-92f5-41b8-953a-c23d05b09a0b
@imfantuan
Copy link
Contributor Author

Is there an easy way to re-sync?

@imfantuan imfantuan mentioned this issue Dec 5, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@imfantuan