volume.tier.upload rpc error #3038

trojansun · 2022-05-11T02:35:35Z

Describe the bug
When I use the volume.tier.upload command, I get an error rpc error: code = Unavailable desc = error reading from server: EOF

System Setup

List the command line to start "weed master", "weed volume", "weed filer", "weed s3", "weed mount".
After the configuration is complete, restart all weed related services.
The firewall of the volume server is fully open
OS version: Linux x64 ubuntu 20.04
output of weed version : version 30GB 3.02 a4ca3ed linux amd64
filer.toml

[postgres2]
enabled = true
createTable = """
  CREATE TABLE IF NOT EXISTS "%s" (
    dirhash   BIGINT,
    name      VARCHAR(65535),
    directory VARCHAR(65535),
    meta      bytea,
    PRIMARY KEY (dirhash, name)
  );
"""
hostname = "192.168.50.123"
port = 5432
username = "seaweedfs"
password = "seaweedfs"
database = "seaweedfs"          # create or use an existing database
schema = ""
sslmode = "disable"
connection_max_idle = 100
connection_max_open = 100
connection_max_lifetime_seconds = 0
# if insert/upsert failing, you can disable upsert or update query syntax to match your RDBMS syntax:
enableUpsert = true
upsertQuery = """INSERT INTO "%[1]s" (dirhash,name,directory,meta) VALUES($1,$2,$3,$4) ON CONFLICT (dirhash,name) DO UPDATE SET meta = EXCLUDED.meta WHERE "%[1]s".meta != EXCLUDED.meta"""

security.toml

[jwt.signing]
key = "seaweedfs"
expires_after_seconds = 30

[jwt.signing.read]
key = ""
expires_after_seconds = 30

[jwt.filer_signing]
key = "seaweedfs"
expires_after_seconds = 30

[jwt.filer_signing.read]
key = "seaweedfs"
expires_after_seconds = 30

[grpc]
ca = "/etc/seaweedfs/out/seaweedfs_CA.crt"

[grpc.volume]
cert = "/etc/seaweedfs/out/volume01.crt"
key = "/etc/seaweedfs/out/volume01.key"

[grpc.master]
cert = "/etc/seaweedfs/out/master01.crt"
key = "/etc/seaweedfs/out/master01.key"

[grpc.filer]
cert = "/etc/seaweedfs/out/filer01.crt"
key = "/etc/seaweedfs/out/filer01.key"

[grpc.client]
cert = "/etc/seaweedfs/out/client01.crt"
key = "/etc/seaweedfs/out/client01.key"

master.toml
s3 is accessible

[storage.backend]
[storage.backend.s3.default]
enabled = true
aws_access_key_id = ""         # if empty, loads from the shared credentials file (~/.aws/credentials).
aws_secret_access_key = ""     # if empty, loads from the shared credentials file (~/.aws/credentials).
region = "ap-beijing"
bucket = "test"    # an existing bucket
endpoint = "xxx.xxx.com"

Expected behavior

Screenshots

Additional context
volumeId exists collection exists

> lock
> volume.tier.upload -dest=s3 -volumeId=22 -collection=ttt
markVolumeReadonly 22 on 192.168.50.5:9005 ...
error: copy dat file for volume 22 on 192.168.50.5:9005 to s3: rpc error: code = Unavailable desc = error reading from server: EOF

The text was updated successfully, but these errors were encountered:

trojansun · 2022-05-11T05:47:40Z

volume weed version: version 30GB 3.02 a4ca3ed linux arm

Further trace volume logs:


panic: unaligned 64-bit atomic operation

goroutine 166 [running]:
runtime/internal/atomic.panicUnaligned()
        /usr/local/go/src/runtime/internal/atomic/unaligned.go:8 +0x24
runtime/internal/atomic.Xadd64(0x39e1ee4, 0x8000)
        /usr/local/go/src/runtime/internal/atomic/atomic_arm.s:258 +0x14
github.com/chrislusf/seaweedfs/weed/storage/backend/s3_backend.(*s3UploadProgressedReader).ReadAt(0x39e1ed8, {0x4d94000, 0x8000, 0x8000}, 0x0)
        /github/workspace/weed/storage/backend/s3_backend/s3_upload.go:86 +0x88
io.(*SectionReader).Read(0x3545fe0, {0x4d94000, 0x8000, 0x8000})
        /usr/local/go/src/io/io.go:512 +0xdc
github.com/aws/aws-sdk-go/aws/request.(*offsetReader).Read(0x3c7dc20, {0x4d94000, 0x8000, 0x8000})
        /go/pkg/mod/github.com/aws/aws-sdk-go@v1.43.45/aws/request/offset_reader.go:47 +0xf4
io.(*LimitedReader).Read(0x3755b10, {0x4d94000, 0x8000, 0x8000})
        /usr/local/go/src/io/io.go:476 +0xc4
io.copyBuffer({0xa6ba6708, 0x37ea310}, {0x217f75c, 0x3755b10}, {0x0, 0x0, 0x0})
        /usr/local/go/src/io/io.go:426 +0x1f8
io.Copy(...)
        /usr/local/go/src/io/io.go:385
io.CopyN({0xa6ba6708, 0x37ea310}, {0xa6ba66d8, 0x3c7dc20}, 0x4000000)
        /usr/local/go/src/io/io.go:361 +0x9c
github.com/aws/aws-sdk-go/aws/signer/v4.makeSha256Reader({0x2182714, 0x3c7dc20})
        /go/pkg/mod/github.com/aws/aws-sdk-go@v1.43.45/aws/signer/v4/v4.go:781 +0x204
github.com/aws/aws-sdk-go/aws/signer/v4.(*signingCtx).buildBodyDigest(0x38eecdc)
        /go/pkg/mod/github.com/aws/aws-sdk-go@v1.43.45/aws/signer/v4/v4.go:713 +0x338
github.com/aws/aws-sdk-go/aws/signer/v4.(*signingCtx).build(0x38eecdc, 0x0)
        /go/pkg/mod/github.com/aws/aws-sdk-go@v1.43.45/aws/signer/v4/v4.go:518 +0x38
github.com/aws/aws-sdk-go/aws/signer/v4.Signer.signWithBody({0x34973b0, 0x0, {0x2179c80, 0x38de478}, 0x0, 0x1, 0x1, 0x1d00ce4, 0x0}, 0x37b5980, ...)
        /go/pkg/mod/github.com/aws/aws-sdk-go@v1.43.45/aws/signer/v4/v4.go:351 +0x250
github.com/aws/aws-sdk-go/aws/signer/v4.SignSDKRequestWithCurrentTime(0x4d92000, 0x1d00ce4, {0x38de498, 0x1, 0x1})
        /go/pkg/mod/github.com/aws/aws-sdk-go@v1.43.45/aws/signer/v4/v4.go:482 +0x230
github.com/aws/aws-sdk-go/aws/signer/v4.BuildNamedHandler.func1(0x4d92000)
        /go/pkg/mod/github.com/aws/aws-sdk-go@v1.43.45/aws/signer/v4/v4.go:437 +0x3c
github.com/aws/aws-sdk-go/aws/request.(*HandlerList).Run(0x4d92104, 0x4d92000)
        /go/pkg/mod/github.com/aws/aws-sdk-go@v1.43.45/aws/request/handlers.go:267 +0x74
github.com/aws/aws-sdk-go/aws/request.(*Request).Sign(0x4d92000)
        /go/pkg/mod/github.com/aws/aws-sdk-go@v1.43.45/aws/request/request.go:445 +0x98
github.com/aws/aws-sdk-go/aws/request.(*Request).Send(0x4d92000)
        /go/pkg/mod/github.com/aws/aws-sdk-go@v1.43.45/aws/request/request.go:550 +0x118
github.com/aws/aws-sdk-go/service/s3.(*S3).UploadPartWithContext(0x38de490, {0x2188e5c, 0x34440c0}, 0x3489c20, {0x37afd28, 0x1, 0x2})
        /go/pkg/mod/github.com/aws/aws-sdk-go@v1.43.45/service/s3/api.go:11019 +0x90
github.com/aws/aws-sdk-go/service/s3/s3manager.(*multiuploader).send(0x350cc00, {{0x2183d1c, 0x3545fe0}, 0x1, 0x1cfdc24})
        /go/pkg/mod/github.com/aws/aws-sdk-go@v1.43.45/service/s3/s3manager/upload.go:718 +0x1fc
github.com/aws/aws-sdk-go/service/s3/s3manager.(*multiuploader).readChunk(0x350cc00, 0x356ffc0)
        /go/pkg/mod/github.com/aws/aws-sdk-go@v1.43.45/service/s3/s3manager/upload.go:696 +0xd8
created by github.com/aws/aws-sdk-go/service/s3/s3manager.(*multiuploader).upload
        /go/pkg/mod/github.com/aws/aws-sdk-go@v1.43.45/service/s3/s3manager/upload.go:592 +0x154

chrislusf · 2022-05-11T06:53:47Z

seems the volume server is not using 3.02. The trace was from 3.00 or earlier.

The specific problem, "unaligned 64-bit atomic operation", would not happen any more after the code change.

trojansun · 2022-05-11T06:57:23Z

volume weed version: version 30GB 3.02 a4ca3ed linux arm

My volume server is using 3.02。

Master weed version: version 30GB 3.02 a4ca3ed linux amd64
Volume1 weed version: version 30GB 3.02 a4ca3ed linux arm

chrislusf · 2022-05-11T06:58:39Z

The line of code, /github/workspace/weed/storage/backend/s3_backend/s3_upload.go:86, does not exist in 3.02

trojansun · 2022-05-11T07:04:15Z

The line of code, /github/workspace/weed/storage/backend/s3_backend/s3_upload.go:86, does not exist in 3.02

OK, thank you

trojansun · 2022-05-11T07:14:50Z

seems the volume server is not using 3.02. The trace was from 3.00 or earlier.

The specific problem, "unaligned 64-bit atomic operation", would not happen any more after the code change.

volume.tier.upload is OK, but volume.tier.download still reports an error. Please help to check again.

Master error message:

error: download dat file for volume 12 to 192.168.50.5:9006: rpc error: code = Unavailable desc = error reading from server: EOF

Volume log:

I0511 15:11:40  7581 volume.go:364] Start Seaweed volume server 30GB 3.02 a4ca3ed1f0c2bcde1d36bb48f81dedc3e1924679 at 192.168.50.5:9006
I0511 15:11:40  7581 volume_grpc_client_to_master.go:50] Volume server start with seed master nodes: [192.168.50.123:9333]
I0511 15:11:40  7581 volume_grpc_client_to_master.go:107] Heartbeat to: 192.168.50.123:9333
panic: unaligned 64-bit atomic operation

goroutine 144 [running]:
runtime/internal/atomic.panicUnaligned()
        /usr/local/go/src/runtime/internal/atomic/unaligned.go:8 +0x24
runtime/internal/atomic.Xadd64(0x4d482c4, 0x2e17)
        /usr/local/go/src/runtime/internal/atomic/atomic_arm.s:258 +0x14
github.com/chrislusf/seaweedfs/weed/storage/backend/s3_backend.(*s3DownloadProgressedWriter).WriteAt(0x4d482b8, {0x4da2000, 0x2e17, 0x8000}, 0x0)
        /github/workspace/weed/storage/backend/s3_backend/s3_download.go:74 +0x88
github.com/aws/aws-sdk-go/service/s3/s3manager.(*dlchunk).Write(0x4d5a5a0, {0x4da2000, 0x2e17, 0x8000})
        /go/pkg/mod/github.com/aws/aws-sdk-go@v1.43.45/service/s3/s3manager/download.go:587 +0xc8
io.copyBuffer({0x218ad64, 0x4d5a5a0}, {0xa6a9fa60, 0x4c8c700}, {0x0, 0x0, 0x0})
        /usr/local/go/src/io/io.go:428 +0x264
io.Copy(...)
        /usr/local/go/src/io/io.go:385
github.com/aws/aws-sdk-go/service/s3/s3manager.(*downloader).tryDownloadChunk(0x3595880, 0x4d441e0, {0x218ad64, 0x4d5a5a0})
        /go/pkg/mod/github.com/aws/aws-sdk-go@v1.43.45/service/s3/s3manager/download.go:467 +0x190
github.com/aws/aws-sdk-go/service/s3/s3manager.(*downloader).downloadChunk(0x3595880, {{0x218c290, 0x4d482b8}, 0x0, 0x4000000, 0x0, {0x0, 0x0}})
        /go/pkg/mod/github.com/aws/aws-sdk-go@v1.43.45/service/s3/s3manager/download.go:428 +0x1d4
github.com/aws/aws-sdk-go/service/s3/s3manager.(*downloader).getChunk(0x3595880)
        /go/pkg/mod/github.com/aws/aws-sdk-go@v1.43.45/service/s3/s3manager/download.go:393 +0x9c
github.com/aws/aws-sdk-go/service/s3/s3manager.(*downloader).download(0x3595880)
        /go/pkg/mod/github.com/aws/aws-sdk-go@v1.43.45/service/s3/s3manager/download.go:313 +0x48
github.com/aws/aws-sdk-go/service/s3/s3manager.Downloader.DownloadWithContext({0x4000000, 0x5, {0x21af878, 0x3934558}, {0x0, 0x0, 0x0}, {0x0, 0x0}}, {0x2199e54, ...}, ...)
        /go/pkg/mod/github.com/aws/aws-sdk-go@v1.43.45/service/s3/s3manager/download.go:219 +0x2c4
github.com/aws/aws-sdk-go/service/s3/s3manager.Downloader.Download({0x4000000, 0x5, {0x21af878, 0x3934558}, {0x0, 0x0, 0x0}, {0x0, 0x0}}, {0x218c290, ...}, ...)
        /go/pkg/mod/github.com/aws/aws-sdk-go@v1.43.45/service/s3/s3manager/download.go:164 +0x70
github.com/chrislusf/seaweedfs/weed/storage/backend/s3_backend.downloadFromS3({0x21af878, 0x3934558}, {0x4bf6cc0, 0x1d}, {0x38ff4f0, 0x4}, {0x37fbb00, 0x24}, 0x4c7e060)
        /github/workspace/weed/storage/backend/s3_backend/s3_download.go:45 +0x398
github.com/chrislusf/seaweedfs/weed/storage/backend/s3_backend.(*S3BackendStorage).DownloadFile(0x37c2f80, {0x4bf6cc0, 0x1d}, {0x37fbb00, 0x24}, 0x4c7e060)
        /github/workspace/weed/storage/backend/s3_backend/s3_backend.go:103 +0x134
github.com/chrislusf/seaweedfs/weed/server.(*VolumeServer).VolumeTierMoveDatFromRemote(0x38fc7e0, 0x4c7c3c0, {0x219e3e0, 0x4c40958})
        /github/workspace/weed/server/volume_grpc_tier_download.go:61 +0x408
github.com/chrislusf/seaweedfs/weed/pb/volume_server_pb._VolumeServer_VolumeTierMoveDatFromRemote_Handler({0x1bcdbe8, 0x38fc7e0}, {0x219d270, 0x4bfcc40})
        /github/workspace/weed/pb/volume_server_pb/volume_server_grpc.pb.go:1512 +0xdc
google.golang.org/grpc.(*Server).processStreamingRPC(0x35aac60, {0x219ff88, 0x3832700}, 0x4bee960, 0x4c66000, 0x30c2d10, 0x0)
        /go/pkg/mod/google.golang.org/grpc@v1.46.0/server.go:1542 +0xe48
google.golang.org/grpc.(*Server).handleStream(0x35aac60, {0x219ff88, 0x3832700}, 0x4bee960, 0x0)
        /go/pkg/mod/google.golang.org/grpc@v1.46.0/server.go:1624 +0xa2c
google.golang.org/grpc.(*Server).serveStreams.func1.2()
        /go/pkg/mod/google.golang.org/grpc@v1.46.0/server.go:922 +0xb0
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /go/pkg/mod/google.golang.org/grpc@v1.46.0/server.go:920 +0x284

trojansun · 2022-05-11T07:36:02Z

I don't know if this error is related to atomic.AddInt64(&w.written, int64(n)) in the weed\storage\backend\s3_backend\s3_download.go:74 file.

thanks

fix #3038 (comment)

chrislusf · 2022-05-11T07:56:09Z

added a fix.

trojansun · 2022-05-11T07:57:33Z

added a fix.

OK, thank you

chrislusf closed this as completed May 11, 2022

chrislusf added a commit that referenced this issue May 11, 2022

fix atomic add int64

3e20336

fix #3038 (comment)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

volume.tier.upload rpc error #3038

volume.tier.upload rpc error #3038

trojansun commented May 11, 2022 •

edited

Loading

trojansun commented May 11, 2022 •

edited

Loading

chrislusf commented May 11, 2022

trojansun commented May 11, 2022

chrislusf commented May 11, 2022

trojansun commented May 11, 2022

trojansun commented May 11, 2022 •

edited

Loading

trojansun commented May 11, 2022

chrislusf commented May 11, 2022

trojansun commented May 11, 2022

volume.tier.upload rpc error #3038

volume.tier.upload rpc error #3038

Comments

trojansun commented May 11, 2022 • edited Loading

trojansun commented May 11, 2022 • edited Loading

chrislusf commented May 11, 2022

trojansun commented May 11, 2022

chrislusf commented May 11, 2022

trojansun commented May 11, 2022

trojansun commented May 11, 2022 • edited Loading

trojansun commented May 11, 2022

chrislusf commented May 11, 2022

trojansun commented May 11, 2022

trojansun commented May 11, 2022 •

edited

Loading

trojansun commented May 11, 2022 •

edited

Loading

trojansun commented May 11, 2022 •

edited

Loading