Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

volume uploading errror: EOF #1592

Closed
kmlebedev opened this issue Nov 2, 2020 · 12 comments
Closed

volume uploading errror: EOF #1592

kmlebedev opened this issue Nov 2, 2020 · 12 comments

Comments

@kmlebedev
Copy link
Contributor

Describe the bug
volume uploading errror: EOF
appears very rarely

System Setup
2.07

Additional context
Created not to forget

seaweedfs/weed/topology/store_replicate.go:150 +0xda
upload_content.go:103] uploading to http://fast-volume-0:8080/257,465d33ef058c35?ts=1604281505&ttl=&type=replicate: upload  0 bytes to http://fast-volume-0:8080/257,465d33ef058c35?ts=1604281505&ttl=&type=replicate: Post "http://fast-volume-0:8080/257,465d33ef058c35?ts=1604281505&ttl=&type=replicate": EOF
seaweedfs/weed/operation/upload_content.go:172 +0x4a1
runtime/debug.PrintStack()
libexec/src/runtime/debug/stack.go:16 +0x22
github.com/chrislusf/seaweedfs/weed/operation.upload_content(0xc000118580, 0x71, 0xc0004cf928, 0x0, 0x0, 0x3fe01, 0x0, 0x0, 0x0, 0xc0004cfba0, ...)
seaweedfs/weed/operation/upload_content.go:99 +0x1ba
/seaweedfs/weed/topology/store_replicate.go:84 +0x660
github.com/chrislusf/seaweedfs/weed/operation.UploadData(...)
github.com/chrislusf/seaweedfs/weed/operation.doUploadData(0xc000118580, 0x71, 0x0, 0x0, 0xc000118500, 0xc001846000, 0x10225, 0x12f10, 0x1, 0x0, ...)
github.com/chrislusf/seaweedfs/weed/operation.retriedUploadData(0xc000118580, 0x71, 0x0, 0x0, 0x0, 0xc001846000, 0x10225, 0x12f10, 0x1bf1601, 0x0, ...)
github.com/chrislusf/seaweedfs/weed/topology.ReplicatedWrite.func1(0xc000047a40, 0x35, 0xc000047a80, 0x35, 0x7154c3, 0xc0005cd540)
github.com/chrislusf/seaweedfs/weed/topology.distributedOperation.func1(0xc003513260, 0xc000047a40, 0x35, 0xc000047a80, 0x35, 0xc000878de0)
@chrislusf
Copy link
Collaborator

debug.PrintStack() was added to track this.

This is only in the logs, no other problems with the retries, correct?

@kmlebedev
Copy link
Contributor Author

kmlebedev commented Nov 2, 2020 •
edited
Loading

debug.PrintStack() was added to track this.

This is only in the logs, no other problems with the retries, correct?

In version 2.07, retries were disabled and client received httpStatus 500

upload  0 bytes to http://fast-volume-0:8080/257,465d33ef058c35?ts=1604281505&ttl=&type=replicate: Post "http://fast-volume-0:8080/257,465d33ef058c35?ts=1604281505&ttl=&type=replicate": EOF
1 s3api_object_handlers.go:317] post to filer: Put "http://172.18.1.2:9090/buckets/bucket/.uploads/392bd995-1b52-4e09-9614-4b5bf6d6c1de/0186.part?collection=bucket": unexpected EOF
common.go:51] response method:PUT URL:/buckets/bucket/.uploads/392bd995-1b52-4e09-9614-4b5bf6d6c1de/0186.part?collection=bucket with httpStatus:500 and JSON:{"error":"read input: unexpected EOF"}

@Suika
Copy link
Contributor

Suika commented Nov 2, 2020

Probably the same type of problem, since I noticed it in the logs not too long ago.

FUSE Mount Logs 
E1101 20:40:19 14672 filehandle.go:271] fh flush create /buckets/asagi-images/a/image/1498/09/1498099848685.jpg: CreateEntry: rpc error: code = Internal desc = grpc: error unmarshalling request: unexpected EOF
E1101 20:40:19 14672 filehandle.go:285] /buckets/asagi-images/a/image/1498/09/1498099848685.jpg fh 9026040808888781237 flush: fh flush create /buckets/asagi-images/a/image/1498/09/1498099848685.jpg: CreateEntry: rpc error: code = Internal desc = grpc: error unmarshalling request: unexpected EOF
E1101 20:40:19 14672 filehandle.go:187] Release doFlush 1498099848685.jpg: input/output error
E1102 07:03:40 14672 file.go:311] UpdateEntry file /buckets/asagi-images/a/image/1521/77/1521770452265.jpg: rpc error: code = Internal desc = grpc: error unmarshalling request: unexpected EOF
E1102 12:59:05 14672 file.go:311] UpdateEntry file /buckets/asagi-images/a/image/1538/24/1538248115400.jpg: rpc error: code = Internal desc = grpc: error unmarshalling request: unexpected EOF
E1102 15:05:38 14672 file.go:311] UpdateEntry file /buckets/asagi-images/a/image/1544/29/1544298049244.jpg: rpc error: code = Internal desc = grpc: error unmarshalling request: unexpected EOF
Filer Logs 
    today at 1:55 AM E1102 01:55:59     1 upload_content.go:236] upload  0 bytes to http://172.27.0.4:8080/1,1eba691e7bf8f6: Post http://172.27.0.4:8080/1,1eba691e7bf8f6: http: server closed idle connection
    today at 1:55 AM goroutine 113 [running]:
    today at 1:55 AM runtime/debug.Stack(0x8d, 0x0, 0x0)
    today at 1:55 AM /usr/lib/go/src/runtime/debug/stack.go:24 +0x9d
    today at 1:55 AM runtime/debug.PrintStack()
    today at 1:55 AM /usr/lib/go/src/runtime/debug/stack.go:16 +0x22
    today at 1:55 AM github.com/chrislusf/seaweedfs/weed/operation.upload_content(0xc00451d680, 0x27, 0xc0040b1ae0, 0x0, 0x0, 0x100, 0x0, 0x0, 0x0, 0x0, ...)
    today at 1:55 AM /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:237 +0xc5d
    today at 1:55 AM github.com/chrislusf/seaweedfs/weed/operation.doUploadData(0xc00451d680, 0x27, 0x0, 0x0, 0xc00451d600, 0xc004822000, 0x6f442, 0x6fdd2, 0x0, 0x0, ...)
    today at 1:55 AM /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:172 +0x4a1
    today at 1:55 AM github.com/chrislusf/seaweedfs/weed/operation.retriedUploadData(0xc00451d680, 0x27, 0x0, 0x0, 0x0, 0xc004822000, 0x6f442, 0x6fdd2, 0x0, 0x0, ...)
    today at 1:55 AM /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:99 +0x1ba
    today at 1:55 AM github.com/chrislusf/seaweedfs/weed/operation.UploadData(...)
    today at 1:55 AM /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:69
    today at 1:55 AM github.com/chrislusf/seaweedfs/weed/filer.(*Filer).assignAndUpload(0xc00002a5a0, 0xc004822000, 0x6f442, 0x6fdd2, 0xc00018cd40, 0x4ef241, 0x3676840, 0x205da20)
    today at 1:55 AM /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify_append.go:67 +0x2a2
    today at 1:55 AM github.com/chrislusf/seaweedfs/weed/filer.(*Filer).appendToFile(0xc00002a5a0, 0xc00451d650, 0x2c, 0xc004822000, 0x6f442, 0x6fdd2, 0x2c, 0x132)
    today at 1:55 AM /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify_append.go:16 +0x67
    today at 1:55 AM github.com/chrislusf/seaweedfs/weed/filer.(*Filer).logFlushFunc(0xc00002a5a0, 0xc04b6b7, 0xed7315e2a, 0x3726da0, 0xffc84a3, 0xed7315e2f, 0x3726da0, 0xc004822000, 0x6f442, 0x6fdd2)
    today at 1:55 AM /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify.go:98 +0x42f
    today at 1:55 AM github.com/chrislusf/seaweedfs/weed/util/log_buffer.(*LogBuffer).loopFlush(0xc0005180d0)
    today at 1:55 AM /go/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:130 +0xfe
    today at 1:55 AM created by github.com/chrislusf/seaweedfs/weed/util/log_buffer.NewLogBuffer
    today at 1:55 AM /go/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:51 +0x187
    today at 1:55 AM W1102 01:55:59     1 upload_content.go:103] uploading to http://172.27.0.4:8080/1,1eba691e7bf8f6: upload  0 bytes to http://172.27.0.4:8080/1,1eba691e7bf8f6: Post http://172.27.0.4:8080/1,1eba691e7bf8f6: http: server closed idle connection
    today at 2:57 AM E1102 02:57:59     1 upload_content.go:236] upload  0 bytes to http://172.27.0.4:8080/1,20cbfcfba69e8d: Post http://172.27.0.4:8080/1,20cbfcfba69e8d: write tcp 172.27.0.5:40462->172.27.0.4:8080: write: broken pipe
    today at 2:57 AM goroutine 113 [running]:
    today at 2:57 AM runtime/debug.Stack(0xa9, 0x0, 0x0)
    today at 2:57 AM /usr/lib/go/src/runtime/debug/stack.go:24 +0x9d
    today at 2:57 AM runtime/debug.PrintStack()
    today at 2:57 AM /usr/lib/go/src/runtime/debug/stack.go:16 +0x22
    today at 2:57 AM github.com/chrislusf/seaweedfs/weed/operation.upload_content(0xc002c9ac30, 0x27, 0xc000015ae0, 0x0, 0x0, 0x100, 0x0, 0x0, 0x0, 0x0, ...)
    today at 2:57 AM /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:237 +0xc5d
    today at 2:57 AM github.com/chrislusf/seaweedfs/weed/operation.doUploadData(0xc002c9ac30, 0x27, 0x0, 0x0, 0xc002c9ac00, 0xc012a6e000, 0x1a5dcd, 0x1a67e5, 0x0, 0x0, ...)
    today at 2:57 AM /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:172 +0x4a1
    today at 2:57 AM github.com/chrislusf/seaweedfs/weed/operation.retriedUploadData(0xc002c9ac30, 0x27, 0x0, 0x0, 0x0, 0xc012a6e000, 0x1a5dcd, 0x1a67e5, 0x0, 0x0, ...)
    today at 2:57 AM /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:99 +0x1ba
    today at 2:57 AM github.com/chrislusf/seaweedfs/weed/operation.UploadData(...)
    today at 2:57 AM /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:69
    today at 2:57 AM github.com/chrislusf/seaweedfs/weed/filer.(*Filer).assignAndUpload(0xc00002a5a0, 0xc012a6e000, 0x1a5dcd, 0x1a67e5, 0xc0008ced40, 0x4ef241, 0x3676840, 0x205da20)
    today at 2:57 AM /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify_append.go:67 +0x2a2
    today at 2:57 AM github.com/chrislusf/seaweedfs/weed/filer.(*Filer).appendToFile(0xc00002a5a0, 0xc000276630, 0x2c, 0xc012a6e000, 0x1a5dcd, 0x1a67e5, 0x2c, 0x132)
    today at 2:57 AM /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify_append.go:16 +0x67
    today at 2:57 AM github.com/chrislusf/seaweedfs/weed/filer.(*Filer).logFlushFunc(0xc00002a5a0, 0x1a471c9d, 0xed7316c83, 0x3726da0, 0x67c1751, 0xed7316cab, 0x3726da0, 0xc012a6e000, 0x1a5dcd, 0x1a67e5)
    today at 2:57 AM /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify.go:98 +0x42f
    today at 2:57 AM github.com/chrislusf/seaweedfs/weed/util/log_buffer.(*LogBuffer).loopFlush(0xc0005180d0)
    today at 2:57 AM /go/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:130 +0xfe
    today at 2:57 AM created by github.com/chrislusf/seaweedfs/weed/util/log_buffer.NewLogBuffer
    today at 2:57 AM /go/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:51 +0x187
    today at 2:57 AM W1102 02:57:59     1 upload_content.go:103] uploading to http://172.27.0.4:8080/1,20cbfcfba69e8d: upload  0 bytes to http://172.27.0.4:8080/1,20cbfcfba69e8d: Post http://172.27.0.4:8080/1,20cbfcfba69e8d: write tcp 172.27.0.5:40462->172.27.0.4:8080: write: broken pipe
    today at 3:00 AM E1102 03:00:59     1 upload_content.go:236] upload  0 bytes to http://172.27.0.4:8080/1,20dc72da815dae: Post http://172.27.0.4:8080/1,20dc72da815dae: write tcp 172.27.0.5:48444->172.27.0.4:8080: write: connection reset by peer
    today at 3:00 AM goroutine 113 [running]:
    today at 3:00 AM runtime/debug.Stack(0xb6, 0x0, 0x0)
    today at 3:00 AM /usr/lib/go/src/runtime/debug/stack.go:24 +0x9d
    today at 3:00 AM runtime/debug.PrintStack()
    today at 3:00 AM /usr/lib/go/src/runtime/debug/stack.go:16 +0x22
    today at 3:00 AM github.com/chrislusf/seaweedfs/weed/operation.upload_content(0xc005c5f8c0, 0x27, 0xc00a921ae0, 0x0, 0x0, 0x100, 0x0, 0x0, 0x0, 0x0, ...)
    today at 3:00 AM /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:237 +0xc5d
    today at 3:00 AM github.com/chrislusf/seaweedfs/weed/operation.doUploadData(0xc005c5f8c0, 0x27, 0x0, 0x0, 0xc005c5f800, 0xc012d32000, 0x386edc, 0x387460, 0x0, 0x0, ...)
    today at 3:00 AM /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:172 +0x4a1
    today at 3:00 AM github.com/chrislusf/seaweedfs/weed/operation.retriedUploadData(0xc005c5f8c0, 0x27, 0x0, 0x0, 0x0, 0xc012d32000, 0x386edc, 0x387460, 0x0, 0x0, ...)
    today at 3:00 AM /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:99 +0x1ba
    today at 3:00 AM github.com/chrislusf/seaweedfs/weed/operation.UploadData(...)
    today at 3:00 AM /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:69
    today at 3:00 AM github.com/chrislusf/seaweedfs/weed/filer.(*Filer).assignAndUpload(0xc00002a5a0, 0xc012d32000, 0x386edc, 0x387460, 0xc0009d3d40, 0x4ef241, 0x3676840, 0x205da20)
    today at 3:00 AM /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify_append.go:67 +0x2a2
    today at 3:00 AM github.com/chrislusf/seaweedfs/weed/filer.(*Filer).appendToFile(0xc00002a5a0, 0xc005afdce0, 0x2c, 0xc012d32000, 0x386edc, 0x387460, 0x2c, 0x132)
    today at 3:00 AM /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify_append.go:16 +0x67
    today at 3:00 AM github.com/chrislusf/seaweedfs/weed/filer.(*Filer).logFlushFunc(0xc00002a5a0, 0x2c9f954b, 0xed7316d2f, 0x3726da0, 0x2be777b1, 0xed7316d6b, 0x3726da0, 0xc012d32000, 0x386edc, 0x387460)
    today at 3:00 AM /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify.go:98 +0x42f
    today at 3:00 AM github.com/chrislusf/seaweedfs/weed/util/log_buffer.(*LogBuffer).loopFlush(0xc0005180d0)
    today at 3:00 AM /go/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:130 +0xfe
    today at 3:00 AM created by github.com/chrislusf/seaweedfs/weed/util/log_buffer.NewLogBuffer
    today at 3:00 AM /go/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:51 +0x187
    today at 3:00 AM W1102 03:00:59     1 upload_content.go:103] uploading to http://172.27.0.4:8080/1,20dc72da815dae: upload  0 bytes to http://172.27.0.4:8080/1,20dc72da815dae: Post http://172.27.0.4:8080/1,20dc72da815dae: write tcp 172.27.0.5:48444->172.27.0.4:8080: write: connection reset by peer
    today at 3:46 AM E1102 03:46:59     1 upload_content.go:236] upload  0 bytes to http://172.27.0.4:8080/1,226e6b2303ec0f: Post http://172.27.0.4:8080/1,226e6b2303ec0f: write tcp 172.27.0.5:49660->172.27.0.4:8080: write: connection reset by peer
    today at 3:46 AM goroutine 113 [running]:
    today at 3:46 AM runtime/debug.Stack(0xb6, 0x0, 0x0)
    today at 3:46 AM /usr/lib/go/src/runtime/debug/stack.go:24 +0x9d
    today at 3:46 AM runtime/debug.PrintStack()
    today at 3:46 AM /usr/lib/go/src/runtime/debug/stack.go:16 +0x22
    today at 3:46 AM github.com/chrislusf/seaweedfs/weed/operation.upload_content(0xc00900ecf0, 0x27, 0xc0007efae0, 0x0, 0x0, 0x100, 0x0, 0x0, 0x0, 0x0, ...)
    today at 3:46 AM /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:237 +0xc5d
    today at 3:46 AM github.com/chrislusf/seaweedfs/weed/operation.doUploadData(0xc00900ecf0, 0x27, 0x0, 0x0, 0xc00900ec00, 0xc010a50000, 0xe308a, 0xe55ea, 0x0, 0x0, ...)
    today at 3:46 AM /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:172 +0x4a1
    today at 3:46 AM github.com/chrislusf/seaweedfs/weed/operation.retriedUploadData(0xc00900ecf0, 0x27, 0x0, 0x0, 0x0, 0xc010a50000, 0xe308a, 0xe55ea, 0x0, 0x0, ...)
    today at 3:46 AM /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:99 +0x1ba
    today at 3:46 AM github.com/chrislusf/seaweedfs/weed/operation.UploadData(...)
    today at 3:46 AM /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:69
    today at 3:46 AM github.com/chrislusf/seaweedfs/weed/filer.(*Filer).assignAndUpload(0xc00002a5a0, 0xc010a50000, 0xe308a, 0xe55ea, 0xc0007efd40, 0x4ef241, 0x3676840, 0x205da20)
    today at 3:46 AM /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify_append.go:67 +0x2a2
    today at 3:46 AM github.com/chrislusf/seaweedfs/weed/filer.(*Filer).appendToFile(0xc00002a5a0, 0xc00900ecc0, 0x2c, 0xc010a50000, 0xe308a, 0xe55ea, 0x2c, 0x132)
    today at 3:46 AM /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify_append.go:16 +0x67
    today at 3:46 AM github.com/chrislusf/seaweedfs/weed/filer.(*Filer).logFlushFunc(0xc00002a5a0, 0x3589141f, 0xed73177fd, 0x3726da0, 0x2f8d38a8, 0xed7317833, 0x3726da0, 0xc010a50000, 0xe308a, 0xe55ea)
    today at 3:46 AM /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify.go:98 +0x42f
    today at 3:46 AM github.com/chrislusf/seaweedfs/weed/util/log_buffer.(*LogBuffer).loopFlush(0xc0005180d0)
    today at 3:46 AM /go/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:130 +0xfe
    today at 3:46 AM created by github.com/chrislusf/seaweedfs/weed/util/log_buffer.NewLogBuffer
    today at 3:46 AM /go/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:51 +0x187
    today at 3:46 AM W1102 03:46:59     1 upload_content.go:103] uploading to http://172.27.0.4:8080/1,226e6b2303ec0f: upload  0 bytes to http://172.27.0.4:8080/1,226e6b2303ec0f: Post http://172.27.0.4:8080/1,226e6b2303ec0f: write tcp 172.27.0.5:49660->172.27.0.4:8080: write: connection reset by peer

@chrislusf
Copy link
Collaborator

FUSE Mount Logs

@Suika do you have logs on the filer side for the corresponding time?

@Suika
Copy link
Contributor

Suika commented Nov 3, 2020

These are the only logs that the filer has.

Filer Logs Complete 
I1101 11:54:58     1 leveldb2_store.go:40] filer store leveldb2 dir: /data/filerldb2
I1101 11:54:58     1 file_util.go:23] Folder /data/filerldb2 Permission: -rwxr-xr-x
I1101 11:54:58     1 filer.go:96] create filer.store.id to -57649120
I1101 11:54:58     1 configuration.go:25] Configure filer for leveldb2
I1101 11:54:58     1 filer.go:172] Start Seaweed Filer 8000GB 2.07 44921220 at 172.27.0.5:8888
I1101 11:54:58     1 filer_grpc_server_sub_meta.go:175] + listener filer:172.27.0.5:8888@172.27.0.5:38728
I1101 11:54:58     1 filer_grpc_server_sub_meta.go:70]  filer:172.27.0.5:8888@172.27.0.5:38728 local subscribe / from 2020-11-01 11:53:58.695326685 +0000 UTC
I1101 11:54:58     1 filer_grpc_server_sub_meta.go:85] after local log reads, filer:172.27.0.5:8888@172.27.0.5:38728 local subscribe / from 2020-11-01 11:53:58.695326685 +0000 UTC
I1101 11:55:14     1 filer_grpc_server_sub_meta.go:175] + listener mount@172.27.0.1:39354
I1101 11:55:14     1 filer_grpc_server_sub_meta.go:26]  mount@172.27.0.1:39354 starts to subscribe / from 2020-11-01 03:01:57.144498292 +0000 UTC
I1101 12:01:57     1 filer_grpc_server_sub_meta.go:175] + listener mount@172.27.0.1:55862
I1101 12:01:57     1 filer_grpc_server_sub_meta.go:26]  mount@172.27.0.1:55862 starts to subscribe / from 2020-11-01 12:01:57.400573138 +0000 UTC
I1101 12:01:59     1 filer_grpc_server_sub_meta.go:166] => client mount@172.27.0.1:39354: rpc error: code = Unavailable desc = transport is closing
E1101 12:01:59     1 filer_grpc_server_sub_meta.go:49] processed to 2020-11-01 12:01:59.566107795 +0000 UTC: rpc error: code = Unavailable desc = transport is closing
I1101 12:02:02     1 filer_grpc_server_sub_meta.go:180] - listener mount@172.27.0.1:39354
E1101 12:48:58     1 upload_content.go:236] upload  0 bytes to http://172.27.0.4:8080/1,024cd91b4e9e67: Post http://172.27.0.4:8080/1,024cd91b4e9e67: write tcp 172.27.0.5:51982->172.27.0.4:8080: write: broken pipe
goroutine 113 [running]:
runtime/debug.Stack(0xa9, 0x0, 0x0)
        /usr/lib/go/src/runtime/debug/stack.go:24 +0x9d
runtime/debug.PrintStack()
        /usr/lib/go/src/runtime/debug/stack.go:16 +0x22
github.com/chrislusf/seaweedfs/weed/operation.upload_content(0xc014928870, 0x27, 0xc0007ebae0, 0x0, 0x0, 0x100, 0x0, 0x0, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:237 +0xc5d
github.com/chrislusf/seaweedfs/weed/operation.doUploadData(0xc014928870, 0x27, 0x0, 0x0, 0xc014928800, 0xc00092e000, 0x7aa08, 0x7ad6e, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:172 +0x4a1
github.com/chrislusf/seaweedfs/weed/operation.retriedUploadData(0xc014928870, 0x27, 0x0, 0x0, 0x0, 0xc00092e000, 0x7aa08, 0x7ad6e, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:99 +0x1ba
github.com/chrislusf/seaweedfs/weed/operation.UploadData(...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:69
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).assignAndUpload(0xc00002a5a0, 0xc00092e000, 0x7aa08, 0x7ad6e, 0xc000b11d40, 0x4ef241, 0x3676840, 0x205da20)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify_append.go:67 +0x2a2
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).appendToFile(0xc00002a5a0, 0xc00cfb8c90, 0x2c, 0xc00092e000, 0x7aa08, 0x7ad6e, 0x2c, 0x131)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify_append.go:16 +0x67
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).logFlushFunc(0xc00002a5a0, 0xf56f468, 0xed730a588, 0x3726da0, 0x17b41ad5, 0xed730a5b3, 0x3726da0, 0xc00092e000, 0x7aa08, 0x7ad6e)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify.go:98 +0x42f
github.com/chrislusf/seaweedfs/weed/util/log_buffer.(*LogBuffer).loopFlush(0xc0005180d0)
        /go/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:130 +0xfe
created by github.com/chrislusf/seaweedfs/weed/util/log_buffer.NewLogBuffer
        /go/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:51 +0x187
W1101 12:48:58     1 upload_content.go:103] uploading to http://172.27.0.4:8080/1,024cd91b4e9e67: upload  0 bytes to http://172.27.0.4:8080/1,024cd91b4e9e67: Post http://172.27.0.4:8080/1,024cd91b4e9e67: write tcp 172.27.0.5:51982->172.27.0.4:8080: write: broken pipe
E1101 15:27:58     1 upload_content.go:236] upload  0 bytes to http://172.27.0.4:8080/1,06977bbbb4f9a8: Post http://172.27.0.4:8080/1,06977bbbb4f9a8: write tcp 172.27.0.5:59416->172.27.0.4:8080: use of closed network connection
goroutine 113 [running]:
runtime/debug.Stack(0xb7, 0x0, 0x0)
        /usr/lib/go/src/runtime/debug/stack.go:24 +0x9d
runtime/debug.PrintStack()
        /usr/lib/go/src/runtime/debug/stack.go:16 +0x22
github.com/chrislusf/seaweedfs/weed/operation.upload_content(0xc006bfc000, 0x27, 0xc002451ae0, 0x0, 0x0, 0x100, 0x0, 0x0, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:237 +0xc5d
github.com/chrislusf/seaweedfs/weed/operation.doUploadData(0xc006bfc000, 0x27, 0x0, 0x0, 0xc006bfc000, 0xc006c06000, 0x949e2, 0x95ab4, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:172 +0x4a1
github.com/chrislusf/seaweedfs/weed/operation.retriedUploadData(0xc006bfc000, 0x27, 0x0, 0x0, 0x0, 0xc006c06000, 0x949e2, 0x95ab4, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:99 +0x1ba
github.com/chrislusf/seaweedfs/weed/operation.UploadData(...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:69
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).assignAndUpload(0xc00002a5a0, 0xc006c06000, 0x949e2, 0x95ab4, 0xc000833d40, 0x4ef241, 0x3676840, 0x205da20)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify_append.go:67 +0x2a2
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).appendToFile(0xc00002a5a0, 0xc006bcccc0, 0x2c, 0xc006c06000, 0x949e2, 0x95ab4, 0x2c, 0x131)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify_append.go:16 +0x67
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).logFlushFunc(0xc00002a5a0, 0xc17c027, 0xed730cad3, 0x3726da0, 0x2c33205c, 0xed730cafe, 0x3726da0, 0xc006c06000, 0x949e2, 0x95ab4)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify.go:98 +0x42f
github.com/chrislusf/seaweedfs/weed/util/log_buffer.(*LogBuffer).loopFlush(0xc0005180d0)
        /go/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:130 +0xfe
created by github.com/chrislusf/seaweedfs/weed/util/log_buffer.NewLogBuffer
        /go/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:51 +0x187
W1101 15:27:58     1 upload_content.go:103] uploading to http://172.27.0.4:8080/1,06977bbbb4f9a8: upload  0 bytes to http://172.27.0.4:8080/1,06977bbbb4f9a8: Post http://172.27.0.4:8080/1,06977bbbb4f9a8: write tcp 172.27.0.5:59416->172.27.0.4:8080: use of closed network connection
E1101 15:52:58     1 upload_content.go:236] upload  0 bytes to http://172.27.0.4:8080/1,06df32ee552b9c: Post http://172.27.0.4:8080/1,06df32ee552b9c: write tcp 172.27.0.5:55642->172.27.0.4:8080: write: broken pipe
goroutine 113 [running]:
runtime/debug.Stack(0xa9, 0x0, 0x0)
        /usr/lib/go/src/runtime/debug/stack.go:24 +0x9d
runtime/debug.PrintStack()
        /usr/lib/go/src/runtime/debug/stack.go:16 +0x22
github.com/chrislusf/seaweedfs/weed/operation.upload_content(0xc00ccedc20, 0x27, 0xc000017ae0, 0x0, 0x0, 0x100, 0x0, 0x0, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:237 +0xc5d
github.com/chrislusf/seaweedfs/weed/operation.doUploadData(0xc00ccedc20, 0x27, 0x0, 0x0, 0xc00ccedc00, 0xc011866000, 0x11b292, 0x11ea7a, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:172 +0x4a1
github.com/chrislusf/seaweedfs/weed/operation.retriedUploadData(0xc00ccedc20, 0x27, 0x0, 0x0, 0x0, 0xc011866000, 0x11b292, 0x11ea7a, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:99 +0x1ba
github.com/chrislusf/seaweedfs/weed/operation.UploadData(...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:69
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).assignAndUpload(0xc00002a5a0, 0xc011866000, 0x11b292, 0x11ea7a, 0xc000017d40, 0x4ef241, 0x3676840, 0x205da20)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify_append.go:67 +0x2a2
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).appendToFile(0xc00002a5a0, 0xc00cd35410, 0x2c, 0xc011866000, 0x11b292, 0x11ea7a, 0x2c, 0x131)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify_append.go:16 +0x67
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).logFlushFunc(0xc00002a5a0, 0x3099bee8, 0xed730d0a3, 0x3726da0, 0x2d4497d3, 0xed730d0da, 0x3726da0, 0xc011866000, 0x11b292, 0x11ea7a)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify.go:98 +0x42f
github.com/chrislusf/seaweedfs/weed/util/log_buffer.(*LogBuffer).loopFlush(0xc0005180d0)
        /go/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:130 +0xfe
created by github.com/chrislusf/seaweedfs/weed/util/log_buffer.NewLogBuffer
        /go/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:51 +0x187
W1101 15:52:58     1 upload_content.go:103] uploading to http://172.27.0.4:8080/1,06df32ee552b9c: upload  0 bytes to http://172.27.0.4:8080/1,06df32ee552b9c: Post http://172.27.0.4:8080/1,06df32ee552b9c: write tcp 172.27.0.5:55642->172.27.0.4:8080: write: broken pipe
E1101 20:22:59     1 upload_content.go:236] upload  0 bytes to http://172.27.0.4:8080/1,13f5b2c53fb872: Post http://172.27.0.4:8080/1,13f5b2c53fb872: write tcp 172.27.0.5:50944->172.27.0.4:8080: write: broken pipe
goroutine 113 [running]:
runtime/debug.Stack(0xa9, 0x0, 0x0)
        /usr/lib/go/src/runtime/debug/stack.go:24 +0x9d
runtime/debug.PrintStack()
        /usr/lib/go/src/runtime/debug/stack.go:16 +0x22
github.com/chrislusf/seaweedfs/weed/operation.upload_content(0xc008096330, 0x27, 0xc004ff3ae0, 0x0, 0x0, 0x100, 0x0, 0x0, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:237 +0xc5d
github.com/chrislusf/seaweedfs/weed/operation.doUploadData(0xc008096330, 0x27, 0x0, 0x0, 0xc008096300, 0xc015894000, 0x260561, 0x260d0f, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:172 +0x4a1
github.com/chrislusf/seaweedfs/weed/operation.retriedUploadData(0xc008096330, 0x27, 0x0, 0x0, 0x0, 0xc015894000, 0x260561, 0x260d0f, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:99 +0x1ba
github.com/chrislusf/seaweedfs/weed/operation.UploadData(...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:69
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).assignAndUpload(0xc00002a5a0, 0xc015894000, 0x260561, 0x260d0f, 0xc00018fd40, 0x4ef241, 0x3676840, 0x205da20)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify_append.go:67 +0x2a2
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).appendToFile(0xc00002a5a0, 0xc008096300, 0x2c, 0xc015894000, 0x260561, 0x260d0f, 0x2c, 0x131)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify_append.go:16 +0x67
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).logFlushFunc(0xc00002a5a0, 0x71ed6db, 0xed7310ffe, 0x3726da0, 0x3607b574, 0xed7311020, 0x3726da0, 0xc015894000, 0x260561, 0x260d0f)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify.go:98 +0x42f
github.com/chrislusf/seaweedfs/weed/util/log_buffer.(*LogBuffer).loopFlush(0xc0005180d0)
        /go/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:130 +0xfe
created by github.com/chrislusf/seaweedfs/weed/util/log_buffer.NewLogBuffer
        /go/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:51 +0x187
W1101 20:22:59     1 upload_content.go:103] uploading to http://172.27.0.4:8080/1,13f5b2c53fb872: upload  0 bytes to http://172.27.0.4:8080/1,13f5b2c53fb872: Post http://172.27.0.4:8080/1,13f5b2c53fb872: write tcp 172.27.0.5:50944->172.27.0.4:8080: write: broken pipe
E1102 00:21:59     1 upload_content.go:236] upload  0 bytes to http://172.27.0.4:8080/1,1c371e540130fd: Post http://172.27.0.4:8080/1,1c371e540130fd: write tcp 172.27.0.5:42600->172.27.0.4:8080: write: connection reset by peer
goroutine 113 [running]:
runtime/debug.Stack(0xb6, 0x0, 0x0)
        /usr/lib/go/src/runtime/debug/stack.go:24 +0x9d
runtime/debug.PrintStack()
        /usr/lib/go/src/runtime/debug/stack.go:16 +0x22
github.com/chrislusf/seaweedfs/weed/operation.upload_content(0xc005be84e0, 0x27, 0xc000877ae0, 0x0, 0x0, 0x100, 0x0, 0x0, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:237 +0xc5d
github.com/chrislusf/seaweedfs/weed/operation.doUploadData(0xc005be84e0, 0x27, 0x0, 0x0, 0xc005be8400, 0xc013026000, 0x146e22, 0x32adb6, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:172 +0x4a1
github.com/chrislusf/seaweedfs/weed/operation.retriedUploadData(0xc005be84e0, 0x27, 0x0, 0x0, 0x0, 0xc013026000, 0x146e22, 0x32adb6, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:99 +0x1ba
github.com/chrislusf/seaweedfs/weed/operation.UploadData(...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:69
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).assignAndUpload(0xc00002a5a0, 0xc013026000, 0x146e22, 0x32adb6, 0xc0008d3d40, 0x4ef241, 0x3676840, 0x205da20)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify_append.go:67 +0x2a2
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).appendToFile(0xc00002a5a0, 0xc005be84b0, 0x2c, 0xc013026000, 0x146e22, 0x32adb6, 0x2c, 0x132)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify_append.go:16 +0x67
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).logFlushFunc(0xc00002a5a0, 0x155f485e, 0xed731480b, 0x3726da0, 0x2d3a2cc4, 0xed7314821, 0x3726da0, 0xc013026000, 0x146e22, 0x32adb6)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify.go:98 +0x42f
github.com/chrislusf/seaweedfs/weed/util/log_buffer.(*LogBuffer).loopFlush(0xc0005180d0)
        /go/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:130 +0xfe
created by github.com/chrislusf/seaweedfs/weed/util/log_buffer.NewLogBuffer
        /go/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:51 +0x187
W1102 00:21:59     1 upload_content.go:103] uploading to http://172.27.0.4:8080/1,1c371e540130fd: upload  0 bytes to http://172.27.0.4:8080/1,1c371e540130fd: Post http://172.27.0.4:8080/1,1c371e540130fd: write tcp 172.27.0.5:42600->172.27.0.4:8080: write: connection reset by peer
E1102 01:55:59     1 upload_content.go:236] upload  0 bytes to http://172.27.0.4:8080/1,1eba691e7bf8f6: Post http://172.27.0.4:8080/1,1eba691e7bf8f6: http: server closed idle connection
goroutine 113 [running]:
runtime/debug.Stack(0x8d, 0x0, 0x0)
        /usr/lib/go/src/runtime/debug/stack.go:24 +0x9d
runtime/debug.PrintStack()
        /usr/lib/go/src/runtime/debug/stack.go:16 +0x22
github.com/chrislusf/seaweedfs/weed/operation.upload_content(0xc00451d680, 0x27, 0xc0040b1ae0, 0x0, 0x0, 0x100, 0x0, 0x0, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:237 +0xc5d
github.com/chrislusf/seaweedfs/weed/operation.doUploadData(0xc00451d680, 0x27, 0x0, 0x0, 0xc00451d600, 0xc004822000, 0x6f442, 0x6fdd2, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:172 +0x4a1
github.com/chrislusf/seaweedfs/weed/operation.retriedUploadData(0xc00451d680, 0x27, 0x0, 0x0, 0x0, 0xc004822000, 0x6f442, 0x6fdd2, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:99 +0x1ba
github.com/chrislusf/seaweedfs/weed/operation.UploadData(...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:69
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).assignAndUpload(0xc00002a5a0, 0xc004822000, 0x6f442, 0x6fdd2, 0xc00018cd40, 0x4ef241, 0x3676840, 0x205da20)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify_append.go:67 +0x2a2
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).appendToFile(0xc00002a5a0, 0xc00451d650, 0x2c, 0xc004822000, 0x6f442, 0x6fdd2, 0x2c, 0x132)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify_append.go:16 +0x67
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).logFlushFunc(0xc00002a5a0, 0xc04b6b7, 0xed7315e2a, 0x3726da0, 0xffc84a3, 0xed7315e2f, 0x3726da0, 0xc004822000, 0x6f442, 0x6fdd2)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify.go:98 +0x42f
github.com/chrislusf/seaweedfs/weed/util/log_buffer.(*LogBuffer).loopFlush(0xc0005180d0)
        /go/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:130 +0xfe
created by github.com/chrislusf/seaweedfs/weed/util/log_buffer.NewLogBuffer
        /go/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:51 +0x187
W1102 01:55:59     1 upload_content.go:103] uploading to http://172.27.0.4:8080/1,1eba691e7bf8f6: upload  0 bytes to http://172.27.0.4:8080/1,1eba691e7bf8f6: Post http://172.27.0.4:8080/1,1eba691e7bf8f6: http: server closed idle connection
E1102 02:57:59     1 upload_content.go:236] upload  0 bytes to http://172.27.0.4:8080/1,20cbfcfba69e8d: Post http://172.27.0.4:8080/1,20cbfcfba69e8d: write tcp 172.27.0.5:40462->172.27.0.4:8080: write: broken pipe
goroutine 113 [running]:
runtime/debug.Stack(0xa9, 0x0, 0x0)
        /usr/lib/go/src/runtime/debug/stack.go:24 +0x9d
runtime/debug.PrintStack()
        /usr/lib/go/src/runtime/debug/stack.go:16 +0x22
github.com/chrislusf/seaweedfs/weed/operation.upload_content(0xc002c9ac30, 0x27, 0xc000015ae0, 0x0, 0x0, 0x100, 0x0, 0x0, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:237 +0xc5d
github.com/chrislusf/seaweedfs/weed/operation.doUploadData(0xc002c9ac30, 0x27, 0x0, 0x0, 0xc002c9ac00, 0xc012a6e000, 0x1a5dcd, 0x1a67e5, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:172 +0x4a1
github.com/chrislusf/seaweedfs/weed/operation.retriedUploadData(0xc002c9ac30, 0x27, 0x0, 0x0, 0x0, 0xc012a6e000, 0x1a5dcd, 0x1a67e5, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:99 +0x1ba
github.com/chrislusf/seaweedfs/weed/operation.UploadData(...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:69
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).assignAndUpload(0xc00002a5a0, 0xc012a6e000, 0x1a5dcd, 0x1a67e5, 0xc0008ced40, 0x4ef241, 0x3676840, 0x205da20)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify_append.go:67 +0x2a2
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).appendToFile(0xc00002a5a0, 0xc000276630, 0x2c, 0xc012a6e000, 0x1a5dcd, 0x1a67e5, 0x2c, 0x132)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify_append.go:16 +0x67
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).logFlushFunc(0xc00002a5a0, 0x1a471c9d, 0xed7316c83, 0x3726da0, 0x67c1751, 0xed7316cab, 0x3726da0, 0xc012a6e000, 0x1a5dcd, 0x1a67e5)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify.go:98 +0x42f
github.com/chrislusf/seaweedfs/weed/util/log_buffer.(*LogBuffer).loopFlush(0xc0005180d0)
        /go/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:130 +0xfe
created by github.com/chrislusf/seaweedfs/weed/util/log_buffer.NewLogBuffer
        /go/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:51 +0x187
W1102 02:57:59     1 upload_content.go:103] uploading to http://172.27.0.4:8080/1,20cbfcfba69e8d: upload  0 bytes to http://172.27.0.4:8080/1,20cbfcfba69e8d: Post http://172.27.0.4:8080/1,20cbfcfba69e8d: write tcp 172.27.0.5:40462->172.27.0.4:8080: write: broken pipe
E1102 03:00:59     1 upload_content.go:236] upload  0 bytes to http://172.27.0.4:8080/1,20dc72da815dae: Post http://172.27.0.4:8080/1,20dc72da815dae: write tcp 172.27.0.5:48444->172.27.0.4:8080: write: connection reset by peer
goroutine 113 [running]:
runtime/debug.Stack(0xb6, 0x0, 0x0)
        /usr/lib/go/src/runtime/debug/stack.go:24 +0x9d
runtime/debug.PrintStack()
        /usr/lib/go/src/runtime/debug/stack.go:16 +0x22
github.com/chrislusf/seaweedfs/weed/operation.upload_content(0xc005c5f8c0, 0x27, 0xc00a921ae0, 0x0, 0x0, 0x100, 0x0, 0x0, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:237 +0xc5d
github.com/chrislusf/seaweedfs/weed/operation.doUploadData(0xc005c5f8c0, 0x27, 0x0, 0x0, 0xc005c5f800, 0xc012d32000, 0x386edc, 0x387460, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:172 +0x4a1
github.com/chrislusf/seaweedfs/weed/operation.retriedUploadData(0xc005c5f8c0, 0x27, 0x0, 0x0, 0x0, 0xc012d32000, 0x386edc, 0x387460, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:99 +0x1ba
github.com/chrislusf/seaweedfs/weed/operation.UploadData(...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:69
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).assignAndUpload(0xc00002a5a0, 0xc012d32000, 0x386edc, 0x387460, 0xc0009d3d40, 0x4ef241, 0x3676840, 0x205da20)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify_append.go:67 +0x2a2
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).appendToFile(0xc00002a5a0, 0xc005afdce0, 0x2c, 0xc012d32000, 0x386edc, 0x387460, 0x2c, 0x132)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify_append.go:16 +0x67
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).logFlushFunc(0xc00002a5a0, 0x2c9f954b, 0xed7316d2f, 0x3726da0, 0x2be777b1, 0xed7316d6b, 0x3726da0, 0xc012d32000, 0x386edc, 0x387460)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify.go:98 +0x42f
github.com/chrislusf/seaweedfs/weed/util/log_buffer.(*LogBuffer).loopFlush(0xc0005180d0)
        /go/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:130 +0xfe
created by github.com/chrislusf/seaweedfs/weed/util/log_buffer.NewLogBuffer
        /go/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:51 +0x187
W1102 03:00:59     1 upload_content.go:103] uploading to http://172.27.0.4:8080/1,20dc72da815dae: upload  0 bytes to http://172.27.0.4:8080/1,20dc72da815dae: Post http://172.27.0.4:8080/1,20dc72da815dae: write tcp 172.27.0.5:48444->172.27.0.4:8080: write: connection reset by peer
E1102 03:46:59     1 upload_content.go:236] upload  0 bytes to http://172.27.0.4:8080/1,226e6b2303ec0f: Post http://172.27.0.4:8080/1,226e6b2303ec0f: write tcp 172.27.0.5:49660->172.27.0.4:8080: write: connection reset by peer
goroutine 113 [running]:
runtime/debug.Stack(0xb6, 0x0, 0x0)
        /usr/lib/go/src/runtime/debug/stack.go:24 +0x9d
runtime/debug.PrintStack()
        /usr/lib/go/src/runtime/debug/stack.go:16 +0x22
github.com/chrislusf/seaweedfs/weed/operation.upload_content(0xc00900ecf0, 0x27, 0xc0007efae0, 0x0, 0x0, 0x100, 0x0, 0x0, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:237 +0xc5d
github.com/chrislusf/seaweedfs/weed/operation.doUploadData(0xc00900ecf0, 0x27, 0x0, 0x0, 0xc00900ec00, 0xc010a50000, 0xe308a, 0xe55ea, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:172 +0x4a1
github.com/chrislusf/seaweedfs/weed/operation.retriedUploadData(0xc00900ecf0, 0x27, 0x0, 0x0, 0x0, 0xc010a50000, 0xe308a, 0xe55ea, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:99 +0x1ba
github.com/chrislusf/seaweedfs/weed/operation.UploadData(...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:69
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).assignAndUpload(0xc00002a5a0, 0xc010a50000, 0xe308a, 0xe55ea, 0xc0007efd40, 0x4ef241, 0x3676840, 0x205da20)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify_append.go:67 +0x2a2
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).appendToFile(0xc00002a5a0, 0xc00900ecc0, 0x2c, 0xc010a50000, 0xe308a, 0xe55ea, 0x2c, 0x132)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify_append.go:16 +0x67
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).logFlushFunc(0xc00002a5a0, 0x3589141f, 0xed73177fd, 0x3726da0, 0x2f8d38a8, 0xed7317833, 0x3726da0, 0xc010a50000, 0xe308a, 0xe55ea)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify.go:98 +0x42f
github.com/chrislusf/seaweedfs/weed/util/log_buffer.(*LogBuffer).loopFlush(0xc0005180d0)
        /go/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:130 +0xfe
created by github.com/chrislusf/seaweedfs/weed/util/log_buffer.NewLogBuffer
        /go/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:51 +0x187
W1102 03:46:59     1 upload_content.go:103] uploading to http://172.27.0.4:8080/1,226e6b2303ec0f: upload  0 bytes to http://172.27.0.4:8080/1,226e6b2303ec0f: Post http://172.27.0.4:8080/1,226e6b2303ec0f: write tcp 172.27.0.5:49660->172.27.0.4:8080: write: connection reset by peer
E1102 17:48:01     1 upload_content.go:236] upload  0 bytes to http://172.27.0.4:8080/1,47066121334f32: Post http://172.27.0.4:8080/1,47066121334f32: write tcp 172.27.0.5:42508->172.27.0.4:8080: write: connection reset by peer
goroutine 113 [running]:
runtime/debug.Stack(0xb6, 0x0, 0x0)
        /usr/lib/go/src/runtime/debug/stack.go:24 +0x9d
runtime/debug.PrintStack()
        /usr/lib/go/src/runtime/debug/stack.go:16 +0x22
github.com/chrislusf/seaweedfs/weed/operation.upload_content(0xc00b3e3b30, 0x27, 0xc0007efae0, 0x0, 0x0, 0x100, 0x0, 0x0, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:237 +0xc5d
github.com/chrislusf/seaweedfs/weed/operation.doUploadData(0xc00b3e3b30, 0x27, 0x0, 0x0, 0xc00b3e3b00, 0xc017a70000, 0xddf20, 0xdea86, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:172 +0x4a1
github.com/chrislusf/seaweedfs/weed/operation.retriedUploadData(0xc00b3e3b30, 0x27, 0x0, 0x0, 0x0, 0xc017a70000, 0xddf20, 0xdea86, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:99 +0x1ba
github.com/chrislusf/seaweedfs/weed/operation.UploadData(...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:69
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).assignAndUpload(0xc00002a5a0, 0xc017a70000, 0xddf20, 0xdea86, 0xc00f7c0d40, 0x4ef241, 0x3676840, 0x205da20)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify_append.go:67 +0x2a2
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).appendToFile(0xc00002a5a0, 0xc00b3e3b00, 0x2c, 0xc017a70000, 0xddf20, 0xdea86, 0x2c, 0x132)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify_append.go:16 +0x67
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).logFlushFunc(0xc00002a5a0, 0xd848203, 0xed7323d1f, 0x3726da0, 0x24334d58, 0xed7323d51, 0x3726da0, 0xc017a70000, 0xddf20, 0xdea86)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify.go:98 +0x42f
github.com/chrislusf/seaweedfs/weed/util/log_buffer.(*LogBuffer).loopFlush(0xc0005180d0)
        /go/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:130 +0xfe
created by github.com/chrislusf/seaweedfs/weed/util/log_buffer.NewLogBuffer
        /go/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:51 +0x187
W1102 17:48:01     1 upload_content.go:103] uploading to http://172.27.0.4:8080/1,47066121334f32: upload  0 bytes to http://172.27.0.4:8080/1,47066121334f32: Post http://172.27.0.4:8080/1,47066121334f32: write tcp 172.27.0.5:42508->172.27.0.4:8080: write: connection reset by peer
E1102 19:34:01     1 upload_content.go:236] upload  0 bytes to http://172.27.0.4:8080/1,4903869fb7f897: Post http://172.27.0.4:8080/1,4903869fb7f897: write tcp 172.27.0.5:55506->172.27.0.4:8080: write: broken pipe
goroutine 113 [running]:
runtime/debug.Stack(0xa9, 0x0, 0x0)
        /usr/lib/go/src/runtime/debug/stack.go:24 +0x9d
runtime/debug.PrintStack()
        /usr/lib/go/src/runtime/debug/stack.go:16 +0x22
github.com/chrislusf/seaweedfs/weed/operation.upload_content(0xc002c9a120, 0x27, 0xc000803ae0, 0x0, 0x0, 0x100, 0x0, 0x0, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:237 +0xc5d
github.com/chrislusf/seaweedfs/weed/operation.doUploadData(0xc002c9a120, 0x27, 0x0, 0x0, 0xc002c9a100, 0xc0157c8000, 0x95c2a, 0xb7b92, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:172 +0x4a1
github.com/chrislusf/seaweedfs/weed/operation.retriedUploadData(0xc002c9a120, 0x27, 0x0, 0x0, 0x0, 0xc0157c8000, 0x95c2a, 0xb7b92, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:99 +0x1ba
github.com/chrislusf/seaweedfs/weed/operation.UploadData(...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:69
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).assignAndUpload(0xc00002a5a0, 0xc0157c8000, 0x95c2a, 0xb7b92, 0xc00f7c2d40, 0x4ef241, 0x3676840, 0x205da20)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify_append.go:67 +0x2a2
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).appendToFile(0xc00002a5a0, 0xc002c9a0f0, 0x2c, 0xc0157c8000, 0x95c2a, 0xb7b92, 0x2c, 0x132)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify_append.go:16 +0x67
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).logFlushFunc(0xc00002a5a0, 0x1a9cb1bf, 0xed73255f9, 0x3726da0, 0x29b3efe8, 0xed7325628, 0x3726da0, 0xc0157c8000, 0x95c2a, 0xb7b92)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify.go:98 +0x42f
github.com/chrislusf/seaweedfs/weed/util/log_buffer.(*LogBuffer).loopFlush(0xc0005180d0)
        /go/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:130 +0xfe
created by github.com/chrislusf/seaweedfs/weed/util/log_buffer.NewLogBuffer
        /go/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:51 +0x187
W1102 19:34:01     1 upload_content.go:103] uploading to http://172.27.0.4:8080/1,4903869fb7f897: upload  0 bytes to http://172.27.0.4:8080/1,4903869fb7f897: Post http://172.27.0.4:8080/1,4903869fb7f897: write tcp 172.27.0.5:55506->172.27.0.4:8080: write: broken pipe
E1102 20:01:01     1 upload_content.go:236] upload  0 bytes to http://172.27.0.4:8080/1,49775858919398: Post http://172.27.0.4:8080/1,49775858919398: write tcp 172.27.0.5:51800->172.27.0.4:8080: write: connection reset by peer
goroutine 113 [running]:
runtime/debug.Stack(0xb6, 0x0, 0x0)
        /usr/lib/go/src/runtime/debug/stack.go:24 +0x9d
runtime/debug.PrintStack()
        /usr/lib/go/src/runtime/debug/stack.go:16 +0x22
github.com/chrislusf/seaweedfs/weed/operation.upload_content(0xc0077e5f20, 0x27, 0xc000017ae0, 0x0, 0x0, 0x100, 0x0, 0x0, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:237 +0xc5d
github.com/chrislusf/seaweedfs/weed/operation.doUploadData(0xc0077e5f20, 0x27, 0x0, 0x0, 0xc0077e5f00, 0xc0173b6000, 0x975b4, 0x97c28, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:172 +0x4a1
github.com/chrislusf/seaweedfs/weed/operation.retriedUploadData(0xc0077e5f20, 0x27, 0x0, 0x0, 0x0, 0xc0173b6000, 0x975b4, 0x97c28, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:99 +0x1ba
github.com/chrislusf/seaweedfs/weed/operation.UploadData(...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:69
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).assignAndUpload(0xc00002a5a0, 0xc0173b6000, 0x975b4, 0x97c28, 0xc000017d40, 0x4ef241, 0x3676840, 0x205da20)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify_append.go:67 +0x2a2
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).appendToFile(0xc00002a5a0, 0xc00791e4b0, 0x2c, 0xc0173b6000, 0x975b4, 0x97c28, 0x2c, 0x132)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify_append.go:16 +0x67
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).logFlushFunc(0xc00002a5a0, 0x2a7aeede, 0xed7325c45, 0x3726da0, 0x283707ab, 0xed7325c7c, 0x3726da0, 0xc0173b6000, 0x975b4, 0x97c28)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify.go:98 +0x42f
github.com/chrislusf/seaweedfs/weed/util/log_buffer.(*LogBuffer).loopFlush(0xc0005180d0)
        /go/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:130 +0xfe
created by github.com/chrislusf/seaweedfs/weed/util/log_buffer.NewLogBuffer
        /go/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:51 +0x187
W1102 20:01:01     1 upload_content.go:103] uploading to http://172.27.0.4:8080/1,49775858919398: upload  0 bytes to http://172.27.0.4:8080/1,49775858919398: Post http://172.27.0.4:8080/1,49775858919398: write tcp 172.27.0.5:51800->172.27.0.4:8080: write: connection reset by peer
E1102 20:05:01     1 upload_content.go:236] upload  0 bytes to http://172.27.0.4:8080/1,497a6536fea2f3: Post http://172.27.0.4:8080/1,497a6536fea2f3: write tcp 172.27.0.5:39490->172.27.0.4:8080: write: broken pipe
goroutine 113 [running]:
runtime/debug.Stack(0xa9, 0x0, 0x0)
        /usr/lib/go/src/runtime/debug/stack.go:24 +0x9d
runtime/debug.PrintStack()
        /usr/lib/go/src/runtime/debug/stack.go:16 +0x22
github.com/chrislusf/seaweedfs/weed/operation.upload_content(0xc008615b00, 0x27, 0xc0007ebae0, 0x0, 0x0, 0x100, 0x0, 0x0, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:237 +0xc5d
github.com/chrislusf/seaweedfs/weed/operation.doUploadData(0xc008615b00, 0x27, 0x0, 0x0, 0xc008615b00, 0xc018e78000, 0xacdfd, 0xb1ed3, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:172 +0x4a1
github.com/chrislusf/seaweedfs/weed/operation.retriedUploadData(0xc008615b00, 0x27, 0x0, 0x0, 0x0, 0xc018e78000, 0xacdfd, 0xb1ed3, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:99 +0x1ba
github.com/chrislusf/seaweedfs/weed/operation.UploadData(...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:69
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).assignAndUpload(0xc00002a5a0, 0xc018e78000, 0xacdfd, 0xb1ed3, 0xc0007ebd40, 0x4ef241, 0x3676840, 0x205da20)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify_append.go:67 +0x2a2
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).appendToFile(0xc00002a5a0, 0xc008757e30, 0x2c, 0xc018e78000, 0xacdfd, 0xb1ed3, 0x2c, 0x132)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify_append.go:16 +0x67
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).logFlushFunc(0xc00002a5a0, 0x27a833b7, 0xed7325d4b, 0x3726da0, 0x20c932d0, 0xed7325d67, 0x3726da0, 0xc018e78000, 0xacdfd, 0xb1ed3)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify.go:98 +0x42f
github.com/chrislusf/seaweedfs/weed/util/log_buffer.(*LogBuffer).loopFlush(0xc0005180d0)
        /go/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:130 +0xfe
created by github.com/chrislusf/seaweedfs/weed/util/log_buffer.NewLogBuffer
        /go/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:51 +0x187
W1102 20:05:01     1 upload_content.go:103] uploading to http://172.27.0.4:8080/1,497a6536fea2f3: upload  0 bytes to http://172.27.0.4:8080/1,497a6536fea2f3: Post http://172.27.0.4:8080/1,497a6536fea2f3: write tcp 172.27.0.5:39490->172.27.0.4:8080: write: broken pipe
E1102 23:43:01     1 upload_content.go:236] upload  0 bytes to http://172.27.0.4:8080/1,4fdc0119fcb657: Post http://172.27.0.4:8080/1,4fdc0119fcb657: write tcp 172.27.0.5:47098->172.27.0.4:8080: use of closed network connection
goroutine 113 [running]:
runtime/debug.Stack(0xb7, 0x0, 0x0)
        /usr/lib/go/src/runtime/debug/stack.go:24 +0x9d
runtime/debug.PrintStack()
        /usr/lib/go/src/runtime/debug/stack.go:16 +0x22
github.com/chrislusf/seaweedfs/weed/operation.upload_content(0xc018fdb770, 0x27, 0xc016ca9ae0, 0x0, 0x0, 0x100, 0x0, 0x0, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:237 +0xc5d
github.com/chrislusf/seaweedfs/weed/operation.doUploadData(0xc018fdb770, 0x27, 0x0, 0x0, 0xc018fdb700, 0xc01e2f0000, 0x2034f0, 0x207568, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:172 +0x4a1
github.com/chrislusf/seaweedfs/weed/operation.retriedUploadData(0xc018fdb770, 0x27, 0x0, 0x0, 0x0, 0xc01e2f0000, 0x2034f0, 0x207568, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:99 +0x1ba
github.com/chrislusf/seaweedfs/weed/operation.UploadData(...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:69
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).assignAndUpload(0xc00002a5a0, 0xc01e2f0000, 0x2034f0, 0x207568, 0xc0179f4d40, 0x4ef241, 0x3676840, 0x205da20)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify_append.go:67 +0x2a2
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).appendToFile(0xc00002a5a0, 0xc018fdb740, 0x2c, 0xc01e2f0000, 0x2034f0, 0x207568, 0x2c, 0x132)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify_append.go:16 +0x67
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).logFlushFunc(0xc00002a5a0, 0x1737ceff, 0xed7329068, 0x3726da0, 0x19d35200, 0xed7329084, 0x3726da0, 0xc01e2f0000, 0x2034f0, 0x207568)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify.go:98 +0x42f
github.com/chrislusf/seaweedfs/weed/util/log_buffer.(*LogBuffer).loopFlush(0xc0005180d0)
        /go/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:130 +0xfe
created by github.com/chrislusf/seaweedfs/weed/util/log_buffer.NewLogBuffer
        /go/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:51 +0x187
W1102 23:43:01     1 upload_content.go:103] uploading to http://172.27.0.4:8080/1,4fdc0119fcb657: upload  0 bytes to http://172.27.0.4:8080/1,4fdc0119fcb657: Post http://172.27.0.4:8080/1,4fdc0119fcb657: write tcp 172.27.0.5:47098->172.27.0.4:8080: use of closed network connection
E1102 23:59:01     1 upload_content.go:236] upload  0 bytes to http://172.27.0.4:8080/1,501b64466a64cf: Post http://172.27.0.4:8080/1,501b64466a64cf: write tcp 172.27.0.5:60730->172.27.0.4:8080: write: connection reset by peer
goroutine 113 [running]:
runtime/debug.Stack(0xb6, 0x0, 0x0)
        /usr/lib/go/src/runtime/debug/stack.go:24 +0x9d
runtime/debug.PrintStack()
        /usr/lib/go/src/runtime/debug/stack.go:16 +0x22
github.com/chrislusf/seaweedfs/weed/operation.upload_content(0xc00eadf8c0, 0x27, 0xc0007efae0, 0x0, 0x0, 0x100, 0x0, 0x0, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:237 +0xc5d
github.com/chrislusf/seaweedfs/weed/operation.doUploadData(0xc00eadf8c0, 0x27, 0x0, 0x0, 0xc00eadf800, 0xc01dc86000, 0x1188ba, 0x1192d8, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:172 +0x4a1
github.com/chrislusf/seaweedfs/weed/operation.retriedUploadData(0xc00eadf8c0, 0x27, 0x0, 0x0, 0x0, 0xc01dc86000, 0x1188ba, 0x1192d8, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:99 +0x1ba
github.com/chrislusf/seaweedfs/weed/operation.UploadData(...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:69
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).assignAndUpload(0xc00002a5a0, 0xc01dc86000, 0x1188ba, 0x1192d8, 0xc0007efd40, 0x4ef241, 0x3676840, 0x205da20)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify_append.go:67 +0x2a2
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).appendToFile(0xc00002a5a0, 0xc00eb150b0, 0x2c, 0xc01dc86000, 0x1188ba, 0x1192d8, 0x2c, 0x132)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify_append.go:16 +0x67
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).logFlushFunc(0xc00002a5a0, 0x103b7b52, 0xed7329419, 0x3726da0, 0x38f703c8, 0xed7329445, 0x3726da0, 0xc01dc86000, 0x1188ba, 0x1192d8)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify.go:98 +0x42f
github.com/chrislusf/seaweedfs/weed/util/log_buffer.(*LogBuffer).loopFlush(0xc0005180d0)
        /go/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:130 +0xfe
created by github.com/chrislusf/seaweedfs/weed/util/log_buffer.NewLogBuffer
        /go/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:51 +0x187
W1102 23:59:01     1 upload_content.go:103] uploading to http://172.27.0.4:8080/1,501b64466a64cf: upload  0 bytes to http://172.27.0.4:8080/1,501b64466a64cf: Post http://172.27.0.4:8080/1,501b64466a64cf: write tcp 172.27.0.5:60730->172.27.0.4:8080: write: connection reset by peer
E1103 01:21:02     1 upload_content.go:236] upload  0 bytes to http://172.27.0.4:8080/1,51986652623882: Post http://172.27.0.4:8080/1,51986652623882: write tcp 172.27.0.5:39628->172.27.0.4:8080: write: broken pipe
goroutine 113 [running]:
runtime/debug.Stack(0xa9, 0x0, 0x0)
        /usr/lib/go/src/runtime/debug/stack.go:24 +0x9d
runtime/debug.PrintStack()
        /usr/lib/go/src/runtime/debug/stack.go:16 +0x22
github.com/chrislusf/seaweedfs/weed/operation.upload_content(0xc00cf10a80, 0x27, 0xc00a925ae0, 0x0, 0x0, 0x100, 0x0, 0x0, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:237 +0xc5d
github.com/chrislusf/seaweedfs/weed/operation.doUploadData(0xc00cf10a80, 0x27, 0x0, 0x0, 0xc00cf10a00, 0xc01e738000, 0xf6f86, 0x2d4c74, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:172 +0x4a1
github.com/chrislusf/seaweedfs/weed/operation.retriedUploadData(0xc00cf10a80, 0x27, 0x0, 0x0, 0x0, 0xc01e738000, 0xf6f86, 0x2d4c74, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:99 +0x1ba
github.com/chrislusf/seaweedfs/weed/operation.UploadData(...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:69
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).assignAndUpload(0xc00002a5a0, 0xc01e738000, 0xf6f86, 0x2d4c74, 0xc00a925d40, 0x4ef241, 0x3676840, 0x205da20)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify_append.go:67 +0x2a2
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).appendToFile(0xc00002a5a0, 0xc00cf10a50, 0x2c, 0xc01e738000, 0xf6f86, 0x2d4c74, 0x2c, 0x133)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify_append.go:16 +0x67
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).logFlushFunc(0xc00002a5a0, 0x22d1dfb3, 0xed732a76a, 0x3726da0, 0xea47ac, 0xed732a77e, 0x3726da0, 0xc01e738000, 0xf6f86, 0x2d4c74)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify.go:98 +0x42f
github.com/chrislusf/seaweedfs/weed/util/log_buffer.(*LogBuffer).loopFlush(0xc0005180d0)
        /go/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:130 +0xfe
created by github.com/chrislusf/seaweedfs/weed/util/log_buffer.NewLogBuffer
        /go/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:51 +0x187
W1103 01:21:02     1 upload_content.go:103] uploading to http://172.27.0.4:8080/1,51986652623882: upload  0 bytes to http://172.27.0.4:8080/1,51986652623882: Post http://172.27.0.4:8080/1,51986652623882: write tcp 172.27.0.5:39628->172.27.0.4:8080: write: broken pipe
E1103 03:08:02     1 upload_content.go:236] upload  0 bytes to http://172.27.0.4:8080/1,547fd1fcc451ae: Post http://172.27.0.4:8080/1,547fd1fcc451ae: EOF
goroutine 113 [running]:
runtime/debug.Stack(0x6d, 0x0, 0x0)
        /usr/lib/go/src/runtime/debug/stack.go:24 +0x9d
runtime/debug.PrintStack()
        /usr/lib/go/src/runtime/debug/stack.go:16 +0x22
github.com/chrislusf/seaweedfs/weed/operation.upload_content(0xc01267aa80, 0x27, 0xc0017f7ae0, 0x0, 0x0, 0x100, 0x0, 0x0, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:237 +0xc5d
github.com/chrislusf/seaweedfs/weed/operation.doUploadData(0xc01267aa80, 0x27, 0x0, 0x0, 0xc01267aa00, 0xc01f13e000, 0x24a3f, 0x12bdfc, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:172 +0x4a1
github.com/chrislusf/seaweedfs/weed/operation.retriedUploadData(0xc01267aa80, 0x27, 0x0, 0x0, 0x0, 0xc01f13e000, 0x24a3f, 0x12bdfc, 0x0, 0x0, ...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:99 +0x1ba
github.com/chrislusf/seaweedfs/weed/operation.UploadData(...)
        /go/src/github.com/chrislusf/seaweedfs/weed/operation/upload_content.go:69
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).assignAndUpload(0xc00002a5a0, 0xc01f13e000, 0x24a3f, 0x12bdfc, 0xc0017f7d40, 0x4ef241, 0x3676840, 0x205da20)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify_append.go:67 +0x2a2
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).appendToFile(0xc00002a5a0, 0xc01267aa50, 0x2c, 0xc01f13e000, 0x24a3f, 0x12bdfc, 0x2c, 0x133)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify_append.go:16 +0x67
github.com/chrislusf/seaweedfs/weed/filer.(*Filer).logFlushFunc(0xc00002a5a0, 0x19360ed0, 0xed732c056, 0x3726da0, 0xb6df62e, 0xed732c091, 0x3726da0, 0xc01f13e000, 0x24a3f, 0x12bdfc)
        /go/src/github.com/chrislusf/seaweedfs/weed/filer/filer_notify.go:98 +0x42f
github.com/chrislusf/seaweedfs/weed/util/log_buffer.(*LogBuffer).loopFlush(0xc0005180d0)
        /go/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:130 +0xfe
created by github.com/chrislusf/seaweedfs/weed/util/log_buffer.NewLogBuffer
        /go/src/github.com/chrislusf/seaweedfs/weed/util/log_buffer/log_buffer.go:51 +0x187
W1103 03:08:02     1 upload_content.go:103] uploading to http://172.27.0.4:8080/1,547fd1fcc451ae: upload  0 bytes to http://172.27.0.4:8080/1,547fd1fcc451ae: Post http://172.27.0.4:8080/1,547fd1fcc451ae: EOF

@flowluap
Copy link

flowluap commented Nov 8, 2020

I think i got a similar problem using Docker. Used different files, "server"-command, separated server/master...

needle_parse_upload.go:143] Reading Content [ERROR] unexpected EOF
common.go:51] response method:POST URL:/6,04084bf0ec with httpStatus:400 and JSON:{"error":"unexpected EOF"}

@chrislusf
Copy link
Collaborator

I think i got a similar problem using Docker. Used different files, "server"-command, separated server/master...

needle_parse_upload.go:143] Reading Content [ERROR] unexpected EOF
common.go:51] response method:POST URL:/6,04084bf0ec with httpStatus:400 and JSON:{"error":"unexpected EOF"}

Retry should help.

@flowluap
Copy link

flowluap commented Nov 8, 2020

I think i got a similar problem using Docker. Used different files, "server"-command, separated server/master...

needle_parse_upload.go:143] Reading Content [ERROR] unexpected EOF
common.go:51] response method:POST URL:/6,04084bf0ec with httpStatus:400 and JSON:{"error":"unexpected EOF"}

Retry should help.

I removed the volume, etc. but nothing helped... Retry didn't either.

@chrislusf
Copy link
Collaborator

I removed the volume, etc. but nothing helped... Retry didn't either.

what are you trying to do?

@kmlebedev
Copy link
Contributor Author

kmlebedev commented Nov 11, 2020 •
edited
Loading

Its maybe
s3: do not close reader too early
73f934d

@chrislusf
Copy link
Collaborator

Related to golang/go#19943

Need to add a {X-,}Idempotency-Key header for http POST.

@chrislusf
Copy link
Collaborator

This is related to Go not retrying the POST requests.

Added a fix. Closing it for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@chrislusf @Suika @kmlebedev @flowluap