Copy objects to minio is unusual slow

[rsoutar]

Copy objects to minio is unusual slow

Expected Behavior

Faster?

Current Behavior

./mc cp --recursive folder/ local/bucket/ is running around 4.5 MB/s

dd if=/dev/zero of=bigfile bs=1m count=1024

./mc cp bigfile minio/test is running 128MB/s

Context

The read speed is around 50-70 MB/s
Files under the bucket are something like BucketName/MediaID/Quality/MovieName/Files

Your Environment

Version: 2017-05-05T01:14:51Z
Supermicro SuperStorage 6028R-E1CR12L
CPU : model name : Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
CORE : 40
Memory : 512 GB
Disks : 12 * 4TB
Ubuntu 16.04

[donatello]

@rsoutar

1. What is the average size of the files being copied into your local minio server?
2. Do you have a very large number of files in a single directory?

[harshavardhana]

What is the average size of the files being copied into your local minio server?
Do you have a very large number of files in a single directory?

Actually this was discussed on #general @donatello we saw copy saturates the network link on a large file copy i.e for multipart transfer.

Copy small files sequentially we don't actually saturate the network, there is a mc github issue to fix this and bring back some amount of parallelism to optimally saturate the network link.

[harshavardhana]

@deekoder this is not a minio server bug per se. We have to open this bug on mc and fix it.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.