s3 sync uses high CPU when wrapped with trickle to limit bandwidth

[jkrauska]

I am attempting to use trickle [1] to limit the upload speed an 'aws s3 sync' uses.

My ISP will charge me for overages if I go above 20Mbps, so I need to limit the throughput.

I have had good luck with trickle in the past for simpler things like ftp, but it seems there's something about how mutlipart uploads work that doesn't play nicely? Eventually several threads begin to get stuck at 100% CPU utilization.

How I'm calling aws-cli:
nice trickle -s -u $UPLIMIT aws s3 sync --recursive $EXCLUDES ${BACKUPDIR}/$dirname s3://${S3BUCKET}/$dirname

With an $UPLIMIT of about 2500 (in KB/s) to give under 20Mbps.

Please advise? Perhaps you have a bwlimit feature in your roadmap for the s3 sync?

1. http://monkey.org/~marius/pages/?page=trickle

[jamesls]

I think this is a great feature request. I've filed #1090 to track this. I tried out trickle with the latest version of the CLI, and it appears to work for me. I didn't see any 100% cpu utilization. I'd try again with the latest version of the AWS CLI.

However, long term, I think we're going to just add support directly in the AWS CLI for limiting bandwidth, which we can track over in #1090.

[lkjangir]

Hi Jamesls,
i am using the latest version of aws cli and what i see during the upload that both s3 cp and s3 sync are consuming all CPU and memory. My current server has 16 GB and s3 sync has consumed all 100% CPU and RAM. is there any solution for this, please let me know.

[bobintetley]

I had the same issue. If you set the time-smoothing and length-smoothing to higher values, trickle will use a lot less CPU.

nice trickle -s -t 5 -l 20 -u $UPLIMIT

worked well for me and reduces CPU usage from 100% down to about 0.7%. The default time smoothing is very aggressive at 0.1 seconds and I think to blame for the high CPU usage.

[bobintetley]

Oh, and that doesn't fix the thread sticking issue, to do that use aws configure to limit to a single thread with:

aws configure set default.s3.max_concurrent_requests 1

[xoroz]

Try reducing the max_concurrent_requests
http://docs.aws.amazon.com/cli/latest/topic/s3-config.html