aws s3 sync used SWAP space in memory

[KinWaiCheung]

Describe the bug

Hi,
has anyone face issued with aws s3 sync from local (on prem) to s3bucket and eats up swap memory?

Even though there is enough physical RAM memory left?
Our monitoring doesn't show RAM being used up completely when SWAP usage increases.

aws-cli/2.11.23
vm.swappiness=10

every week I schedule a job to sync backup files from local to s3 bucket and the swap memory is decreased a couple hundred of MB each week.

s3 is configured with 20 concurrent requests
Each file has an avg size of 10GB to 70 GB and total amount that we sync to s3 on is between 3TB-4TB from this source

free -m
total used free shared buff/cache available
Mem: 515215 424992 75334 3634 14888 82955
Swap: 3999 3482 517

Expected Behavior

no swap being used

Current Behavior

swap usage increases every week when we execute sync and doesn't seem to decrease after that

Reproduction Steps

BACKUP_DAY=date +%Y%m%d
(full) backupfiles created with $BACKUP_DAY in filename
and then we sync them to S3 bucket

aws s3 sync $BACKUP_DIR $S3_PATH --exclude '' --include '_'$BACKUP_DAY'*' --only-show-errors >> $LOGFILE

Possible Solution

No response

Additional Information/Context

No response

CLI version used

2.11.23

Environment details (OS name and version, etc.)

OEL7.9

[tim-finnigan]

Hi @KinWaiCheung thanks for reaching out. I found one other issue (#5876) which involved discussion on swap usage which could be worth reviewing. Is adding more swap space an option for you? In addition you might consider looking into optimizations for uploading large files to s3 such as those described here: https://repost.aws/knowledge-center/s3-upload-large-files.

[KinWaiCheung]

Hi @tim-finnigan ,

sure will review the previous mentioned issue.

Concerning adding more swap space, it's currently set to 4GB and we can add more but the problem it's never decreasing after each weekly upload of the full backup large files.
After each upload it's decreasing by around 500 MB. (doing a swapoff and swapon will put everything back in the RAM, and there is enough free, even during the upload)

[tim-finnigan]

Hi @KinWaiCheung thanks for following up. Do you have any updates as far as what you've tried? Were there any optimizations from the article I linked that you were able to implement?

[KinWaiCheung]

swap usage is still seen after updating vm.swappiness from 10 to 1 but it never decreases after the aws s3 sync is done.
In our case the swap usage looks to be less used than before but the main point is that we see swap being used when aws s3 sync is done even though there is enough free ram available.

workaround is doing a swapoff/swapon (if enough physical ram is available/free, which is the case for us)
but over time it looks like it will happen again when the sync is ongoing using bit by bit of the swap instead of ram.

Another optimization that we are using is testing the multipart_chunksize value. This has according to your doc a default size of 8MB -> tested with 50/75/100/150/200/500MB and it looks like 75MB improved the upload time.
(concurrent requests is already optimized to 20 instead of 10)
Not sure yet if this decrease the swap usage or not

[tim-finnigan]

Thanks for following up. I'm going to link the discussion you opened as well: #7936. As mentioned there, we may want to close either the issue or discussion just to consolidate the discussion.

I think some further review and testing is needed here, and maybe others can contribute what they have observed. If multipart_chunksize or other optimizations help the issue please let us know.

[github-actions]

Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.

[KinWaiCheung]

changing multipart_chunksize didn't prevent swap utilization (there is enough physical memory).