High memory usage when copying files to bucket

[OliverDBA]

Hi,

on a Redhat 7.9 server with gcsfuse-0.32.0-1.x86_64.rpm and google-cloud-sdk-319.0.0-1.x86_64.rpm, I experiance very high memory usage of the gcsfuse process, when writing to a mount from two different processes:

• filesize about 1.5 GB

• 3 files written per minute

• gcsfuse memory usage increases to 20GB

• No files written to temporary directory

Mount:

gcsfuse --implicit-dirs -o allow_other --max-retry-sleep 2m0s --log-file /home/user/gcsfuse.log --temp-dir /u02/stage --key-file /home/user/mykey.json mybucket /u02/targetdir

If there are four parallel processes the memory usage increases to more then 30 GB

top - 21:43:56 up 10:54, 3 users, load average: 1.62, 0.66, 0.27
Tasks: 386 total, 1 running, 385 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.1 us, 1.8 sy, 0.0 ni, 77.9 id, 20.1 wa, 0.0 hi, 0.0 si, 0.1 st
KiB Mem : 92381128 total, 449948 free, 90713360 used, 1217820 buff/cache
KiB Swap: 10305528 total, 9543672 free, 761856 used. 940420 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
18669 testuser 20 0 36.5g 27.4g 3912 S 2.7 31.1 33:48.84 gcsfuse

[sand8080]

Hi,

Have the same problem on Alpine Docker golang:1.15.4-alpine3.12 after upgrade on the gcsfue latest version.
Gcsfuse build such way:

FROM golang:1.15.4-alpine3.12 AS builder

# Installing gcsfuse
RUN apk --update add build-base git fuse fuse-dev \
    && go get -u github.com/googlecloudplatform/gcsfuse \
    && cp /go/bin/gcsfuse /bin

Writing files 1.1Gb size to the mounted GCS bucket. Memory consumption on writing 4 files reaches 8-10Gb.
I've tried to change --stat-cache-capacity, --limit-ops-per-sec, --stat-cache-ttl, --type-cache-ttl, --temp-dir with no effect on memory consumption.

I have the old one version of gcsfuse. Unfortunatelly can't say the exact version, but can suggest it is v0.28.1. This version works without memory consumption.

[OliverDBA]

I can't downgrade as I do use CMEK encrypted buckets and this doesn't work with 0.28
We did some tests with 0.28 on Google encrypted buckets some months ago and there we also didn't face memory leaks

[OliverDBA]

Increased memory consumption starts with release 0.31. , I did some tests today.

[VengefulAncient]

Not sure if this is related, but it might be. Recently, a memory leak similar to the one in #204 has appeared on our backend using gcsfuse. We're not uploading anything big, just an occasional image, but it crawls steadily over a few days. This was also not happening with previous versions of gcsfuse, it ran fine for months.

[kovan]

I am also experiencing this. After 20 seconds gcsfuse is using 40% of the RAM (of 2GB total), and after a minute or so it gets killed by OOM killer.

FYI am running it with gcsfuse --disable-http2 --debug_gcs --foreground mybucket /mnt/bucket/, and in the output nothing is shown, only "Killed".

Using version 0.33.1 (Go version go1.15.7)

[kovan]

I have downgraded to 0.32.0 and it is better but still leaks and is killed by OOM killer.
Now I am using 0.30.0 and it is working very well, it doesn't leak at all.
I didn't test 0.31, because it is not in the apt repo.

[ToxicWar]

I tested two scenarios:

1. Copy files into container mounted dir (kubectl cp ...)
2. Copy files in mounted dir between subdirs

0.33.1 has issues with memory (with CMEK support)

0.28.1 works good (without CMEK support)

[lezh]

This is not a memory leak.

The memory spikes are the consequence of the files written to GCS being loaded entirely into memory. This is an implementation in the retry library (a part of the dependencies) to avoid file corruption.

Since the retry was introduced in #414 and released in 0.31 or later, these memory spikes were not seen before 0.30.

While a fix is planned, please specify

--max-retry-sleep=0

to avoid such issue now if you are using 0.31 or later.

[hayorov]

as @lezh mentioned it's not a memory leak, it's a "buffer/cache" in memory between sync/upload retries. (*retryBucket) createObject -> ... -> ioutil ReadAll -> bytes makeSlice

This is how memory usage looks like

With --max-retry-sleep=0 there is no buffer/cache

[gaul]

Duplicate of #505. Please test the proposed PR referenced there.