Memory grows constantly on Docker #198
Comments
|
What version are you using? I'm not seeing similar trends with 1.0.15 |
|
1.0.15 +1 |
|
Hello, same problem here. How to reproduce
Output: Health api Output: ENV OS Docker Engine - Community Imaginary |
|
I can confirm that My test setup: $ docker stats$ docker run --rm -it -p 9000:9000 h2non/imaginary -enable-url-source -concurrency 20$ wrk -t 2 -c 2 -d 60m "http://localhost:9000/resize?height=50&weight=50&url=.."
...
Requests/sec: 416.17
Transfer/sec: 163.60KB$ watch -n1 "curl -sq http://127.0.0.1:9000/health" |
|
I don't know if this matter, but my instance of Imaginary is actually working with POST API: The divergence between Docker stats and the health status api is still present in my installation:
|
|
I've performed a very rudimentary analysis, but I'm not detecting any leakage so far. The test took nearly 45 minutes and ever second I've taken a sample from the SetupVersion detailsDocker ImaginaryStarted imaginary with a low concurrency and a high burst to not engage the throttler (HTTP 429's). $ ./imaginary -concurrency 2 -burst 10000000000PayloadThe payload sent during these tests was along the lines of: $ curl -v -s -XPOST "http://localhost:9000/resize?height=50&width=50&type=jpeg" -T ./img.png 1>/dev/nullThe patchdiff --git health.go health.go
index c6dc53c..b3d9508 100644
--- health.go
+++ health.go
@@ -15,7 +15,12 @@ type HealthStats struct {
AllocatedMemory float64 `json:"allocatedMemory"`
TotalAllocatedMemory float64 `json:"totalAllocatedMemory"`
Goroutines int `json:"goroutines"`
+ GCCycles uint32 `json:"completedGCCycles"`
NumberOfCPUs int `json:"cpus"`
+ HeapSys float64 `json:"maxHeapUsage"`
+ HeapAllocated float64 `json:"heapInUse"`
+ ObjectsInUse uint64 `json:"objectsInUse"`
+ OSMemoryObtained float64 `json:"OSMemoryObtained"`
}
func GetHealthStats() *HealthStats {
@@ -28,6 +33,11 @@ func GetHealthStats() *HealthStats {
TotalAllocatedMemory: toMegaBytes(mem.TotalAlloc),
Goroutines: runtime.NumGoroutine(),
NumberOfCPUs: runtime.NumCPU(),
+ GCCycles: mem.NumGC,
+ HeapSys: toMegaBytes(mem.HeapSys),
+ HeapAllocated: toMegaBytes(mem.HeapAlloc),
+ ObjectsInUse: mem.Mallocs - mem.Frees,
+ OSMemoryObtained: toMegaBytes(mem.Sys),
}
}Available as PR #227 The resultsTotal allocated memoryThis is a cumulative size (In MiB) of heap objects. It makes sense this is a growing number. Objects in useThe amount of live objects Maximum heap usageThis graph shows the estimated largest size (in MiB) the heap has had. (runtime's Memstats.HeapSys) --edit----- |
|
In a second test I recorded Docker's memory usage and again I've reached a similar conclusion that I'm not detecting any leaks. Next I'll try different operations, so far I've only used the Docker's memory over time (test took around an hour), similar setup as before only this time I've "stressed" it with 5 concurrent requests, which I controlled client-side. Imaginary was set to a concurrency value of 20: $ hey -z 1h \
-c 5 \
-m POST \
-D ~/img.png \
'http://localhost:9000/resize?width=30&height=20&type=jpeg'
Summary:
Total: 3600.0736 secs
Slowest: 1.1892 secs
Fastest: 0.0225 secs
Average: 0.0805 secs
Requests/sec: 62.0851
Total data: 1032620820 bytes
Size/request: 4620 bytes
Response time histogram:
0.023 [1] |
0.139 [213654] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
0.256 [9686] |■■
0.373 [142] |
0.489 [12] |
0.606 [1] |
0.723 [0] |
0.839 [0] |
0.956 [1] |
1.073 [8] |
1.189 [6] |
Latency distribution:
10% in 0.0526 secs
25% in 0.0620 secs
50% in 0.0749 secs
75% in 0.0909 secs
90% in 0.1108 secs
95% in 0.1325 secs
99% in 0.1935 secs
Details (average, fastest, slowest):
DNS+dialup: 0.0000 secs, 0.0225 secs, 1.1892 secs
DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0078 secs
req write: 0.0017 secs, 0.0003 secs, 0.1556 secs
resp wait: 0.0784 secs, 0.0219 secs, 1.1887 secs
resp read: 0.0003 secs, 0.0000 secs, 0.0287 secs
Status code distribution:
[200] 223511 responsesExtracted samples with: $ docker stats ${containerId} \
--no-stream \
--format "{\"container\":\"{{ .Container }}\",\"memory\":{\"raw\":\"{{ .MemUsage }}\",\"percent\":\"{{ .MemPerc }}\"},\"cpu\":\"{{ .CPUPerc }}\"}" \(Note: This changed the sample rate, which was now around 3s). Resulting plot: As you can see it's a stable graph. |
|
More testing. This time using pipelining and using an enlarge and crop: Imaginary statisticsThe objectsInUse has an interesting end, when we're reaching the hour the graph becomes more erratic. Docker memory statisticsIn MiB. Payload$ hey -z 1h -c 5 -m POST -D ~/img.png 'http://localhost:9000/pipeline?operations=%5B%7B%22operation%22%3A%22enlarge%22%2C%22params%22%3A%7B%22force%22%3A%20true%2C%20%22width%22%3A2000%2C%22height%22%3A2000%2C%22quality%22%3A90%7D%7D%2C%7B%22operation%22%3A%22crop%22%2C%22params%22%3A%7B%22width%22%3A200%2C%22height%22%3A600%2C%22quality%22%3A80%7D%7D%5D'The payload pretty-printed [
{
"operation": "enlarge",
"params": {
"force": true,
"width": 2000,
"height": 2000,
"quality": 90
}
},
{
"operation": "crop",
"params": {
"width": 200,
"height": 600,
"quality": 80
}
}
] |
|
I've also investigated processing related to fetching remote resources (The So basically. Without more information. I can't reproduce :-/ |
|
@Dynom Sorry unrelated question. What script/tools are you using for /health sampling and plotting? Im interested to use it! |
|
I'm slightly embarrassed to say that it's just a couple of one-liners, JSON parsing with jq and plotting with asciigraph. However, here you go. Reading Docker's stats:The cd /tmp;
ts=$(date | md5 );
containerId="5a52d8b236f6"; # The right container, find it with `docker ps`, otherwise you might include stats from other containers as well.
for i in $(seq 1 3600); do
outfile="docker-plot-data-${i}-${ts}.json";
docker stats ${containerId} \
--no-stream \
--format "{\"container\":\"{{ .Container }}\",\"memory\":{\"raw\":\"{{ .MemUsage }}\",\"percent\":\"{{ .MemPerc }}\"},\"cpu\":\"{{ .CPUPerc }}\"}" \
| jq . > ${outfile};
cat ${outfile};
doneReading /health infocd /tmp;
ts=$(date | md5 );
for i in $(seq 1 3700); do
outfile="plot-data-${i}-${ts}.json";
curl -sq http://127.0.0.1:9000/health | jq . > ${outfile};
cat ${outfile};
sleep 1;
doneCollecting the info for plotting#!/usr/bin/env bash
for metric in allocatedMemory maxHeapUsage objectsInUse; do
kpi="";
samples="$(ls -1 plot-data-* | wc -l)";
for value in $(ls -1 plot-data-* | sort -V | xargs cat | jq ".${metric}"); do
kpi="${kpi} ${value}";
done
echo ${kpi} | asciigraph -w 180 -h 30 -c "${metric} (1s interval) samples: ${samples}"
done
metric="";
kpi="";
samples="$(ls -1 docker-plot-data-* | wc -l)"
for value in $(ls -1 docker-plot-data-* | sort -V | xargs cat | jq '.memory.raw' | awk '{ print substr($0, 2, index($0, "M")-2)}'); do
kpi="${kpi} ${value}"
done
echo ${kpi} | asciigraph -w 180 -h 30 -c "Docker stats memory.raw (~2s interval) samples: ${samples}"Attack commandsUsing POST $ hey -z 1h -c 5 -m POST -D ~/img.png 'http://localhost:9000/pipeline?operations=%5B%7B%22operation%22%3A%22enlarge%22%2C%22params%22%3A%7B%22force%22%3A%20true%2C%20%22width%22%3A2000%2C%22height%22%3A2000%2C%22quality%22%3A90%7D%7D%2C%7B%22operation%22%3A%22crop%22%2C%22params%22%3A%7B%22width%22%3A200%2C%22height%22%3A600%2C%22quality%22%3A80%7D%7D%5D'Using $ hey -z 1h -c 5 -m GET 'http://localhost:9000/pipeline?operations=%5B%7B%22operation%22%3A%22enlarge%22%2C%22params%22%3A%7B%22force%22%3A%20true%2C%20%22width%22%3A2000%2C%22height%22%3A2000%2C%22quality%22%3A90%7D%7D%2C%7B%22operation%22%3A%22crop%22%2C%22params%22%3A%7B%22width%22%3A200%2C%22height%22%3A600%2C%22quality%22%3A80%7D%7D%5D&url=http://x.x.x.x:8888/img.png' |
|
Thank you for quick reply. I'll try it later!! 👍 Edit: |
|
Hello!
Code of #!/bin/bash
for i in {1..10}
do
for j in {1..100}
do
curl -s -F "file=@testdata/large.jpg" -X POST "http://localhost:8088/crop?width=500&height=200&gravity=smart" > /dev/null &
done
wait
sleep 40
docker stats imaginary_imaginary_1 --no-stream
doneOutput of CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
6116984c063b imaginary_imaginary_1 0.00% 301.5MiB / 15.55GiB 1.89% 88.2MB / 2.83MB 0B / 14.2MB 106
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
6116984c063b imaginary_imaginary_1 0.00% 357.8MiB / 15.55GiB 2.25% 177MB / 5.65MB 0B / 17.1MB 107
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
6116984c063b imaginary_imaginary_1 0.00% 349.7MiB / 15.55GiB 2.20% 265MB / 8.44MB 0B / 23MB 107
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
6116984c063b imaginary_imaginary_1 0.00% 375.9MiB / 15.55GiB 2.36% 353MB / 11.3MB 0B / 23MB 107
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
6116984c063b imaginary_imaginary_1 0.00% 394.9MiB / 15.55GiB 2.48% 441MB / 14MB 0B / 23MB 107
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
6116984c063b imaginary_imaginary_1 0.00% 422MiB / 15.55GiB 2.65% 529MB / 16.8MB 0B / 23MB 108
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
6116984c063b imaginary_imaginary_1 0.00% 424.1MiB / 15.55GiB 2.66% 618MB / 19.6MB 0B / 23MB 108
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
6116984c063b imaginary_imaginary_1 0.00% 450.2MiB / 15.55GiB 2.83% 706MB / 22.4MB 0B / 23MB 109
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
6116984c063b imaginary_imaginary_1 0.00% 451.4MiB / 15.55GiB 2.83% 794MB / 25.2MB 0B / 23MB 109
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
6116984c063b imaginary_imaginary_1 0.00% 461.5MiB / 15.55GiB 2.90% 882MB / 28MB 0B / 23MB 110Result of REPOSITORY TAG IMAGE ID CREATED SIZE
h2non/imaginary latest dfde555a07ac 7 weeks ago 172MBHope this helps a bit in finding a problem. |
|
Can this issue have a link with this one: h2non/bimg#282 ? |
|
I was able to find the reason why my script gave the following results. This is not related to either
If you add this option to |
|
Ah, thanks for the followup! |
|
The link to bimg#282 from @ncarlier above lead me to this doc here for imgproxy/libvips. After setting
The 24 hours before show the memory growth, as well as a growth spike towards the end where the process swapped, before it was restarted. Leaving the comment to save a some clicking for future visitors. I suspect why I have more acute memory growth is simply because of real world traffic that's hitting a 2TB image archive, so it tends to add up quickly. As to what may be the cause of the memory leaks, it would just be guessing but libvips does handle png's poorly, as well as exif metadata. Those are the issues that bubble up to the surface from the error log :( |
|
Oh, to add, I'm limiting memory usage on docker with mem_limit to 8g - but as I don't have swap accounting turned on, it doesn't limit the process further. If I did, it would possibly be OOM killed sooner than when the host runs out of memory. With the MALLOC_ARENA_MAX=2 environment, it seems to avoid eating swap memory. If this is due to memory allocation fragmentation or something else, I wouldn't know, but either way - the situation is currently noticeably improved. |
|
It seems like this was fine for about 2-3 days. Didn't seem to have any lasting effect.
|
|
running imaginary with apiVersion: apps/v1
kind: Deployment
...
spec:
...
template:
...
spec:
...
containers:
- image: h2non/imaginary:1.2.2
resources:
requests:
cpu: "10m"
memory: "400Mi"
limits:
memory: "600Mi"
cpu: "20m"
env:
- name: MALLOC_ARENA_MAX
value: "2" |
|
This still happens to me no matter the configuration. I have not tried the |
Hello,
I am testing the library as it seems a very nice one, but I noticed that memory does not get released completely, even with a short
mreleasevalue.In order to reproduce this just run the docker container:
The look at the container's stats:
I can observe memory spiking up at every new resize request, but it does not go down.
Am I doing something wrong, or is there a way to fix this?
The text was updated successfully, but these errors were encountered: