Cache control

[dmitrytretyakov]

Is it possible to use cache_set_max, cache_set_max_mem, etc in lua-vips?
I use lua-vips with openresty (nginx) and i see that source files (vips.Image.new_from_file) keep in memory.

[jcupitt]

Hello @dmitrytretyakov

No, those calls are not exposed in lua-vips. I agree, they are useful in a server environment. I'll have a look at adding them.

[dmitrytretyakov]

Thank you!
It would be cool!

[jcupitt]

OK, git master should have the calls, would you be able to test?

-- set number of cached operations
vips.set_max(100)
-- set maximum cache memory use
vips.set_max_mem(10 * 1024 * 1024)
-- set maximum number of open files
vips.set_max_files(10)

[dmitrytretyakov]

It's cool! I didn't expect such fast support
Thank you very much!
Directives works. But problems stay with keep in memory (memory leaking) with nginx.
I test magick and vips:

init_by_lua_block {
vips = require "vips"
vips.set_max(0)
}
content_by_lua_block {
local path = "/path/to/files/image.jpg"
local base = vips.Image.new_from_file(path, {access = sequential})
base:write_to_file("/new/path/test_image.jpg")
}

init_by_lua_block {
magick = require "magick"
}
content_by_lua_block {
local path = "/path/to/files/image.jpg"
local base = magick.load_image(path)
base:write("/new/path/test_image.jpg")
}

and i see memory growth with every request with vips.
Possibly, it is not problem of vips.

[jcupitt]

Here's the program I used for leak testing:

#!/usr/bin/env luajit
        
vips = require 'vips'
        
-- benchmark thumbnail via a memory buffer
function via_memory(filename, thumbnail_width)
    print("memory processing ", filename)

    local fp = assert(io.open(filename, "rb"))
    local data = fp:read("*all")

    local thumb = vips.Image.thumbnail_buffer(data, thumbnail_width,
        {crop = "centre"})

    return thumb:write_to_buffer(".jpg")
end

-- benchmark thumbnail via files
function via_files(filename, thumbnail_width)
    print("file processing ", filename)
    
    local thumb = vips.Image.thumbnail(filename, thumbnail_width,
        {crop = "centre"})
    
    return thumb:write_to_buffer(".jpg")
end 
    
for i = 1, #arg do
    collectgarbage() 
    -- thumb = via_memory(arg[i], 500)
    thumb = via_files(arg[i], 500)
end

If I run like this:

$ mkdir samples
$ for i in {1..1000}; do cp ~/pics/k2.jpg samples/$i.jpg; done
$ time ./soak.lua samples/*
real	0m46.222s
user	1m19.727s
sys	0m2.503s

It runs in a steady 150mb of memory, looking at RES in top. If I add vips.set_max(0), it runs in about 45mb. If I comment out the collectgarbage()it runs in about 90mb.

So I think lua-vips does not leak, at least in this case. The libvips cache needs about 100mb, and the luajit garbage collector about 50mb.

I've not used openresty, I don't know if lua-vips could have problems integrating with it.

[jcupitt]

How would I set up a minimal test case for openresty? Is there an idiot's guide somewhere? How did you configure it for your test?

[dmitrytretyakov]

I installed openresty from apt repository: https://openresty.org/en/linux-packages.html, luarocks (from base ubuntu repo) and lua-vips (libvips 8.6.3)
I added into default sections /usr/local/nginx/conf/nginx.conf next:

http {
user www-data;
...
init_by_lua_block {
        vips = require "vips"
        vips.set_max(0)
    }
...
server {
...
lua_code_cache off;
...
location ~ /vips/ {
    default_type 'text/plain';
    content_by_lua_block {
        local path = "/path/to/files/image.jpg"
        local base = vips.Image.new_from_file(path, {access = sequential})
        base:write_to_file("/path/to/files/new.jpg")
        ngx.say("Ok")
    }
}
}
}

Give access openresty to /path/to/files/: chgrp www-data /path/to/files/; chmod 775 /path/to/files/
Request: curl http://localhost/vips/test

[jcupitt]

It seems to work without leaks for me. I made this conf file:

worker_processes  1;
error_log logs/error.log;
events {
    worker_connections 1024;
}
http {
    init_by_lua_block {
        vips = require "vips"
        vips.set_max(0)
    }

    server {
        listen 8080;
        location / {
            default_type text/html;
            content_by_lua_block {
                local path = "/home/john/pics/nina.jpg"
                local image = vips.Image.new_from_file(path,
                    {access = sequential})
                image:write_to_file("/tmp/new.jpg")
                ngx.say("Ok")
            }
        }
    }
}

Where nina.jpg is 6000 x 4000 pixel rgb jpg, and tested with:

$ for i in {1..10000}; do echo $i ; curl http://localhost:8080/; done

I let it run for about 5000 iterations and memuse (as seen by top) stayed in the 500mb - 1.8gb range. It does seem rather high, but at least it didn't seem to increase. I guess the bouncing up and down was with LuaJIT's GC.

[jcupitt]

... that was with libvips from the Ubuntu 17.10 package, openresty from their PPA, and lua-vips from the rock.

[dmitrytretyakov]

Thank you very much, @jcupitt !

Looks like my fail. Memory growth limited.
But I noticed different memory allocation for different images.
Configuration from comment and comment

Permanent state after 200 iteration, 1st image :
lua-vips: www-data 764 96.0 0.1 765452 26336 ? S 08:37 0:02 nginx: worker process
magick: www-data 20850 62.7 0.0 101280 14952 ? S 08:49 0:05 nginx: worker process

Permanent state after 200 iteration, 2nd image :
lua-vips: www-data 30746 26.6 1.5 766672 250440 ? S 08:33 0:42 nginx: worker process
magick: www-data 19982 84.8 0.0 96940 11068 ? S 08:47 1:13 nginx: worker process

Permanent state after 200 iteration, 3rd image
lua-vips: www-data 17215 37.9 0.2 767060 45164 ? S 08:41 1:34 nginx: worker process
magick: www-data 22498 29.0 0.0 96784 10720 ? S 08:50 3:09 nginx: worker process

I will be grateful for any advice.

[jcupitt]

Sorry, I don't know. Something about how openresty works I guess.

I tried two small programs:

#!/usr/bin/env luajit

local vips = require 'vips'

-- vips.set_max(0)
    
for i = 1, #arg do
    -- collectgarbage()
    print(arg[i] .. " ...")
    local thumb = vips.Image.thumbnail(arg[i], 64)
    thumb:write_to_file("x.jpg")
end

and

#!/usr/bin/env luajit

local magick = require "magick"

for i = 1, #arg do
    -- collectgarbage()
    print(arg[i] .. " ...")
    magick.thumb(arg[i], "64x64", "x.jpg")
end

With 1000 jpg images, for magick, I see:

        real    1m5.991s
        user    2m59.921s
        sys     0m0.711s
        peak mem 90mb

If I add collectgarbage() on each loop, I see a peak mem of 33mb.

For lua-vips I see:

        real    0m14.306s
        user    0m13.774s
        sys     0m0.684s
        peak mem 50mb

If I add collectgarbage() I see a peak mem of about 48mb. If I add set_max(0) I see a peak mem of about 24mb.

[jcupitt]

I suppose I don't understand how openresty handles threads, processes and initialization. How often is the init block run, how is the result of init shared for parallel requests, how is the result of init reused for repeated requests, when does the luajit VM start up and shut down, when is the GC triggered, are there hooks for request end that rocks can attach to, and so on.

I did notice that nginx memory use went up and down a great deal. Rather than looking after exactly 200 iterations, I think I would run top and watch RES over several thousand iterations. I wouldn't expect a large difference in memory use over a long period of time.