Leofs_gateway do not cache large(1.2GB) files, although configured

[minhchuduc]

Environments:
Leofs version 1.2.6, installed using .deb file
Erlang R16B03-1 (erts-5.10.4) [source] [64-bit halfword] [smp:16:16] [async-threads:10] [kernel-poll:false]
Ubuntu 14.04 on Baremetal
Kernel 3.13.0-44-generic #73-Ubuntu SMP Tue Dec 16 00:22:43 UTC 2014 x86_64 GNU/Linux

Problem:
10MB file is cached on gateway, when download again, I see leofs-gateway does not need to load from storage-node again (i monitored network traffic between nodes)
But when download 1.2GB, it's not cached whether i configured on leofs_gateway.conf:

## --------------------------------------------------------------------
## GATEWAY - Large Object
## --------------------------------------------------------------------
## Total number of chunked objects
large_object.max_chunked_objs = 1000

## Length of a chunked object
large_object.chunked_obj_len = 5242880

## Threshold of length of a chunked object
large_object.threshold_of_chunk_len = 5767168

## Reading length of a chuncked object
##   * If happening timeout when copying a large object,
##     you will solve to set this value as less than 5MB.
##   * default: "large_object.chunked_obj_len" (5242880 - 5MB)
large_object.reading_chunked_obj_len = 5242880

## --------------------------------------------------------------------
## GATEWAY - Cache
## --------------------------------------------------------------------
## If this parameter is 'true', Gateway turns on HTTP-based cache server, like Varnish OR Squid.
## If this parameter is 'false', Stores objects into the Gateway’s memory.
## When operating READ, the Etag of the cache is compared with a backend storage’s Etag.
## cache.http_cache = false

## A number of cache workers
## cache.cache_workers = 16

## Memory cache capacity in bytes
cache.cache_ram_capacity  = 6123456789

## Disk cache capacity in bytes
cache.cache_disc_capacity = 30123456789

## When the length of the object exceeds this value, store the object on disk
cache.cache_disc_threshold_len = 20123456

## Directory for the disk cache data
cache.cache_disc_dir_data    = ./cache/data

## Directory for the disk cache journal
cache.cache_disc_dir_journal = ./cache/journal

## Cache Expire in seconds
cache.cache_expire = 600

## Cache Max Content Length in bytes
cache.cache_max_content_len = 3123456789

Output of my "leofs-adm status gateway_0@172.x.y.z":

-------------------------------+------------------
             Item              |       Value
-------------------------------+------------------
 Config-1: basic
--------------------------------------------------
 [basic]
-------------------------------+------------------
                       version | 1.2.6
                using protocol | s3
                       log dir | ./log/erlang
-------------------------------+------------------
 [http server related for rest/s3 api]
-------------------------------+------------------
                listening port | 80
            listening ssl port | 443
           number of_acceptors | 256
-------------------------------+------------------
 [cache-related]
-------------------------------+------------------
       http cache [true|false] | false
       number of cache_workers | 16
                  cache expire | 600
         cache max content len | 3123456789
            ram cache capacity | 6123456789
        disk cache capacity    | 30123456789
        disk cache threshold   | 20123456
        disk cache data dir    | ./cache/data
        disk cache journal dir | ./cache/journal
-------------------------------+------------------
 [large object related]
-------------------------------+------------------
      max number of chunk objs | 1000
           chunk object length | 5242880
             max object length | 5242880000
     reading  chunk obj length | 5242880
     threshold of chunk length | 5767168
-------------------------------+------------------
 Config-2: watchdog
-------------------------------+------------------
 [rex(rpc-proc)]               |
             watch interval(s) | 5
        threshold mem capacity | 335544320
-------------------------------+------------------
 [cpu]                         |
              watchdog eanbled | disabled
             watch interval(s) | 5
        threshold cpu load avg | 5.0
         threshold cpu util(%) | 100
-------------------------------+------------------
 [io]                          |
              watchdog enabled | enabled
             watch interval(s) | 1
        threshold input size/s | 134217728
       threshold output size/s | 134217728
-------------------------------+------------------
 Status-1: RING hash
-------------------------------+------------------
             current ring hash | 9d2edfe6
            previous ring hash | 9d2edfe6
-------------------------------+------------------
 Status-2: Erlang VM
-------------------------------+------------------
                    vm version | 5.10.4
               total mem usage | 75322280
              system mem usage | 54630464
               procs mem usage | 20690424
                 ets mem usage | 5585976
                         procs | 728/1048576
                   kernel_poll | true
              thread_pool_size | 32
-------------------------------+------------------

[minhchuduc]

I do more tests for sure by creating some big files to test how big Leofs_gateway will cache by:

# dd if=/dev/urandom of=100MB_file bs=1M count=100 conv=fdatasync
# dd if=/dev/urandom of=300MB_file bs=1M count=300 conv=fdatasync
# dd if=/dev/urandom of=900MB_file bs=1M count=900 conv=fdatasync

I upload 3 files to leofs, then download full files few times to make sure Leofs_gateway will cache it.
Then i check the cache folder:

ls -lRh /usr/local/leofs/1.2.6/leo_gateway/cache/data/ | less

And i found that with above configuration, only 2 files are cached:

...
/usr/local/leofs/1.2.6/leo_gateway/cache/data/10:
total 300M
-rw-r----- 1 root root 300M Mar  4 11:18 hehe1%2F300MB_file
....
/usr/local/leofs/1.2.6/leo_gateway/cache/data/4:
total 108M
-rw-r----- 1 root root 100M Mar  4 11:11 hehe1%2F100MB_file
-rw-r----- 1 root root 7.6M Mar  4 11:05 hehe1%2F8.07.14_MegaCLI.zip
.... (other dirs are empty)

[minhchuduc]

On downloading 900MB_file, i see the cache file is created, but it's auto-removed when the download finish:

/usr/local/leofs/1.2.6/leo_gateway/cache/data/15:
total 155M
-rw-r----- 1 root root 155M Mar  4 11:39 hehe1%2F900MB_file.tmp

[yosukehara]

Thank you for sharing the issue. We'll check it soon.

[mocchira]

@minhchuduc Thank you for reporting this issue.
We found out that there is a another MAX disk size limit factor.
In your case,

name	value
disk cache capacity	30123456789
cache.cache_workers	16

Since the number of directories to store cache files will be equal to cache.cache_workers AND
the max size per one directory has been determined by
disk cache capacity / cache.cache_workers ,
so there is also around 1.8G disk size limits.

You might hit this problem.
We will append this limitation to the document.

[yosukehara]

[minhchuduc]

Hi, but my disk cache capacity = 30123456789 (~ 30GB)
--> max capacity per worker is: 30GB /16 = 1.8GB
And my cache directory is almost empty:

#ls -lRh /usr/local/leofs/1.2.6/leo_gateway/cache/data/  
/usr/local/leofs/1.2.6/leo_gateway/cache/data/:
total 64K
drwxr-x--- 2 root root 4.0K Mar  4 10:12 1
drwxr-x--- 2 root root 4.0K Mar  4 12:00 10
drwxr-x--- 2 root root 4.0K Mar  4 10:12 11
drwxr-x--- 2 root root 4.0K Mar  4 10:12 12
drwxr-x--- 2 root root 4.0K Mar  4 10:12 13
drwxr-x--- 2 root root 4.0K Mar  4 11:15 14
drwxr-x--- 2 root root 4.0K Mar  4 11:39 15
drwxr-x--- 2 root root 4.0K Mar  4 11:34 16
drwxr-x--- 2 root root 4.0K Mar  4 10:12 2
drwxr-x--- 2 root root 4.0K Mar  4 10:12 3
drwxr-x--- 2 root root 4.0K Mar  4 11:11 4
drwxr-x--- 2 root root 4.0K Mar  4 11:58 5
drwxr-x--- 2 root root 4.0K Mar  4 10:12 6
drwxr-x--- 2 root root 4.0K Mar  4 10:12 7
drwxr-x--- 2 root root 4.0K Mar  4 10:12 8
drwxr-x--- 2 root root 4.0K Mar  4 11:33 9
/usr/local/leofs/1.2.6/leo_gateway/cache/data/1:
total 0
/usr/local/leofs/1.2.6/leo_gateway/cache/data/10:
total 300M
-rw-r----- 1 root root 300M Mar  4 11:18 hehe1%2F300MB_file
-rw-r----- 1 root root    0 Mar  4 12:00 hehe1%2F300MB_file.tmp
/usr/local/leofs/1.2.6/leo_gateway/cache/data/11:
total 0
/usr/local/leofs/1.2.6/leo_gateway/cache/data/12:
total 0
/usr/local/leofs/1.2.6/leo_gateway/cache/data/13:
total 0
/usr/local/leofs/1.2.6/leo_gateway/cache/data/14:
total 0
/usr/local/leofs/1.2.6/leo_gateway/cache/data/15:
total 0
/usr/local/leofs/1.2.6/leo_gateway/cache/data/16:
total 200M
-rw-r----- 1 root root 200M Mar  4 11:34 hehe1%2F200MB_file
/usr/local/leofs/1.2.6/leo_gateway/cache/data/2:
total 0
/usr/local/leofs/1.2.6/leo_gateway/cache/data/3:
total 0
/usr/local/leofs/1.2.6/leo_gateway/cache/data/4:
total 108M
-rw-r----- 1 root root 100M Mar  4 11:11 hehe1%2F100MB_file
-rw-r----- 1 root root 7.6M Mar  4 11:05 hehe1%2F8.07.14_MegaCLI.zip
/usr/local/leofs/1.2.6/leo_gateway/cache/data/5:
total 0
/usr/local/leofs/1.2.6/leo_gateway/cache/data/6:
total 0
/usr/local/leofs/1.2.6/leo_gateway/cache/data/7:
total 0
/usr/local/leofs/1.2.6/leo_gateway/cache/data/8:
total 0
/usr/local/leofs/1.2.6/leo_gateway/cache/data/9:
total 0

then i tried to increased disk cache capacity to 300123456789 (~ 300GB), the result is the same (not cached)

[mocchira]

@minhchuduc Thank you for clarification.
We found out a bug.
https://github.com/leo-project/leo_cache/blob/develop/src/leo_cache_server_dcerl.erl#L56
The value to be set to disk cache size has been initialized with ram cache size.

We will fix this issue before long.

[mocchira]

@minhchuduc I filed this issue at leo-project/leofs#322,
because we've managed all issues regarding LeoFS on the LeoFS repository's issue tracker for our centralized management policy.
So could you file a issue on the LeoFS repository from next time?

Anyway. thanks for your contribution.

[minhchuduc]

Thank you! I will file next issue on LeoFS repo.