timeout is not working as expected in shared dict configuration

[coolkingh]

Compiled the latest code from master

Configuration in /etc/nginx/stream.conf file

js_shared_dict_zone zone=foo:10M timeout=30s type=number;

server {
        listen 127.0.0.1:9007 ;
        js_periodic stream.periodics_worker interval=60s jitter=1s;
        proxy_pass $set_shared_dict;
}

Code in stream.js

function set_shared_dict(s) {

 ngx.shared.foo.set("34567", 122);
 ngx.shared.foo.set("34568", 123);
 ngx.shared.foo.set("34566", 121);
 ngx.shared.foo.set("34565", 125);

}

function periodics_worker() {
        if (ngx.worker_id != 0) {
                /* using ngx.worker_id to run handler only in one worker. */
                return;
        }
        ngx.log(ngx.INFO, "*********** periodics_worker called after 60s ****************");
        var keys = ngx.shared.foo.keys();
        ngx.log(ngx.INFO, keys[0] + " : " + keys[1] + " : " + keys[2] + " : " + keys[3]);
        ngx.log(ngx.INFO, " get value 0: " + ngx.shared.foo.get(keys[0]));
        ngx.log(ngx.INFO, " get value 1: " + ngx.shared.foo.get(keys[1]));
        ngx.log(ngx.INFO, " get value 2: " + ngx.shared.foo.get(keys[2]));
        ngx.log(ngx.INFO, " get value 3: " + ngx.shared.foo.get(keys[3]));
}

Logs are as follows

2023/08/30 08:19:02 [info] 25718#25718: js: *********** periodics_worker called after 60s ****************
2023/08/30 08:19:02 [info] 25718#25718: js: 34566 : 34567 : 34565 : 34568 
2023/08/30 08:19:02 [info] 25718#25718: js:  get value 0: undefined
2023/08/30 08:19:02 [info] 25718#25718: js:  get value 1: undefined
2023/08/30 08:19:02 [info] 25718#25718: js:  get value 2: undefined
2023/08/30 08:19:02 [info] 25718#25718: js:  get value 3: undefined

As per the specification of timeout, it removes key-value pairs after 30 seconds of inactivity but actually it doesn't remove it just return unspecified.

ngx.shared.foo.keys() function return all the keys which are also expired.

I looked into the code and found that when we call ngx.shared.foo.get() function it doesn't actually delete the expired keys. It checks the expiry and return not_found. Below is the code from nginx/ngx_js_shared_dict.c

static ngx_int_t
ngx_js_dict_get(njs_vm_t *vm, ngx_js_dict_t *dict, njs_str_t *key,
    njs_value_t *retval)
{
    ngx_int_t            rc;
    ngx_msec_t           now;
    ngx_time_t          *tp;
    ngx_js_dict_node_t  *node;

    ngx_rwlock_rlock(&dict->sh->rwlock);

    node = ngx_js_dict_lookup(dict, key);

    if (node == NULL) {
        goto not_found;
    }

    if (dict->timeout) {
        tp = ngx_timeofday();
        now = tp->sec * 1000 + tp->msec;

        if (now >= node->expire.key) {
            goto not_found;
        }
    }

    rc = ngx_js_dict_copy_value_locked(vm, dict, node, retval);
    ngx_rwlock_unlock(&dict->sh->rwlock);

    return rc;

not_found:

I waited for like one hour but the keys are still there.

If the data of expired keys is still there in the memory it will occupy the space and the shared_dict space will be full.

[coolkingh]

I have also observed that keys are only getting removed if any new key is inserted in ngx_js_dict_add function. Therefore, It will not fill the space allocated.

The only question that remains is
key() function is returning all the keys including expired keys.

Is that a expected behaviour ?

[xeioex]

@coolkingh

If the data of expired keys is still there in the memory it will occupy the space and the shared_dict space will be full.

No, the idea is not to touch the shared memory until there is an access to it. The expired nodes are tracked and removed here at the moment a new item is added.
So it should prevent from overflow even if no evict= is enabled.

[xeioex]

@coolkingh

The only question that remains is key() function is returning all the keys including expired keys.

Yes, this looks like an issue, it will be fixed. Thank you for reporting it.