[0.9.2] Memory leak issue with njs 0.9.2 parsing JWT tokens #971
Description
Memory issue
On Sunday we updates NJS from 0.9.1 to 0.9.2 using the APT package on Ubuntu 22.04.
It took about 12 hours for the app server to start getting memory alerts and after we investigated we found that after the deployment nginx memory was climbing slowly and never coming back down. The only change in the configuration was the update for NJS.
This was deployed on all our environments (8 in total) and all exhibited the same behavior.
The NJS code we have parses a JWT token from the Authorization header and extracts a couple of values from the payload to use in logging. The code extracts the attributes "uid" and "user_uuid" from the JWT token. There are defaults if they aren't present. It's pretty simple code (below).
We reverted the NJS module to 0.9.1 and the issue did not occur.
To reproduce
Steps to reproduce the behavior:
- JS script
function jwt(data) {
var parts = data.split('.').slice(0,2)
.map(v=>Buffer.from(v, 'base64url').toString())
.map(JSON.parse);
return { headers:parts[0], payload: parts[1] };
}
function jwt_payload_uid(r) {
var uid;
try {
uid = jwt(r.headersIn.Authorization.slice(7)).payload.uid;
}
catch(e) {
uid = 0;
}
return uid;
}
function jwt_payload_user_uuid(r) {
var user_uuid;
try {
user_uuid = jwt(r.headersIn.Authorization.slice(7)).payload.user_uuid;
}
catch(e) {
user_uuid = 'anonymous';
}
return user_uuid;
}
export default {jwt_payload_uid, jwt_payload_user_uuid}- NGINX configuration
load_module "modules/ngx_http_js_module.so";
user www-data;
worker_processes auto;
pid /run/nginx.pid;
events {
worker_connections 2048;
}
http {
js_path "/etc/nginx/njs/";
js_import main from jwt.js;
js_set $jwt_payload_uid main.jwt_payload_uid;
js_set $jwt_payload_user_uuid main.jwt_payload_user_uuid;
log_format main 'request_time=$time_local '
'base_url="$scheme://$host" request="$request_method $request_uri" response_code=$status '
'http_referrer="$http_referrer" http_user_agent="$http_user_agent" '
'ip=$http_true_client_ip cdn=$remote_addr remote_user=$remote_user '
'bytes_sent=$body_bytes_sent rt=$request_time urt=$upstream_response_time '
'user_uid=$jwt_payload_uid user_uuid="$jwt_payload_user_uuid"';
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
gzip on;
gzip_comp_level 6;
gzip_vary on;
gzip_min_length 1000;
gzip_proxied any;
gzip_types text/plain text/css application/json application/x-javascript te$
gzip_buffers 16 8k;
types_hash_max_size 2048;
client_max_body_size 500m;
# /var/log/supervisor is shared between the host and the guest
access_log /var/log/supervisor/nginx-access.log main;
error_log /var/log/supervisor/nginx-error.log;
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
set_real_ip_from 172.16.0.0/12;
real_ip_header X-Forwarded-For;
real_ip_recursive on;
fastcgi_buffers 8 16k; # increase the buffer size for PHP-FTP
fastcgi_buffer_size 32k; # increase the buffer size for PHP-FTP
fastcgi_connect_timeout 60;
fastcgi_send_timeout 300;
fastcgi_read_timeout 300;
}
### Expected behavior
No memory leak.
### Your environment
- NJS version 0.9.2
- NGINX 1.28.0
- OS: Ubuntu 22.04
### Additional context
I asked ClaudeCode to review the changes especially with reference the JWT script:
Based on my analysis of the differences between njs 0.9.1 and 0.9.2, I found several significant memory management changes that could cause memory leaks in per-request JWT processing:
Key Memory Leak Sources
1. QuickJS Temporary String Memory Leak (commit a9dabca1)
Most Likely Culprit - This commit fixes missing JS_FreeCString() calls throughout the QuickJS code. When JavaScript strings are converted to C strings using JS_ToCStringLen(), they must be freed with JS_FreeCString(), or they leak memory.
Affected areas relevant to JWT processing:
- nginx/ngx_qjs_fetch.c: Header operations, string conversions
- nginx/ngx_js_shared_dict.c: Dictionary key/value operations
- nginx/ngx_stream_js_module.c: Variable and event handling
Impact: Every request processing JWT headers would leak string memory if using QuickJS engine.
2. Heap-Use-After-Free Fix (commit 4eeb4c52)
Changed ngx_qjs_string() to allocate from nginx pools instead of QuickJS engine pools. The signature changed from:
ngx_qjs_string(JSContext *cx, JSValueConst val, ngx_str_t *dst)
to:
ngx_qjs_string(JSContext *cx, ngx_pool_t *pool, JSValueConst val, ngx_str_t *dst)
Impact: If any nginx module code still uses the old calling convention, it would cause memory allocation issues. Also, the switch in memory pools could change deallocation behavior.
3. Simplified nginx Values Access (commit 51326759)
Refactored how nginx configuration is accessed from JS engines - now stores a direct pointer to ngx_js_loc_conf_t instead of calling functions. The fetch
code now stores:
http->conf = conf; // Direct pointer to config structure
Impact: If the config structure lifecycle doesn't match the http object lifecycle, this could prevent proper cleanup.
4. Removed Extra Buffer Copy (commit 6842e1b8)
Removed memory allocation/copy when passing buffers to JavaScript:
// OLD: Made a copy
p = ngx_pnalloc(r->pool, len);
ngx_memcpy(p, b->pos, len);
arguments[1] = ngx_qjs_prop(cx, jlcf->buffer_type, p, len);
// NEW: Direct buffer reference
arguments[1] = ngx_qjs_prop(cx, jlcf->buffer_type, b->pos, len);
Impact: Combined with other changes, if the buffer lifecycle management isn't correct, this could cause issues.
Recommendation
The most likely cause is commit a9dabca1 (QuickJS: freeing temporary strings). If you're using the QuickJS engine (not NJS engine), version 0.9.2 introduced
many string operations that now properly free temporary strings - but if there are any missed conversions or if your JWT script triggers code paths that
weren't covered by these fixes, you'd see a leak on every request.
To verify: Check which JavaScript engine you're using (NJS vs QuickJS). If using QuickJS, the string freeing fixes in 0.9.2 might have exposed other unfixed
code paths, or inadvertently introduced new issues in the complex interplay between nginx memory pools and QuickJS memory management.