http client object is GC-collected when http request object is alive #9453

ligurio · 2023-12-07T08:23:22Z

(initially reported by Oleg Babin in #9341 (comment))

Bug description

http client object is GC-collected when http request object is alive

OS: Linux
OS Version: Ubuntu 22.04
Architecture: [amd64, arm]

Place tarantool --version output here.

Steps to reproduce

reproducer script:

url = 'https://api.open-meteo.com/v1/forecast?latitude=52.52&longitude=13.41&current=temperature_2m,wind_speed_10m&hourly=temperature_2m,relative_humidity_2m,wind_speed_10m'

c = require('http.client').new()

r1 = c:get(url, {chunked = true})
r2 = c:get(url, {chunked = true})

r1:read(20)
r2:read(20)

r1 = nil
collectgarbage()
collectgarbage()

r2 = nil

collectgarbage()
collectgarbage()

r3 = c:get(url, {chunked = true})
r4 = c:get(url, {chunked = true})

c = nil
collectgarbage()
collectgarbage()

r3:read(20) -- fails here
r3 = nil
collectgarbage()
collectgarbage()

data = r4:read(20)
print(data)

r4 = nil
collectgarbage()
collectgarbage()

os.exit()

How to run:

Just type tarantool test.lua.

Actual behavior

IllegalParams: io: request must be io
fatal error, exiting the event loop

Expected behavior

Successful read.

The text was updated successfully, but these errors were encountered:

`http.client` has an incorrect behaviour: HTTP-client object could be GC-collected before GC-collecting an active chunked HTTP request. The patch adds a test with an incorrect behaviour. Following commits will fix this behaviour. Needed for tarantool#9453 NO_CHANGELOG=testing NO_DOC=testing

There is a problem with destroying http objects in `httpc` module triggered by GC. `httpc` module has two GC-finalizators: first one for a Lua http client (`luaT_httpc_cleanup`) and second one for a Lua http io (`luaT_httpc_io_cleanup`). In some circumstances GC-finalizators have a race that leads to use-after-free errors. In a stacktrace the problem looks as below: ``` 0x55ca7d47652e in crash_collect+256 0x55ca7d476f6a in crash_signal_cb+100 0x7fb876c42520 in __sigaction+80 0x55ca7d641e51 in curl_slist_free_all+35 0x55ca7d441498 in httpc_request_delete+45 0x55ca7d4653f1 in httpc_io_destroy+27 0x55ca7d4674bc in luaT_httpc_io_cleanup+36 0x55ca7d4e00c7 in lj_BC_FUNCC+70 0x55ca7d4f8364 in gc_call_finalizer+668 0x55ca7d4f8946 in gc_finalize+1387 0x55ca7d4f91e2 in gc_onestep+864 0x55ca7d4f9716 in lj_gc_fullgc+276 ... ``` The patch introduces two functions: `httpc_env_finish` and `curl_env_finish`, that prepares curl and httpc environments for destruction. HTTP client's GC finilizer now calls `httpc_env_finish` instead of `httpc_env_destroy`, this prevents from destroying memory that could be in use by HTTP requests. Additionally `httpc_env_finish` sets a flag `cleanup`. HTTP environment destroying is called when flag `cleanup` is set and all HTTP requests were deleted. Fixes tarantool#9346 Fixes tarantool#9453 NO_DOC=bugfix

`httpc` module has two GC-finalizers: the first one for a Lua http client (C function `luaT_httpc_cleanup`) and the second one for a Lua http chunked requests (C function `luaT_httpc_io_cleanup`). In a C implementation HTTP requests depends on structures of HTTP client and there is a problem with destroying Lua objects in `httpc` module - these GC-finalizers are not synchronized. This could lead to at least two problems: There is a race with GC-finalization that leads to use-after-free errors when HTTP client is collected before collecting HTTP request. In a stacktrace the problem looks as below: ``` 0x55ca7d47652e in crash_collect+256 0x55ca7d476f6a in crash_signal_cb+100 0x7fb876c42520 in __sigaction+80 0x55ca7d641e51 in curl_slist_free_all+35 0x55ca7d441498 in httpc_request_delete+45 0x55ca7d4653f1 in httpc_io_destroy+27 0x55ca7d4674bc in luaT_httpc_io_cleanup+36 0x55ca7d4e00c7 in lj_BC_FUNCC+70 0x55ca7d4f8364 in gc_call_finalizer+668 0x55ca7d4f8946 in gc_finalize+1387 0x55ca7d4f91e2 in gc_onestep+864 0x55ca7d4f9716 in lj_gc_fullgc+276 ... ``` Lua object `http.client` could be GC-collected when chunked HTTP request is alive. This will lead to an error "IllegalParams: io: request must be io" because we call a method when Lua object is already a `nil`. ```lua local url = 'https://bronevichok.ru/' local c = require('http.client').new() local r = c:get(url, {chunked = true}) c = nil collectgarbage() collectgarbage() r:read(1) -- IllegalParams: io: request must be io ``` The patch introduces two functions: `httpc_env_finish` and `curl_env_finish`, that prepares curl and httpc environments for destruction. HTTP client's GC finalizer now calls `httpc_env_finish` instead of `httpc_env_destroy`, this prevents from destroying memory that could be in use by HTTP requests. Additionally `httpc_env_finish` sets a flag `cleanup`. HTTP environment destroying is called when flag `cleanup` is set and a there are no active HTTP requests. The main idea of the patch is a synchronization of descructors for HTTP client and HTTP chunked request. Unfortunately, GC will eventually collect HTTP client object after calling its `__gc`. To prevent this we put a reference to a userdata with HTTP client object to a `LUA_REGISTRYINDEX` and remove this reference only when GC-finalizer for HTTP client is called and there are no active HTTP chunked requests. Fixes tarantool#9346 Fixes tarantool#9453

`httpc` module has two GC-finalizers: the first one for a Lua http client (C function `luaT_httpc_cleanup`) and the second one for a Lua http chunked requests (C function `luaT_httpc_io_cleanup`). In a C implementation HTTP requests depends on structures of HTTP client and there is a problem with destroying Lua objects in `httpc` module - these GC-finalizers are not synchronized. This could lead to at least two problems: There is a race with GC-finalization that leads to use-after-free errors when HTTP client is collected before collecting HTTP request. In a stacktrace the problem looks as below: ``` 0x55ca7d47652e in crash_collect+256 0x55ca7d476f6a in crash_signal_cb+100 0x7fb876c42520 in __sigaction+80 0x55ca7d641e51 in curl_slist_free_all+35 0x55ca7d441498 in httpc_request_delete+45 0x55ca7d4653f1 in httpc_io_destroy+27 0x55ca7d4674bc in luaT_httpc_io_cleanup+36 0x55ca7d4e00c7 in lj_BC_FUNCC+70 0x55ca7d4f8364 in gc_call_finalizer+668 0x55ca7d4f8946 in gc_finalize+1387 0x55ca7d4f91e2 in gc_onestep+864 0x55ca7d4f9716 in lj_gc_fullgc+276 ... ``` Lua object `http.client` could be GC-collected when chunked HTTP request is alive. This will lead to an error "IllegalParams: io: request must be io" because we call a method when Lua object is already a `nil`. ```lua local url = 'https://bronevichok.ru/' local c = require('http.client').new() local r = c:get(url, {chunked = true}) c = nil collectgarbage() collectgarbage() r:read(1) -- IllegalParams: io: request must be io ``` The patch introduces two functions: `httpc_env_finish` and `curl_env_finish`, that prepares curl and httpc environments for destruction. HTTP client's GC finalizer now calls `httpc_env_finish` instead of `httpc_env_destroy`, this prevents from destroying memory that could be in use by HTTP requests. Additionally `httpc_env_finish` sets a flag `cleanup`. HTTP environment destroying is called when flag `cleanup` is set and a there are no active HTTP requests. The main idea of the patch is a synchronization of descructors for HTTP client and HTTP chunked request. Unfortunately, GC will eventually collect HTTP client object after calling its `__gc`. To prevent this we put a reference to a userdata with HTTP client object to a `LUA_REGISTRYINDEX` and remove this reference only when GC-finalizer for HTTP client is called and there are no active HTTP chunked requests. Fixes tarantool#9346 Fixes tarantool#9453 NO_DOC=bugfix

`httpc` module has two GC-finalizers: the first one for a Lua http client (C function `luaT_httpc_cleanup`) and the second one for a Lua http chunked requests (C function `luaT_httpc_io_cleanup`) introduced in commit 417c6cb ("httpc: introduce stream input/output interface"). In a C implementation HTTP requests depends on structures of HTTP client and there is a problem with destroying Lua objects in `httpc` module - these GC-finalizers are not synchronized. This could lead to at least two problems: There is a race with GC-finalization that leads to use-after-free errors when HTTP client is collected before collecting HTTP request. In a stacktrace the problem looks as below: ``` 0x55ca7d47652e in crash_collect+256 0x55ca7d476f6a in crash_signal_cb+100 0x7fb876c42520 in __sigaction+80 0x55ca7d641e51 in curl_slist_free_all+35 0x55ca7d441498 in httpc_request_delete+45 0x55ca7d4653f1 in httpc_io_destroy+27 0x55ca7d4674bc in luaT_httpc_io_cleanup+36 0x55ca7d4e00c7 in lj_BC_FUNCC+70 0x55ca7d4f8364 in gc_call_finalizer+668 0x55ca7d4f8946 in gc_finalize+1387 0x55ca7d4f91e2 in gc_onestep+864 0x55ca7d4f9716 in lj_gc_fullgc+276 ... ``` Lua object `http.client` could be GC-collected when chunked HTTP request is alive. This will lead to an error "IllegalParams: io: request must be io" because we call a method when Lua object is already a `nil`. ```lua local url = 'https://bronevichok.ru/' local c = require('http.client').new() local r = c:get(url, {chunked = true}) c = nil collectgarbage() collectgarbage() r:read(1) -- IllegalParams: io: request must be io ``` The patch introduces two functions: `httpc_env_finish` and `curl_env_finish`, that prepares curl and httpc environments for destruction. HTTP client's GC finalizer now calls `httpc_env_finish` instead of `httpc_env_destroy`, this prevents from destroying memory that could be in use by HTTP requests. Additionally `httpc_env_finish` sets a flag `cleanup`. HTTP environment destroying is called when flag `cleanup` is set and a there are no active HTTP requests. The main idea of the patch is a synchronization of descructors for HTTP client and HTTP chunked request. Unfortunately, GC will eventually collect HTTP client object after calling its `__gc`. To prevent this we put a reference to a userdata with HTTP client object to a `LUA_REGISTRYINDEX` and remove this reference only when GC-finalizer for HTTP client is called and there are no active HTTP chunked requests. Fixes tarantool#9346 Fixes tarantool#9453 NO_DOC=bugfix

`httpc` module has two GC-finalizers: the first one for a Lua http client (C function `luaT_httpc_cleanup`) and the second one for a Lua http chunked requests (C function `luaT_httpc_io_cleanup`) introduced in commit 417c6cb ("httpc: introduce stream input/output interface"). In a C implementation HTTP requests depends on structures of HTTP client and there is a problem with destroying Lua objects in `httpc` module - these GC-finalizers are not synchronized. This could lead to at least two problems: There is a race with GC-finalization that leads to use-after-free errors when HTTP client is collected before collecting HTTP request. In a stacktrace the problem looks as below: ``` 0x55ca7d47652e in crash_collect+256 0x55ca7d476f6a in crash_signal_cb+100 0x7fb876c42520 in __sigaction+80 0x55ca7d641e51 in curl_slist_free_all+35 0x55ca7d441498 in httpc_request_delete+45 0x55ca7d4653f1 in httpc_io_destroy+27 0x55ca7d4674bc in luaT_httpc_io_cleanup+36 0x55ca7d4e00c7 in lj_BC_FUNCC+70 0x55ca7d4f8364 in gc_call_finalizer+668 0x55ca7d4f8946 in gc_finalize+1387 0x55ca7d4f91e2 in gc_onestep+864 0x55ca7d4f9716 in lj_gc_fullgc+276 ... ``` Lua object `http.client` could be GC-collected when chunked HTTP request is alive. This will lead to an error "IllegalParams: io: request must be io" because we call a method when Lua object is already a `nil`. ```lua local url = 'https://bronevichok.ru/' local c = require('http.client').new() local r = c:get(url, {chunked = true}) c = nil collectgarbage() collectgarbage() r:read(1) -- IllegalParams: io: request must be io ``` The patch introduces two functions: `httpc_env_finish` and `curl_env_finish`, that prepares curl and httpc environments for destruction. HTTP client's GC finalizer now calls `httpc_env_finish` instead of `httpc_env_destroy`, this prevents from destroying memory that could be in use by HTTP requests. Additionally `httpc_env_finish` sets a flag `cleanup`. HTTP environment destroying is called when flag `cleanup` is set and a there are no active HTTP requests. The main idea of the patch is a synchronization of destructors for HTTP client and HTTP chunked requests. Unfortunately, GC will eventually collect HTTP client object after calling its `__gc`. To prevent this we put a reference to a Curl's userdata in Lua objects with HTTP chunked requests and HTTP default client. Fixes tarantool#9346 Fixes tarantool#9453 NO_DOC=bugfix

`httpc` module has two GC-finalizers: the first one for a Lua http client (C function `luaT_httpc_cleanup`) and the second one for a Lua http chunked requests (C function `luaT_httpc_io_cleanup`) introduced in commit 417c6cb ("httpc: introduce stream input/output interface"). In a C implementation HTTP requests depends on structures of HTTP client and there is a problem with destroying Lua objects in `httpc` module - these GC-finalizers are not synchronized. This could lead to at least two problems: There is a race with GC-finalization that leads to use-after-free errors when HTTP client is collected before collecting HTTP request. In a stacktrace the problem looks as below: ``` 0x55ca7d47652e in crash_collect+256 0x55ca7d476f6a in crash_signal_cb+100 0x7fb876c42520 in __sigaction+80 0x55ca7d641e51 in curl_slist_free_all+35 0x55ca7d441498 in httpc_request_delete+45 0x55ca7d4653f1 in httpc_io_destroy+27 0x55ca7d4674bc in luaT_httpc_io_cleanup+36 0x55ca7d4e00c7 in lj_BC_FUNCC+70 0x55ca7d4f8364 in gc_call_finalizer+668 0x55ca7d4f8946 in gc_finalize+1387 0x55ca7d4f91e2 in gc_onestep+864 0x55ca7d4f9716 in lj_gc_fullgc+276 ... ``` Lua object `http.client` could be GC-collected when chunked HTTP request is alive. This will lead to an error "IllegalParams: io: request must be io" because we call a method when Lua object is already a `nil`. ```lua local url = 'https://bronevichok.ru/' local c = require('http.client').new() local r = c:get(url, {chunked = true}) c = nil collectgarbage() collectgarbage() r:read(1) -- IllegalParams: io: request must be io ``` The patch introduces two functions: `httpc_env_finish` and `curl_env_finish`, that prepares curl and httpc environments for destruction. HTTP client's GC finalizer now calls `httpc_env_finish` instead of `httpc_env_destroy`, this prevents from destroying memory that could be in use by HTTP requests. Additionally `httpc_env_finish` sets a flag `cleanup`. HTTP environment destroying is called when flag `cleanup` is set and a there are no active HTTP requests. The main idea of the patch is a synchronization of destructors for HTTP client and HTTP chunked requests. Unfortunately, GC will eventually collect HTTP client object after calling its `__gc`. To prevent this we put a reference to a Curl's userdata in Lua objects with HTTP chunked requests and HTTP default client. Fixes tarantool#9346 Fixes tarantool#9453 NO_DOC=bugfix (cherry picked from commit 17e9c6f)

`httpc` module has two GC-finalizers: the first one for a Lua http client (C function `luaT_httpc_cleanup`) and the second one for a Lua http chunked requests (C function `luaT_httpc_io_cleanup`) introduced in commit 417c6cb ("httpc: introduce stream input/output interface"). In a C implementation HTTP requests depends on structures of HTTP client and there is a problem with destroying Lua objects in `httpc` module - these GC-finalizers are not synchronized. This could lead to at least two problems: There is a race with GC-finalization that leads to use-after-free errors when HTTP client is collected before collecting HTTP request. In a stacktrace the problem looks as below: ``` 0x55ca7d47652e in crash_collect+256 0x55ca7d476f6a in crash_signal_cb+100 0x7fb876c42520 in __sigaction+80 0x55ca7d641e51 in curl_slist_free_all+35 0x55ca7d441498 in httpc_request_delete+45 0x55ca7d4653f1 in httpc_io_destroy+27 0x55ca7d4674bc in luaT_httpc_io_cleanup+36 0x55ca7d4e00c7 in lj_BC_FUNCC+70 0x55ca7d4f8364 in gc_call_finalizer+668 0x55ca7d4f8946 in gc_finalize+1387 0x55ca7d4f91e2 in gc_onestep+864 0x55ca7d4f9716 in lj_gc_fullgc+276 ... ``` Lua object `http.client` could be GC-collected when chunked HTTP request is alive. This will lead to an error "IllegalParams: io: request must be io" because we call a method when Lua object is already a `nil`. ```lua local url = 'https://bronevichok.ru/' local c = require('http.client').new() local r = c:get(url, {chunked = true}) c = nil collectgarbage() collectgarbage() r:read(1) -- IllegalParams: io: request must be io ``` The patch introduces two functions: `httpc_env_finish` and `curl_env_finish`, that prepares curl and httpc environments for destruction. HTTP client's GC finalizer now calls `httpc_env_finish` instead of `httpc_env_destroy`, this prevents from destroying memory that could be in use by HTTP requests. Additionally `httpc_env_finish` sets a flag `cleanup`. HTTP environment destroying is called when flag `cleanup` is set and a there are no active HTTP requests. The main idea of the patch is a synchronization of destructors for HTTP client and HTTP chunked requests. Unfortunately, GC will eventually collect HTTP client object after calling its `__gc`. To prevent this we put a reference to a Curl's userdata in Lua objects with HTTP chunked requests and HTTP default client. Fixes #9346 Fixes #9453 NO_DOC=bugfix (cherry picked from commit 17e9c6f)

ligurio added the bug Something isn't working label Dec 7, 2023

ligurio mentioned this issue Dec 7, 2023

Fix cleanup in http.client #9341

Merged

igormunkin closed this as completed in 17e9c6f Dec 22, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

http client object is GC-collected when http request object is alive #9453

http client object is GC-collected when http request object is alive #9453

ligurio commented Dec 7, 2023

http client object is GC-collected when http request object is alive #9453

http client object is GC-collected when http request object is alive #9453

Comments

ligurio commented Dec 7, 2023