textadept_11.4_alpha get stuck on filter_through large file

[amitbha]

OS: Manjaro 21.2.5
Version: textadept_11.4_alpha

The file The_Mutt_manual.md has 6900+ lines. While press Ctrl+| to filter_through cat -s to reduce blank lines, the textadept get stuck.

Use strace to check what happened:

• cat -s

strace: Process 648747 attached
write(1, "\200\234To:\342\200\235 header field when you h"..., 65536

• textadept (the last lines)

...
poll([{fd=3, events=POLLIN}, {fd=4, events=POLLIN}, {fd=8, events=POLLIN}], 3, -1) = 1 ([{fd=3, revents=POLLIN}])
recvmsg(3, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="U\2D\376{\355W\5\3\20\0\0\20\0\0\0\0\0\0\20\20\20\20\20\0\0\3\37%\3\0\0", iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 32
recvmsg(3, {msg_namelen=0}, 0)          = -1 EAGAIN (Resources temporarily unavailable)
recvmsg(3, {msg_namelen=0}, 0)          = -1 EAGAIN (Resources temporarily unavailable)
poll([{fd=3, events=POLLIN}, {fd=4, events=POLLIN}, {fd=8, events=POLLIN}], 3, -1) = 1 ([{fd=3, revents=POLLIN}])
recvmsg(3, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="W\0D\376\1\0\200\2\v\1`\2\1\0\0\0\264\tX\5\263\tX\5\0\0\0\0\0\0\0\0", iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 32
recvmsg(3, {msg_namelen=0}, 0)          = -1 EAGAIN (Resources temporarily unavailable)
recvmsg(3, {msg_namelen=0}, 0)          = -1 EAGAIN (Resources temporarily unavailable)
poll([{fd=3, events=POLLIN}, {fd=4, events=POLLIN}, {fd=8, events=POLLIN}], 3, -1) = 1 ([{fd=3, revents=POLLIN}])
recvmsg(3, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="W\0D\376\1\0\200\2\257\2\240\2\1\0\0\0e\304X\5_\304X\5\0\0\0\0\0\0\0\0", iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 32
recvmsg(3, {msg_namelen=0}, 0)          = -1 EAGAIN (Resources temporarily unavailable)
recvmsg(3, {msg_namelen=0}, 0)          = -1 EAGAIN (Resources temporarily unavailable)
poll([{fd=3, events=POLLIN}, {fd=4, events=POLLIN}, {fd=8, events=POLLIN}], 3, -1) = 1 ([{fd=3, revents=POLLIN}])
recvmsg(3, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="W\0D\376\1\0\200\2\257\2\240\2\1\0\0\0\267\365X\5\263\365X\5\0\0\0\0\0\0\0\0", iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 32
recvmsg(3, {msg_namelen=0}, 0)          = -1 EAGAIN (Resources temporarily unavailable)
recvmsg(3, {msg_namelen=0}, 0)          = -1 EAGAIN (Resources temporarily unavailable)

[orbitalquark]

Thanks for the report. I'll take a look when I have some time.

[orbitalquark]

After looking into this, I have concluded that it is an OS-specific issue with GTK's process spawning capabilities with no clear workaround. For commands that emit stdout while reading stdin (as opposed to emitting stdout after stdin is closed), if the stdout pipe/buffer gets full, input can no longer be read. In my tests on Linux, Textadept is hanging on writing input to the process after that process has emitted a little over 100K to stdout. I don't know if there is a heuristic that can be applied such that stdin is broken up into chunks (e.g. 64K blocks) and stdout read every now and then vs. the existing approach of passing all stdin and reading stdout after the fact. I don't know what to do here, sorry :(

[amitbha]

Thanks awfully!
Any possible to use multithreading for write and read? Passing all stdin and reading stdout at the same time.
I am new to lua, can Coroutine be a solution?

[orbitalquark]

I have committed some warnings on this by the way: 2d48870

[orbitalquark]

For what it's worth the Qt version of Textadept does not have this issue. It is currently available as a nightly release and will be generally available in the next 12.0 release.

[orbitalquark]

I think the closest you can come to async read/write would be to do something like this (untested): local stdout = {} local function handle_stdout(s) stdout[#stdout + 1] = s end local p = os.spawn('cat -s', handle_stdout) for i = 1, #input, 65535 do p:write(input:sub(i, i + 65535)) ui.update() -- will call handle_stdout if any is available end p:wait() stdout = table.concat(stdout) `input` is a really large string, and `stdout` ends up holding the entire stdout of the process. This will only work if the spawned process emits stdout while reading stdin. The above example emulates coroutines because `ui.update()` passes control to GTK, which invokes the process stdout handler, and then passes back control to Lua when done.

[amitbha]

I need some time to digest it.

This will only work if the spawned process emits stdout while reading stdin.

Problem is related to specific commands?

[orbitalquark]

Oops, I forgot that you need `p:close()` before `p:wait()`. This ends an EOF to the process stdin.

[orbitalquark]

That is correct. Only those types of commands need to be handled specially.