Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Race conditions may occur when multiple RRDtool processes are running #4347

Closed
dwpaulx opened this issue Jul 14, 2021 · 2 comments
Closed

Race conditions may occur when multiple RRDtool processes are running #4347

dwpaulx opened this issue Jul 14, 2021 · 2 comments
Labels
bug Undesired behaviour resolved A fixed issue
Milestone

Comments

@dwpaulx
Copy link
Contributor

dwpaulx commented Jul 14, 2021

Describe the bug

There is a race in proc_open in PHP that is currently not avoided by
setting FD_CLOEXEC on the pipe file descriptors (with the 'e' flag for
the pipes). What happens, in a multithreaded Apache environment, when
multiple rrdtools are attempted to be launched (for example, when a lot
of graphs are requested at once) is this:

Thread 1: creates pipes for stdin/stdout
Thread 2: creates pipes for stdin/stdout
Thread 1: forks process x
Thread 2: forks process y
Process x: inherits pipes from Thread 1 and Thread 2
Process y: inherits pipes from Thread 1 and Thread 2
Process x: closes 'other side' of pipes from Thread 1
Process x: executes rrdtool
Thread 1: closes 'other side' of its pipes
Thread 1: writes command to rrdtool (no trailing newline)
Thread 1: closes stdin pipe to Process x
Process x: receives command from Thread 1
Process x: waits for more data, or stdin to close
Process x: hangs, because the pipe is still open in Process y
Thread 1: waits for data from rrdtool, which never comes
(meanwhile, the same thing happens between Thread 2 and Process y)

This is really a PHP bug, and I'll see if there's a possibility to get
that fixed there, but this at least fixes the problem by allowing
rrdtool to exit, itself, instead of waiting for stdin to be closed.

It seems related to this PHP issue:
https://bugs.php.net/bug.php?id=70932
but the PHP FD_CLOEXEC implementation also has races and does not fix this problem.

To Reproduce

Steps to reproduce the behavior:

Open a graph display page with many graphs. Set the number of columns to 6, maximum graphs to 100.

Wait.

Eventually, with the automatic refresh, there will be hanging processes. strace on them show they are waiting on read on stdin. The graphs related to these do not load, and the refresh will stall.

Expected behavior

Graphs to be plotted correctly, and for rrdtool processes not to hang.

Desktop (please complete the following information)

Gentoo Linux, Apache 2.4.48 with event MPM
PHP 7.4.21

Additional context

Possibly related to this forum thread : https://forums.cacti.net/viewtopic.php?f=2&t=61643

I will be sending a pull request soon with a workaround that seems to be working for me.

@dwpaulx dwpaulx added bug Undesired behaviour unverified Some days we don't have a clue labels Jul 14, 2021
dwpaulx added a commit to dwpaulx/cacti that referenced this issue Jul 14, 2021
…l processes

There is a race in proc_open in PHP that is currently not avoided by
setting FD_CLOEXEC on the pipe file descriptors (with the 'e' flag for
the pipes). What happens, in a multithreaded Apache environment, when
multiple rrdtools are attempted to be launched (for example, when a lot
of graphs are requested at once) is this:

Thread 1: creates pipes for stdin/stdout
Thread 2: creates pipes for stdin/stdout
Thread 1: forks process x
Thread 2: forks process y
Process x: inherits pipes from Thread 1 and Thread 2
Process y: inherits pipes from Thread 1 and Thread 2
Process x: closes 'other side' of pipes from Thread 1
Process x: executes rrdtool
Thread 1: closes 'other side' of its pipes
Thread 1: writes command to rrdtool (no trailing newline)
Thread 1: closes stdin pipe to Process x
Process x: receives command from Thread 1
Process x: waits for more data, or stdin to close
Process x: hangs, because the pipe is still open in Process y
Thread 1: waits for data from rrdtool, which never comes
(meanwhile, the same thing happens between Thread 2 and Process y)

This is really a PHP bug, and I'll see if there's a possibility to get
that fixed there, but this at least fixes the problem by allowing
rrdtool to exit, itself, instead of waiting for stdin to be closed.
@dwpaulx
Copy link
Contributor Author

dwpaulx commented Jul 14, 2021

I forgot to mention that you can see this happening when there are hung rrdtool processes by looking in /proc/PID/fd ... and you can see the stdin pipes from the other rrdtool processes from the 'hung group' open in the other rrdtool processes, even though they have been closed in the Apache processes (by the fclose after the fwrite)

@netniV netniV added resolved A fixed issue and removed unverified Some days we don't have a clue labels Sep 5, 2021
@netniV netniV added this to the 1.2.19 milestone Sep 5, 2021
netniV pushed a commit that referenced this issue Sep 5, 2021
…cesses (#4348)

There is a race in proc_open in PHP that is currently not avoided by
setting FD_CLOEXEC on the pipe file descriptors (with the 'e' flag for
the pipes). What happens, in a multithreaded Apache environment, when
multiple rrdtools are attempted to be launched (for example, when a lot
of graphs are requested at once) is this:

Thread 1: creates pipes for stdin/stdout
Thread 2: creates pipes for stdin/stdout
Thread 1: forks process x
Thread 2: forks process y
Process x: inherits pipes from Thread 1 and Thread 2
Process y: inherits pipes from Thread 1 and Thread 2
Process x: closes 'other side' of pipes from Thread 1
Process x: executes rrdtool
Thread 1: closes 'other side' of its pipes
Thread 1: writes command to rrdtool (no trailing newline)
Thread 1: closes stdin pipe to Process x
Process x: receives command from Thread 1
Process x: waits for more data, or stdin to close
Process x: hangs, because the pipe is still open in Process y
Thread 1: waits for data from rrdtool, which never comes
(meanwhile, the same thing happens between Thread 2 and Process y)

This is really a PHP bug, and I'll see if there's a possibility to get
that fixed there, but this at least fixes the problem by allowing
rrdtool to exit, itself, instead of waiting for stdin to be closed.
netniV pushed a commit that referenced this issue Sep 5, 2021
…cesses (#4348)

There is a race in proc_open in PHP that is currently not avoided by
setting FD_CLOEXEC on the pipe file descriptors (with the 'e' flag for
the pipes). What happens, in a multithreaded Apache environment, when
multiple rrdtools are attempted to be launched (for example, when a lot
of graphs are requested at once) is this:

Thread 1: creates pipes for stdin/stdout
Thread 2: creates pipes for stdin/stdout
Thread 1: forks process x
Thread 2: forks process y
Process x: inherits pipes from Thread 1 and Thread 2
Process y: inherits pipes from Thread 1 and Thread 2
Process x: closes 'other side' of pipes from Thread 1
Process x: executes rrdtool
Thread 1: closes 'other side' of its pipes
Thread 1: writes command to rrdtool (no trailing newline)
Thread 1: closes stdin pipe to Process x
Process x: receives command from Thread 1
Process x: waits for more data, or stdin to close
Process x: hangs, because the pipe is still open in Process y
Thread 1: waits for data from rrdtool, which never comes
(meanwhile, the same thing happens between Thread 2 and Process y)

This is really a PHP bug, and I'll see if there's a possibility to get
that fixed there, but this at least fixes the problem by allowing
rrdtool to exit, itself, instead of waiting for stdin to be closed.
@netniV
Copy link
Member

netniV commented Sep 5, 2021

This has now been committed and I've manually applied it to 1.2.x too.

@netniV netniV closed this as completed Sep 5, 2021
@netniV netniV changed the title Filehandle race with proc_open results in blocked rrdtool processes while plotting many graphs Race conditions may occur when multiple RRDtool processes are running Oct 3, 2021
@github-actions github-actions bot locked and limited conversation to collaborators Jan 2, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Undesired behaviour resolved A fixed issue
Projects
None yet
Development

No branches or pull requests

2 participants