FIFO pipe code will hang indefinitely preventing any logging output

[hans2520]

Originally posted by @hans2520 in #10489 (comment)

In debugging the above issue, I wrote some code that could lead to better reliability with the open_fifo_write method as well as debuggability.

This method is critical in the startup of an AWX playbook job, and if it hangs, it will give the user no indication whatsoever as to what is happening.

The below code was just my start at improving the method, before the ultimate root cause in my issue (a recent Crowdstrike policy update deploying script-based execution monitoring that blocked the FIFO pipe in the first place) was discovered. The basic goal of the improvement is to force-kill the job if it doesn't finish after a certain allotted time, capture the error, and thereby give the user something useful to work with as to the cause. Also, the flushing here should help reliability some especially with larger keys.

But it's a work in progress for someone else to finish. In particular, the join will not have the intended effect if indeed the pipe hasn't finished its writing task. Use of the 'daemon' mode may be required here, in addition, consider using multiprocessing library here as opposed to Thread, as that is more likely better suited for this task.

def open_fifo_write(path, data):
    '''open_fifo_write opens the fifo named pipe in a new thread.
    This blocks the thread until an external process (such as ssh-agent)
    reads data from the pipe.
    '''
    logger.debug(f'Pipe write! path: {path}')
    try:
        os.mkfifo(path, stat.S_IRUSR | stat.S_IWUSR)
        def write_data(p, d):
            with open(p, 'wb') as f:
                f.write(d)
                f.flush()
        t1 = threading.Thread(target=write_data, args=(path, data))
        t1.start()
        t1.join(3)
        logger.debug(f"thread {t1} finished writing")
    except Exception as e:
       logger.error(e)
       logger.exception(e)

[craph]

I have a similar issue #11518 and it's seems liked to crowdstrike too but only when AWX is on K8S. With an older version (17.x) installed in local Docker and crowdstrike running too the project update is working.

Do you have any idea of the changes between version 17.x and 19.x about the playbook processing for the project updates ?

[hans2520]

This area of code is unchanged from version 5.0.0, where it was originally found

[craph]

This area of code is unchanged from version 5.0.0, where it was originally found

Why are you saying it was found in version 5.0.0 ? I don't see this in the referenced issue. It refers version 19.2.1 in #10489

[hans2520]

The code in this issue was found in v5.0.0 and is unchanged thru the latest code. It is likely in even an earlier version as well.

…

On Tue, Jan 11, 2022, 2:34 PM craph ***@***.***> wrote: This area of code is unchanged from version 5.0.0, where it was originally found Why are you saying it was found in version 5.0.0 ? I don't see this in the referenced issue. It refers version 19.2.1 in #10489 <#10489> — Reply to this email directly, view it on GitHub <#11453 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIINJPAK7SBFHZTOYGV25OTUVSH6ZANCNFSM5J7MIUQA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[craph]

@hans2520 Please can you give me the file where the issue is ?

[hans2520]

The file is linked in the issue description