Add --interval to launch monitor command

[gamuniz]

SUMMARY

This allows passing in an integer(seconds) to the launch option in awxkit to rate limit the api calls for longer running jobs or to avoid overwhelming the api

example:

awx workflow launch 8 --monitor --interval 5

nginx logs:

10.244.0.1 - - [01/Jun/2023:03:12:13 +0000] "GET /api/v2/unified_jobs/?order_by=finished&unified_job_node__workflow_job=22 HTTP/1.1" 200 5981 "-" "python-requests/2.28.1" "-"

10.244.0.1 - - [01/Jun/2023:03:12:18 +0000] "GET /api/v2/workflow_jobs/22/ HTTP/1.1" 200 1740 "-" "python-requests/2.28.1" "-"

10.244.0.1 - - [01/Jun/2023:03:12:24 +0000] "GET /api/v2/unified_jobs?order_by=finished&unified_job_node__workflow_job=22 HTTP/1.1" 301 5 "-" "python-requests/2.28.1" "-"

ISSUE TYPE

• New or Enhanced Feature

COMPONENT NAME

• CLI

AWX VERSION

make VERSION 
awx: 22.3.1.dev19+gb0068caf41

ADDITIONAL INFORMATION

[hluk]

I have a bit different solution in #14069. It additionally defines default and minimum value.

I think having a larger default interval would help us making most clients behave nicely to our server.

[hluk]

The unit is not mentioned here. Assuming it is in seconds but the default value type below is float.

[AlanCoding]

In either case, I see that hard-coded 0.25 and strongly support fixing that. Up to @gamuniz how to move forward with it.

[hluk]

👍 This would fix the problems I'm experiencing.

One minor issue is that user can set (for whatever reason) a negative value for the interval which would cause ValueError in time.sleep(). Simple fix would be to limit it to values>0 (though I would still prefer a higher limit if possible): time.sleep(max(0.0, interval))

[john-westcott-iv]

Do you really want a float instead of an int here? It should work, just feels weird.

[hluk]

With the new update and the minimum value set to 2.5, I think it would be a good idea to make the argument int (and minimum an integer larger than 0). Hopefully, nobody needs faster updates - it would make printing output faster/smoother but potentially overwhelming the server again.

[jangel97]

Hello team,

Please, could this be reviewed my some maintainer when possible? This feature would be very interesting for many teams who are using awxkit to span jobs from their CICD pipelines.

Thank you!

[jay-steurer]

Running some tests, I will merge once is done.

[rmahroua]

Hi chiming in -- we've had some performance issues with large systems and the best way I've found to alleviate the load is to implement retries with exponential backoff.
Pseudo-implementation:

def retry_backoff(retries: int = 15, backoff: float = 1.0, debug: bool = False) -> object:
    def rb(f):
        def wrapper(**kwargs):
            count = 0
            while True:
                try:
                    return f(**kwargs), None
                except <API EXCEPTION HERE> as e:
                    if count == retries - 1:
                        return False, None
                    else:
                        sleep = (count ** 1.5 * backoff + random.uniform(0, 1))
                        time.sleep(sleep)
                        count += 1
        return wrapper
    return rb

With that, the AWX API should be able to handle a higher volumes of transaction. Let me know if you'd be open to go this route and I'd be happy to send a MR that builds on @gamuniz's work.