Image requests in Engine backing up

[donaldgray]

The logs indicate that a flurry of synchronous ingest requests for Images will start to backup and take longer and longer to respond to subsequent requests.

The following logs are from API and show the results of a series of /reingest - each subsequent request takes longer and longer to respond.

2024-05-01 12:45:16.920 responded 500 in 59996.1585 ms
2024-05-01 12:45:16.917 responded 500 in 60000.0734 ms
2024-05-01 12:45:16.911 responded 500 in 60007.7099 ms
2024-05-01 12:44:58.464 responded 500 in 60002.2894 ms
2024-05-01 12:44:58.439 responded 500 in 59999.4567 ms
2024-05-01 12:44:58.431 responded 500 in 59999.2968 ms
2024-05-01 12:44:55.782 responded 500 in 59999.5084 ms
2024-05-01 12:44:55.115 responded 200 in 59334.3579 ms
2024-05-01 12:44:55.043 responded 200 in 59270.2620 ms
2024-05-01 12:44:53.923 responded 200 in 58142.8098 ms
2024-05-01 12:44:48.831 responded 200 in 55516.8583 ms
2024-05-01 12:44:45.012 responded 200 in 55328.3890 ms
2024-05-01 12:44:44.335 responded 200 in 54682.0420 ms
2024-05-01 12:44:39.844 responded 200 in 54221.9620 ms
2024-05-01 12:44:38.832 responded 200 in 53194.3398 ms
2024-05-01 12:44:36.025 responded 200 in 50911.8482 ms
2024-05-01 12:44:32.211 responded 200 in 47141.3037 ms
2024-05-01 12:44:30.105 responded 200 in 45021.3106 ms
2024-05-01 12:44:28.145 responded 200 in 43076.4135 ms
2024-05-01 12:44:25.541 responded 200 in 40469.5709 ms
2024-05-01 12:44:22.012 responded 200 in 36939.2480 ms
2024-05-01 12:44:20.011 responded 200 in 38042.9406 ms
2024-05-01 12:44:16.511 responded 200 in 34526.9843 ms
2024-05-01 12:44:15.908 responded 200 in 33889.4540 ms
2024-05-01 12:44:11.863 responded 200 in 29895.1371 ms
2024-05-01 12:44:06.808 responded 200 in 24847.6208 ms
2024-05-01 12:44:05.807 responded 200 in 26680.2945 ms
2024-05-01 12:44:05.250 responded 200 in 23272.6060 ms
2024-05-01 12:44:04.212 responded 200 in 25085.6742 ms
2024-05-01 12:43:56.815 responded 200 in 17693.2700 ms
2024-05-01 12:43:53.624 responded 200 in 14484.6532 ms
2024-05-01 12:43:50.260 responded 200 in 11136.7257 ms

I suspect that the issue with appetiser - possibly how it makes subproc calls to kdu. We'll need to investigate to find out what is happening. If it is appetiser, can we fix? Or is there anything we can do in the Engine or API? E.g. rate-limiting synchronous ingest requests for per customer? Over a period of time, or X number allowed concurrently?

Is this in any way related to #839?

[donaldgray]

if Appetiser related, was it better in version 1.3.0? 1.4.0 bumped a few pypi packages