Concurrent ingest requests result in 1 request failing

[donaldgray]

We need to validate that this is an issue, it looks to be the case from the logs. The final error stored for an image is something like:

Appetiser Error: [Errno 2] No such file or directory: '/scratch/2/5/b22365965_0078.jp2/output/thumbs/b22365965_0078.jp2_400.jpg'

If there are multiple consecutive synchronous ingest requests for an Image sent to the same engine instance then there's a possibility of one ending in error as they both share the same local folder structure.

I suspect what's happening is:

1. Request1: ImageServerClient creates directories for image processing (for user with Appetiser)
   • Default pattern is {root}/{customer}/{space}/{image}/output/
2. Request1: call appetiser + initiate processing
3. Request2: ImageServerClient creates local directories for image processing (for user with Appetiser)
   • Default pattern is {root}/{customer}/{space}/{image}/output/
4. Request1: appetiser finished, process output.
   • ImageServerClient has a finally block that will delete {root}/{customer}/{space}/{image}/output/
5. Request2: appetiser finished, process output.
   • Expected directory is deleted. Exception. This results in "Error" being set in the Asset despite it successfully processing moments ago.

The route issue is that the folder generated at (1) and (3) is the same. We should add some sort of random slug to the path, which would allow multiple synchronous ingests.

Q: Do we want to reject requests for assets that are "Ingesting"? If so could an asset be stuck in an "Ingesting" state?
A: No, do not reject requests. Handle them - even if we're donig extra work.

This is not an issue for asynchronous ingests as messages are processed 1 at a time.

ImageServerClient was previously AppetiserClient

[tomcrane]

The random slug could be DateTime.Now.Ticks

[tomcrane]

Acceptance criteria:

Demonstrate that the error can be triggered ("We need to validate that this is an issue") by scripting simultaneous ingests on the same asset. This can be a standalone python or JS or whatever.

If that is the case, implement the change suggested (unique directory name)

Then re-test to see if the problem has gone.