Chowda sometimes does not finish processing media files, but reports no error

[owencking]

Description

When I run a CLAMS app on a media file, Chowda should report success only on the condition that the media file was completely processed. Right now, I can download MMIF even if the CLAMS app in question only got through part of the file.

Reproduction steps

From my Slack message about this:

A few days ago I ran ten identical single-guid batches in Chowda, because the Brandeis team and I were trying to measure how much indeterminacy there was in our image classification model. Those were batches 39-47.

Well, I was just going through the results. The output MMIF from those batches differs a ton, and not due to variability in the classifier. The MMIF varies just in terms of how much of the video was completed. The video is about 31 minutes long. But some of the runs didn't make it past 5 minutes, and only one batch out of the ten made it all the way through the video.

Expected behavior

Ideally, it should get fully through every file. However, in cases where it doesn't, it should not fail silently. It should alert the Clammer that it didn't get all the way through.

Screenshots

No response

Browsers

No response

OS

No response

Additional context

By my lights, the most serious thing about this issue is that it's hard to detect. The main reason I noticed it was that I just happened to be comparing several runs that should have had very similar output. If it hadn't been for that, it might have been awhile before I noticed. It doesn't result in any overt failures or error messages.

[mrharpo]

As far as I can tell, there's no indication any clams apps are erroring. Each app is responding with a 200 code, which means exceptions are being caught somewhere instead of raised. My guess is in the Python SDK or the app code.