Filter out bad caption files

[dphoria]

Feature Description

{"generator": "CDP WebVTT Conversion -- CDP v3.2.3", "confidence": 0.97, "session_datetime": "2022-09-14T10:00:00-04:00", "created_datetime": "2022-09-17T15:51:32.537666", "sentences": [{"index": 0, "confidence": 0.97, "start_time": 0.0, "end_time": 0.0, "words": [], "text": "", "speaker_index": 0, "speaker_name": null, "annotations": null}], "annotations": null}

Note end_time in the above output from a PIttsburgh, PA captions processing. Turns out that event does have a caption file in the Legistar data structure for the event but it is in fact empty.

We want to filter these out and leave Session.caption_uri = None.

Use Case

Better to avoid wasting resources to process these invalid caption files.

Solution

Compare the lengths of the video and the caption file. If they differ by more than some threshold, e.g. 20%, throw away the caption file.
This means the scraper can no longer just hand off the caption file URL as-is.

Alternatives

• Throw away caption files with file size less than some threshold, e.g. 100 bytes.
• Throw away caption files less than ~1 minute.

[dphoria]

The more I think about this, I prefer not comparing against the associated video. Mostly, I don't want to have to handle all the different video formats. So I prefer just to do some very simple stupid validation on the caption file, alone.

[evamaxfield]

I would say do this on the cdp-backend side where we already have ffmpeg stuff enabled.

which means that something like this should work: https://stackoverflow.com/a/3844467

[dphoria]

I would say do this on the cdp-backend side where we already have ffmpeg stuff enabled.

Ah, good idea.

[dphoria]

CouncilDataProject/cdp-backend#213