Video Pipeline/Training Support

[BryonLewis]

Not the best way to solve the issue but the least intrusive way for the time being.

Pipeline Running Updates:

• image-sequence for training will now filter out .json files
• video type will now load the folder filter out any non compatible video files and then look for the video file which doesn't have the codec meta tag on it. I would like to share the validMediaFiles with the constants folder or maybe do this filtering before calling the task, I just didn't want to go with too big of a change right now with just getting video training to work.

CSV Metadata Update:

• Added in the writing of the metadata to the header. Right not it only writes the fps value for the training and other items to user. More stuff can be added in the future, just trying to keep it fairly simple right now.
• NOTE!!!! - The resetting of fps = None is there because of a training bug right now. Once fixed that will be removed.

Training Updates:

• In the beginning of the training I updated the CSV creation to be more like export and include stuff like the fps so it knows to write it.
• Training will use the same function to find the source video file and place that in the training manifest file. This means the file is directly referenced with the included extensions as well.

[subdavis]

All the conventional wisdom I'm aware of is telling me that judging files by their extension is a bad idea. We've done a large amount of work to make sure whatever garbage files a user might upload get forced through our sieve and come out sanitized, safe video, and this is a step in the wrong direction.

Proposal

If we want to move to using the raw uploaded video, we should go through the process of having ffprobe verify the file (we already do this), tag that file, and then use it.

Then, the logic could look like this:

if tagged safe original video exists,
  send it to training,
else if tagged safe transcoded video exists:
  send it to training
else 
  throw an error about no safe videos.

[subdavis]

Unnecessary branching/default. Could we just make metadata a required positional arg and remove the if?

[subdavis]

This function is kinda dangerous and should be documented

[subdavis]

Might as well make it a set instead of a list so that when the refactor happens, it's a drop-in replacement.

[subdavis]

I don't like the strategy of "choose the thing that isn't tagged but has the right file extension".

It's fragile. I think the transcoded file should be the thing we actually transmit, not the original source.

If a user uploads a txt file named .mp4, this will explode. However, if they used the tagged transcoded video, it's basically bulletproof because transcoding will only succeed on a well-formed video.

Transcoding is basically a litmus test for valid video before training runs, so I think we should use the "safer" version.

We can slightly relax our compression on transcoding if that would help. I really don't think this is a good idea.

[BryonLewis]

So I removed the extension checking and now it checks for:

• meta: {source_video: true}
• Then meta: {codec: 'h264'}
• Then it will raise an exception if no video is found

Added the itemId to the convert_video, it was just the easiest way to add the metadata for {source_video:True} if it properly converts the video instead of iterating over the folder and finding the item by name/path.

Updated the get_source_video_filename function to specifically just get the filename of the source video or fall back to the transcoded version if the source one can't be found. Leave it up to where it is used to raise an error and handle the filepath joining.

My Testing process:

• Importing new videos and making sure the metadata was tagged properly.
• Running pipeline on old video and on new video. Checking the job output to make sure it referenced the proper file.
• Running training on both old and newly uploaded videos. Again checked the job output to make sure each folder was using the proper video (transcoded for old video, source video for new ones)
• Deleted the source_video and codec metadata and attempted to run pipelines and training. Error was properly raised in both instances
• Also double checked the metadata output when exporting detections and that seemed to reflect the fps that was set properly. The updated one will include a # metadata - line for non videos but I don't think that should be a problem.

Only issue I see is if there is a super old video which doesn't have the {codec: 'h264'} tag on it but I think that will be rare.

[subdavis]

Seems like it would be prudent to go ahead and tag every video we've verified with codec.

Now, codec can just plainly mean codec, instead of carrying the extra implied "BTW this is the trancoded video".

Note that you also have to explicitly query for source_video does not exist in _get_clip_meta

item = Item().findOne(
            {
                'folderId': folder['_id'],
                'meta.codec': 'h264',
                'meta.source_video': { '$exists': False },
            }
        )

[subdavis]

Suggested change

	gc.addMetadataToItem(itemId, {"source_video": True})
	gc.addMetadataToItem(
	itemId,
	{
	"source_video": True,
	"codec": videostream[0]["codec_name"],
	},
	)

[subdavis]

I'm testing this myself now, but I have to download the newer gpu algorithms.