How can I watch the videos result?

[Ayelets]

and also how can i use the model for summarize new videos?
many thanks

[ok1zjf]

Hi Ayelets,

To summarize your own video you need to first extract the video frames as images and generate change points (cps) with the KTS method.

VASNet/cpd_auto.py

Line 4 in c378753

def cpd_auto(K, ncp, vmax, desc_rate=1, **kwargs):

Then subsample the video sequence to 2fps and for each frame extract CNN features (1024 dim from the pool5 layer) with the GoogLeNet trained on ImageNet. Features for the entire video sequence are then provided as the input to the model.

To produce the video summarization, first use the model to generate frame importance scores y

VASNet/main.py

Line 294 in c378753

y, att_vec = self.model(seq, seq.shape[1])

and then generate keyshot summarization by calling

VASNet/main.py

Line 326 in c378753

machine_summary = generate_summary(probs, cps, num_frames, nfps, positions)

The machines_summary is a binary array indicating which video frame is a part of the summary. You can
use this information to create the summarized video with your favourite image/video processing toolkit,
such as OpenCV or moviepy. I hope this helps.

Kind Regards
Jiri

[wjb123]

@ok1zjf how to use KTS to generate change points ? I use the official KTS code and employ CNN feature for each frame, but get same number of segments for every video. Is there any problem?

[BarCodeReader]

Hi Ayelets,

To summarize your own video you need to first extract the video frames as images and generate change points (cps) with the KTS method.

VASNet/cpd_auto.py

Line 4 in c378753

def cpd_auto(K, ncp, vmax, desc_rate=1, **kwargs):

Then subsample the video sequence to 2fps and for each frame extract CNN features (1024 dim from the pool5 layer) with the GoogLeNet trained on ImageNet. Features for the entire video sequence are then provided as the input to the model.
To produce the video summarization, first use the model to generate frame importance scores y

VASNet/main.py

Line 294 in c378753

y, att_vec = self.model(seq, seq.shape[1])

and then generate keyshot summarization by calling

VASNet/main.py

Line 326 in c378753

machine_summary = generate_summary(probs, cps, num_frames, nfps, positions)

The machines_summary is a binary array indicating which video frame is a part of the summary. You can
use this information to create the summarized video with your favourite image/video processing toolkit,
such as OpenCV or moviepy. I hope this helps.
Kind Regards
Jiri

i think we need to input GoogLeNet feature map into VASNet to generate frame score first, then feed in the score (1 dimensional vector) into KTS to do segmentation???

[ok1zjf]

Yes, you need to extract the video frames as images (e.g. jpg) and then generate the change points with KTS. The input to the KTS are indeed the GoogLeNet features.
Best
Jiri

[AdithyaSireesh]

Then subsample the video sequence to 2fps and for each frame extract CNN features (1024 dim from the pool5 layer) with the GoogLeNet trained on ImageNet. Features for the entire video sequence are then provided as the input to the model

Hi,

for the probabilities parameter in the generate_summary function, are we meant to input the probabilities for subsampled frames, or the whole video?

Thanks,
Adi