-
Notifications
You must be signed in to change notification settings - Fork 455
Labels
Description
Koala-36M proposes a significantly improved model for scene transition detection (paper: HTML or PDF)
Of interest is that this algorithm should be able to handle dissolve transitions more effectively, and assuming the numbers in the paper are correct, has vastly improved precision and recall.
See section 4.1 which uses an SVM classifier. The performance degredation is just over 2x slowdown, however the accuracy, precision, and recall show marked improvements across the board that likely warrant this change for the majority of users.
Metadata
Metadata
Assignees
Labels
Projects
Milestone
Relationships
Development
Select code repository
Activity
wjs018 commentedon Oct 17, 2024
The table of results looks promising and the approach is interesting. They haven't published their code yet (coming soon according to their homepage). Ideally, they could also release a pre-trained SVM model that could simply be imported and used.
Detection Algorithm
The bit about this that I don't understand is how they are using temporal information. Looking through this, here is my understanding of how this algorithm works:
d_color
metric is simply the correlation between sequential frames' rgb histograms. This is the exact same metric used byHistogramDetector
in PySceneDetect.d_struct
is calculated by combining the maximum value of each pixel for the current frame of either the grayscale image or the image after a Canny edge filter (equation 3 in section 4.1). Then, the result of this is compared to the previous frame's equation 3 output by way of SSIM (I thinkskimage
has an implementation of this).What I don't understand is the temporal information. They write:
So, I get that they are looking backwards X number of frames, finding the standard deviation, and calculating the current frame's Z-score. However, are they calculating the Z-score for both of the above metrics? An average of the two? Some other metric? It isn't clear. Additionally, it doesn't seem like this temporal information is used by the SVM since they explicitly say that the SVM only takes the two parameters calculated above. So, is something marked as a scene if the SVM classifies it as such or the frame's Z-score exceeds 3? I am not sure how to incorporate the temporal information here.
SVM Training
This is an area that would be quite onerous to do on our own, so if they do release a model, it would be a tremendous help. When describing how their SVM is trained, they write (from 4.1):
I am assuming here that they are using their giant dataset for this. If I am trying to extract their data generation method from their very brief description, then it would be something like this:
d_color
andd_struct
)d_color
andd_struct
)The same method could be used to generate your test data as well. This would only be possible with a curated dataset like theirs that consist of single-scene videos.
Their dataset seems to be a giant list of youtube videos with timestamps denoting the start and stop points of each of those youtube videos, segmenting what part of the video is included in the dataset.
If we wanted to train our own SVM based on this dataset, it would be a huge task to reconstruct their dataset based off of the youtube urls and timestamps. Additionally, they don't really give any insight on the SVM parameters. I am far from an ML expert, so having some additional information on any of the options used for the SVM would be helpful for replication.
Breakthrough commentedon Oct 18, 2024
I'm curious what it would look like if we plotted the values for
d_color
andd_struct
on a few videos to see if any obvious patterns emerge. If the data can be fitted by a typical kernel function we may just need to get the coefficients correct. I also found across the additive chi squared kernel which seems like a feature map which can be trained in linear time, but does require training. The scikit SSIM function isskimage.metrics.structural_similarity
.This section is also very unclear to me as well, and you raise some good questions. Hopefully they will publish some more information soon.
wjs018 commentedon Oct 18, 2024
I am pretty sure that
d_color
is literally the same metric thatHistogramDetector
already uses. Calculatingd_struct
doesn't look too bad, but would probably need to add anskimage
dependency.In doing some searching around about this, I also ran across CLIP. This is a pre-trained deep-learning transformer that can measure similarity between two images. I found a Stack Overflow answer that has a great explanation with some examples. This might be an alternative similarity metric to SSIM. It would also require new dependencies though, and I have no idea what the computational efficiency would be.
[detectors] Implement Koala-36M
[detectors] Implement Koala-36M
[detectors] Implement Koala-36M
[detectors] Implement Koala-36M
[detectors] Implement Koala-36M
[detectors] Implement Koala-36M
Breakthrough commentedon Nov 24, 2024
I have something up and running in #459 if folks are interested in testing it out. You can find Python packages and Windows builds in that PR. The new detector is called
detect-koala
. Note that it will say there are 0 scenes detected until the very end, but it should still provide results.Anyone willing to test this out is encouraged to do so and provide feedback here. You can use the new detector the same as any other one, e.g.
scenedetect -i video.mp4 detect-koala
On most of our test videos it provides identical results to the other fast-cut detectors:
Note that via the Python API the detector is called
KoalaDetector
.[detectors] Implement Koala-36M
[-]Determine Feasibility of Koala-36M method for detector[/-][+]Koala-36M Detection Algorithm[/+][detectors] Implement Koala-36M
[detectors] Implement Koala-36M