How to Prepare data which likes ava dataset to train Spatial Temporal Action Detection SlowFast model ？

[ZJU-lishuang]

I think the best way is to get annotation format familiar with the ava .
But the parameter of timestamp is not exist in custom dataset.
The SampleAVAFrames function has the parameter of timestamp_start and timestamp_end.

center_index = fps * (timestamp - timestamp_start) + 1

And we can not guarantee that every video is the same.
How to deal with it?

Maybe we can set timestamp_start=0,how about timestamp_end?

shot_info = (0, (self.timestamp_end - self.timestamp_start) * self._FPS)

[innerlee]

cc. @kennymckormick

[ZJU-lishuang]

Now I try to convert custom dataset to ava format.I got a csv like ava.
And I find the csv file already has the box coordinates.
What is the function of proposals in ava_dense_proposals_train.FAIR.recall_93.9.pkl during training slwofast?

[irvingzhang0512]

related issue #652

[ZJU-lishuang]

@innerlee ava dataset rawframes is begin from img_00001.jpg.
But shot_info start from 0 in AVADataset funtction.

shot_info = (0, (self.timestamp_end - self.timestamp_start) *
                             self._FPS)

When I use my dataset.There is a error in "don't find img_0000.jpg".
Should shot_info start from 1 in AVADataset funtction？

shot_info = (1, (self.timestamp_end - self.timestamp_start) *
                             self._FPS)

[irvingzhang0512]

Yes, and this var is used to clip frame indexes frame_inds = np.clip(frame_inds, shot_info[0], shot_info[1] - 1). I agree with @ZJU-lishuang

[innerlee]

@irvingzhang0512 @dreamerlin We need to sort out the start-index related issues.

[irvingzhang0512]

Maybe we need a tutorial about customizing ava-like dataset(or introducing the big picture of ava dataset and related configs) and update related codes. All these start-index/center-index issues(maybe more) are part of this.
There are at least 3 issues, asking for how to customize ava-like dataset.

Haven't studied all the details about ava dataset yet, but I believe for now we are not fully prepared for this customizing issue.

[innerlee]

@kennymckormick is planning to write such material

[eliethesaiyan]

@kennymckormick , Any progress on the tutorial materials?

[ZJU-lishuang]

Right now I have trained slowfast model in custom dataset for spatical teporal action.
So I meet a lot of problems during training own dataset.
I'm interested in the tutorial,but I'm poor in writing skill and I'm busy in a actual project.
Here is my script tools for convert custom dataset to ava format.
Hope it helps.
The custom dataset annotation format comes from cvat.

I think mmaction2 is a great project in the world.
Yes.
Answer myself.

[innerlee]

Thanks!

[dnvnxg]

@ZJU-lishuang Can I ask, which dataset format you used when exporting from CVAT to make this work?

[eliethesaiyan]

@ZJU-lishuang
How did you deal with videos that don't have the same length?

[arvindchandel]

@ZJU-lishuang @innerlee @kennymckormick Hi, I need some inputs from you related to Spatial Temporal Action Detection Slow Fast on custom data. details are below:

1. I have prepared the custom data in AVA format for 3 classes viz Sweeping, Walking and Standing.
2. Current data size of each class respectively in Train/Validation is 2153/561 for sweeping, 1079/163 (walking) , 1743/305 (standing).
3. Trained the SlowFast model initially with 25 epochs on above data and the results ranges in 0.55 to 0.67 mAP on various epochs.
4. I tested the model with 0.67 mAP on sample video and results were ok not upto expectation, as there were some false positives as well.

Now i want your inputs on below points:

1. For further increasing performance, i will add more data to retrain it.
2. While testing the sample video on trained model, any action that were detected with confidence score in range 0.45 to 0.55, either its true positive or false positive, but no action is getting better score like say sweeping detected with 0.85 score, not even once. that i am not getting why it is so. seeks you input on this as i am in process of training custom actions and finetuing the model.

[yikoudamifan]

@arvindchandel
I hope you can answer my question: how do you make ava custom data sets. thanks