Resnet3DRS documentation and checkpoints

[JapinderSandhu]

Prerequisites
Please answer the following question for yourself before submitting an issue.

I checked to make sure that this issue has not been filed already.

1. The entire URL of the documentation with the issue
   https://github.com/tensorflow/models/blob/master/official/vision/beta/MODEL_GARDEN.md#video-classification

2. Describe the issue
   I dont understand how to find the 3D-ResNet or ResNet-RS or R3D model code for video classification purposes and use it to train on my own data.

https://github.com/tensorflow/models/blob/master/official/vision/beta/configs/backbones_3d.py

I saw this model from the following paper:
"REVISITING 3D RESNETS FOR VIDEO RECOGNITION"
https://arxiv.org/pdf/2109.01696.pdf

In the abstract it says
“Code is available at: https://github.com/tensorflow/models/tree/master/official.”

I see that they reference their model as
ResNet-RS in the README.md
https://github.com/tensorflow/models/blob/master/official/vision/beta/MODEL_GARDEN.md

when further clicking on ResNet-RS
It takes me to the MODEL_GARDEN.md page
https://github.com/tensorflow/models/blob/master/official/vision/beta/MODEL_GARDEN.md#video-classification

here at the bottom of the MODEL_GARDEN.md page I see Video Classification
it shows the 3D-ResNet (R3D) model
however I am confused on how to actually get the code

all I get is this k400_3d-resnet50_tpu.yaml file:
https://github.com/tensorflow/models/blob/master/official/vision/beta/configs/experiments/video_classification/k400_3d-resnet50_tpu.yaml

I dont know what to do with this yaml file, also I dont have a TPU, I use a GPU.

Here there is a PyTorch 3D-ResNets example which is very clear on how to fine-tune, download the weights and all.
https://github.com/kenshohara/3D-ResNets-PyTorch

my hope was there could be a Tensorflow version of this PyTorch model so I found this 3D-ResNet or ResNet-RS or R3D model code but I cant find out how to actually get the code from Tensorflow/models

[saberkun]

@xianzhidu could you help response this question? Thanks

[xianzhidu]

Hi, we are still in the process of releasing checkpoints/documentation. The implementation is checked in. Here are some pointers:

ResNet-3D-RS config: https://github.com/tensorflow/models/blob/master/official/vision/beta/configs/experiments/video_classification/k400_resnet3drs_50_tpu.yaml

Build the model:
https://github.com/tensorflow/models/blob/master/official/vision/beta/modeling/backbones/resnet_3d.py#L418

[saberkun]

af554fd is supposed to add this.

[JapinderSandhu]

Hi, we are still in the process of releasing checkpoints/documentation. The implementation is checked in. Here are some pointers:

ResNet-3D-RS config: https://github.com/tensorflow/models/blob/master/official/vision/beta/configs/experiments/video_classification/k400_resnet3drs_50_tpu.yaml

Build the model: https://github.com/tensorflow/models/blob/master/official/vision/beta/modeling/backbones/resnet_3d.py#L418

Hi thank you for the reply and providing more information.

Here there is a PyTorch 3D-ResNets example which is very clear on how to fine-tune, download the weights and even providing examples on how to train and test.
https://github.com/kenshohara/3D-ResNets-PyTorch

Is there any way you could provide examples on how to execute build, train and test scripts/commands on a dataset like kinetics? The PyTorch example is very well documented and easy to understand, my hope is that there could be clear documentation like PyTorch example.

I still do not fully understand how to actually train the model and what to do with the ResNet-3D-RS config: file.

I am trying to train this Tensorflow 3D-ResNet model on kinetics dataset and then fine-tune it on my own dataset.

The PyTorch example gave clear examples on how to fine-tune as well.

Is it required to train this Tensorflow 3D-ResNet model with a TPU? Can i train this model with a GPU?

[xianzhidu]

@yeqingli will there be any tutorials on how to train 3D models on Kinetics datasets?