kinetics_dataset.py to generate rgb and flow arrays for evaluate_sample.py

[ss-github-code]

Hi,
I am trying to figure out the steps needed to generate rgb and flow arrays that I can use in the evaluate_sample.py script in kinetics-i3d. If I setup VIDEO_PATH to v_CricketShot_g04_c01.avi (which is used as the sample data in kinetics-i3d), how do I get the rgb and flow files?

Perhaps, I am missing a step.
Thanks

[jiuqiant]

Hi, I can successfully generate the feature tfrecord for v_CricketShot_g04_c01.avi.

First, Deepmind recently updated their the 700 dateset, I need to modify the kietics_dataset.py accordingly (we will have the fix in the new release):

diff --git a/mediapipe/examples/desktop/media_sequence/kinetics_dataset.py b/mediapipe/examples/desktop/media_sequence/kinetics_dataset.py
index 1d7f5b6..f15f93e 100644
--- a/mediapipe/examples/desktop/media_sequence/kinetics_dataset.py
+++ b/mediapipe/examples/desktop/media_sequence/kinetics_dataset.py
@@ -93,15 +93,15 @@ FILEPATTERN = "kinetics_700_%s_25fps_rgb_flow"
 SPLITS = {
     "train": {
         "shards": 1000,
-        "examples": 541632
+        "examples": 541523
     },
     "validate": {
         "shards": 100,
-        "examples": 34727
+        "examples": 34719
     },
     "test": {
         "shards": 100,
-        "examples": 69347
+        "examples": 69327
     },
     "custom": {
         "csv": None,  # Add a CSV for your own data here.

Then, run the following commands:

CUSTOM_CSV=/tmp/custom_kinetics.csv
VIDEO_PATH=/path/to/your/v_CricketShot_g04_c01.avi 
echo -e "video,time_start,time_end,split\n${VIDEO_PATH},0,10,custom" > ${CUSTOM_CSV}
bazel build -c opt mediapipe/examples/desktop/media_sequence:media_sequence_demo   --define MEDIAPIPE_DISABLE_GPU=1
python -m mediapipe.examples.desktop.media_sequence.kinetics_dataset   --alsologtostderr   --splits_to_process=custom   --path_to_custom_csv=${CUSTOM_CSV}   --video_path_format_string={video}   --path_to_kinetics_data=/tmp/ms/kinetics/   --path_to_mediapipe_binary=bazel-bin/mediapipe/examples/desktop/media_sequence/media_sequence_demo    --path_to_graph_directory=mediapipe/graphs/media_sequence/

Then, you can read the tfrecord in Python

import tensorflow as tf
for example in tf.python_io.tf_record_iterator('/tmp/ms/kinetics/kinetics_700_custom_25fps_rgb_flow-00000-of-00001'):
  print(tf.train.SequenceExample.FromString(example))

# It should print 
WARNING:tensorflow:From <stdin>:1: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and: 
`tf.data.TFRecordDataset(path)`
context {
  feature {
    key: "FORWARD_FLOW/image/channels"
    value {
      int64_list {
        value: 3
      }
    }
  }
  feature {
    key: "FORWARD_FLOW/image/format"
    value {
      bytes_list {
        value: "JPEG"
      }
    }
  }
  feature {
    key: "FORWARD_FLOW/image/frame_rate"
    value {
      float_list {
        value: 25.0
      }
    }
  }
  feature {
    key: "FORWARD_FLOW/image/height"
    value {
      int64_list {
        value: 256
      }
    }
  }
  feature {
    key: "FORWARD_FLOW/image/width"
    value {
      int64_list {
        value: 340
      }
    }
  }
  feature {
    key: "clip/alternative_media_id"
    value {
      bytes_list {
        value: "custom"
      }
    }
  }
  feature {
    key: "clip/data_path"
    value {
      bytes_list {
        value: "/Users/jqtang/Desktop/v_CricketShot_g04_c01.avi"
      }
    }
  }
  feature {
    key: "clip/media_id"
    value {
      bytes_list {
        value: "/Users/jqtang/Desktop/v_CricketShot_g04_c01.avi"
      }
    }
  }
  feature {
    key: "example/id"
    value {
      bytes_list {
        value: "/Users/jqtang/Desktop/v_CricketShot_g04_c01.avi_0"
      }
    }
  }
  feature {
    key: "image/channels"
    value {
      int64_list {
        value: 3
      }
    }
  }
  feature {
    key: "image/format"
    value {
      bytes_list {
        value: "JPEG"
      }
    }
  }
  feature {
    key: "image/frame_rate"
    value {
      float_list {
        value: 25.0
      }
    }
  }
  feature {
    key: "image/height"
    value {
      int64_list {
        value: 256
      }
    }
  }
  feature {
    key: "image/width"
    value {
      int64_list {
        value: 340
      }
    }
  }
}
feature_lists {
  feature_list {
    key: "FORWARD_FLOW/image/encoded"
    value {
      feature {
        bytes_list {

[ss-github-code]

Thanks a lot for your quick response and follow up,

Then in order to use the FORWARD_FLOW/image/encoded and image/encoded, how do I convert into a flow_input and rgb_input tensor that is used in the evaluate_sample.py. Do I have to first decode the JPEG bytes_list. Please do share the steps that are required to take this output over to evaluate_sample.py.

[PatrickHytla]

I am having the same issue. I am able to create the tfrecord via mediapipe, which seems to be RGB feature and FLOW feature wrapped up in the same record. I'd like to be able to separate those features into individual files, one for RGB and one for FLOW (preferrably .npy, but I'll take whatever). Does anyone have any guidance on how to do that?

[ss-github-code]

Are we supposed to integrate these steps into evaluate_sample.py so that we don't need to go through the middle step? Just a guess... I really hope that Jiuqiang will chime in. Please!

[bryanseybold]

The Kinetics.as_dataset() function is what you can use to do this. It creates a tf.data.Dataset that can be used to read the TFRecord file and extract the relevant tensors. I think this is the syntax you need, but let me know if you have problems with this. This example should work:

path_to_data = "path/to/data"
split = "custom"
data = Kinetics(path_to_data)
dataset = data.as_dataset(split)

In tf1 the next line of code would be

tensors = dataset.make_one_shot_iterator().get_next()
images = tensors["images"]
flow = tensors["flow"]

In tf2 the next line of code would be

tensors = next(iter(dataset))   # rather than using next(iter(...)), you can just use a for loop to iterate.
images = tensors["images"]
flow = tensors["flow"]