Skip to content
Permalink
Browse files

final code with examples and intermediate readmen updates

  • Loading branch information...
griffbr committed Mar 27, 2019
1 parent 362acd3 commit 121c68ce5d7ca076aa458a04a8bd1ce1d4c88a43
Showing with 105 additions and 119 deletions.
  1. +1 −0 .gitignore
  2. +19 −10 README.md
  3. +5 −7 bubblenets_select_frame.py
  4. BIN data/rawData/motocross-jump/ground_truth/00000.png
  5. BIN data/rawData/motocross-jump/ground_truth/00001.png
  6. BIN data/rawData/motocross-jump/ground_truth/00002.png
  7. BIN data/rawData/motocross-jump/ground_truth/00003.png
  8. BIN data/rawData/motocross-jump/ground_truth/00004.png
  9. BIN data/rawData/motocross-jump/ground_truth/00005.png
  10. BIN data/rawData/motocross-jump/ground_truth/00006.png
  11. BIN data/rawData/motocross-jump/ground_truth/00007.png
  12. BIN data/rawData/motocross-jump/ground_truth/00008.png
  13. BIN data/rawData/motocross-jump/ground_truth/00009.png
  14. BIN data/rawData/motocross-jump/ground_truth/00010.png
  15. BIN data/rawData/motocross-jump/ground_truth/00011.png
  16. BIN data/rawData/motocross-jump/ground_truth/00012.png
  17. BIN data/rawData/motocross-jump/ground_truth/00013.png
  18. BIN data/rawData/motocross-jump/ground_truth/00014.png
  19. BIN data/rawData/motocross-jump/ground_truth/00015.png
  20. BIN data/rawData/motocross-jump/ground_truth/00016.png
  21. BIN data/rawData/motocross-jump/ground_truth/00017.png
  22. BIN data/rawData/motocross-jump/ground_truth/00018.png
  23. BIN data/rawData/motocross-jump/ground_truth/00019.png
  24. BIN data/rawData/motocross-jump/ground_truth/00020.png
  25. BIN data/rawData/motocross-jump/ground_truth/00021.png
  26. BIN data/rawData/motocross-jump/ground_truth/00022.png
  27. BIN data/rawData/motocross-jump/ground_truth/00023.png
  28. BIN data/rawData/motocross-jump/ground_truth/00024.png
  29. BIN data/rawData/motocross-jump/ground_truth/00025.png
  30. BIN data/rawData/motocross-jump/ground_truth/00026.png
  31. BIN data/rawData/motocross-jump/ground_truth/00027.png
  32. BIN data/rawData/motocross-jump/ground_truth/00028.png
  33. BIN data/rawData/motocross-jump/ground_truth/00029.png
  34. BIN data/rawData/motocross-jump/ground_truth/00030.png
  35. BIN data/rawData/motocross-jump/ground_truth/00031.png
  36. BIN data/rawData/motocross-jump/ground_truth/00032.png
  37. BIN data/rawData/motocross-jump/ground_truth/00033.png
  38. BIN data/rawData/motocross-jump/ground_truth/00034.png
  39. BIN data/rawData/motocross-jump/ground_truth/00035.png
  40. BIN data/rawData/motocross-jump/ground_truth/00036.png
  41. BIN data/rawData/motocross-jump/ground_truth/00037.png
  42. BIN data/rawData/motocross-jump/ground_truth/00038.png
  43. BIN data/rawData/motocross-jump/ground_truth/00039.png
  44. BIN data/rawData/motocross-jump/src/00000.jpg
  45. BIN data/rawData/motocross-jump/src/00001.jpg
  46. BIN data/rawData/motocross-jump/src/00002.jpg
  47. BIN data/rawData/motocross-jump/src/00003.jpg
  48. BIN data/rawData/motocross-jump/src/00004.jpg
  49. BIN data/rawData/motocross-jump/src/00005.jpg
  50. BIN data/rawData/motocross-jump/src/00006.jpg
  51. BIN data/rawData/motocross-jump/src/00007.jpg
  52. BIN data/rawData/motocross-jump/src/00008.jpg
  53. BIN data/rawData/motocross-jump/src/00009.jpg
  54. BIN data/rawData/motocross-jump/src/00010.jpg
  55. BIN data/rawData/motocross-jump/src/00011.jpg
  56. BIN data/rawData/motocross-jump/src/00012.jpg
  57. BIN data/rawData/motocross-jump/src/00013.jpg
  58. BIN data/rawData/motocross-jump/src/00014.jpg
  59. BIN data/rawData/motocross-jump/src/00015.jpg
  60. BIN data/rawData/motocross-jump/src/00016.jpg
  61. BIN data/rawData/motocross-jump/src/00017.jpg
  62. BIN data/rawData/motocross-jump/src/00018.jpg
  63. BIN data/rawData/motocross-jump/src/00019.jpg
  64. BIN data/rawData/motocross-jump/src/00020.jpg
  65. BIN data/rawData/motocross-jump/src/00021.jpg
  66. BIN data/rawData/motocross-jump/src/00022.jpg
  67. BIN data/rawData/motocross-jump/src/00023.jpg
  68. BIN data/rawData/motocross-jump/src/00024.jpg
  69. BIN data/rawData/motocross-jump/src/00025.jpg
  70. BIN data/rawData/motocross-jump/src/00026.jpg
  71. BIN data/rawData/motocross-jump/src/00027.jpg
  72. BIN data/rawData/motocross-jump/src/00028.jpg
  73. BIN data/rawData/motocross-jump/src/00029.jpg
  74. BIN data/rawData/motocross-jump/src/00030.jpg
  75. BIN data/rawData/motocross-jump/src/00031.jpg
  76. BIN data/rawData/motocross-jump/src/00032.jpg
  77. BIN data/rawData/motocross-jump/src/00033.jpg
  78. BIN data/rawData/motocross-jump/src/00034.jpg
  79. BIN data/rawData/motocross-jump/src/00035.jpg
  80. BIN data/rawData/motocross-jump/src/00036.jpg
  81. BIN data/rawData/motocross-jump/src/00037.jpg
  82. BIN data/rawData/motocross-jump/src/00038.jpg
  83. BIN data/rawData/motocross-jump/src/00039.jpg
  84. +30 −0 data/rawData/scooter-black/ResNet_preprocess.pk
  85. +3 −0 data/rawData/scooter-black/frame_selection/BN0.txt
  86. +3 −0 data/rawData/scooter-black/frame_selection/BNLF.txt
  87. +5 −0 data/rawData/scooter-black/frame_selection/all.txt
  88. +27 −0 data/rawData/scooter-black/frame_selection/frame_classification.pk
  89. BIN data/rawData/scooter-black/src/00000.jpg
  90. BIN data/rawData/scooter-black/src/00001.jpg
  91. BIN data/rawData/scooter-black/src/00002.jpg
  92. BIN data/rawData/scooter-black/src/00003.jpg
  93. BIN data/rawData/scooter-black/src/00004.jpg
  94. BIN data/rawData/scooter-black/src/00005.jpg
  95. BIN data/rawData/scooter-black/src/00006.jpg
  96. BIN data/rawData/scooter-black/src/00007.jpg
  97. BIN data/rawData/scooter-black/src/00008.jpg
  98. BIN data/rawData/scooter-black/src/00009.jpg
  99. BIN data/rawData/scooter-black/src/00010.jpg
  100. BIN data/rawData/scooter-black/src/00011.jpg
  101. BIN data/rawData/scooter-black/src/00012.jpg
  102. BIN data/rawData/scooter-black/src/00013.jpg
  103. BIN data/rawData/scooter-black/src/00014.jpg
  104. BIN data/rawData/scooter-black/src/00015.jpg
  105. BIN data/rawData/scooter-black/src/00016.jpg
  106. BIN data/rawData/scooter-black/src/00017.jpg
  107. BIN data/rawData/scooter-black/src/00018.jpg
  108. BIN data/rawData/scooter-black/src/00019.jpg
  109. BIN data/rawData/scooter-black/src/00020.jpg
  110. BIN data/rawData/scooter-black/src/00021.jpg
  111. BIN data/rawData/scooter-black/src/00022.jpg
  112. BIN data/rawData/scooter-black/src/00023.jpg
  113. BIN data/rawData/scooter-black/src/00024.jpg
  114. BIN data/rawData/scooter-black/src/00025.jpg
  115. BIN data/rawData/scooter-black/src/00026.jpg
  116. BIN data/rawData/scooter-black/src/00027.jpg
  117. BIN data/rawData/scooter-black/src/00028.jpg
  118. BIN data/rawData/scooter-black/src/00029.jpg
  119. BIN data/rawData/scooter-black/src/00030.jpg
  120. BIN data/rawData/scooter-black/src/00031.jpg
  121. BIN data/rawData/scooter-black/src/00032.jpg
  122. BIN data/rawData/scooter-black/src/00033.jpg
  123. BIN data/rawData/scooter-black/src/00034.jpg
  124. BIN data/rawData/scooter-black/src/00035.jpg
  125. BIN data/rawData/scooter-black/src/00036.jpg
  126. BIN data/rawData/scooter-black/src/00037.jpg
  127. BIN data/rawData/scooter-black/src/00038.jpg
  128. BIN data/rawData/scooter-black/src/00039.jpg
  129. BIN data/rawData/scooter-black/src/00040.jpg
  130. BIN data/rawData/scooter-black/src/00041.jpg
  131. BIN data/rawData/scooter-black/src/00042.jpg
  132. BIN data/rawData/scooter-black/usrAnnotate/00032.png
  133. BIN data/rawData/soapbox/src/00000.jpg
  134. BIN data/rawData/soapbox/src/00001.jpg
  135. BIN data/rawData/soapbox/src/00002.jpg
  136. BIN data/rawData/soapbox/src/00003.jpg
  137. BIN data/rawData/soapbox/src/00004.jpg
  138. BIN data/rawData/soapbox/src/00005.jpg
  139. BIN data/rawData/soapbox/src/00006.jpg
  140. BIN data/rawData/soapbox/src/00007.jpg
  141. BIN data/rawData/soapbox/src/00008.jpg
  142. BIN data/rawData/soapbox/src/00009.jpg
  143. BIN data/rawData/soapbox/src/00010.jpg
  144. BIN data/rawData/soapbox/src/00011.jpg
  145. BIN data/rawData/soapbox/src/00012.jpg
  146. BIN data/rawData/soapbox/src/00013.jpg
  147. BIN data/rawData/soapbox/src/00014.jpg
  148. BIN data/rawData/soapbox/src/00015.jpg
  149. BIN data/rawData/soapbox/src/00016.jpg
  150. BIN data/rawData/soapbox/src/00017.jpg
  151. BIN data/rawData/soapbox/src/00018.jpg
  152. BIN data/rawData/soapbox/src/00019.jpg
  153. BIN data/rawData/soapbox/src/00020.jpg
  154. BIN data/rawData/soapbox/src/00021.jpg
  155. BIN data/rawData/soapbox/src/00022.jpg
  156. BIN data/rawData/soapbox/src/00023.jpg
  157. BIN data/rawData/soapbox/src/00024.jpg
  158. BIN data/rawData/soapbox/src/00025.jpg
  159. BIN data/rawData/soapbox/src/00026.jpg
  160. BIN data/rawData/soapbox/src/00027.jpg
  161. BIN data/rawData/soapbox/src/00028.jpg
  162. BIN data/rawData/soapbox/src/00029.jpg
  163. BIN data/rawData/soapbox/src/00030.jpg
  164. BIN data/rawData/soapbox/src/00031.jpg
  165. BIN data/rawData/soapbox/src/00032.jpg
  166. BIN data/rawData/soapbox/src/00033.jpg
  167. BIN data/rawData/soapbox/src/00034.jpg
  168. BIN data/rawData/soapbox/src/00035.jpg
  169. BIN data/rawData/soapbox/src/00036.jpg
  170. BIN data/rawData/soapbox/src/00037.jpg
  171. BIN data/rawData/soapbox/src/00038.jpg
  172. BIN data/rawData/soapbox/src/00039.jpg
  173. BIN data/rawData/soapbox/src/00040.jpg
  174. BIN data/rawData/soapbox/src/00041.jpg
  175. BIN data/rawData/soapbox/src/00042.jpg
  176. BIN data/rawData/soapbox/src/00043.jpg
  177. BIN data/rawData/soapbox/src/00044.jpg
  178. BIN data/rawData/soapbox/src/00045.jpg
  179. BIN data/rawData/soapbox/src/00046.jpg
  180. BIN data/rawData/soapbox/src/00047.jpg
  181. BIN data/rawData/soapbox/src/00048.jpg
  182. BIN data/rawData/soapbox/src/00049.jpg
  183. BIN data/rawData/soapbox/src/00050.jpg
  184. BIN data/rawData/soapbox/src/00051.jpg
  185. BIN data/rawData/soapbox/src/00052.jpg
  186. BIN data/rawData/soapbox/src/00053.jpg
  187. BIN data/rawData/soapbox/src/00054.jpg
  188. BIN data/rawData/soapbox/src/00055.jpg
  189. BIN data/rawData/soapbox/src/00056.jpg
  190. BIN data/rawData/soapbox/src/00057.jpg
  191. BIN data/rawData/soapbox/src/00058.jpg
  192. BIN data/rawData/soapbox/src/00059.jpg
  193. BIN data/rawData/soapbox/src/00060.jpg
  194. BIN data/rawData/soapbox/src/00061.jpg
  195. BIN data/rawData/soapbox/src/00062.jpg
  196. BIN data/rawData/soapbox/src/00063.jpg
  197. BIN data/rawData/soapbox/src/00064.jpg
  198. BIN data/rawData/soapbox/src/00065.jpg
  199. BIN data/rawData/soapbox/src/00066.jpg
  200. BIN data/rawData/soapbox/src/00067.jpg
  201. BIN data/rawData/soapbox/src/00068.jpg
  202. BIN data/rawData/soapbox/src/00069.jpg
  203. BIN data/rawData/soapbox/src/00070.jpg
  204. BIN data/rawData/soapbox/src/00071.jpg
  205. BIN data/rawData/soapbox/src/00072.jpg
  206. BIN data/rawData/soapbox/src/00073.jpg
  207. BIN data/rawData/soapbox/src/00074.jpg
  208. BIN data/rawData/soapbox/src/00075.jpg
  209. BIN data/rawData/soapbox/src/00076.jpg
  210. BIN data/rawData/soapbox/src/00077.jpg
  211. BIN data/rawData/soapbox/src/00078.jpg
  212. BIN data/rawData/soapbox/src/00079.jpg
  213. BIN data/rawData/soapbox/src/00080.jpg
  214. BIN data/rawData/soapbox/src/00081.jpg
  215. BIN data/rawData/soapbox/src/00082.jpg
  216. BIN data/rawData/soapbox/src/00083.jpg
  217. BIN data/rawData/soapbox/src/00084.jpg
  218. BIN data/rawData/soapbox/src/00085.jpg
  219. BIN data/rawData/soapbox/src/00086.jpg
  220. BIN data/rawData/soapbox/src/00087.jpg
  221. BIN data/rawData/soapbox/src/00088.jpg
  222. BIN data/rawData/soapbox/src/00089.jpg
  223. BIN data/rawData/soapbox/src/00090.jpg
  224. BIN data/rawData/soapbox/src/00091.jpg
  225. BIN data/rawData/soapbox/src/00092.jpg
  226. BIN data/rawData/soapbox/src/00093.jpg
  227. BIN data/rawData/soapbox/src/00094.jpg
  228. BIN data/rawData/soapbox/src/00095.jpg
  229. BIN data/rawData/soapbox/src/00096.jpg
  230. BIN data/rawData/soapbox/src/00097.jpg
  231. BIN data/rawData/soapbox/src/00098.jpg
  232. +1 −1 methods/annotate_suggest/BubbleNets_frame_select.py
  233. BIN methods/annotate_suggest/BubbleNets_frame_select.pyc
  234. +1 −1 methods/annotate_suggest/ResNet_preprocess.py
  235. BIN methods/annotate_suggest/ResNet_preprocess.pyc
  236. BIN methods/annotate_suggest/annotation_suggester.pyc
  237. +2 −2 methods/annotate_suggest/color_hist_frame_select.py
  238. BIN methods/annotate_suggest/color_hist_frame_select.pyc
  239. +2 −7 methods/fgbgSeg/osvos_marshal_seg.py
  240. BIN methods/fgbgSeg/osvos_marshal_seg.pyc
  241. +0 −41 methods/preprocess/bresenham.m
  242. +0 −37 methods/preprocess/imageSetting.m
  243. +6 −13 osvos_segment_video.py
@@ -1 +1,2 @@
.DS_Store
.pyc
@@ -20,37 +20,46 @@ Please cite our paper if you find it useful for your research.

## Method

__Video Description:__ https://youtu.be/hlog5FV9RLs

[![IMAGE ALT TEXT HERE](https://img.https://youtu.be/0kNmm8SBnnU/0.jpg)](https://youtu.be/0kNmm8SBnnU)
__Video Description:__ https://youtu.be/0kNmm8SBnnU

[![IMAGE ALT TEXT HERE](https://img.youtube.com/vi/0kNmm8SBnnU/0.jpg)](https://youtu.be/0kNmm8SBnnU)

__HSR Segmenting Objects at Various Heights.__ HSR's grasp camera faces downward (left) and only collects RGB data for objects in the scene (top right). However, using active perception and video object segmentation (bottom right), HSR can locate and grasp a variety of objects in real time.
![alt text](https://github.com/griffbr/VOSVS/blob/master/figure/annotation_example.png "VOS-based Visual Servo Control, Active Depth Estimation, and Mobile Robot Grasping")
<br />


## Setup

Download [resnet_v2_50.ckpt](https://www.dropbox.com/s/gn5uvc6foz10lab/resnet_v2_50.ckpt?dl=0) and add to ``./methods/annotate_suggest/ResNet/``.

Add new data to ``./data/rawData/`` folder following the examples already provided.
Add new data to ``./data/rawData/`` folder following the examples provided.
``scooter-black`` is an example with BubbleNets and annotation already complete, and ``soapbox`` is a completely unprocessed folder example.
Each folder in rawData will be used to train a separate segmentation model using the corresponding annotated training data.
Remove folders from rawData if you do not need to train a new model for them.

## Execution Process

Run video_annotate_marshalv2_9.py [native Python, has scikit dependency, requires TensorFlow]
Uses automatic annotation frame selection with previous GrabCut-based annotation tool.

Run videoSegmentation_Marshalv2_9.py [native Python, requires TensorFlow]
New multi-annotation segmentation tool.
Run ``./bubblenets_select_frame.py`` <br />
Uses automatic BubbleNets annotation frame selection with GrabCut-based user annotation tool. BubbleNet selects are stored in a text file (e.g., ``./rawData/scooter-black/frame_selection/BN0.txt``), so using another annotation tool is also possible. <br />
[native Python, has scikit dependency, requires TensorFlow]

Run ``./osvos_segment_video.py`` <br />
Runs OSVOS segmentation given user-provided annotated training frames. Trained OSVOS models are stored in ``./data/models/``. Results are timestamped and will appear in the ``./results/`` folder. <br />
[native Python, requires TensorFlow]

## Included External Files

S. Caelles*, K.K. Maninis*, J. Pont-Tuset, L. Leal-Taixé, D. Cremers, and L. Van Gool
S. Caelles*, K.K. Maninis*, J. Pont-Tuset, L. Leal-Taixé, D. Cremers, and L. Van Gool. <br />
One-Shot Video Object Segmentation, Computer Vision and Pattern Recognition (CVPR), 2017.<br />
Video Object Segmentation. <br />
https://github.com/scaelles/OSVOS-TensorFlow

K. He, X. Zhang, S. Ren and J. Sun. <br />
Deep Residual Learning for Image Recognition, Computer Vision and Pattern Recognition (CVPR), 2016. <br />
Image preprocessing. <br />
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/slim/python/slim/nets/resnet_v2.py

## Use

This code is available for non-commercial research purposes only.
@@ -1,4 +1,4 @@
# Quickly annotate multiple images per video using BubbleNets suggested frames.
# Quickly annotate images using BubbleNets suggested frames.
# Brent Griffin, 181127
# Questions? griffb@umich.edu
import numpy as np; import cv2; import IPython; import copy; import glob; import os
@@ -9,13 +9,11 @@
from grabCutClass import *
from videoProcessor import *
from annotation_suggester import *
from color_hist_frame_select import *
from ResNet_preprocess import *
from BubbleNets_frame_select import *
from color_hist_frame_select import *
from BubbleNets import bn_utils

# TODO:

user_scale = True
user_select = True

@@ -25,7 +23,7 @@ def get_user_annotation(videoDir):
annotationDir = os.path.join(videoDir, 'usrAnnotate')
if not os.path.isdir(annotationDir):
os.makedirs(annotationDir)
imageDir = os.path.join(videoDir, 'srcSegmentation')
imageDir = os.path.join(videoDir, 'src')
imageFiles = glob.glob(os.path.join(imageDir,'*'))
imageFiles.sort()
# Get list of suggested annotations.
@@ -76,7 +74,7 @@ def get_user_annotation(videoDir):
break
except:
print ('Image ') + str(annotationImageIdx) + (' does not exist!')
outputMaskDir = os.path.join(annotationDir,os.path.basename(imageDir))
outputMaskDir = os.path.join(annotationDir,os.path.basename(imageDir).split('.')[0] + '.png')
# Let user annotate selected image.
GrabCutter(imageDir, outputMaskDir, windowx, windowy, scale)
save_extra_image_copy(imageDir, videoDir, nAntImgs)
@@ -111,7 +109,7 @@ def main():
# Get BubbleNets suggested annotated frame.
BubbleNets_sort(rawDataDir, model='BNLF')
BubbleNets_sort(rawDataDir, model='BN0')
color_hist_frame_select(rawDataDir, annotate_rate=50)
color_hist_frame_select(rawDataDir, annotate_rate=int(10e6))
# Cycle through each video.
for i, videoName in enumerate(videoList):
# Misc. setup.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.

Large diffs are not rendered by default.

@@ -0,0 +1,3 @@
BN0
32
00032.jpg
@@ -0,0 +1,3 @@
BNLF
22
00022.jpg
@@ -0,0 +1,5 @@
2 suggested annotations for scooter-black
22
00022.jpg
32
00032.jpg

Large diffs are not rendered by default.

Diff not rendered.
Diff not rendered.
Diff not rendered.