-
Notifications
You must be signed in to change notification settings - Fork 0
/
index.htm
477 lines (405 loc) · 21.7 KB
/
index.htm
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<meta name="description" content="">
<meta name="author" content="">
<title>EPIC-KITCHENS Dataset</title>
<link href="static/vendor/bootstrap/css/bootstrap.min.css" rel="stylesheet">
<link href="https://fonts.googleapis.com/css?family=Montserrat:400,700|Kaushan+Script|Droid+Serif:400,700,400italic,700italic|Roboto+Slab:400,100,300,700" rel="stylesheet" type="text/css">
<link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.3.1/css/solid.css" integrity="sha384-VGP9aw4WtGH/uPAOseYxZ+Vz/vaTb1ehm1bwx92Fm8dTrE+3boLfF1SpAtB1z7HW" crossorigin="anonymous">
<link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.3.1/css/fontawesome.css" integrity="sha384-1rquJLNOM3ijoueaaeS5m+McXPJCGdr5HcA03/VHXxcp2kX2sUrQDmFc3jR5i/C7" crossorigin="anonymous">
<!-- Custom styles for this template -->
<link href="static/css/agency.css" rel="stylesheet">
<body id="page-top">
<!-- Navigation -->
<nav class="navbar navbar-expand-lg navbar-dark fixed-top" id="mainNav">
<div class="container">
<a class="navbar-brand js-scroll-trigger" href="#page-top"> <img style="width:10em;" src="static/img/logo/epic-kitchens-logo-red-side.svg" alt="Logo"></a>
<button class="navbar-toggler navbar-toggler-right" type="button" data-toggle="collapse" data-target="#navbarResponsive" aria-controls="navbarResponsive" aria-expanded="false" aria-label="Toggle navigation">
<i class="fa fa-bars fa-2x" aria-hidden="true"></i>
</button>
<div class="collapse navbar-collapse" id="navbarResponsive">
<ul class="navbar-nav text-uppercase ml-auto">
<li class="nav-item">
<a class="nav-link js-scroll-trigger" href="#trailer">Trailer</a>
</li>
<li class="nav-item">
<a class="nav-link js-scroll-trigger" href="#about">About</a>
</li>
<li class="nav-item">
<a class="nav-link js-scroll-trigger" href="#stats">Explore</a>
</li>
<li class="nav-item">
<a class="nav-link js-scroll-trigger" href="#downloads">Downloads</a>
</li>
<li class="nav-item">
<a class="nav-link js-scroll-trigger" href="#team">Team</a>
</li>
</ul>
</div>
</div>
</nav>
<!-- Header -->
<header class="masthead">
<div class="container">
<div class="intro-text">
</div>
</div>
</header>
<section id="trailer">
<div class="container">
<div class="row text-center">
<div class="col text-center">
<h2 class="section-heading text-uppercase">Watch the Trailer</h2>
</div>
</div>
<div class="Container">
<div class="row align-items-center">
<div class="col text-center">
<iframe width="840" height="473" src="https://www.youtube.com/embed/yGodQAbYW_E" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
</div>
</div>
</div></div>
</section>
<!-- About -->
<section class="bg-light" id="about">
<div class="container">
<div class="row">
<div class="col" align='center'>
<h2 class="service-heading">EPIC-KITCHENS VISOR</h2>
<p>
We are proud to announce the EPIC-KITCHENS VISOR, a new dataset of
pixel annotations and a benchmark suite for segmenting hands and active
objects in egocentric video. VISOR annotates videos from
EPIC-KITCHENS, which comes with a new set of challenges not encountered
in current video segmentation datasets. Specifically, we need to ensure
both short- and long-term consistency of pixel-level annotations as
objects undergo transformative interactions, e.g. an onion is peeled,
diced and cooked - where we aim to obtain accurate pixel-level
annotations of the peel, onion pieces, chopping board, knife, pan, as
well as the acting hands. VISOR introduces an annotation pipeline,
AI-powered in parts, for scalability and quality, and introduces:
</p>
</div>
</div>
<div class="row">
<div class="col" align='center'>
<h4 class="service-heading">Sparse Annotations</h4>
<img src='static/img/sparse.jpg' style='width:70%'>
<p>271K masks covering 36 hours of untrimmed video</p>
</div>
<div class="col" align='center'>
<h4 class="service-heading">Dense Annotations</h4>
<img src='static/img/dense.jpg' style='width:70%'>
<p>14.9M high quality automatic interpolations</p>
</div>
<div class="row">
<div class="col" align='center'>
<h4 class="service-heading">Video Object Segmentation</h4>
<p><b>Goal:</b> Track segments through video and occlusion</p>
<img src='static/img/c1.png' style='width:100%'>
</div>
<div class="col" align='center'>
<h4 class="service-heading">Hand Object Segmentation</h4>
<p><b>Goal:</b> Identify contact with 67K in-hand object masks </p>
<img src='static/img/c2.png' style='width:100%'>
</div>
<div class="col" align='center'>
<h4 class="service-heading">Where Did This Come From?</h4>
<p><b>Goal:</b> Name and point to where things came from with 222 test cases</p>
<img src='static/img/c3.png' style='width:100%'>
</div>
</div>
</section>
<!-- Stats -->
<section id="stats" class="bg-light">
<div class="container">
<div class="row">
<div class="col-lg-12 text-center">
<h2 class="section-heading text-uppercase">Explore EPIC-KITCHENS VISOR</h2>
<h3 class="section-subheading text-muted">To get a sense of the data, feel free to explore some of the data in VISOR!</h3>
</div>
</div>
</div>
<!-- Segments -->
<script type='text/javascript' src='segmentwidget.js'></script>
<script type='text/javascript' src='static/all_spans.js'></script>
<div class="container">
<div class="row align-items-center">
<div class="col-lg-12 text-center">
<h4 class="section-heading text-uppercase"><span style='color:red;'>Interactive! </span>Watch a Segment.</h4>
<p>
You can click through and see the annotations for a full sequence. We show the image on the left, the annotations on the right, and the legend for the annotation below.
</p>
<br>
<span id='framecount' style='border:0.1em solid black; padding:0.1em 0.5em 0.1em 0.5em; font-size:2em;'>Frame 1 / 402</span><br><br>
</div>
</div>
<div class="row align-items-center">
<div class="col-lg-1 col-md-1 mb-1 text-center"><input type="button" style="cursor:pointer;border:0.1em solid black;padding:0em 0.5em 0.1em 0.5em;font-size:2em;" onclick="adjustBN(-1)" value="<<"></div>
<div class="col-lg-5 col-md-5 mb-4">
<center>Image</center>
<img id="widgetimage" src="https://epicwidget.s3.amazonaws.com/P22_107/P22_107_frame_0000000213.jpg" class="img-fluid mb-4" alt="">
</div>
<div class="col-lg-5 col-md-5 mb-4">
<center>Annotation</center>
<img id="widgetanno" src="https://epicwidget.s3.amazonaws.com/P22_107/segmentations/P22_107_frame_0000000213.png" class="img-fluid mb-4" alt="">
</div>
<div class="col-lg-1 col-md-1 mb-1 text-center"><input type="button" style="cursor:pointer;border:0.1em solid black;padding:0em 0.5em 0.1em 0.5em;font-size:2em;" onclick="adjustBN(1)" value=">>"></div>
</div>
<div class="row justify-content-center">
<div class="col text-center" id='legend'>
<span style='background-color:#a08000;border:1px solid black'> </span> drawer <span style='background-color:#a0c0c0;border:1px solid black'> </span> left hand <span style='background-color:#006040;border:1px solid black'> </span> right hand
</div>
</div>
</div>
<div class="container"><div class="row"><div class="col"><br/><br/><br/><br/></div></div></div>
<div class="container">
<div class="row">
<div class="col-lg-12 text-center">
<h4 class="section-heading text-uppercase"><span style='color:red;'>Interactive! </span>See Our Dense Annotations.</h4>
<p>
Part of VISOR is a new collection of 14.9M new masks that are interpolated between our sparse annotation.<br/>
<b>Click</b> on
any of the images below to see some clips of new dense annotations.
</p>
</div>
</div>
<script>
function updateVid(i){
document.getElementById('vid'+i).innerHTML = "<video width='100%' controls autoplay=true loop><source src='video/cut_"+i+".webm' type='video/webm'><source src='video/cut_"+i+".mp4' type='video/mp4'></video>";
}
</script>
<style>
.imagemouseover{
-webkit-filter: brightness(100%);
-moz-filter: brightness(100%);
-o-filter: brightness(100%);
-ms-filter: brightness(100%);
filter: brightness(100%);
}
.imagemouseover:hover{
-webkit-filter: brightness(70%);
-moz-filter: brightness(70%);
-o-filter: brightness(70%);
-ms-filter: brightness(70%);
filter: brightness(70%);
}
</style>
<div class="row justify-content-center">
<div class="col" align='center' id='vid0' onclick="updateVid(0)"> <img src='video/cut_0.jpg' width="100%" class='imagemouseover'> </div>
<div class="col" align='center' id='vid1' onclick="updateVid(1)"> <img src='video/cut_1.jpg' width="100%" class='imagemouseover'> </div>
</div>
<div class="row"><div class='col'><br/></div></div>
<div class="row justify-content-center">
<div class="col" align='center' id='vid2' onclick="updateVid(2)"> <img src='video/cut_2.jpg' width="100%" class='imagemouseover'> </div>
<div class="col" align='center' id='vid3' onclick="updateVid(3)"> <img src='video/cut_3.jpg' width="100%" class='imagemouseover'> </div>
</div>
</div>
<div class="container"><div class="row"><div class="col"><br/><br/><br/><br/><br/><br/></div></div></div>
<!-- Hands in action -->
<div class="container">
<div class="row">
<div class="col-lg-12 text-center">
<h4 class="section-heading text-uppercase"><span style='color:red;'>Interactive! </span>What are Hands Doing?</h4>
<p>
Mouseover an image and you can see what hands are up to in EPIC-KITCHENS. We'll show you a
hand that's at your mouse cursor.
</p>
</div>
</div>
<div class="row">
<div class="col-lg-12 text-center">
<div style='align:center;display:inline' id='dselectors'></div><br/><br/>
<canvas id='visualizer' width=675 height=375 style='border:2px solid black;'></canvas><br/>
<img style="width:5em; margin-top:-2em;" src="static/img/finger.png" /> Move your mouse here! <br>
<script type='text/javascript'>
var colorActive = '#aaf'; var colorInactive = '#ddd';
var gridBinShow = new Array(); var gridHandSets = new Array(); var gridHandSetNames = new Array();
function doSetups(){ setupHandMove() }
window.onload = doSetups;
</script>
<script src='locdata.js'></script>
<script src='locationDriver.js'></script>
</div>
</section>
<section id="downloads">
<div class="container">
<div class="row">
<div class="col-md-12">
<h2 class="section-heading text-uppercase">Download Data</h2>
<p style='font-size:150%'>
<a href="https://doi.org/10.5523/bris.2v6cgv1x04ol22qp9rm9x2j6a7">VISOR is now available for download</a>. <br/></p>
<p>Annotation and sparse frames are available at the University of Bristol data repository, data.bris, at <a href="https://doi.org/10.5523/bris.2v6cgv1x04ol22qp9rm9x2j6a7">https://doi.org/10.5523/bris.2v6cgv1x04ol22qp9rm9x2j6a7</a><br/><br/>
</p>
<h4 class="section-subheading">Code</h4>
We make the following codes now public, which replicate the VISOR paper's baseline and provide visualisation support for the annotations
<ul>
<li><a href="https://github.com/epic-kitchens/VISOR-VIS">VISOR-VIS</a>: Code to visualise segmentations</li>
<li><a href="https://github.com/epic-kitchens/VISOR-FrameExtraction">VISOR-FrameExtraction</a>: Code to extract frames for dense annotations from the original video</li>
<li><a href="https://github.com/epic-kitchens/VISOR-VOS">VISOR-VOS</a>: Code to perform semi-supervised video object segmentation. Models and code replicate our first benchmark</li>
<li><a href="https://github.com/epic-kitchens/VISOR-HOS">VISOR-HOS</a>: Code to perform in-frame hand and active object segmentations. Models and code replicate our second baseline</li>
<li><a href="https://github.com/epic-kitchens/VISOR-WDTCF">VISOR-WDTCF</a>: Code to replicate our taster benchmark: <i>Where did <b>this</b> come from?</i></li>
</ul>
<p>The above repos contain everything you need to replicate our paper's results and visualise annotations. We are not releasing any further code or models.</p>
<h4 class="section-subheading">Paper and Citation</h4>
<p>
Read our NeurIPS 2022 paper EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations on <a href="http://arxiv.org/abs/2209.13064">ArXiv</a> and <a href="https://openreview.net/forum?id=djnKHOjpb7I">Open Review</a>
</p>
<p>When using these annotations, cite our <a href="http://arxiv.org/abs/2209.13064">EPIC-KITCHENS VISOR Benchmark</a> paper:</p>
<pre class="bibtex"><code>@inproceedings{VISOR2022,
title={EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations},
author={Darkhalil, Ahmad and Shan, Dandan and Zhu, Bin and Ma, Jian and Kar, Amlan and Higgins, Richard and Fidler, Sanja and Fouhey, David and Damen, Dima},
booktitle = {Proceedings of the Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks},
year = {2022}
} </code></pre>
Also cite the <a href="https://link.springer.com/content/pdf/10.1007/s11263-021-01531-2.pdf">EPIC-KITCHENS-100</a> paper where the videos originate:
<pre class="bibtex"><code>@ARTICLE{Damen2022RESCALING,
title={Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100},
author={Damen, Dima and Doughty, Hazel and Farinella, Giovanni Maria and and Furnari, Antonino
and Ma, Jian and Kazakos, Evangelos and Moltisanti, Davide and Munro, Jonathan
and Perrett, Toby and Price, Will and Wray, Michael},
journal = {International Journal of Computer Vision (IJCV)},
year = {2022},
volume = {130},
pages = {33–55},
Url = {https://doi.org/10.1007/s11263-021-01531-2}
} </code></pre>
</div>
</div>
<div class="row">
<div class="col-md-12">
<h4 class="section-subheading">Disclaimer </h4>
<p>The underlying data that power VISOR, EPIC-KITCHENS-55 and EPIC-KITCHENS-100, were collected as a tool for research in computer vision. The dataset may have unintended biases (including those of a societal, gender or racial nature).</p>
</div>
</div>
<div class="row">
<div class="col-md-12">
<h4 class="section-subheading">Copyright <img alt="Creative Commons License" style="border-width:1px;float:left;margin-right:15px;margin-bottom:0px;" src="https://i.creativecommons.org/l/by-nc/3.0/88x31.png"/></h4>
<p>
The VISOR dataset is copyright by us and published under the <a rel="license" href="https://creativecommons.org/licenses/by-nc/4.0/">Creative Commons Attribution-NonCommercial 4.0 International</a> License. This means that you must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use. You may not use the material for commercial purposes.
</p>
<p>For commercial licenses of EPIC-KITCHENS and VISOR annotations, email us at <a href="mailto:uob-epic-kitchens@bristol.ac.uk">uob-epic-kitchens@bristol.ac.uk</a></p>
</div>
</div>
</div>
</section>
<section id="team" class="bg-light">
<div class="container">
<div class="col-lg-12 text-center">
<h2 class="section-heading text-uppercase">The Team</h2>
</div>
<div class="row">
<div class="col-md-12 text-center">
<p>VISOR is the result of a collaboration of the Universities of <a href='http://www.bristol.ac.uk/'>Bristol</a>, <a href='https://umich.edu/'>Michigan</a>, and <a href='https://www.utoronto.ca/'>Toronto</a>.</p>
</div>
</div>
<div class="row justify-content-center">
<div class="col-md-2">
<div class="team-member">
<img class="mx-auto rounded-circle" src="static/img/profile/ad.png" />
<h4>Ahmad Dar Khalil*</h4></a>
<h6 class="text-muted">University of Bristol</h6>
</div>
</div> <!--Ahmad-->
<div class="col-md-2">
<div class="team-member">
<img class="mx-auto rounded-circle" src="static/img/profile/ds.jpg" />
<h4>Dandan Shan*</h4></a>
<h6 class="text-muted">University of Michigan</h6>
</div>
</div> <!--Dandan-->
<div class="col-md-2">
<div class="team-member">
<img class="mx-auto rounded-circle" src="static/img/profile/Bin.jpg" />
<h4>Bin Zhu*</h4></a>
<h6 class="text-muted">University of Bristol</h6>
</div>
</div> <!--Bin-->
<div class="col-md-2">
<div class="team-member">
<img class="mx-auto rounded-circle" src="static/img/profile/jm2-min.jpg" />
<h4>Jian Ma*</h4></a>
<h6 class="text-muted">University of Bristol</h6>
</div>
</div> <!--Jian-->
<div class="col-md-2">
<div class="team-member">
<img class="mx-auto rounded-circle" src="static/img/profile/ak.jpg" />
<h4>Amlan Kar</h4></a>
<h6 class="text-muted">University of Toronto</h6>
</div>
</div> <!--Amlan-->
<div class="col-md-2">
<div class="team-member">
<img class="mx-auto rounded-circle" src="static/img/profile/rh.png" />
<h4>Richard Higgins</h4></a>
<h6 class="text-muted">University of Michigan</h6>
</div>
</div> <!--Richard-->
<div class="col-md-2">
<div class="team-member">
<img class="mx-auto rounded-circle" src="static/img/profile/sf.jpg" />
<h4>Sanja Fidler</h4></a>
<h6 class="text-muted">University of Toronto</h6>
</div>
</div> <!--Sanja-->
<div class="col-md-2">
<div class="team-member">
<img class="mx-auto rounded-circle" src="static/img/profile/df.jpg" />
<h4>David Fouhey</h4></a>
<h6 class="text-muted">University of Michigan</h6>
</div>
</div> <!--David-->
<div class="col-md-2">
<div class="team-member">
<img class="mx-auto rounded-circle" src="static/img/profile/dd-min.jpg" />
<h4>Dima Damen</h4></a>
<h6 class="text-muted">University of Bristol</h6>
</div>
</div> <!--Dima-->
</div>
<div class="container">
<div class="row">
<div class="col-lg-12">
<h2 class="section-heading text-uppercase">Research Funding</h2>
<div class="text-muted">
<p> The work on VISOR was supported by the following:</p>
<ul class="text-muted">
<li>Segmentation annotations were funded by charitable unrestricted donation from Procter and Gamble as well as charitable unrestricted donation from DeepMind.
<li>Research at the University of Bristol is supported by UKRI Engineering and Physical Sciences Research Council (EPSRC) Doctoral Training Program (DTP), EPSRC Fellowship UMPIRE (EP/T004991/1) and EPSRC Program Grant Visual AI (EP/T028572/1).
<li>The project acknowledges the use of the ESPRC funded Tier 2 facility, JADE and University of Bristol's Blue Crystal 4 facility.
<li>Research at the University of Michigan is based upon work supported by the National Science Foundation under Grant No. 2006619.
<li>Research at the University of Toronto is in part sponsored by NSERC. S.F. also acknowledges support through the Canada CIFAR AI Chair program.
</ul>
</div>
</div>
</div>
</div>
</section>
<!-- Footer -->
<footer style="background-color:#373435ff;">
<div class="container">
<div class="row">
<div class="col-md-4">
<img alt="Creative Commons License" style="border-width:1px;float:left;margin-right:15px;margin-bottom:0px;" src="http://i.creativecommons.org/l/by-nc/3.0/88x31.png"/>
<span class="copyright" style="color:#eee;">Copyright © EPIC KITCHENS 2022</span>
</div>
<div class="col-md-8">
<p style="color:#eee;">For general enquiries, email us at
<a href="mailto:uob-epic-kitchens@bristol.ac.uk"> uob-epic-kitchens@bristol.ac.uk</a></p>
</div>
</div>
</div>
</footer>
<!-- Bootstrap core JavaScript -->
<script src="static/vendor/jquery/jquery.min.js"></script>
<script src="static/vendor/bootstrap/js/bootstrap.bundle.min.js"></script>
<!-- Plugin JavaScript -->
<script src="static/vendor/jquery-easing/jquery.easing.min.js"></script>
<!-- Custom scripts for this template -->
<script src="static/js/agency.min.js"></script>
</body>
</html>