-
Notifications
You must be signed in to change notification settings - Fork 1
/
index.html
231 lines (197 loc) · 13.4 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
<!DOCTYPE html>
<html>
<head lang="en">
<meta name="keywords" content="DomainBridgingNav">
<title>Real-world Instance-specific Image Goal Navigation for Service Robots: Bridging the Domain Gap with Contrastive Learning</title>
<script src="https://polyfill.io/v3/polyfill.min.js?features=es6"></script>
<script id="MathJax-script" async src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>
<style>
.container {
display: flex;
justify-content: center;
align-items: center;
height: 70px;
flex-direction: column;
text-align: center;
}
.fixed-width {
width: 700px;
margin: 10px auto;
}
.center-align {
text-align: center;
}
.text-justify {
text-align: justify;
}
video {
width: 700px;
height: auto;
display: block;
margin: 10px auto;
}
.col-md-8 {
width: 700px;
margin: 10px auto;
}
.github-link {
display: flex;
justify-content: center;
}
.github-link a {
margin: 10px;
}
.image-container {
display: flex;
justify-content: center;
}
.image-container {
display: flex;
justify-content: center;
align-items: center; /* 追加: 要素を横軸中心に配置 */
flex-direction: column; /* 追加: 子要素を縦に並べる */
width: 600px; /* 必要に応じて調整 */
margin: auto;
}
.image-container img {
width: 100%; /* 子要素の幅を親要素に合わせる */
height: auto;
}
.image-caption {
text-align: center;
width: 100%; /* キャプションの幅を親要素に合わせる */
margin-top: 10px; /* キャプションの上の余白を追加 */
}
.video-container {
margin: 10px;
display: flex;
justify-content: center;
}
.video-container iframe {
width: 700px;
height: 394px;
}
</style>
</head>
<body>
<div class="container" id="main">
<div class="row">
<h2 class="col-md-12">
Real-world Instance-specific Image Goal Navigation for Service Robots<br>via Bridging Domain Gap Based on Contrastive Learning
</h2>
</div>
</div>
<p class="center-align">Taichi Sakaguchi, <a href="https://scholar.google.com/citations?hl=en&user=jtB7J0AAAAAJ" target=“_blank” rel=“noopener noreferrer”>Akira Taniguchi</a>, <a href="https://scholar.google.com/citations?hl=en&user=Y4qjYvMAAAAJ" target=“_blank” rel=“noopener noreferrer”>Yoshinobu Hagiwara</a>, <a href="https://scholar.google.co.jp/citations?user=tsm7qaQAAAAJ&hl" target=“_blank” rel=“noopener noreferrer”>Lotfi El Hafi</a>, <a href="https://scholar.google.co.jp/citations?user=KPxSCJUAAAAJ&hl" target=“_blank” rel=“noopener noreferrer”>Shoichi Hasegawa</a>, <a href="https://scholar.google.com/citations?hl=en&user=dPOCLQEAAAAJ" target=“_blank” rel=“noopener noreferrer”>Tadahiro Taniguchi</a></p>
<div class="github-link">
<a href="https://github.com/EmergentSystemLabStudent/DomainBridgingNav" target=“_blank” rel=“noopener noreferrer”>Github (Codes will be coming soon...)</a>
<a href="https://arxiv.org/abs/2404.09645" target=“_blank” rel=“noopener noreferrer”>Paper</a>
<!-- <a href="" target=“_blank” rel=“noopener noreferrer”>Slide</a> -->
</div>
<video src="IROS24_2258_VI_i.mp4" style="width: 700px; height: auto;" controls></video>
<!-- <div class="video-container">
<iframe src="https://www.youtube.com/embed/n8se-MgPi50?si=B5B60-8XXMO-j2CC" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
</div> -->
<h2 class="fixed-width">Abstract</h2>
<p class="center-align fixed-width text-justify">
Improving instance-specific image goal navigation (InstanceImageNav), which locates the identical object in a real-world environment from a query image, is essential for robotic systems to assist users in finding desired objects.
The challenge lies in the domain gap between low-quality images observed by the moving robot, characterized by motion blur and low-resolution, and high-quality query images provided by the user.
Such domain gaps could significantly reduce the task success rate but have not been the focus of previous work.
To address this, we propose a novel method called <b> Few-shot Cross-quality Instance-aware Adaptation (CrossIA)</b>, which employs contrastive learning with an instance classifier to align features between massive low- and few high-quality images.
This approach effectively reduces the domain gap by bringing the latent representations of cross-quality images closer on an instance basis.
Additionally, the system integrates an object image collection with a pre-trained deblurring model to enhance the observed image quality.
Our method fine-tunes the SimSiam model, pre-trained on ImageNet, using CrossIA.
We evaluated our method's effectiveness through an InstanceImageNav task with 20 different types of instances, where the robot identifies the same instance in a real-world environment as a high-quality query image.
Our experiments showed that our method improves the task success rate by up to three times compared to the baseline, a conventional approach based on SuperGlue.
These findings highlight the potential of leveraging contrastive learning and image enhancement techniques to bridge the domain gap and improve object localization in robotic applications. </p>
<h2 class="fixed-width">Approach</h2>
<p class="center-align fixed-width text-justify">
As shown in Figure 1, the proposed system is divided into three main modules.
Firstly, the Data Collection Module constructs the 3D semantic map of the environment and then collects object images.
Secondly, the Fine-tuning Module fine-tunes the pre-trained models using the collected object images and few-shot high-quality images provided by the user through contrastive learning.
Our contrastive learning is called <b> Few-shot Cross-quality Instance-aware Adaptation (CrossIA) </b>.
Lastly, the Navigation Module leverages the fine-tuned model and the semantic map to locate objects identical to the query image. </p>
<div class="image-container">
<img src="./images/real_approach.svg" alt="Overview of our method">
<div class="image-caption">Figure 1: Overview of our method</div>
</div>
<h2 class="fixed-width">Data Collection Module</h2>
<p class="center-align fixed-width text-justify">
The Data Collection Module constructs a 3D semantic map from sequence data of RGBD images and camera poses collected by the robot moving in 3D space.
First, RGB images collected by the robot exploring the environment are inputted into the Multi-Scale Stage Network for deblurring.
Next, they are inputted into the FastSAM for image segmentation.
Then 3D semantic map is constructed using the segmentation results, depth images, and camera poses.
Next, this module collects object images using a constructed 3D semantic map.
During this process, 2D segmentation masks images from the 3D semantic map by ray tracing.
Ray tracing generates a mask image from an arbitrary camera pose by sending pseudo-rays in the depth direction from the camera and capturing the instance ID of the first collision on the 3D map.
This process ensures the consistent generation of pseudo-labels for images of the same instance.
Then, the generated mask images are transformed into bounding boxes (BBoxes), and the regions of the transformed BBoxes are extracted from the RGB images to collect object images. </p>
<h2 class="fixed-width">Fine-tuning Module</h2>
<p class="center-align fixed-width text-justify">
This module fine-tunes a pre-trained image encoder by the contrastive task between low-quality images observed by the robot and few-shot high-quality images provided by the user.
SimSiam, a negative free contrastive learning method, is shown to be capable of learning with small batch size.
In contrast, contrastive learning methods require negative pairs learning with a huge batch size.
Therefore, we utilize SimSiam for fine-tuning.
Additionally, for training, the user needs to capture few-shot images of objects they want the robot to locate with a mobile device and provide them to the robot.
This requires the user to instruct the robot, but since it involves only a few images, it could not be considered a significant burden for the user.
In this study, we assume that the system has up to five high-quality images for each instance and evaluate the experiments accordingly.
Furthermore, it is shown that by minimizing the loss of contrastive learning and the loss function of the linear classifier for a pre-trained image encoder through fine-tuning, the variance of feature vectors between images with the same label decreases.
This characteristic leads to an improvement in the accuracy of image classification tasks. </p>
<div class="image-container" style="width: 600px; margin: auto;">
<img src="./images/fine_tuning_module.jpg" style="width: 100%; height: auto;">
<p class="center-align" style="text-align: center;">Figure 2: Fine-tuning module</p>
</div>
<p class="center-align fixed-width text-justify">
Therefore, we also conduct fine-tuning of the image encoder by adding a linear classifier to SimSiam as shown in Figure 2.
The label of instances required for calculating the loss of the linear classifier is obtained using pseudo-labels automatically generated by the data collection module.
\( p_i^l \) and \( z_i^{l} \) are the outputs of SimSiam's projector and predictor of the object image \( x_i^l \) observed by the robot, respectively.
\( p_i^{\prime l} \) and \( z_i^{\prime l} \) are the outputs of SimSiam's projector and predictor of the object image \( x_i^{\prime l} \) where \( x_i^{l} \) and \( x_i^{\prime l} \) are different images of the same object generated by data augmentation.
\( y_{i, pred}^l \) is the prediction result when \( x_i^l \) is classified.
\( p_i^h \) and \( z_i^h \) are the outputs of projector and predictor, respectively, when the high-quality image \( x_i^h \) is input to SimSiam, and \( y_{i, pred} ^h \) is the prediction result when \( x_i^h \) is classified.
Therefore, fine-tuning the image encoder involves contrastive learning between high-quality and low-quality images, and the contrastive learning between images the robot observed.
In this paper,
\( \{(x_i^l, x_i^h, y_{i, true})\}_{i =1}^{N_c} \)
is the set of high-quality image, low-quality image and their label and
\( \{(x_i^l, y_{i_{true}})\}_{i=1}^{N_r} \)
is the set of images and their labels of low-quality image.
<h2 class="fixed-width">Navigation Module</h2>
<p class="center-align fixed-width text-justify">
The Navigation Module locates the object identical to the given query image.
This is conducted by leveraging images of objects collected by the Data Collection Module, the 3D semantic map, and feature vectors extracted from the fine-tuned image encoder.
First, the fine-tuned image encoder is utilized to compute the feature vectors
\( x_{obs} \) and \( x_{query} \) for the object images collected by the Data Collection Module and the query image, respectively.
Using these feature vectors, the similarity \( s \) between the query image and the observed object images is calculated using the cosine similarity.
Then, the instance with the highest similarity to the query image is identified.
Next, the module identifies the centroid coordinates of the object on the 3D map that is most similar to the query image, and the robot moves to close it.
</p>
<div class="col-md-8">
<h2 class="text-center">Citation</h2>
<p class="text-justify">
This paper is under review on IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS2024), and the BibTeX for the paper is below.
<textarea id="bibtex" class="form-control" readonly rows="6" cols="105">
@inproceedings{domainbridgingnav2024,
title={Real-world Instance-specific Image Goal Navigation for Service Robots via Bridging Domain Gap Based on Contrastive Learning},
author={Taichi Sakaguchi and Akira Taniguchi and Yoshinobu Hagiwara and Lotfi El Hafi and Shoichi Hasegawa and Tadahiro Taniguchi},
booktitle={IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
year={2024 UnderReview}
}</textarea>
</p>
</div>
<div class="col-md-8">
<h2 class="text-center">Other links</h2>
<p class="text-justify">
<ul>
<li><a href="http://www.em.ci.ritsumei.ac.jp/" target=“_blank” rel=“noopener noreferrer”>Laboratory website</a></li>
<li><a href="" target=“_blank” rel=“noopener noreferrer”>Demo video of this research</a></li>
<li><a href="https://www.youtube.com/watch?v=UBgZGRG00eA" target=“_blank” rel=“noopener noreferrer”>Demo video of related research</a></li>
</ul>
</p>
</div>
<div class="col-md-8">
<h2 class="text-center">Acknowledgements</h2>
<p class="text-justify">
This work was supported by JSPS KAKENHI Grants-in-Aid for Scientific Research (Grant Numbers JP23K16975, 22K12212) JST Moonshot Research & Development Program (Grant Number JPMJMS2011).
</p>
</div>
<script src="script.js"></script>
</body>
</html>