Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Explore Street View depth maps #23

Closed
vict0rsch opened this issue Apr 23, 2019 · 29 comments
Closed

Explore Street View depth maps #23

vict0rsch opened this issue Apr 23, 2019 · 29 comments
Assignees
@sashavor @gcosne @Vahe987 @bsahil29
Labels
domain:data priority:high
Milestone
Summer

Comments

@vict0rsch
Copy link
Contributor

  • formulate needs if we get in touch with Google
@vict0rsch vict0rsch added this to the Summer milestone May 16, 2019
@vict0rsch
Copy link
Contributor Author

The Elevation API is not yet open to the public/research. May happen in a few months. Can't really count on it.

@vict0rsch
Copy link
Contributor Author

vict0rsch commented May 21, 2019 •
edited by unfurl-links bot
Loading

In the mean time, this is still open: https://maps.google.com/cbk?output=xml&ll=45.508457,-73.532738&dm=1

And we can generate such an image using https://github.com/cc-ai/floods-gans/blob/master/various_scripts/depth_map.py

Figure_1

Now open questions:

  • can we map an exact StreetView position to such a query?
    • can we get the original image from which this is computed?
    • can we specify pitch, camera view, angle and so on
  • is the code fast enough + scalable?
  • how to compute a water plane from this data?
  • how to embed said plane into original image?
GitHub
Using GANs to simulate how a house would look like, had it suffered from a flood - cc-ai/floods-gans

@gcosne
Copy link
Member

gcosne commented May 22, 2019

Before going into matching depth-map and street view images automatically, I would like to know if you found GPS coordinates where you managed to match it qualitatively playing on streetview? If so could you please communicate them? I can't find a single place where I can match it visually.

@vict0rsch
Copy link
Contributor Author

I agree I have not and it is an open question since the API is not documented (or is it?)

@Vahe987
Copy link

Vahe987 commented May 22, 2019

@DatCorno on the backend we are using the same API or it's different?

@gcosne
Copy link
Member

gcosne commented May 29, 2019

Reconstructing the spherical img
There are some steps to automatically reconstruct the spherical image relative to a pano_id that should be developped but in the meantime this App do the work for us.

Reconstructing 3D point cloud
Here is a notebook colab to explore depth-map where I tested if we can superpose depth map to the panoramic img without success and where I translated some medium advices to generate the 3D point cloud (there is work to do before having a height in meter)

@gcosne gcosne assigned sashavor and unassigned corneauf May 29, 2019
@vict0rsch
Copy link
Contributor Author

vict0rsch commented May 30, 2019 •
edited
Loading

Ok so this is not good enough for us to work with right? This API is still open but basically it's not as informative as we expected?

@gcosne
Copy link
Member

gcosne commented May 30, 2019

I would not say that, I believe we should write our own code following the steps in this paper explaining how to reconstruct the panorama ourself from the tiles and see if it match the depth map: There is a lot of potential in this approach.

@Vahe987
Copy link

Vahe987 commented May 30, 2019

@gcosne how long it will take in your opinion, if we write our own code?

@gcosne
Copy link
Member

gcosne commented May 30, 2019 •
edited
Loading

Somehow I find a panorama that almost match the depth map :

match

Remark : I discover one thing: Height of the Lidar is about 2.5 meters. Using the formula Normalized/minimum(height) * Lidar-Height allow to translate the normalized coordinates in meters.

Then I tried to" flood "the image until a certain height but I think the lidar doesn't have a lot of points below 1.5m. Hence the segmentation map generated is good when height >2.m but get sparse when we go below 1.7m.
Flood up to 1.7m
flood_17

Flood up to 1.3m
flood_13

@vict0rsch
Copy link
Contributor Author

Pas mal du tout. La question qui nous reste sur les bras c'est: est-ce qu'on peut travailler avec ces panoramiques? i.e. est-ce que c'est suffisamment "relatable" pour en faire qqch et est-ce qu'on peut l'avoir pour n'importe quel endroit / si non lesquels?

Il faut en parler à Alice et mettre au point un premier groupe de testing utilisateur pour se faire un avis, imho

@vict0rsch
Copy link
Contributor Author

oups French, I meant: this is not bad at all! we ought to research that with users to understand if that's usable / relatable for our purpose + figure out if there are restrictions on localizations for which we can get this depth map + original panorama

@sashavor
Copy link
Contributor

sashavor commented May 31, 2019 via email

@Vahe987
Copy link

Vahe987 commented May 31, 2019

very interesting, good job, so you found the magic number I was asking :)
for the flooding part maybe we can find another technick, like reconstructing the scene in 3D and flood afterwards ... but not sure for now, maybe too heavy, we can think more about it

@gcosne
Copy link
Member

gcosne commented Jun 3, 2019

Testing with depth estimation : Using very light single image depth estimator rather than 'google api', we almost have usable masks.

To Be Done:

  • Identify if we have region where google api works for us.
  • Calibrating depth estimator with google api.

Qualitative Results flooding up to 1.5m :
2
1
0
3

@Vahe987
Copy link

Vahe987 commented Jun 3, 2019

well, the flood part doesn't work ...
btw, what about people on street view images? should we "drown" them?

@sashavor
Copy link
Contributor

sashavor commented Jun 3, 2019 via email

@gcosne
Copy link
Member

gcosne commented Jun 3, 2019

Update :
I translated DownloadPanorama C++ function from this incredible project in python.

We can now automatically extract xml information and generate both depthmap and panoramic images within a script. I think I will explore more of PaulWagener project, I sent him an email.

Cropping the image so they are the same size rather than resizing them actually work better. Not perfect yet but I will fix that soon.

pano_img
depth1

@sashavor
Copy link
Contributor

sashavor commented Jun 3, 2019 via email

@gcosne
Copy link
Member

gcosne commented Jun 3, 2019

Not yet but I do trust it will, I have to play with yaw heading.

@sashavor
Copy link
Contributor

sashavor commented Jun 3, 2019 via email

@gcosne
Copy link
Member

gcosne commented Jun 4, 2019

Update : We can almost call it a success.
seg14
seg11

I said almost because the size of the tiles are not consistent, sometimes the 360° equirectangular image is already built (no overlapping between tiles), sometimes it's not. The good news are that we can identify it by the amount of black padding in the image.

          360° panorama without overlapping                                     More than 360° panoramic image
amount_of_black

@sashavor
Copy link
Contributor

sashavor commented Jun 4, 2019 via email

@gcosne
Copy link
Member

gcosne commented Jun 4, 2019 •
edited
Loading

I assume so, nothing of an issue here but just a thing to be aware.

Next steps :

  • Finish the work in low resolution.
  • Create the stitching function for already built panoramic images in higher resolution.

@sashavor
Copy link
Contributor

sashavor commented Jun 4, 2019 via email

@Vahe987
Copy link

Vahe987 commented Jun 4, 2019

cool, good job ;)

@gcosne
Copy link
Member

gcosne commented Jun 4, 2019 •
edited
Loading

Done: Low and High resolution are working. I will clean and commit the code soon.

Next steps : Make it work with the 3D.

  • Flood the ground.
  • Flood a certain height.

@sashavor
Copy link
Contributor

sashavor commented Jun 4, 2019 via email

@gcosne
Copy link
Member

gcosne commented Jun 4, 2019

3D reconstruction: There is an echo around the car that makes the ground plane estimation tricky
3D_cloud
echo

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sashavor sashavor

@gcosne gcosne

@Vahe987 Vahe987

@bsahil29 bsahil29

Labels
domain:data priority:high
Projects
None yet
Milestone
Summer
Development

No branches or pull requests
6 participants
@corneauf @vict0rsch @sashavor @gcosne @Vahe987 @bsahil29