WebGPU based 3D reconstruction in browser (COLMAP style)

[MatiasHiltunen]

Hello, I've been building for a while webgpu based image 3D reconstruction utility and I think it could be feasible to port the reconstruction pipeline to rust/burn and perhaps integrate it directly to brush as well. I have a live demo in https://websfm.onrender.com/ which is in state of "works on my machine" , I have yet to find params that would work the best in certain scenarios.

Reconstruction in the image is done with the 128 image dataset from https://demuc.de/colmap/datasets/ "South Building" COLMAP test set:

I'll soon add a feature to be able to export the reconstruction in a way that it supports brush directly.

[MatiasHiltunen]

From image set to splat, 100% computed in browser! On my HP Omnibook 5 Laptop... :)

[rowild]

That caught my eye! I tested it with the same image set, and I got this message:

Track quality: 17319 points observed in 3+ images, mean track length 4.58, guided observations 13612
Diagnostics: 3 adjacent links failed. Split the sequence around the first failed pair or increase features before export.

The result is upside-down and the canvas cannot be rotated completely...

I am sure I misunderstood a lot. Are you planning to make this a "real thing"? With documentation, github repo or whatever? – Thanks for sharing anyway!! +1

[rowild]

..."Flip up" certainly helps :-) Question: Why is the result flipped initially? Is that depending on the image set?

[MatiasHiltunen]

Nice! It seems to be working on someone else's machine which is great! :D The orientation can be almost anything, depends on the initial successful component that in which orientation it has been found how the end result is going to be build around it (most likely). And yes, I'm planning publishing the repo at some point as open source once I have got the necessary parts working correctly, much of the logic can still be moved to gpu side and I'm aiming to reduce costly allocations to minimum. I'm working on a patch to allow exporting brush compatible zip bundle directly after the run

[MatiasHiltunen]

Try adding the focal override from my earlier screenshot, that should help to maintain the track in line!

[MatiasHiltunen]

The feature to export as "brush dataset.zip" is now live https://websfm.onrender.com/

Fixed also a couple of bugs and now the produced 3D reconstruction is quite okay. Some images from the dataset are not accepted and it is actually better when the bad confidence images are left out of the end result. Focal correction needs to be manually input for these test datasets and 2600 is used in this run. By default fe. 1:1 typical pin-hole phone images are fine.

After the recent fixes to reconstruction pipeline the performance took a hit but its still quite a bit faster than running COLMAP on my laptop CPU.

[MatiasHiltunen]

Some features of a coming version here: https://www.youtube.com/watch?v=IKJER9dWWr8 @rowild

[rowild]

@MatiasHiltunen This is truly impressive!!! Will try it soon! Thank you for sharking and pointing it out to me! :-)