Replies: 1 comment
-
As an update, I also tried using the image registrator, and it registers the PROJ camera in just a couple minutes, but it's always in a wrong, way too close spot. you can also see the weird streak of graycode points that are shared between the projector and the two cams CAM A and CAM B which seems to be quite off. I set "ignore_two_view_tracks" to off |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hi! I have been struggling with a task for a while, and it seems that I am very close, so i'm hoping maybe someone out there will be able to help me!
I am trying to basically use colmap to help with a projection mapping system. I want to scan a scene (twice, once with photogrammetry, and once with graycode projected from a projector). I then want to trick colmap into using the millions of graycode correspondances to
A) improve the overall scan
B) most-importantly, to calibrate and localize the projector in the 3D scene.
Here's how it currently works:
Photogrammetry collection
-We have a pair of stereo cameras (CamA and CamB) that takes 50 photos of a scene from different angles, all around.
-Then we set those stereo cameras down in a final position and take one last canonical "bridging" set of images of the scene
Structured light collection
-Next we run a structured light program with those cameras in the same place and a projector blasting the scene with graycode
-the graycode image set is decoded and its correspondences between CamA and the projector pixels and CamB and the projector pixels are saved out to a CSV.
-Now we have a set of two extra images with ~2.5 million extra points
Processing
-We then run colmap on just the Photogrammetry collection of images. It generally works terrifically, and we end up with decently well calibration parameters of CamA and CamB along with a sparse reconstruction model of the scene with something generally like 25K points. We have a database of the Photogrammetry scan and a folder of the resulting model.
Merging
My goal is to bring in the 2.5 million graycode correspondances from the structured light scan and merge it into the photogrammetry scan.
I wrote a little script that goes to a colmap database and adds extra keypoints, matches, and two-view geometries from the graycode scan.
(You can actually see the graycode database injector script i made here https://github.com/quitmeyer/GraycodeToPLYConverter )
In it, what happens is, I create an extra "virtual camera" that is actually the projector (called PROJ) (in the database it is now camera 3). I actually put a new all white image into the IMG folder that the colmap database is looking at with the same dimensions as the projector.
Then i find those two "bridging" photos (Cam A and Cam B) from the existing photogrammetry scan and I add the extra keypoints from the graycode scan to augment each of the existing sets of SIFT features.
I then augment the existing photogrammetry "matches" and "two-view geometries" between CAM A and CAM B with the new graycode matches. I also add matches and TVG between CAM A and PROJ and CAM B and PROJ
and after lots of tweaking, i think this part is all pretty good! The database loads well in colmap, I can preview the keypoints and matches i added in the database management option in the colmap GUI.
You can even see an image synthesized when you look at the virtual "PROJ" image that lines up great with the projector's point of view! (I have even reprojected this image of just red correspondances back, and it lines up great!)
So i think I have all the information that colmap needs, I just now need to get colmap to put it all together correctly.
Now here's the part where I need help.
If i run an entirely new reconstruction with the augmented database and virtual projector image it will sometimes correctly localize all the camera images AND the projector image . But this seems to only have happened with lower-resolution projectors (1080p and below) and only in graycode scans were there were fewer points (like 200k points and below). it also takes sometimes 20 hours to try to reconstruct everything with all these extra points. Also most often is tries to reconstruct everything but filters out the PROJ image.
Instead, I want to somehow tell colmap:
"please look at this new projector image (that doesn't have any true SIFT features) and look at the millions of correspondances it has with CAMA and CAMB (which are already localized in the scene), and please correctly refine the extrinsics and intrinsic of the PROJ image.
One thing I have recently tried to do is, after I have merged the databases is run the point_triangulator and then Mapper (but it hasn't really worked)
What i have to do is
go to the model of the original photogrammetry reconstruction and blank out the lines in the images.txt of the model underneath CAMA and CAMB (or else the point triangulator and mapper get angry at me)
run point triangulator like this
colmap.exe point_triangulator --database_path C:\Users\andre\Desktop\colmapPy\demoGrayCode\SSI_B.dbnew.db --image_path C:\Users\andre\Desktop\colmapPy\demoGrayCode\img --input_path C:\Users\andre\Desktop\colmapPy\demoGrayCode\ModelwithMattNoGooch_ManEdit2 --output_path C:\Users\andre\Desktop\colmapPy\demoGrayCode\ModelwithMattYesGooched --Mapper.tri_ignore_two_view_tracks 0 --clear_points 1
making sure "clear points" is enabled
colmap.exe mapper --database_path C:\Users\andre\Desktop\colmapPy\demoGrayCode\SSI_B.dbnew.db --image_path C:\Users\andre\Desktop\colmapPy\demoGrayCode\img --input_path C:\Users\andre\Desktop\colmapPy\demoGrayCode\ModelwithMattYesGooched --output_path C:\Users\andre\Desktop\colmapPy\demoGrayCode\ModelwithMattYesGoochedMapped --Mapper.tri_ignore_two_view_tracks 0
but the result almost always filters out the PROJ image. I have drawn in where the projector virtual camera image should be in this scene.
You can also see that if I set the "track length" to 2 in the render view options
you can see where the projector had shown graycode onto the scene!
Anyway, it seems like I am close to getting this right, but im not a super colmap expert and there's perhaps a better way to be doing this!
Beta Was this translation helpful? Give feedback.
All reactions