odm_georef fails with large point clouds (exit code 139, seg fault)

[thsant]

I have gotten a Segmentation fault during odm_georef execution:

[DEBUG] running /code/build/bin/odm_georef -bundleFile /datasets/code/opensfm/bundle_r000.out -inputTransformFile /datasets/code/opensfm/geocoords_transformation.txt -inputCoordFile /datasets/code/odm_georeferencing/coords.txt -inputFile /datasets/code/odm_texturing/odm_textured_model.obj -outputFile /datasets/code/odm_texturing/odm_textured_model_geo.obj -inputPointCloudFile /datasets/code/smvs/smvs_dense_point_cloud.ply -outputPointCloudFile /datasets/code/odm_georeferencing/odm_georeferenced_model.ply -logFile /datasets/code/odm_georeferencing/odm_georeferencing_log.txt -outputTransformFile /datasets/code/odm_georeferencing/odm_georeferencing_transform.txt -georefFileOutputPath /datasets/code/odm_georeferencing/odm_georeferencing_model_geo.txt

According to the output, the child process returned a 139 code (128 + 11 = 139, 11 is the code for a segmentation fault, SIGSEV):

Segmentation fault (core dumped)
Traceback (most recent call last):
  File "/code/run.py", line 47, in <module>
    plasm.execute(niter=1)
  File "/code/scripts/odm_georeferencing.py", line 127, in process
    '-logFile {log} -outputTransformFile {transform_file} -georefFileOutputPath {geo_sys}'.format(**kwargs))
  File "/code/opendm/system.py", line 34, in run
    raise Exception("Child returned {}".format(retcode))
Exception: Child returned 139

I have checked the input files:

-bundleFile /datasets/code/opensfm/bundle_r000.out 
-inputTransformFile /datasets/code/opensfm/geocoords_transformation.txt 
-inputCoordFile /datasets/code/odm_georeferencing/coords.txt 
-inputFile /datasets/code/odm_texturing/odm_textured_model.obj 
-inputPointCloudFile /datasets/code/smvs/smvs_dense_point_cloud.ply 

All of them look fine, but the smvs_dense_point_cloud.ply is an amazing PLY presenting 7.8 Gb. The machine running the Docker application is exclusively used for ODM and it has 252 Gb RAM. The header of this PLY looks fine:

$ head -16 smvs/smvs_dense_point_cloud.ply 
ply
format binary_little_endian 1.0
comment Export generated by libmve
element vertex 236821827
property float x
property float y
property float z
property float nx
property float ny
property float nz
property uchar red
property uchar green
property uchar blue
property float confidence
property float value
end_header

The dataset is composed by 703 images (6000 x 4000 pixels) captured by a Sony ILCE-6000. The georeferenced textured mesh odm_textured_model_geo.obj was successfully built and I'm able to inspect it in Meshlab. ODM was started using:

docker run -ti --rm -v /home/odm/laurimar:/datasets/code opendronemap/opendronemap --project-path /datasets --dsm --dtm --build-overviews

Any tip of advice?

UPDATE: Using ODM 0.4.0 here

[thsant]

Considering the possible link to Issue #730 , the GPS info in EXIF looks fine:

$ exiftool images/DSC00001_geotag.JPG | grep GPS

GPS Latitude Ref                : South
GPS Longitude Ref               : West
GPS Altitude                    : 100.45 m
GPS Latitude                    : 12 deg 9' 4.82" S
GPS Longitude                   : 56 deg 1' 9.05" W
GPS Position                    : 12 deg 9' 4.82" S, 56 deg 1' 9.05" W

[pierotofy]

Looks like this issue: http://community.opendronemap.org/t/exception-child-returned-139-when-running-georeferencing-with-opensfm-transformation-matrix/1319/3

Are you using the docker images?

[thsant]

@pierotofy yes, I'm using Docker.

[pierotofy]

If you re-process the dataset using --use-opensfm-dense does processing continue or do you get the same error? My current guess is a bug in the PCL library that happens when reading large point clouds.

[thsant]

@pierotofy , your current guess looks right: odm_georef finished successfuly:

[INFO] Running ODM Georeferencing Cell - Finished

[pierotofy]

👍 do you know how many points the 7.8GB PLY had? I'm just trying to narrow down the problem and possibly study a fix for it.

[thsant]

The smvs_dense_point_cloud.ply cloud has 236,821,827 points.

[pierotofy]

Possibly related: PointCloudLibrary/pcl#2145

[pierotofy]

@thsant would you be able to share (even privately) these files:

opensfm/bundle_r000.out
opensfm/geocoords_transformation.txt
odm_georeferencing/coords.txt
smvs/smvs_dense_point_cloud.ply

and every file in odm_texturing/?

You can use Dropbox or Google Drive.

[thsant]

Sure, @pierotofy

The files are available at Dropbox

This is PLY file header :

ply
format binary_little_endian 1.0
comment Export generated by libmve
element vertex 236821827
property float x
property float y
property float z
property float nx
property float ny
property float nz
property uchar red
property uchar green
property uchar blue
property float confidence
property float value
end_header

[pierotofy]

Possibly another dataset to test this problem: http://community.opendronemap.org/t/failure-on-georeferencing-with-larger-dataset/1464

[pierotofy]

Well, today I just re-confirmed this with a 1066 images dataset.

Segmentation fault (core dumped)
Traceback (most recent call last):
File "/code/run.py", line 47, in <module>
plasm.execute(niter=1)
File "/code/scripts/odm_georeferencing.py", line 127, in process
'-logFile {log} -outputTransformFile {transform_file} -georefFileOutputPath {geo_sys}'.format(**kwargs))
File "/code/opendm/system.py", line 34, in run
raise Exception("Child returned {}".format(retcode))
Exception: Child returned 139

[smathermather]

It'll still need a fix regardless, as it will break at some higher number, but I wonder if using mve instead of smvs would help (can be tested here or at the pull request @thsant)?

[thsant]

Is a Docker image there for the mve branch or should I do a local build?

[pierotofy]

@smathermather yeah MVE will produce less points, but it's an error that could still happen. I'm thinking that we should have a --max-points flag to place a ceiling on the number of points in our point cloud outputs. The other option would probably require modifying PCL, or remove it from the part where it causes odm_georef to seg fault.

[smathermather]

Yes @pierotofy -- just hoping it fixes @thsant's problem in the meantime. I'd prefer we fix the problem, either by upstream patch or elimination of the library from odm_georef rather than adding a flag. An alternative would be to multi-thread the georeferencing, which would require breaking the point cloud into parts.

[smathermather]

@thsant -- that's either a local build or you can build your own docker image from the branch: https://github.com/smathermather/OpenDroneMap/blob/mvs_and_smvs/README.md#build-and-run-using-docker

[smathermather]

Yes @pierotofy -- just hoping it fixes @thsant's problem in the meantime. I'd prefer we fix the problem, either by upstream patch or elimination of the library from odm_georef rather than adding a flag. An alternative would be to multi-thread the georeferencing, which would require breaking the point cloud into parts.

@pierotofy -- we could of course just subsample our point cloud as you say with the --max-points flag. The challenge with doing so is maintaining features, ala what pdal does with poisson sampling. Of course, the challenge with that approach is estimating what the distance for the sampling needs set to in order to get near the number of points you are targeting.

[pierotofy]

I was thinking of a more naive approach such as https://pdal.io/stages/filters.decimation.html, but as you say, it would be best to allow odm_georef to handle larger inputs.

[smathermather]

The challenge with decimation is feature retention: https://pdal.io/tutorial/sampling/index.html, which could have a real effect on retaining ground points. With lidar, it tends to emphasize ground points as there are more of them. With photogrammetrically derived point clouds, the effect will be the opposite, reducing the quality of the DTM.

[pierotofy]

Mm, yes that could be a problem. My concern is that after odm_georef is fixed, we might still encounter other issues in other parts of the pipeline (DSM calculation for example). Placing a ceiling could give users the option to sacrifice some quality for getting at least some output. I'll still go ahead and attempt to fix odm_georef first, keeping this option as a plan-B.

[pierotofy]

This is still a problem. The comment from YouYue123 in PointCloudLibrary/pcl#2145 holds true. Points are read, but are all set to zero. This causes odm_georef to output a point cloud with zeros all over. Symptoms include process freezing at the cropping area estimation instead of failing with code 139.

[pierotofy]

And the actual fix: OpenDroneMap/pcl#1

[smathermather]

That was quick. 😺

[pierotofy]

This is now fixed for good.

[smathermather]

Nice. Fixed upstream and merged I see. 👍

[manik9000]

How can I reflect these changes from OpenDroneMap/pcl to OpenDroneMap/ODM ?
Note that, I am using docker for ODM. So, do I need to modify something in Docker file to force it to pull PCL from OpenDroneMap/pcl ?

[pierotofy]

You don't need to upgrade PCL, the final fix ended up using PDAL to write the point cloud properly. Just make sure you update ODM to the latest version.