Crash when trying to convert a vpc into a raster with raster_tin

[flemmens]

Hello,

I would like to report an issue:

I have a bunch of .laz files which I would like to convert into a raster.

To begin I create a vpc, which works fine:

pdal_wrench build_vpc --output=myterrain.vpc --input-file-list=my-laz-tiles.txt

Then I want to convert the vpc into a raster tif using the raster_tin method:

pdal_wrench to_raster_tin --resolution=0.2 --input=myterrain.vpc --output=myterrain.tif

If I have only a few .laz files in my-laz-tiles.txt, everything works well.

However, if I have more files (let's say 100+), the process starts and begins to create the temporary directory, but after a while it crashes

0...Killed

I tried with different files on several computers, it always crash after a while if there are too many files in the vpc. Sometimes it goes up to 100 but then it stucks creating the final tif (probably because it don't use bigtiff and is over the 4Gb limit of the regular tiffs).

[flemmens]

Note: when using to_raster instead of to_raster_tin, it usually goes a bit further but eventualy also crash. I noticed that all cpus are used at 100%, which is probably not good as I can't even log on my machine from another terminal..

[wonder-sk]

hi @flemmens

I think the problem in your case will be that you run out of memory and the pdal_wrench process gets killed. By default, pdal_wrench runs as computation in multiple threads - as many as the number of logical CPUs. The raster export via TIN is quite memory intensive, so try limiting the number of threads with e.g. --threads 4 command line option and see if that helps.

Bigtiff should not be a problem - the raster creation is done by GDAL which will use bigtiff automatically if needed.

As for to_raster also crashing, that can be a different problem. A backtrace of the crash would help...

[flemmens]

Thank you for your reply @wonder-sk, I was not aware about this --threads option.

I made a new test with a small VLC of 219 laz files (24Gb) and ran the process on a 96Gb RAM computer with 48 cores using only 4 threads.

The process last a lot longer but finaly crashes at 30%, so it still does not work. However, the subtiles created in the temporary directory are OK, I was able to join them using gdal and got a GeoTIFF of about 1/3 of my terrain.

All the .laz used for the test have been checked, I can process them with a classic pdal pipeline without having any issue. If you wish do the test by yourself I can provide a S3 link with the files.