Spark Package to read and write PLY, LAS and XYZ lidar point clouds using Spark SQL.
Scala
Switch branches/tags
Nothing to show
Clone or download
pfxuan and mbredif Should not close file system inside of map call.
Since the file system handler is shared by other map threads.
Latest commit 3a2070e Mar 29, 2017

README.md

Spark SQL IQmulus Library

A library for reading and writing Lidar point cloud collections in PLY, LAS and XYZ formats from Spark SQL.

Build Status codecov.io

Requirements

This Spark package is for Spark 1.6.2

Usage

The spark-iqmulus library is published as a Spark package, which can be added to Spark jobs launched through spark-shell or spark-submit by using the --packages command line option. For example, to include it when starting the spark shell:

$ bin/spark-shell --packages IGNF:spark-iqmulus:0.1.0-s_2.10

Unlike using --jars, using --packages ensures that this library and its dependencies will be added to the classpath. The --packages argument can also be used with bin/spark-submit. For now this library is only published for Scala 2.10.

Scala API

Examples below require the creation of an SQLContext and the following imports, depending on which aspect of the library is being used:

val sqlContext = new SQLContext(sc)

import fr.ign.spark.iqmulus.ply._
import fr.ign.spark.iqmulus.las._
import fr.ign.spark.iqmulus.xyz._

Reading files as Spark dataframes:

val df1 = sqlContext.read.ply("myfile.ply")
val df2 = sqlContext.read.las("myfile.las")
val df3 = sqlContext.read.xyz("myfile.xyz")

Writing back Spark dataframes as files:

similarly, for XYZ and LAS :

val df = ...
df.write.ply("/tmp/ply")
df.write.las("/tmp/las")
df.write.xyz("/tmp/xyz")

Acknowledgments

IGN France IQmulus The development of this library is partly supported by:

  • the EU FP7 Project IQmulus, which leverages the information hidden in large heterogeneous geospatial data sets and make them a practical choice to support reliable decision making (N. ICT- 2011-318787).
  • IGN, the French National Mapping Agency and its MATIS research lab.

References

To our knowledge this library has been used in the following research: