Skip to content
FITS data source for Spark SQL and DataFrames
Scala Python Shell Java
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
docs Bump from 0.8.2 to 0.8.3 Jun 12, 2019
examples Bump version 0.7.1 -> 0.7.2 Dec 3, 2018
pic
project Upgrade Spark version to 2.4.3. Add support for Scala 2.12 May 13, 2019
src
.gitignore
.travis.yml
CONTRIBUTING.rst Update the README and the CONTRIBUTING sections May 25, 2018
LICENSE Initial commit Jan 31, 2018
README.md
build.sbt
run_image_python.sh
run_java.sh Bump from 0.8.2 to 0.8.3 Jun 12, 2019
run_python.sh
run_python_cluster.sh Bump from 0.8.2 to 0.8.3 Jun 12, 2019
run_python_cori.sh
run_scala-2.11.sh
run_scala-2.12.sh
run_scala_cluster.sh
run_scala_cluster_image.sh
run_scala_cori.sh
test_scala-2.11.sh
test_scala-2.12.sh

README.md

FITS Data Source for Apache Spark

Build Status codecov Maven Central Arxiv

Latest news

  • [01/2018] Launch: project starts!
  • [03/2018] Release: version 0.3.0
  • [04/2018] Paper: Arxiv
  • [05/2018] Release: version 0.4.0
  • [06/2018] New location: spark-fits is an official project of AstroLab!
  • [07/2018] Release: version 0.5.0, 0.6.0
  • [10/2018] Release: version 0.7.0, 0.7.1
  • [12/2018] Release: version 0.7.2
  • [03/2019] Release: version 0.7.3
  • [05/2019] Release: version 0.8.0, 0.8.1, 0.8.2
  • [06/2019] Release: version 0.8.3

spark-fits

This library provides two different tools to manipulate FITS data with Apache Spark:

  • A Spark connector for FITS file.
  • A Scala library to manipulate FITS file.

The user interface has been done to be the same as other built-in Spark data sources (CSV, JSON, Avro, Parquet, etc). Note that spark-fits follows Apache Spark Data Source V1 (plan to migrate to V2). See our website for more information. To include spark-fits in your job:

# Scala 2.11
spark-submit --packages "com.github.astrolabsoftware:spark-fits_2.11:0.8.3" <...>

# Scala 2.12
spark-submit --packages "com.github.astrolabsoftware:spark-fits_2.12:0.8.3" <...>

or you can link against this library in your program at the following coordinates in your build.sbt

// Scala 2.11
libraryDependencies += "com.github.astrolabsoftware" % "spark-fits_2.11" % "0.8.3"

// Scala 2.12
libraryDependencies += "com.github.astrolabsoftware" % "spark-fits_2.12" % "0.8.3"

Currently available:

  • Read fits file and organize the HDU data into DataFrames.
  • Automatically distribute bintable rows over machines.
  • Automatically distribute image rows over machines.
  • Automatically infer DataFrame schema from the HDU header.

Header Challenge!

The header tested so far are very simple, and not so exotic. Over the time, we plan to add many new features based on complex examples (see here). If you use spark-fits, and encounter errors while reading a header, tell us (issues or PR) so that we fix the problem asap!

TODO list

  • Define custom Hadoop InputFile.
  • Migrate to Spark DataSource V2

Support

You can’t perform that action at this time.