Skip to content
A new arguably faster implementation of Apache Spark from scratch in Rust
Rust Other
  1. Rust 98.7%
  2. Other 1.3%
Branch: master
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
config_files automatically dropping executors done(#26) Nov 4, 2019
docker Enable env var for ssh user Nov 13, 2019
examples Enable env var for ssh user Nov 13, 2019
src fold and aggregate test Nov 15, 2019
tests fold and aggregate test Nov 15, 2019
userGuide Ignore code blocks for tests Nov 14, 2019
.dockerignore Docker image buidl scripts Nov 12, 2019
.gitignore
.travis.yml Update .travis.yml Nov 14, 2019
Cargo.lock Add GapSamplingReplacement Nov 5, 2019
Cargo.toml Add more tests and fix minor errors with sampler instantiation Nov 6, 2019
LICENSE Initial commit Oct 22, 2019
README.md Add intro to documentation Nov 14, 2019
build.rs initial commit Oct 23, 2019
rust-toolchain Add rust-toolchain file closes #2 Oct 23, 2019

README.md

native_spark

Join the chat at https://gitter.im/fast_spark/community Build Status License

Documentation

A new, arguably faster, implementation of Apache Spark from scratch in Rust. WIP

Framework tested only on Linux, requires nightly Rust. Read how to get started in the documentation.

ToDo

  • Error Handling(Priority)
  • Fault tolerance

RDD

Most of these except file reader and writer are trivial to implement

  • map
  • flat_map
  • filter
  • group_by
  • reduce_by
  • distinct
  • count
  • take_sample
  • union
  • glom
  • cartesian
  • pipe
  • map_partitions
  • for_each
  • collect
  • reduce
  • fold
  • aggregate
  • take
  • first
  • sample
  • zip
  • save_as_text_file (can save only as text file in executors local file system)

Config Files

  • Replace hard coded values
You can’t perform that action at this time.