Real Time Analytics and Data Pipelines based on Spark Streaming

HDFS compress tar zip snappy gzip uncompress untar codec hadoop spark

Hadoop splittable InputFormat for ROS. Process rosbag with Hadoop Spark and other HDFS compatible systems.

A set of connectors for Monix. 🔛

Declarative text based tool for data analysts and engineers to extract, load, transform and orchestrate their data pipelines.

全套大数据基础学习教程，包含最基础的centos、maven。大数据主要包含hdfs、mr、yarn、hbase、kafka、scala、sparkcore、sparkstreaming、sparksql。教程包含所有的源代码演示以及在线文档说明。

零基础学习spark，大数据学习

A light Kafka to HDFS/S3 ETL library based on Apache Spark

Components for building stream loaders from Kafka to arbitrary storages

全国大数据竞赛三等奖解决方案，省赛二等奖解决方案。一键安装大数据环境脚本，自动部署集群环境，包括zookeeper、hadoop、mysql、hive、spark以及一些基础环境。已通过实际服务器测试，效果极佳，仅需要输入密码等少量人为干预。解放安装部署配置所需人力。并添加若干scala案例，结合spark用以进行数据准备。

WASP is a framework to build complex real time big data applications. It relies on a kind of Kappa/Lambda architecture mainly leveraging Kafka and Spark. If you need to ingest huge amount of heterogeneous data and analyze them through complex pipelines, this is the framework for you.

Bucketing and partitioning system for Parquet

FITS data source for Spark SQL and DataFrames

EasterBunny数据分析

A bunch of low-level basic methods for data processing and monitoring with Scala Spark

An sbt plugin for publishing artifacts to HDFS.

spark hdfs tika

Spark, Spark Streaming and Kafka Streaming examples

Change data capture realization using Spark and Sqoop