diff --git a/docs/backend/spark-architecture.md b/docs/backend/spark-architecture.md new file mode 100644 index 00000000..3d131920 --- /dev/null +++ b/docs/backend/spark-architecture.md @@ -0,0 +1,64 @@ +# Spark architecture + +This section describes how the [Backend architecture](../index.md#backend) is implemented in Spark Streaming. + +## Data Flow Graph + +Bullet Spark implements the backend piece from the full [Architecture](../index.md#architecture). It is implemented with Spark Streaming: + +![Bullet Spark DAG](../img/spark-dag.png) + +The components in the [Architecture](../index.md#architecture) have direct counterparts here. The Query Receiver reading from the PubSub layer using plugged-in PubSub consumers and the Query Unioning make up the Request Processor. The Filter Streaming and your plugin for your source of data make up the Data Processor. The Join Streaming and the Result Emitter make up the Combiner. + +The red lines are the path for the queries that come in through the PubSub, the orange lines are the path for the signals and the blue lines are for the data from your data source. The shapes of the boxes denote the type of transformation/action being executed in the boxes. + +### Data processing + +Bullet can accept arbitrary sources of data as long as they can be ingested by Spark. They can be Kafka, Flume, Kinesis, and TCP sockets etc. In order to hook up your data to Bullet Spark, you just need to implement the [Data Producer Trait](https://github.com/bullet-db/bullet-spark/blob/master/src/main/scala/com/yahoo/bullet/spark/DataProducer.scala). In your implementation, you can either: +* Use [Spark Streaming built-in sources](https://spark.apache.org/docs/latest/streaming-programming-guide.html#input-dstreams-and-receivers) to receive data. Below is a quick example for a direct Kafka source in Scala. You can also write it in Java: + +```scala +import com.yahoo.bullet.spark.DataProducer +import org.apache.spark.streaming.kafka010.{ConsumerStrategies, KafkaUtils, LocationStrategies} +// import all other necessary packages + +class DirectKafkaProducer extends DataProducer { + override def getBulletRecordStream(ssc: StreamingContext, config: BulletSparkConfig): DStream[BulletRecord] = { + val topics = Array("test") + val kafkaParams = Map[String, AnyRef]( + "bootstrap.servers" -> "server1, server2", + "group.id" -> "mygroup", + "key.deserializer" -> classOf[StringDeserializer], + "value.deserializer" -> classOf[ByteArrayDeserializer] + // Other kafka params + ) + + val directKafkaStream = KafkaUtils.createDirectStream[String, Array[Byte]]( + ssc, + LocationStrategies.PreferConsistent, + ConsumerStrategies.Subscribe[String, Array[Byte]](topics, kafkaParams)) + + directKafkaStream.map(record => { + // Convert your record to BulletRecord + }) + } +} +``` + +* Write a [custom receiver](https://spark.apache.org/docs/latest/streaming-custom-receivers.html) to receive data from any arbitrary data source beyond the ones for which it has built-in support (that is, beyond Flume, Kafka, Kinesis, files, sockets, etc.). See [example](https://github.com/bullet-db/bullet-db.github.io/tree/src/examples/spark/src/main/scala/com/yahoo/bullet/spark/examples). + +After receiving your data, you can do any transformations like joins or type conversions in your implementation before emitting to the Filter Streaming stage. + +The Filter Streaming stage checks every record from your data source against every query from Query Unioning stage to see if it matches and emits partial results to the Join Streaming stage. + +### Request processing + +The Query Receiver fetches Bullet queries and signals through the PubSub layer using the Subscribers provided by the plugged in PubSub layer. The queries received through the PubSub also contain information about the query such as its unique identifier, potentially other metadata and signals. The Query Unioning collects all active queries by the stateful transformation [updateStateByKey](https://spark.apache.org/docs/latest/streaming-programming-guide.html#updatestatebykey-operation) and broadcasts all the collected queries to every executor for the Filter Streaming stage. + +The Query Unioning also sends all active queries and signals to the Join Streaming stage. + +### Combining + +The Filter Streaming combines all the partial results from the Filter Streaming by the stateful transformation [mapWithState](https://spark.apache.org/docs/2.3.0/api/scala/index.html#org.apache.spark.streaming.dstream.PairDStreamFunctions@mapWithState[StateType,MappedType](spec:org.apache.spark.streaming.StateSpec[K,V,StateType,MappedType])(implicitevidence$2:scala.reflect.ClassTag[StateType],implicitevidence$3:scala.reflect.ClassTag[MappedType]):org.apache.spark.streaming.dstream.MapWithStateDStream[K,V,StateType,MappedType]) and produces final results. + +The Result Emitter uses the particular publisher from the plugged in PubSub layer to send back results/loop signals. diff --git a/docs/backend/spark-setup.md b/docs/backend/spark-setup.md new file mode 100644 index 00000000..8a725b79 --- /dev/null +++ b/docs/backend/spark-setup.md @@ -0,0 +1,93 @@ +# Bullet on Spark + +This section explains how to set up and run Bullet on Spark. + +## Configuration + +Bullet is configured at run-time using settings defined in a file. Settings not overridden will default to the values in [bullet_spark_defaults.yaml](https://github.com/bullet-db/bullet-spark/blob/master/src/main/resources/bullet_spark_defaults.yaml). You can find out what these settings do in the comments listed in the defaults. + +## Installation + +Download the Bullet Spark standalone jar from [JCenter](http://jcenter.bintray.com/com/yahoo/bullet/bullet-spark/). + +If you are using Bullet Kafka as pluggable PubSub, you can download the fat jar from [JCenter](http://jcenter.bintray.com/com/yahoo/bullet/bullet-kafka/). Otherwise, you need to plug in your own PubSub jar or use the RESTPubSub built-into bullet-core and turned on in the API. + +To use Bullet Spark, you need to implement your own [Data Producer Trait](https://github.com/bullet-db/bullet-spark/blob/master/src/main/scala/com/yahoo/bullet/spark/DataProducer.scala) with a JVM based project. You have two ways to implement it as described in the [Spark Architecture](spark-architecture.md#data-processing) section. You include the Bullet artifact and Spark dependencies in your pom.xml or other equivalent build tools. The artifacts are available through JCenter. Here is an example if you use Scala and Maven: + +```xml + + + + false + + central + bintray + http://jcenter.bintray.com + + +``` + +```xml + + 2.11.7 + 2.11 + 2.3.0 + 0.1.1 + + + + org.scala-lang + scala-library + ${scala.version} + provided + + + + org.apache.spark + spark-streaming_${scala.dep.version} + ${spark.version} + provided + + + + org.apache.spark + spark-core_${scala.dep.version} + ${spark.version} + provided + + + + com.yahoo.bullet + bullet-spark + ${bullet.spark.version} + +``` + +You can also add ```sources``` or ```javadoc``` if you want the sources or javadoc. + +## Launch + +After you have implemented your own data producer and built a jar, you could launch your Bullet Spark application. Here is an example command for a [YARN cluster](https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html). + +```bash +./bin/spark-submit \ + --master yarn \ + --deploy-mode cluster \ + --class com.yahoo.bullet.spark.BulletSparkStreamingMain \ + --queue \ + --executor-memory 12g \ + --executor-cores 2 \ + --num-executors 200 \ + --driver-cores 2 \ + --driver-memory 12g \ + --conf spark.streaming.backpressure.enabled=true \ + --conf spark.default.parallelism=20 \ + ... # other Spark settings + --jars /path/to/your-data-producer.jar,/path/to/your-pubsub.jar \ + /path/to/downloaded-bullet-spark-standalone.jar \ + --bullet-spark-conf /path/to/your-settings.yaml +``` + +You can pass other Spark settings by adding ```--conf key=value``` to the command. For more settings, you can refer to the [Spark Configuration](https://spark.apache.org/docs/latest/configuration.html). + +For other platforms, you can find the commands from the [Spark Documentation](https://spark.apache.org/docs/latest/submitting-applications.html). diff --git a/docs/img/spark-dag.png b/docs/img/spark-dag.png new file mode 100644 index 00000000..86ad4dc3 Binary files /dev/null and b/docs/img/spark-dag.png differ diff --git a/docs/releases.md b/docs/releases.md index 7bd7f638..82318b87 100644 --- a/docs/releases.md +++ b/docs/releases.md @@ -16,9 +16,9 @@ The core Bullet logic (a library) that can be used to implement Bullet on differ | | | | ------------------------- | --------------- | -| **Repository** | [https://github.com/yahoo/bullet-core](https://github.com/yahoo/bullet-core) | -| **Issues** | [https://github.com/yahoo/bullet-core/issues](https://github.com/yahoo/bullet-core/issues) | -| **Last Tag** | [![Latest tag](https://img.shields.io/github/release/yahoo/bullet-core/all.svg)](https://github.com/yahoo/bullet-core/tags) | +| **Repository** | [https://github.com/bullet-db/bullet-core](https://github.com/bullet-db/bullet-core) | +| **Issues** | [https://github.com/bullet-db/bullet-core/issues](https://github.com/bullet-db/bullet-core/issues) | +| **Last Tag** | [![Latest tag](https://img.shields.io/github/release/bullet-db/bullet-core/all.svg)](https://github.com/bullet-db/bullet-core/tags) | | **Latest Artifact** | [![Download](https://api.bintray.com/packages/yahoo/maven/bullet-core/images/download.svg)](https://bintray.com/yahoo/maven/bullet-core/_latestVersion) | | **Package Manager Setup** | [Setup for Maven, Gradle etc](https://bintray.com/bintray/jcenter?filterByPkgName=bullet-core) | @@ -26,14 +26,14 @@ The core Bullet logic (a library) that can be used to implement Bullet on differ | Date | Release | Highlights | | ------------ | --------------------------------------------------------------------------------- | ---------- | -| 2017-10-04 | [**0.2.5**](https://github.com/yahoo/bullet-core/releases/tag/bullet-core-0.2.5) | Supports an in-memory BufferingSubscriber implementation for reliable subscribing | -| 2017-10-03 | [**0.2.4**](https://github.com/yahoo/bullet-core/releases/tag/bullet-core-0.2.4) | Helpers added to Config, PubSubMessage, Metadata and JSONFormatter. FAIL signal in Metadata. PubSubMessage is JSON serializable | -| 2017-09-20 | [**0.2.3**](https://github.com/yahoo/bullet-core/releases/tag/bullet-core-0.2.3) | PubSub is no longer required to be Serializable. Makes PubSubMessage fully serializable. Utility classes and checked exceptions for PubSub | -| 2017-08-30 | [**0.2.2**](https://github.com/yahoo/bullet-core/releases/tag/bullet-core-0.2.2) | Helper methods to PubSubMessage and Config | -| 2017-08-23 | [**0.2.1**](https://github.com/yahoo/bullet-core/releases/tag/bullet-core-0.2.1) | Removes PubSubConfig, adds defaults methods to Publisher/Subscriber interfaces and improves PubSubException | -| 2017-08-16 | [**0.2.0**](https://github.com/yahoo/bullet-core/releases/tag/bullet-core-0.2.0) | PubSub interfaces and classes to implement custom communication between API and backend | -| 2017-06-27 | [**0.1.2**](https://github.com/yahoo/bullet-core/releases/tag/bullet-core-0.1.2) | Changes to the BulletConfig interface previously used in Bullet Storm. Users now use BulletStormConfig instead but YAML config is the same | -| 2017-06-27 | [**0.1.1**](https://github.com/yahoo/bullet-core/releases/tag/bullet-core-0.1.1) | First stable release containing the core of Bullet as a library including parsing, implementing queries, creating results, DataSketches etc | +| 2017-10-04 | [**0.2.5**](https://github.com/bullet-db/bullet-core/releases/tag/bullet-core-0.2.5) | Supports an in-memory BufferingSubscriber implementation for reliable subscribing | +| 2017-10-03 | [**0.2.4**](https://github.com/bullet-db/bullet-core/releases/tag/bullet-core-0.2.4) | Helpers added to Config, PubSubMessage, Metadata and JSONFormatter. FAIL signal in Metadata. PubSubMessage is JSON serializable | +| 2017-09-20 | [**0.2.3**](https://github.com/bullet-db/bullet-core/releases/tag/bullet-core-0.2.3) | PubSub is no longer required to be Serializable. Makes PubSubMessage fully serializable. Utility classes and checked exceptions for PubSub | +| 2017-08-30 | [**0.2.2**](https://github.com/bullet-db/bullet-core/releases/tag/bullet-core-0.2.2) | Helper methods to PubSubMessage and Config | +| 2017-08-23 | [**0.2.1**](https://github.com/bullet-db/bullet-core/releases/tag/bullet-core-0.2.1) | Removes PubSubConfig, adds defaults methods to Publisher/Subscriber interfaces and improves PubSubException | +| 2017-08-16 | [**0.2.0**](https://github.com/bullet-db/bullet-core/releases/tag/bullet-core-0.2.0) | PubSub interfaces and classes to implement custom communication between API and backend | +| 2017-06-27 | [**0.1.2**](https://github.com/bullet-db/bullet-core/releases/tag/bullet-core-0.1.2) | Changes to the BulletConfig interface previously used in Bullet Storm. Users now use BulletStormConfig instead but YAML config is the same | +| 2017-06-27 | [**0.1.1**](https://github.com/bullet-db/bullet-core/releases/tag/bullet-core-0.1.1) | First stable release containing the core of Bullet as a library including parsing, implementing queries, creating results, DataSketches etc | ## Bullet Storm @@ -49,10 +49,10 @@ The implementation of Bullet on Storm. Due to major API changes between Storm <= | | | | ----------------------------- | --------------- | -| **Storm-1.0+ Repository** | [https://github.com/yahoo/bullet-storm](https://github.com/yahoo/bullet-storm) | -| **Storm-0.10- Repository** | [https://github.com/yahoo/bullet-storm/tree/storm-0.10](https://github.com/yahoo/bullet-storm/tree/storm-0.10) | -| **Issues** | [https://github.com/yahoo/bullet-storm/issues](https://github.com/yahoo/bullet-storm/issues) | -| **Last Tag** | [![Latest tag](https://img.shields.io/github/release/yahoo/bullet-storm/all.svg)](https://github.com/yahoo/bullet-storm/tags) | +| **Storm-1.0+ Repository** | [https://github.com/bullet-db/bullet-storm](https://github.com/bullet-db/bullet-storm) | +| **Storm-0.10- Repository** | [https://github.com/bullet-db/bullet-storm/tree/storm-0.10](https://github.com/bullet-db/bullet-storm/tree/storm-0.10) | +| **Issues** | [https://github.com/bullet-db/bullet-storm/issues](https://github.com/bullet-db/bullet-storm/issues) | +| **Last Tag** | [![Latest tag](https://img.shields.io/github/release/bullet-db/bullet-storm/all.svg)](https://github.com/bullet-db/bullet-storm/tags) | | **Latest Artifact** | [![Download](https://api.bintray.com/packages/yahoo/maven/bullet-storm/images/download.svg)](https://bintray.com/yahoo/maven/bullet-storm/_latestVersion) | | **Package Manager Setup** | [Setup for Maven, Gradle etc](https://bintray.com/bintray/jcenter?filterByPkgName=bullet-storm) | @@ -60,19 +60,38 @@ The implementation of Bullet on Storm. Due to major API changes between Storm <= | Date | Storm 1.0 | Storm 0.10 | Highlights | | ------------ | ---------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | ---------- | -| 2017-10-24 | [**0.6.2**](https://github.com/yahoo/bullet-storm/releases/tag/bullet-storm-0.6.2) | [**0.6.2**](https://github.com/yahoo/bullet-storm/releases/tag/bullet-storm-0.10-0.6.2) | Adds a fat jar for using the DRPC PubSub in the Web Service | -| 2017-10-18 | [**0.6.1**](https://github.com/yahoo/bullet-storm/releases/tag/bullet-storm-0.6.1) | [**0.6.1**](https://github.com/yahoo/bullet-storm/releases/tag/bullet-storm-0.10-0.6.1) | DRPC PubSub | -| 2017-08-30 | [**0.6.0**](https://github.com/yahoo/bullet-storm/releases/tag/bullet-storm-0.6.0) | [**0.6.0**](https://github.com/yahoo/bullet-storm/releases/tag/bullet-storm-0.10-0.6.0) | New PubSub architecture, removes DRPC components and settings | -| 2017-06-27 | [**0.5.0**](https://github.com/yahoo/bullet-storm/releases/tag/bullet-storm-0.5.0) | [**0.5.0**](https://github.com/yahoo/bullet-storm/releases/tag/bullet-storm-0.10-0.5.0) | Pulled out Bullet Core. BulletConfig to BulletStormConfig | -| 2017-06-09 | [**0.4.3**](https://github.com/yahoo/bullet-storm/releases/tag/bullet-storm-0.4.3) | [**0.4.3**](https://github.com/yahoo/bullet-storm/releases/tag/bullet-storm-0.10-0.4.3) | Adding rounding for DISTRIBUTION. Latency metric | -| 2017-04-28 | [**0.4.2**](https://github.com/yahoo/bullet-storm/releases/tag/bullet-storm-0.4.2) | [**0.4.2**](https://github.com/yahoo/bullet-storm/releases/tag/bullet-storm-0.10-0.4.2) | Strict JSON output and fix for no data distributions | -| 2017-04-26 | [**0.4.1**](https://github.com/yahoo/bullet-storm/releases/tag/bullet-storm-0.4.1) | [**0.4.1**](https://github.com/yahoo/bullet-storm/releases/tag/bullet-storm-0.10-0.4.1) | Result Metadata Concept name mismatch fix | -| 2017-04-21 | [**0.4.0**](https://github.com/yahoo/bullet-storm/releases/tag/bullet-storm-0.4.0) | [**0.4.0**](https://github.com/yahoo/bullet-storm/releases/tag/bullet-storm-0.10-0.4.0) | DISTRIBUTION and TOP K release. Configuration renames. | -| 2017-03-13 | [**0.3.1**](https://github.com/yahoo/bullet-storm/releases/tag/bullet-storm-0.3.1) | [**0.3.1**](https://github.com/yahoo/bullet-storm/releases/tag/bullet-storm-0.10-0.3.1) | Extra records accepted after query expiry bug fix | -| 2017-02-27 | [**0.3.0**](https://github.com/yahoo/bullet-storm/releases/tag/bullet-storm-0.3.0) | [**0.3.0**](https://github.com/yahoo/bullet-storm/releases/tag/bullet-storm-0.10-0.3.0) | Metrics interface, config namespace, NPE bug fix | -| 2017-02-15 | [**0.2.1**](https://github.com/yahoo/bullet-storm/releases/tag/bullet-storm-0.2.1) | [**0.2.1**](https://github.com/yahoo/bullet-storm/releases/tag/bullet-storm-0.10-0.2.1) | Acking support, Max size and other bug fixes | -| 2017-01-26 | [**0.2.0**](https://github.com/yahoo/bullet-storm/releases/tag/bullet-storm-0.2.0) | [**0.2.0**](https://github.com/yahoo/bullet-storm/releases/tag/bullet-storm-0.10-0.2.0) | GROUP (DISTINCT, SUM, COUNT, MIN, MAX, AVG) | -| 2017-01-09 | [**0.1.0**](https://github.com/yahoo/bullet-storm/releases/tag/bullet-storm-0.1.0) | [**0.1.0**](https://github.com/yahoo/bullet-storm/releases/tag/bullet-storm-0.10-0.1.0) | COUNT DISTINCT and micro-batching | +| 2017-10-24 | [**0.6.2**](https://github.com/bullet-db/bullet-storm/releases/tag/bullet-storm-0.6.2) | [**0.6.2**](https://github.com/bullet-db/bullet-storm/releases/tag/bullet-storm-0.10-0.6.2) | Adds a fat jar for using the DRPC PubSub in the Web Service | +| 2017-10-18 | [**0.6.1**](https://github.com/bullet-db/bullet-storm/releases/tag/bullet-storm-0.6.1) | [**0.6.1**](https://github.com/bullet-db/bullet-storm/releases/tag/bullet-storm-0.10-0.6.1) | DRPC PubSub | +| 2017-08-30 | [**0.6.0**](https://github.com/bullet-db/bullet-storm/releases/tag/bullet-storm-0.6.0) | [**0.6.0**](https://github.com/bullet-db/bullet-storm/releases/tag/bullet-storm-0.10-0.6.0) | New PubSub architecture, removes DRPC components and settings | +| 2017-06-27 | [**0.5.0**](https://github.com/bullet-db/bullet-storm/releases/tag/bullet-storm-0.5.0) | [**0.5.0**](https://github.com/bullet-db/bullet-storm/releases/tag/bullet-storm-0.10-0.5.0) | Pulled out Bullet Core. BulletConfig to BulletStormConfig | +| 2017-06-09 | [**0.4.3**](https://github.com/bullet-db/bullet-storm/releases/tag/bullet-storm-0.4.3) | [**0.4.3**](https://github.com/bullet-db/bullet-storm/releases/tag/bullet-storm-0.10-0.4.3) | Adding rounding for DISTRIBUTION. Latency metric | +| 2017-04-28 | [**0.4.2**](https://github.com/bullet-db/bullet-storm/releases/tag/bullet-storm-0.4.2) | [**0.4.2**](https://github.com/bullet-db/bullet-storm/releases/tag/bullet-storm-0.10-0.4.2) | Strict JSON output and fix for no data distributions | +| 2017-04-26 | [**0.4.1**](https://github.com/bullet-db/bullet-storm/releases/tag/bullet-storm-0.4.1) | [**0.4.1**](https://github.com/bullet-db/bullet-storm/releases/tag/bullet-storm-0.10-0.4.1) | Result Metadata Concept name mismatch fix | +| 2017-04-21 | [**0.4.0**](https://github.com/bullet-db/bullet-storm/releases/tag/bullet-storm-0.4.0) | [**0.4.0**](https://github.com/bullet-db/bullet-storm/releases/tag/bullet-storm-0.10-0.4.0) | DISTRIBUTION and TOP K release. Configuration renames. | +| 2017-03-13 | [**0.3.1**](https://github.com/bullet-db/bullet-storm/releases/tag/bullet-storm-0.3.1) | [**0.3.1**](https://github.com/bullet-db/bullet-storm/releases/tag/bullet-storm-0.10-0.3.1) | Extra records accepted after query expiry bug fix | +| 2017-02-27 | [**0.3.0**](https://github.com/bullet-db/bullet-storm/releases/tag/bullet-storm-0.3.0) | [**0.3.0**](https://github.com/bullet-db/bullet-storm/releases/tag/bullet-storm-0.10-0.3.0) | Metrics interface, config namespace, NPE bug fix | +| 2017-02-15 | [**0.2.1**](https://github.com/bullet-db/bullet-storm/releases/tag/bullet-storm-0.2.1) | [**0.2.1**](https://github.com/bullet-db/bullet-storm/releases/tag/bullet-storm-0.10-0.2.1) | Acking support, Max size and other bug fixes | +| 2017-01-26 | [**0.2.0**](https://github.com/bullet-db/bullet-storm/releases/tag/bullet-storm-0.2.0) | [**0.2.0**](https://github.com/bullet-db/bullet-storm/releases/tag/bullet-storm-0.10-0.2.0) | GROUP (DISTINCT, SUM, COUNT, MIN, MAX, AVG) | +| 2017-01-09 | [**0.1.0**](https://github.com/bullet-db/bullet-storm/releases/tag/bullet-storm-0.1.0) | [**0.1.0**](https://github.com/bullet-db/bullet-storm/releases/tag/bullet-storm-0.10-0.1.0) | COUNT DISTINCT and micro-batching | + +## Bullet Spark + +The implementation of Bullet on Spark Streaming. + +| | | +| ------------------------- | --------------- | +| **Repository** | [https://github.com/bullet-db/bullet-spark](https://github.com/bullet-db/bullet-spark) | +| **Issues** | [https://github.com/bullet-db/bullet-spark/issues](https://github.com/bullet-db/bullet-spark/issues) | +| **Last Tag** | [![Latest tag](https://img.shields.io/github/release/bullet-db/bullet-spark/all.svg)](https://github.com/bullet-db/bullet-spark/tags) | +| **Latest Artifact** | [![Download](https://api.bintray.com/packages/yahoo/maven/bullet-spark/images/download.svg)](https://bintray.com/yahoo/maven/bullet-spark/_latestVersion) | +| **Package Manager Setup** | [Setup for Maven, Gradle etc](https://bintray.com/bintray/jcenter?filterByPkgName=bullet-spark) | + +### Releases + +| Date | Release | Highlights | +| ------------ | --------------------------------------------------------------------------------- | ---------- | +| 2018-06-08 | [**0.1.1**](https://github.com/bullet-db/bullet-spark/releases/tag/bullet-spark-0.1.1) | Adds a command flag to pass custom setting file | +| 2018-05-25 | [**0.1.0**](https://github.com/bullet-db/bullet-spark/releases/tag/bullet-spark-0.1.0) | The first release | ## Bullet Web Service @@ -84,9 +103,9 @@ The Web Service implementation that can serve a static schema from a file and ta | | | | -------------------------- | --------------- | -| **Repository** | [https://github.com/yahoo/bullet-service](https://github.com/yahoo/bullet-service) | -| **Issues** | [https://github.com/yahoo/bullet-service/issues](https://github.com/yahoo/bullet-service/issues) | -| **Last Tag** | [![Latest tag](https://img.shields.io/github/release/yahoo/bullet-service/all.svg)](https://github.com/yahoo/bullet-service/tags) | +| **Repository** | [https://github.com/bullet-db/bullet-service](https://github.com/bullet-db/bullet-service) | +| **Issues** | [https://github.com/bullet-db/bullet-service/issues](https://github.com/bullet-db/bullet-service/issues) | +| **Last Tag** | [![Latest tag](https://img.shields.io/github/release/bullet-db/bullet-service/all.svg)](https://github.com/bullet-db/bullet-service/tags) | | **Latest Artifact** | [![Download](https://api.bintray.com/packages/yahoo/maven/bullet-service/images/download.svg)](https://bintray.com/yahoo/maven/bullet-service/_latestVersion) | | **Package Manager Setup** | [Setup for Maven, Gradle etc](https://bintray.com/bintray/jcenter?filterByPkgName=bullet-service) | @@ -94,8 +113,8 @@ The Web Service implementation that can serve a static schema from a file and ta | Date | Release | Highlights | | ------------ | -------------------------------------------------------------------------------------- | ---------- | -| 2017-10-19 | [**0.1.1**](https://github.com/yahoo/bullet-service/releases/tag/bullet-service-0.1.1) | New PubSub architecture. Switching to Spring Boot and executable JAR instead of WAR | -| 2016-12-16 | [**0.0.1**](https://github.com/yahoo/bullet-service/releases/tag/bullet-service-0.0.1) | The first release with support for DRPC and the file-based schema | +| 2017-10-19 | [**0.1.1**](https://github.com/bullet-db/bullet-service/releases/tag/bullet-service-0.1.1) | New PubSub architecture. Switching to Spring Boot and executable JAR instead of WAR | +| 2016-12-16 | [**0.0.1**](https://github.com/bullet-db/bullet-service/releases/tag/bullet-service-0.0.1) | The first release with support for DRPC and the file-based schema | !!! note "Want to directly download jars?" @@ -107,23 +126,23 @@ The Bullet UI that lets you build, run, save and visualize results from Bullet. | | | | ------------------- | --------------- | -| **Repository** | [https://github.com/yahoo/bullet-ui](https://github.com/yahoo/bullet-ui) | -| **Issues** | [https://github.com/yahoo/bullet-ui/issues](https://github.com/yahoo/bullet-ui/issues) | -| **Last Tag** | [![GitHub release](https://img.shields.io/github/tag/yahoo/bullet-ui.svg)](https://github.com/yahoo/bullet-ui/tags) | -| **Latest Artifact** | [![GitHub release](https://img.shields.io/github/release/yahoo/bullet-ui.svg)](https://github.com/yahoo/bullet-ui/releases/latest) | +| **Repository** | [https://github.com/bullet-db/bullet-ui](https://github.com/bullet-db/bullet-ui) | +| **Issues** | [https://github.com/bullet-db/bullet-ui/issues](https://github.com/bullet-db/bullet-ui/issues) | +| **Last Tag** | [![GitHub release](https://img.shields.io/github/tag/bullet-db/bullet-ui.svg)](https://github.com/bullet-db/bullet-ui/tags) | +| **Latest Artifact** | [![GitHub release](https://img.shields.io/github/release/bullet-db/bullet-ui.svg)](https://github.com/bullet-db/bullet-ui/releases/latest) | ### Releases | Date | Release | Highlights | | ------------ | -------------------------------------------------------------------------------------- | ---------- | -| 2017-08-22 | [**0.4.0**](https://github.com/yahoo/bullet-ui/releases/tag/v0.4.0) | Query sharing, collapsible Raw view, and unsaved/error indicators. Settings rename and other bug fixes| -| 2017-05-22 | [**0.3.2**](https://github.com/yahoo/bullet-ui/releases/tag/v0.3.2) | Exporting to TSV in Pivot table. Fixes unselectability bug in Raw view | -| 2017-05-15 | [**0.3.1**](https://github.com/yahoo/bullet-ui/releases/tag/v0.3.1) | Adds styles to the Pivot table. Fixes some minor UI interactions | -| 2017-05-10 | [**0.3.0**](https://github.com/yahoo/bullet-ui/releases/tag/v0.3.0) | Adds Charting and Pivoting support. Migrations enhanced. Support for overriding nested default settings | -| 2017-05-03 | [**0.2.2**](https://github.com/yahoo/bullet-ui/releases/tag/v0.2.2) | Fixes maxlength of the input for points | -| 2017-05-02 | [**0.2.1**](https://github.com/yahoo/bullet-ui/releases/tag/v0.2.1) | Fixes a bug with a dependency that broke sorting the Filters | -| 2017-05-01 | [**0.2.0**](https://github.com/yahoo/bullet-ui/releases/tag/v0.2.0) | Release for Top K and Distribution. Supports Bullet Storm 0.4.2+ | -| 2017-02-21 | [**0.1.0**](https://github.com/yahoo/bullet-ui/releases/tag/v0.1.0) | The first release with support for all features included in Bullet Storm 0.2.1+ | +| 2017-08-22 | [**0.4.0**](https://github.com/bullet-db/bullet-ui/releases/tag/v0.4.0) | Query sharing, collapsible Raw view, and unsaved/error indicators. Settings rename and other bug fixes| +| 2017-05-22 | [**0.3.2**](https://github.com/bullet-db/bullet-ui/releases/tag/v0.3.2) | Exporting to TSV in Pivot table. Fixes unselectability bug in Raw view | +| 2017-05-15 | [**0.3.1**](https://github.com/bullet-db/bullet-ui/releases/tag/v0.3.1) | Adds styles to the Pivot table. Fixes some minor UI interactions | +| 2017-05-10 | [**0.3.0**](https://github.com/bullet-db/bullet-ui/releases/tag/v0.3.0) | Adds Charting and Pivoting support. Migrations enhanced. Support for overriding nested default settings | +| 2017-05-03 | [**0.2.2**](https://github.com/bullet-db/bullet-ui/releases/tag/v0.2.2) | Fixes maxlength of the input for points | +| 2017-05-02 | [**0.2.1**](https://github.com/bullet-db/bullet-ui/releases/tag/v0.2.1) | Fixes a bug with a dependency that broke sorting the Filters | +| 2017-05-01 | [**0.2.0**](https://github.com/bullet-db/bullet-ui/releases/tag/v0.2.0) | Release for Top K and Distribution. Supports Bullet Storm 0.4.2+ | +| 2017-02-21 | [**0.1.0**](https://github.com/bullet-db/bullet-ui/releases/tag/v0.1.0) | The first release with support for all features included in Bullet Storm 0.2.1+ | ## Bullet Record @@ -131,9 +150,9 @@ The AVRO container that you need to convert your data into to be consumed by Bul | | | | -------------------------- | --------------- | -| **Repository** | [https://github.com/yahoo/bullet-record](https://github.com/yahoo/bullet-record) | -| **Issues** | [https://github.com/yahoo/bullet-record/issues](https://github.com/yahoo/bullet-record/issues) | -| **Last Tag** | [![Latest tag](https://img.shields.io/github/release/yahoo/bullet-record/all.svg)](https://github.com/yahoo/bullet-record/tags) | +| **Repository** | [https://github.com/bullet-db/bullet-record](https://github.com/bullet-db/bullet-record) | +| **Issues** | [https://github.com/bullet-db/bullet-record/issues](https://github.com/bullet-db/bullet-record/issues) | +| **Last Tag** | [![Latest tag](https://img.shields.io/github/release/bullet-db/bullet-record/all.svg)](https://github.com/bullet-db/bullet-record/tags) | | **Latest Artifact** | [![Download](https://api.bintray.com/packages/yahoo/maven/bullet-record/images/download.svg)](https://bintray.com/yahoo/maven/bullet-record/_latestVersion) | | **Package Manager Setup** | [Setup for Maven, Gradle etc](https://bintray.com/bintray/jcenter?filterByPkgName=bullet-record) | @@ -141,9 +160,9 @@ The AVRO container that you need to convert your data into to be consumed by Bul | Date | Release | Highlights | | ------------ | ------------------------------------------------------------------------------------ | ---------- | -| 2017-05-19 | [**0.1.2**](https://github.com/yahoo/bullet-record/releases/tag/bullet-record-0.1.2) | Reduces the memory footprint needed to serialize itself by a factor of 128 for small records | -| 2017-04-17 | [**0.1.1**](https://github.com/yahoo/bullet-record/releases/tag/bullet-record-0.1.1) | Helper methods to remove, rename, check presence and count fields in the Record | -| 2017-02-09 | [**0.1.0**](https://github.com/yahoo/bullet-record/releases/tag/bullet-record-0.1.0) | Map constructor | +| 2017-05-19 | [**0.1.2**](https://github.com/bullet-db/bullet-record/releases/tag/bullet-record-0.1.2) | Reduces the memory footprint needed to serialize itself by a factor of 128 for small records | +| 2017-04-17 | [**0.1.1**](https://github.com/bullet-db/bullet-record/releases/tag/bullet-record-0.1.1) | Helper methods to remove, rename, check presence and count fields in the Record | +| 2017-02-09 | [**0.1.0**](https://github.com/bullet-db/bullet-record/releases/tag/bullet-record-0.1.0) | Map constructor | ## Bullet Kafka @@ -151,9 +170,9 @@ A PubSub implementation using Kafka as the backing PubSub. Can be used with any | | | | -------------------------- | --------------- | -| **Repository** | [https://github.com/yahoo/bullet-record](https://github.com/yahoo/bullet-kafka) | -| **Issues** | [https://github.com/yahoo/bullet-record/issues](https://github.com/yahoo/bullet-kafka/issues) | -| **Last Tag** | [![Latest tag](https://img.shields.io/github/release/yahoo/bullet-kafka/all.svg)](https://github.com/yahoo/bullet-kafka/tags) | +| **Repository** | [https://github.com/bullet-db/bullet-record](https://github.com/bullet-db/bullet-kafka) | +| **Issues** | [https://github.com/bullet-db/bullet-record/issues](https://github.com/bullet-db/bullet-kafka/issues) | +| **Last Tag** | [![Latest tag](https://img.shields.io/github/release/bullet-db/bullet-kafka/all.svg)](https://github.com/bullet-db/bullet-kafka/tags) | | **Latest Artifact** | [![Download](https://api.bintray.com/packages/yahoo/maven/bullet-kafka/images/download.svg)](https://bintray.com/yahoo/maven/bullet-kafka/_latestVersion) | | **Package Manager Setup** | [Setup for Maven, Gradle etc](https://bintray.com/bintray/jcenter?filterByPkgName=bullet-kafka) | @@ -161,6 +180,6 @@ A PubSub implementation using Kafka as the backing PubSub. Can be used with any | Date | Release | Highlights | | ------------ | ------------------------------------------------------------------------------------ | ---------- | -| 2017-10-19 | [**0.2.0**](https://github.com/yahoo/bullet-kafka/releases/tag/bullet-kafka-0.2.0) | Refactors and re-releases. Pass-through settings to Kafka. Manual offset committing bug fix | -| 2017-09-27 | [**0.1.2**](https://github.com/yahoo/bullet-kafka/releases/tag/bullet-kafka-0.1.2) | Fixes a bug with config loading | -| 2017-09-22 | [**0.1.1**](https://github.com/yahoo/bullet-kafka/releases/tag/bullet-kafka-0.1.1) | First release using the PubSub interfaces | +| 2017-10-19 | [**0.2.0**](https://github.com/bullet-db/bullet-kafka/releases/tag/bullet-kafka-0.2.0) | Refactors and re-releases. Pass-through settings to Kafka. Manual offset committing bug fix | +| 2017-09-27 | [**0.1.2**](https://github.com/bullet-db/bullet-kafka/releases/tag/bullet-kafka-0.1.2) | Fixes a bug with config loading | +| 2017-09-22 | [**0.1.1**](https://github.com/bullet-db/bullet-kafka/releases/tag/bullet-kafka-0.1.1) | First release using the PubSub interfaces | diff --git a/mkdocs.yml b/mkdocs.yml index 720885cb..89bbc5ed 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -16,6 +16,9 @@ pages: - Architecture: backend/storm-architecture.md - Setup: backend/storm-setup.md - Performance: backend/storm-performance.md + - Spark: + - Architecture: backend/spark-architecture.md + - Setup: backend/spark-setup.md - PubSub: - Architecture: pubsub/architecture.md - Kafka: pubsub/kafka.md