Skip to content

Home

Jump to bottom
Vibert Bounyasit edited this page Aug 7, 2022 · 14 revisions

Welcome to My Data Framework wiki!

This section will teach you everything you have to know about DataFlow, and will help you get started and write your own Spark application based on this Framework.

If you haven't done it yet, before you start, you should import the framework library into your dependencies in your pom.xml file :

    <dependency>
        <groupId>io.github.vbounyasit</groupId>
        <artifactId>dataflow</artifactId>
        <version>1.1.0-SNAPSHOT</version>
    </dependency>

After familiarizing yourself with the ins and out of this Framework, you should be able to have a project ready for production to be packaged into a jar and sent to your Spark cluster via a spark-submit command.

So go ahead and get started right now !

Want a concrete example of a project created from scratch using the framework? You can head to the MyDataFramework-projects repository, where I have made an example project showcasing most of the features offered by my framework. This can help you have a more practical point of view on this.

Any questions ?

If you have any questions or feedback about my work on it, feel free to contact me through this mailing address : vibert.bounyasit@gmail.com

If you are interested in knowing who I am, you can head to my Linkedin page

Overview

Getting started

Writing your Spark application

Writing Unit and Integration-tests

Defining custom parameters for your jobs

Clone this wiki locally