Installation

Installation Instructions

Note

If you are using Windows, please install Spark and set environment variables as mentioned here before going through the installation below. You'll need to set the environment variables given here.

• Clone the repository (with all the submodules):

  $ git clone --recurse-submodules https://github.com/RADAR-base/radarpipeline.git

• Change the directory to radarpipeline:

  $ cd radarpipeline

• Checkout the development branch:

  $ git checkout dev

• Create a virtual environment and activate it. The instructions here use virtualenv but feel free to use any python virtual environment manager of your choice (e.g. https://docs.python.org/3/library/venv.html).

  • Install the virtualenv package:

    $ python -m pip install --user virtualenv

  • Create a python virtual environment:

    $ python -m virtualenv env

  • Activate the virtual environment:

    On Windows, run:

    $ .\env\Scripts\activate

    On Linux or MacOS, run:

    $ source ./env/bin/activate

• Install the dependencies:

  $ python -m pip install -r requirements.txt

• Install the module as a python package by running the command

  $ python -m pip install -e .

• To verify the installations, run the following command in the project root directory to run the pipeline:

  $ python .

• The pipeline would do a mock run and ingest the data in the mock-data directory. You can see some outputs in the CLI and if the project is installed correctly, the mock pipeline would run without errors and save the data to the output directory.

git submodules

The project uses git submodules to fetch additional resources such as the mock data. While cloning the repo for the first time, they are downloaded with the repo, if the --recurse-submodules flag is provided to the git clone command.

If the submodule needs to be updated, run the following command in the project root directory:

$ git submodule update --init --recursive