RapidRepoPull

Created and Maintained by

• Talal Balouch (Twitter @xtbalz)

Description

• This program uses Python to clone/maintain multiple security related repos using threading and multiprocessing

Goal

• The goal of this program is to quickly pull and install repos from its list

Use cases

• Quickly install your favorite Security repos on a new system

• Kick off multiple concurrent git clone tasks utilizing Python

• Add/remove Security related Github repos to the default.xt list as needed in order to address indivudual use case/project needs

• Clone repos to review offline while traveling without internet connectivity

Requirements

• This program was tested with Python3 version 3.8.9 64-bit

• Ensure the Python3 virtual environment package is installed (Ubuntu)

  sudo apt-get install python3-venv

• Ensure git is installed (Ubuntu)

  sudo apt-get install git

Usage Option 1 Automatic (Docker)

This option does not allow a user to specify the number of threads or a text file including additional repos

• Clone code repo

  git clone https://github.com/tbalz2319/RapidRepoPull.git

• Change directory into RapidRepoPull

  cd RapidRepoPull

• The script will run in a minimal Alpine Docker container (126 MB) and extract the dirs in the current working dir

  docker-compose up --build

Usage Option 2 Local Install

• Clone code repo

  git clone https://github.com/tbalz2319/RapidRepoPull.git

• Change directory into RapidRepoPull

  cd RapidRepoPull

• Execute the script below

  ./install.sh

Usage Option 3 Manual

• Clone code repo

  git clone https://github.com/tbalz2319/RapidRepoPull.git

• Change directory into RapidRepoPull

  cd RapidRepoPull

• Create a virtual Python3 environment to run this code

  python3 -m venv venv

• Activate the virual enivornment

  source venv/bin/activate

• Install requirements

  pip install -r requirements.txt

• Run program

  python rapid.py

Usage Option 4 Custom

• Clone code repo

  git clone https://github.com/tbalz2319/RapidRepoPull.git

• Change directory into RapidRepoPull

  cd RapidRepoPull

• Execute the script below

  ./option4.sh

• The script will exit while leaving the program in a python virtual env indicated by (venv)

• Use the help menu below to navigate options

  Usage: rapid.py [OPTIONS]

     Options:
     -v, --verbose      Will print verbose messages.
     -f, --file TEXT    Specify a text file with a list of user selected Github repos
     -t, --thread TEXT  Specify the number of CPU threads to use
     -u, --url TEXT     Specify a url to scrape containing Github repos to clone
     --help             Show this message and exit.```
  
  

• Example specifying a custom text file with a list of repos and how many threads the script should use

• The text file must reside in the same directory as rapid.py and can be called anything

• The example file "bob.txt" is there to testing an example purposes

  python rapid.py -f bob.txt -t 25

• The example below with scrape a url containing Github to clone and provide verbose output, utilizing 75 threads

  python rapid.py -v -u https://apsdehal.in/awesome-ctf -t 75

Update Program

• Run the following script

  ./update.sh

Mass Update all existing repos

• Run the command to maintain all existing repos by attempting to pull latest version

  ./update_repos.sh

To do

• Add error handling

• Clean up code

• Implement new features to enhance functionality via cli

Contribute

• The code for this project is constantly being cleaned up, refactored and refined, there are most likely lots of bugs that need to fixed and identified.

• Any help with the project is greatly appreciated

• Pull requests welcomed