This project focuses on creating a distributed Raspberry Pi Cluster
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
01_logging_liveness
02_packaging_common_functionality
03_talking_to_master
04_config_files
05_sending_data_with_json
06_sending_slave_details_to_master
07_sending_data_to_the_slave
08_slave_helper_functions
xx_daemon_process
xx_slave_tasks
.gitignore
LICENSE
README.md

README.md

Raspberry Pi Cluster project

This repository holds the code for the tutorials for creating a Raspberry Pi Cluster in Python.

Here I go through the basics of creating a simple logger to a distributed task assigned system.

Each folder relates to a new topic of creating a Raspberry Pi Cluster. If you are familiar with python the first few topics should be short topics to revise. Later we will get onto more complicated topics such as reaching a consensus of who must "lead" the cluster.

Questions comments and suggestions can be raised the specific blog post or by using issues here.

Current Examples

  1. Logging Liveness Full tutorial - To start with we are going to make a really simple file to start logging data. This will set up a python logger and log data to it every minute.

  2. Packaging common functionality Full tutorial - Here I introduce python packages and the reason why we use them. I move some of the common functionality into a package so more easily use across scripts.

  3. Basic node communication between two nodes Full tutorial - Here I introduce sockets as a simple way of sending data between two concurrently running scripts.

  4. Configuration files Full tutorial - Here I introduce configuration files and why we use them. I change the previous script to use configuration files.

  5. Sending data with JSON Full tutorial - Here I improve I the cluster sends data around by creating a JSON based message format. The implementation is abstracted into a module to allow easy modification without changing scripts that use it.

  6. Sending Slave Details to the Master Full tutorial - This tutorial focuses on modifying the original data sending functions to use a more generic format. This will allow the master to read what messages are being received and determine what action it needs to take for each message type.

  7. Sending data to the Slave Full tutorial - This tutorial focuses on improving the slave by being able to request data from the master.

  8. Slave Helper Functions Full tutorial - This tutorial focuses on creating a number of slave helper functions to begin the process of fully automating the slaves.

ToDo - List of things that will come in the future

  • Socket talking - Master Broadcast
  • Socket talking - Replay attacks
  • Socket talking - Encryption
  • Config loading - JSON
  • Daemon Process -
  • Master running simple webserver to offer files
  • Slave /Master version/updates
  • Multiple Slaves
  • Periodic updates of the system
  • Storing state of the cluster node
  • Report Health metrics back (uptime, updates pending, time correctness, disk space)
  • Rejoining the cluster after a master restart
  • Smart slave restart
  • Restarting slave once rebooted
  • Keeping track of the running code revision
  • Why keeping track of the time is important
  • Slave git stash, pull, stash pop, resolving.