TaskRunner Inputs:
- List of machines
- Path to private key
- Path to script to execute (it should take input item, output directory, log file as inputs)
- Input file (one line per input item)
- Marker directory
- Output directory
- Log file path
Steps
- Find a master node and deploy the script there.
- Start the script with the above mentioned inputs
- Screen session should start automatically and start logging the progress
Script's works as follows,
- Split the input into reasonable sized chunks
- Creates a map of files to be uploaded to the servers
- Uploads the files to all the servers
- In each of the machines it starts a screen session
- Runs the script in the screen session passing one line of input at a time.
- Logs the marker (line number) in a marker directory
- Cats the log file and assembles them in the master node's log file.