Exception.md

Process Exception

This section shows you how we addressed exceptions. We primarily measured against exceptions that occur when processes terminate abnormally.

Multi-Process

In main.py, we use multiprocessing to publish separate processes for each D-Bus and other function. Additionally, we have published monitor thread to monitor these process all the time.

The reason we used thread to monitor processes instead of process is that a sibling process can't monitor and check if their sibling process are alive or not. A process can only monitor and check for a child process. That's why we used threads that act as parent's avatars.

For sure, if main.py process is killed by SIGKILL, the monitor thread doesn't make sence because you cannot cleanup and restart main.py again. So we develop a monitor that is checking if main.py (python3_main_process) is alive or not. That is a shell script.

main.py

# import libraries (omitted)

def terminate_processes(processes):
    for p in processes:
        p.terminate()

if __name__ == '__main__':
  piracer         = PiRacerStandard()
  shanwan_gamepad = ShanWanGamepad()

  car_control_process = Process(target=car_control, args=(piracer, shanwan_gamepad), name='python3_car_control')
  car_control_process.start()

  battery_process = Process(target=battery_service_process, args=(piracer,), name='python3_battery_process')
  setproctitle("python3_battery_process")
  battery_process.start()

  dbus_process = Process(target=dbus_service_process, name='python3_dbus_process')
  setproctitle("python3_dbus_process")
  dbus_process.start()

  processes = [car_control_process, battery_process, dbus_process]
  monitor_thread = threading.Thread(target=monitor_thread, args=(processes, piracer, shanwan_gamepad), name='monitor_thread')
  monitor_thread.start()

  setproctitle("python3_main_process")

  try:
    car_control_process.join()
    battery_process.join()
    dbus_process.join()
  except KeyboardInterrupt:
    print("Ctrl + C detected. Terminating processes...")
    terminate_processes(processes)

monitor.py

# import libraries (omitted)

def restart_process(target, args, name):
    new_process = Process(target=target, args=args, name=name)
    new_process.start()
    return new_process

def monitor_thread(processes, piracer, shanwan_gamepad):
     while True:
        for p in processes:
            if not p.is_alive():
                print(f"Process {p.name} has terminated unexpectedly!")
                if p.name == 'python3_car_control':
                    piracer.set_steering_percent(0)
                    piracer.set_throttle_percent(0)
                    new_process = restart_process(target=car_control, args=(piracer, shanwan_gamepad), name=p.name)
                    setproctitle("python3_car_control")
                elif p.name == 'python3_battery_process':
                    new_process = restart_process(target=battery_service_process, args=(piracer,), name=p.name)
                    setproctitle("python3_battery_process")
                elif p.name == 'python3_dbus_process':
                    new_process = restart_process(target=dbus_service_process, args=(), name=p.name)
                    setproctitle("python3_dbus_process")
                processes.remove(p)
                processes.append(new_process)
        time.sleep(1)

monitor_main.sh

#!/bin/bash
process_name="python3_main_process"
process_path="/home/seame01/workspace/DES02-PiRacer-instrument/app/d-bus/script/restart.sh"
inter=3
wait=3

while true; do
  isAliveProcess=$(ps -ef | grep "$process_name" |
                  grep -v grep | wc -l)

  if [ $isAliveProcess -eq 1 ]; then
    echo "o:${process_name} process"
  else
    echo "x:${process_name} process"
    /bin/bash $process_path &
  fi

  sleep $inter
done