tetragon: fix hang on error in tetragonExecute

There has been a longstanding bug where if Tetragon encounters an error inside of
tetragonExecute, the process will hang instead of exiting as expected. When looking at the
goroutine stacktrace dump provided by the runtime on SIGABRT, we can immediately see the
problem. The main thread is stuck on a channel send inside of observer.RemoveSensors().
Further investigation reveals that the channel is never opened because InitSensorManager()
is waiting on the waitChan to be closed, which does not happen until we have loaded the
base sensor.

To fix this issue, we simply need to move the defer call into observer.RemoveSensors() to
after we indicate that InitSensorManager() is cleared to run. This patch does exactly
that. Since we haven't loaded any BPF progs yet until the base sensor has been loaded
anyway, this should be safe to do.

Signed-off-by: William Findlay <will@isovalent.com>

cmd/tetragon/main.go

@@ -324,9 +324,6 @@ func tetragonExecute() error {
 	if err := obs.InitSensorManager(sensorMgWait); err != nil {
 		return err
 	}
-	defer func() {
-		observer.RemoveSensors(ctx)
-	}()
 
 	/* Remove any stale programs, otherwise feature set change can cause
 	 * old programs to linger resulting in undefined behavior. And because
@@ -472,6 +469,9 @@ func tetragonExecute() error {
 	close(sensorMgWait)
 	sensorMgWait = nil
 	observer.GetSensorManager().LogSensorsAndProbes(ctx)
+	defer func() {
+		observer.RemoveSensors(ctx)
+	}()
 
 	err = loadTpFromDir(ctx, option.Config.TracingPolicyDir)
 	if err != nil {