[Feature] Ability to reliably test overseer

[rafaeljusto]

I was trying to create some unit tests to my app that is using overseer. The test need to change the os.Args so it can test different scenarios. The problem is that overseer starts a new sub-process (as it should) and the Go test framework fails to parse the modified arguments, as it doesn't recognize them. Check the following simplified example:

program.go

package main

import (
    "os"
    "strings"
    "time"

    "github.com/jpillora/overseer"
    "github.com/jpillora/overseer/fetcher"
)

var value = ""

func main() {
    overseer.Run(overseer.Config{
        Program: prog,
        Address: ":3000",
        Fetcher: &fetcher.HTTP{
            URL:      "http://localhost:4000/binaries/myapp",
            Interval: 1 * time.Second,
        },
    })
}

func prog(state overseer.State) {
    println("Running program")

    if len(os.Args) == 2 {
        if argParts := strings.Split(os.Args[1], "="); len(argParts) == 2 && argParts[0] == "--value" {
            value = argParts[1]
        }
    }
}

program_test.go

package main

import (
    "os"
    "testing"
)

func Test_main(t *testing.T) {
    os.Args = os.Args[:1]
    os.Args = append(os.Args, "--value=2")
    main()

    if value != "2" {
        t.Error("expected value 2, got %s", value)
    }
}

output

flag provided but not defined: -value
Usage of /tmp/go-build073924981/labs/program/_test/program.test:
  -test.bench string
        regular expression to select benchmarks to run
  -test.benchmem
        print memory allocations for benchmarks
  -test.benchtime duration
        approximate run time for each benchmark (default 1s)
  -test.blockprofile string
        write a goroutine blocking profile to the named file after execution
  -test.blockprofilerate int
        if >= 0, calls runtime.SetBlockProfileRate() (default 1)
  -test.count n
        run tests and benchmarks n times (default 1)
  -test.coverprofile string
        write a coverage profile to the named file after execution
  -test.cpu string
        comma-separated list of number of CPUs to use for each test
  -test.cpuprofile string
        write a cpu profile to the named file during execution
  -test.memprofile string
        write a memory profile to the named file after execution
  -test.memprofilerate int
        if >=0, sets runtime.MemProfileRate
  -test.outputdir string
        directory in which to write profiles
  -test.parallel int
        maximum test parallelism (default 8)
  -test.run string
        regular expression to select tests and examples to run
  -test.short
        run smaller test suite to save time
  -test.timeout duration
        if positive, sets an aggregate time limit for all tests
  -test.trace string
        write an execution trace to the named file after execution
  -test.v
        verbose: print additional output
exit status 2
FAIL    labs/program    0.019s

One solution would be instead of modifying the os.Args, set an environment variable and detect it inside the sub-process execution:

program_test.go

package main

import (
    "os"
    "testing"
)

func Test_main(t *testing.T) {
    if arg := os.Getenv("TEST_ARG"); arg != "" {
        os.Args = os.Args[:1]
        os.Args = append(os.Args, arg)
        main()
        return
    }

    os.Setenv("TEST_ARG", "--value=2")

    main()

    if value != "2" {
        t.Errorf("expected value 2, got %s", value)
    }
}

output

Running program
ok      labs/program    0.019s

Is there any better solution for this situation? If not, this issue can be closed and could be used in the future for others that have the same problem that I had.

[jpillora]

Sorry Rafael, swamped with work, though you raise a good point - this package should have proper tests. No time now, though will leave open so it's not forgotten :)

[rafaeljusto]

Thanks @jpillora!

I'm now thinking that testing a server with a non-stop feature isn't so trivial. After the sub-process exits, the overseer framework will try to restart it or will execute the os.Exit, and in both situations we can't check the results.

proc_master.go (fork)

        mp.debugf("prog exited with %d", code)
        //if a restarts are disabled or if it was an
        //unexpected creash, proxy this exit straight
        //through to the main process
        if mp.NoRestart || !mp.restarting {
            os.Exit(code)
        }

We could have a test flag in the configuration, as we have for the debug mode, that could avoid calling os.Exit in the main process and run the sub-process only once. Don't know yet how it would work, I need to study better the library. 😄

[rafaeljusto]

With this PR the test is now easier:

program.go

package main

import (
    "time"

    "github.com/rafaeljusto/overseer"
    "github.com/rafaeljusto/overseer/fetcher"
)

var value = ""

func main() {
    overseer.Run(overseer.Config{
        Program: prog,
        Address: ":3000",
        Fetcher: &fetcher.HTTP{
            URL:      "http://localhost:4000/binaries/myapp",
            Interval: 1 * time.Second,
        },
        Test: true,
    })
}

func prog(state overseer.State) {
    println("Running program")
    value = "2"
}

program_test.go

package main

import "testing"

func Test_main(t *testing.T) {
    main()

    if value != "2" {
        t.Errorf("expected value 2, got %s", value)
    }
}