Timing as a metric

[stephanbuys]

Would it make sense to add Time as a factor for test success/failure?

I'm thinking that a global timer would be very interesting to find serious regressions in overall test runtime.

Also command could have a timeout/expected time? (Again thinking about finding regressions in tests).

[aelsabbahy]

Command having a timeout is a great idea.

I'm thinking that a global timer would be very interesting to find serious regressions in overall test runtime.

Do you mean something along the lines of goss validate --timeout=1m, where if the tests take longer than 1 minute it's considered a failure? If so, should the test suite stop immediately or finish running but still mark result as a failure due to timeout?

[stephanbuys]

Hi, yes - that would be great. And report total run time for some of the outputs?

[stephanbuys]

Actually, thinking about it a bit more:

--timeout=1m would timeout, quit and report fail
--expecttime=1m would keep running but report failure

Something like that?

[aelsabbahy]

All fomrats have a total time report at the end of the run. Per result time is available today in the json formatter.

$ goss validate --format json
{
    "results": [
        {
            "duration": 663138,
            "err": null,
            "expected": [
                "true"
            ],
            "found": [
                "true"
            ],
            "property": "exists",
            "resource-type": "User",
            "successful": true,
            "summary-line": "User: nobody: exists: matches expectation: [true]",
            "test-type": 0,
            "title": "nobody"
        }
    ],
    "summary": {
        "failed-count": 0,
        "summary-line": "Count: 1, Failed: 0, Duration: 0.009s",
        "test-count": 1,
        "total-duration": 9326791
    }
}

[stephanbuys]

Maybe a "global" flag --failafter=1m (for the whole job) as well as per-section { "failafter": "1m", ... } and then a global --stopOnFail would cover 90% of use-cases?

--stopOnFail is useful for many scenarios, especially during CD when you want to fail as fast as possible and give feedback to the user?

[aelsabbahy]

I created a separate issue to track command timeout. #20

I'm holding off on global timeout, stoponfail and fail after for now to better understand the scope.

Out of curiosity, have you had any test suite go over ~10 seconds, yet?

[stephanbuys]

Not yet, but some items like "docker pull", etc will definitely break through that barrier - I'm starting to flesh out the test suites now.

[aelsabbahy]

Curious if #20 was sufficient for your usecase.

For global timeout, I'm still leaning towards offloading that to other tools, ex: http://man7.org/linux/man-pages/man1/timeout.1.html

[aelsabbahy]

Going to close this in favor of using external tools for global timeout. Per-test timeout is within the goss "blackbox" and has been addressed by #20.