Skip to content

Commit

Permalink
Browse files Browse the repository at this point in the history
Introduction of 'MustPassRepeatedly' decorator (#1051)
* introduction of RepeatAttempts decorator

* first review

* Refactor: renamed 'test' to 'spec' in various parts of the doc and code

* remove repeated FAILEd log; correct entry point to FLAKEY TEST log

* removing Focus from test

* remove multipleExecutionDecorator type switch

* remove multipleExecutionDecorator type switch - pt 2
  • Loading branch information
felipe88alves committed Oct 19, 2022
1 parent 4442772 commit 047c02f
Show file tree
Hide file tree
Showing 22 changed files with 505 additions and 87 deletions.
12 changes: 10 additions & 2 deletions decorator_dsl.go
Expand Up @@ -13,13 +13,21 @@ You can learn more about decorators here: https://onsi.github.io/ginkgo/#decorat
type Offset = internal.Offset

/*
FlakeAttempts(uint N) is a decorator that allows you to mark individual specs or spec containers as flaky. Ginkgo will run them up to `N` times until they pass.
FlakeAttempts(uint N) is a decorator that allows you to mark individual specs or spec containers as flaky. Ginkgo will run them up to `N` times until they pass.
You can learn more here: https://onsi.github.io/ginkgo/#repeating-spec-runs-and-managing-flaky-specs
You can learn more here: https://onsi.github.io/ginkgo/#the-flakeattempts-decorator
You can learn more about decorators here: https://onsi.github.io/ginkgo/#decorator-reference
*/
type FlakeAttempts = internal.FlakeAttempts

/*
MustPassRepeatedly(uint N) is a decorator that allows you to repeat the execution of individual specs or spec containers. Ginkgo will run them up to `N` times until they fail.
You can learn more here: https://onsi.github.io/ginkgo/#the-mustpassrepeatedly-decorator
You can learn more about decorators here: https://onsi.github.io/ginkgo/#decorator-reference
*/
type MustPassRepeatedly = internal.MustPassRepeatedly

/*
Focus is a decorator that allows you to mark a spec or container as focused. Identical to FIt and FDescribe.
Expand Down
54 changes: 49 additions & 5 deletions docs/index.md
Expand Up @@ -2487,7 +2487,21 @@ One quick note on `--repeat`: when you invoke `ginkgo --repeat=N` Ginkgo will ru

Both `--until-it-fails` and `--repeat` help you identify flaky specs early. Doing so will help you debug flaky specs while the context that introduced them is fresh.

However. There are times when the cost of preventing and/or debugging flaky specs simply is simply too high and specs simply need to be retried. While this should never be the primary way of dealing with flaky specs, Ginkgo is pragmatic about this reality and provides a mechanism for retrying specs.
A more granular approach to repeating specs is by decorating individual subject or container nodes with the MustPassRepeatedly(N) decorator:

```go
Describe("Storing books", func() {
It("can save books to the central library", MustPassRepeatedly(3), func() {
// this spec has been marked and will be retried up to 3 times
})

It("can save books locally", func() {
// this spec has not been marked and will not be retired
})
})
```

However, There are times when the cost of preventing and/or debugging flaky specs simply is simply too high and specs simply need to be retried. While this should never be the primary way of dealing with flaky specs, Ginkgo is pragmatic about this reality and provides a mechanism for retrying specs.

You can retry all specs in a suite via:

Expand All @@ -2497,7 +2511,7 @@ ginkgo --flake-attempts=N

Now, when a spec fails Ginkgo will not automatically mark the suite as failed. Instead it will attempt to rerun the spec up to `N` times. If the spec succeeds during a retry, Ginkgo moves on and marks the suite as successful but reports that the spec needed to be retried.

You can take a more granular approach by decorating individual subject nodes or container nodes as potentially flaky with the `FlakeAttempts(N)` decorator:
A more granular approach is also provided for this functionality with the use of the `FlakeAttempts(N)` decorator:

```go
Describe("Storing books", func() {
Expand Down Expand Up @@ -4799,9 +4813,9 @@ In addition to `Offset`, users can decorate nodes with a `types.CodeLocation`.
Passing a `types.CodeLocation` decorator in has the same semantics as passing `Offset` in: it only applies to the node in question.
#### The FlakeAttempts Decorator
The `FlakeAttempts(uint)` decorator applies container and subject nodes. It is an error to apply `FlakeAttempts` to a setup node.
The `FlakeAttempts(uint)` decorator applies to container and subject nodes. It is an error to apply `FlakeAttempts` to a setup node.
`FlakeAttempts` allows the user to flag specific tests or groups of tests as potentially flaky. Ginkgo will run tests up to the number of times specified in `FlakeAttempts` until they pass. For example:
`FlakeAttempts` allows the user to flag specs trees as potentially flaky. Ginkgo will retry the spec up to the number of times specified in `FlakeAttempts` until they pass. For example:
```go
Describe("flaky tests", FlakeAttempts(3), func() {
Expand All @@ -4825,8 +4839,38 @@ Describe("flaky tests", FlakeAttempts(3), func() {
With this setup, `"is flaky"` and `"is also flaky"` will run up to 3 times. `"is _really_ flaky"` will run up to 5 times. `"is _not_ flaky"` will run only once. Note that if multiple `FlakeAttempts` appear in a spec's hierarchy, the most deeply nested `FlakeAttempts` wins. If multiple `FlakeAttempts` are passed into a given node, the last one wins.
If `ginkgo --flake-attempts=N` is set the value passed in by the CLI will override all the decorated values. Every test will now run up to `N` times.
If `ginkgo --flake-attempts=N` is set the value passed in by the CLI will override all the decorated values. Every spec in the test suite will now run up to `N` times.
#### The MustPassRepeatedly Decorator
The `MustPassRepeatedly(uint)` decorator applies to container and subject nodes. It is an error to apply `MustPassRepeatedly` to a setup node.
the `MustPassRepeatedly` flag allows the user to repeatedly run specs in a controlled manner. Ginkgo will repeatedly run specs up to the number of times specified in `MustPassRepeatedly` or until they fail. For example:
```go
Describe("repeated specs", MustPassRepeatedly(3), func() {
It("is repeated", func() {
...
})
It("is also repeated", func() {
...
})
It("is repeated even more", MustPassRepeatedly(5) func() {
...
})
It("is repeated less", MustPassRepeatedly(1), func() {
...
})
})
```
With this setup, `"is repeated"` and `"is also repeated"` will run up to 3 times. `"is repeated even more"` will run up to 5 times. `"is repeated less"` will run only once. Note that if multiple `MustPassRepeatedly` appear in a spec's hierarchy, the most deeply nested `MustPassRepeatedly` wins. If multiple `MustPassRepeatedly` are passed into a given node, the last one wins.
The `ginkgo --repeat=N` value passed in by the CLI has no relation with the `MustPassRepeatedly` decorator. If the `--repeat` CLI flag is used and a container or subject node also contains the `MustPassRepeatedly` decorator, then the spec will run up to `N*R` times, where `N` is the values passed to the `--repeat` CLI flag and `R` is the value passed to the MustPassRepeatedly decorator.
If the `MustPassRepeatedly` decorator is set, it will override the `ginkgo --flake-attempts=N` CLI config. The specs that do not contain the `MustPassRepeatedly(R)` decorator will still run up to `N` times, in accordance to the `ginkgo --flake-attempts=N` CLI config.
#### The SuppressProgressOutput Decorator
Expand Down
3 changes: 2 additions & 1 deletion dsl/decorators/decorators_dsl.go
@@ -1,7 +1,7 @@
/*
Ginkgo is usually dot-imported via:
import . "github.com/onsi/ginkgo/v2"
import . "github.com/onsi/ginkgo/v2"
however some parts of the DSL may conflict with existing symbols in the user's code.
Expand All @@ -18,6 +18,7 @@ import (

type Offset = ginkgo.Offset
type FlakeAttempts = ginkgo.FlakeAttempts
type MustPassRepeatedly = ginkgo.MustPassRepeatedly
type Labels = ginkgo.Labels
type PollProgressAfter = ginkgo.PollProgressAfter
type PollProgressInterval = ginkgo.PollProgressInterval
Expand Down
@@ -0,0 +1,33 @@
package decorations_fixture_test

import (
"fmt"
"testing"

. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
)

func TestDecorationsFixture(t *testing.T) {
RegisterFailHandler(Fail)
RunSpecs(t, "DecorationsFixture Suite")
}

var countFlake = 0
var countRepeat = 0

var _ = Describe("some decorated specs", func() {
It("passes eventually", func() {
countFlake += 1
if countFlake < 3 {
Fail("fail")
}
}, FlakeAttempts(3))

It("fails eventually", func() {
countRepeat += 1
if countRepeat >= 3 {
Fail(fmt.Sprintf("failed on attempt #%d", countRepeat))
}
}, MustPassRepeatedly(3))
})

This file was deleted.

@@ -0,0 +1,19 @@
package invalid_decorations_flakeattempts_mustpassrepeatedly_test

import (
"testing"

. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
)

func TestInvalidDecorations(t *testing.T) {
RegisterFailHandler(Fail)
RunSpecs(t, "InvalidDecorations Suite - MustPassRepeatedly and FlakeAttempts")
}

var _ = Describe("invalid decorators: mustpassrepeatedly and flakeattempts", FlakeAttempts(3), MustPassRepeatedly(3), func() {
It("never runs", func() {

})
})
@@ -0,0 +1,19 @@
package invalid_decorations_focused_pending_test

import (
"testing"

. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
)

func TestInvalidDecorations(t *testing.T) {
RegisterFailHandler(Fail)
RunSpecs(t, "InvalidDecorations Suite - Focused and Pending")
}

var _ = Describe("invalid decorators: focused and pending", Focus, Pending, func() {
It("never runs", func() {

})
})
Expand Up @@ -12,9 +12,10 @@ func TestDecorationsFixture(t *testing.T) {
RunSpecs(t, "DecorationsFixture Suite")
}

var count = 0
var countFlake = 0
var countRepeat = 0

var _ = Describe("some decorated tests", func() {
var _ = Describe("some decorated specs", func() {
Describe("focused", Focus, func() {
OffsetIt()
})
Expand All @@ -23,13 +24,6 @@ var _ = Describe("some decorated tests", func() {

})

It("passes eventually", func() {
count += 1
if count < 3 {
Fail("fail")
}
}, FlakeAttempts(3))

It("focused it", Focus, func() {
Ω(true).Should(BeTrue())
})
Expand Down
31 changes: 25 additions & 6 deletions integration/decorations_test.go
Expand Up @@ -13,23 +13,42 @@ var _ = Describe("Decorations", func() {
fm.MountFixture("decorations")
})

It("processes the various decorations", func() {
session := startGinkgo(fm.PathTo("decorations"), "-vv", "--no-color")
It("processes the Offset, Focus and Pending decorations", func() {
session := startGinkgo(fm.PathTo("decorations", "offset_focus_pending"), "-vv", "--no-color")
Eventually(session).Should(gexec.Exit(types.GINKGO_FOCUS_EXIT_CODE))

out := string(session.Out.Contents())
Ω(out).Should(MatchRegexp(
`P \[PENDING\]
some decorated tests
.*decorations_fixture_suite_test.go:\d+
some decorated specs
.*offset_focus_pending_fixture_suite_test.go:\d+
pending it`,
))

Ω(out).ShouldNot(ContainSubstring("never_see_this_file"))
})

It("exits with a clear error if decorations are misconfigured", func() {
session := startGinkgo(fm.PathTo("decorations", "invalid_decorations"), "-v", "--no-color")
It("processes the FlakeAttempts and the MustPassRepeatedly decorations", func() {
session := startGinkgo(fm.PathTo("decorations", "flaky_repeated"), "-vv", "--no-color")
Eventually(session).Should(gexec.Exit(1))

Ω(session).Should(gbytes.Say("Ginkgo: Attempt #1 Failed. Retrying..."))
Ω(session).Should(gbytes.Say("Ginkgo: Attempt #2 Failed. Retrying..."))

Ω(session).Should(gbytes.Say("Ginkgo: Attempt #1 Passed. Repeating..."))
Ω(session).Should(gbytes.Say("Ginkgo: Attempt #2 Passed. Repeating..."))
Ω(session).Should(gbytes.Say("failed on attempt #3"))
})

It("exits with a clear error if decorations are misconfigured - focus and pending error", func() {
session := startGinkgo(fm.PathTo("decorations", "invalid_decorations_focused_pending"), "-v", "--no-color")
Eventually(session).Should(gexec.Exit(1))
Ω(session).Should(gbytes.Say("Invalid Combination of Decorators: Focused and Pending"))
})

It("exits with a clear error if decorations are misconfigured - flakeattempts and mustpassrepeatedly error", func() {
session := startGinkgo(fm.PathTo("decorations", "invalid_decorations_flakeattempts_mustpassrepeatedly"), "-v", "--no-color")
Eventually(session).Should(gexec.Exit(1))
Ω(session).Should(gbytes.Say("Invalid Combination of Decorators: FlakeAttempts and MustPassRepeatedly"))
})
})
32 changes: 27 additions & 5 deletions internal/group.go
Expand Up @@ -118,6 +118,8 @@ func (g *group) initialReportForSpec(spec Spec) types.SpecReport {
ParallelProcess: g.suite.config.ParallelProcess,
IsSerial: spec.Nodes.HasNodeMarkedSerial(),
IsInOrderedContainer: !spec.Nodes.FirstNodeMarkedOrdered().IsZero(),
MaxFlakeAttempts: spec.Nodes.GetMaxFlakeAttempts(),
MaxMustPassRepeatedly: spec.Nodes.GetMaxMustPassRepeatedly(),
}
}

Expand Down Expand Up @@ -299,16 +301,29 @@ func (g *group) run(specs Specs) {

g.suite.currentSpecReport.StartTime = time.Now()
if !skip {
maxAttempts := max(1, spec.FlakeAttempts())
if g.suite.config.FlakeAttempts > 0 {

var maxAttempts = 1

if g.suite.currentSpecReport.MaxMustPassRepeatedly > 0 {
maxAttempts = max(1, spec.MustPassRepeatedly())
} else if g.suite.config.FlakeAttempts > 0 {
maxAttempts = g.suite.config.FlakeAttempts
g.suite.currentSpecReport.MaxFlakeAttempts = maxAttempts
} else if g.suite.currentSpecReport.MaxFlakeAttempts > 0 {
maxAttempts = max(1, spec.FlakeAttempts())
}

for attempt := 0; attempt < maxAttempts; attempt++ {
g.suite.currentSpecReport.NumAttempts = attempt + 1
g.suite.writer.Truncate()
g.suite.outputInterceptor.StartInterceptingOutput()
if attempt > 0 {
fmt.Fprintf(g.suite.writer, "\nGinkgo: Attempt #%d Failed. Retrying...\n", attempt)
if g.suite.currentSpecReport.MaxMustPassRepeatedly > 0 {
fmt.Fprintf(g.suite.writer, "\nGinkgo: Attempt #%d Passed. Repeating...\n", attempt)
}
if g.suite.currentSpecReport.MaxFlakeAttempts > 0 {
fmt.Fprintf(g.suite.writer, "\nGinkgo: Attempt #%d Failed. Retrying...\n", attempt)
}
}

g.attemptSpec(attempt == maxAttempts-1, spec)
Expand All @@ -318,8 +333,15 @@ func (g *group) run(specs Specs) {
g.suite.currentSpecReport.CapturedGinkgoWriterOutput += string(g.suite.writer.Bytes())
g.suite.currentSpecReport.CapturedStdOutErr += g.suite.outputInterceptor.StopInterceptingAndReturnOutput()

if g.suite.currentSpecReport.State.Is(types.SpecStatePassed | types.SpecStateSkipped | types.SpecStateAborted | types.SpecStateInterrupted) {
break
if g.suite.currentSpecReport.MaxMustPassRepeatedly > 0 {
if g.suite.currentSpecReport.State.Is(types.SpecStateFailureStates | types.SpecStateSkipped) {
break
}
}
if g.suite.currentSpecReport.MaxFlakeAttempts > 0 {
if g.suite.currentSpecReport.State.Is(types.SpecStatePassed | types.SpecStateSkipped | types.SpecStateAborted | types.SpecStateInterrupted) {
break
}
}
}
}
Expand Down

0 comments on commit 047c02f

Please sign in to comment.