Skip to content
An implementation of failpoints for Golang.
Branch: master
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.github *: add github templates Apr 5, 2019
LICENSES README: explain the failpoint usage and implementation details (#4) Apr 10, 2019
code code: merge modifications back to the original file (#32) May 12, 2019
failpoint-ctl update comment Apr 16, 2019
media README: explain the failpoint usage and implementation details (#4) Apr 10, 2019
tools adjust gitignore Apr 17, 2019
.codecov.yml adjust CI Apr 17, 2019
.gitignore adjust gitignore Apr 17, 2019
.travis.yml adjust CI Apr 17, 2019
CONTRIBUTING.md *: add github templates Apr 5, 2019
LICENSE Initial commit Apr 2, 2019
MAINTAINERS.md failpoint: implement the basic logic of enable/disable and all statem… Apr 7, 2019
Makefile delete interfacer Apr 17, 2019
README.md README: refine quick start guide (#34) May 16, 2019
failpoint.go failpoint: change the semantic of 'EvalContext' Apr 29, 2019
failpoint_test.go failpoint: change the semantic of 'EvalContext' Apr 29, 2019
go.mod code: merge modifications back to the original file (#32) May 12, 2019
go.sum code: merge modifications back to the original file (#32) May 12, 2019
http.go *: add comment to make linter happy (#18) Apr 16, 2019
http_test.go tests: fix unit test (#16) Apr 15, 2019
marker.go code: add support for failpoint.Inject("failpoint-name", nil) and Ret… Apr 17, 2019
runtime.go code: improve expr_rewriter test coverage (#15) Apr 15, 2019
runtime_test.go runtime: fix return value sequence of failpoint.Eval (#26) Apr 22, 2019
terms.go *: add comment to make linter happy (#18) Apr 16, 2019
terms_test.go code: improve expr_rewriter test coverage (#15) Apr 15, 2019

README.md

failpoint

LICENSE Language Go Report Card Build Status Coverage Status Mentioned in Awesome Go

An implementation of failpoints for Golang.

Quick Start

  1. Build failpoint-ctl from source

    git clone https://github.com/pingcap/failpoint.git
    cd failpoint
    make
    ls bin/failpoint-ctl
  2. Inject failpoints to your program, eg:

    package main
    
    import "github.com/pingcap/failpoint"
    
    func main() {
        failpoint.Inject("testPanic", func() {
            panic("failpoint triggerd")
        })
    }
  3. Transfrom your code with failpoint-ctl enable

  4. Build with go build

  5. Enable failpoints with GO_FAILPOINTS environment variable

    GO_FAILPOINTS="main/testPanic=return(true)" ./your-program
  6. If you use go run to run the test, don't forget to add the generated binding__failpoint_binding__.go in your command, like:

    GO_FAILPOINTS="main/testPanic=return(true)" go run your-program.go binding__failpoint_binding__.go

Design principles

  • Define failpoint in valid Golang code, not comments or anything else

  • Failpoint does not have any extra cost

    • Will not take effect on regular logic
    • Will not cause regular code performance regression
    • Failpoint code will not appear in the final binary
  • Failpoint routine is writable/readable and should be checked by a compiler

  • Generated code by failpoint definition is easy to read

  • Keep the line numbers same with the injecting codes(easier to debug)

  • Support parallel tests with context.Context

Key concepts

  • Failpoint

    Faillpoint is a code snippet that is only executed when the corresponding failpoint is active. The closure will never be executed if failpoint.Disable("failpoint-name-for-demo") is executed.

    var outerVar = "declare in outer scope"
    failpoint.Inject("failpoint-name-for-demo", func(val failpoint.Value) {
        fmt.Println("unit-test", val, outerVar)
    })
  • Marker functions

    • It is just an empty function

      • To hint the rewriter to rewrite with an equality statement
      • To receive some parameters as the rewrite rule
      • It will be inline in the compiling time and emit nothing to binary (zero cost)
      • The variables in external scope can be accessed in closure by capturing, and the converted code is still legal because all the captured-variables location in outer scope of IF statement.
    • It is easy to write/read

    • Introduce a compiler check for failpoints which cannot compile in the regular mode if failpoint code is invalid

  • Marker funtion list

    • func Inject(fpname string, fpblock func(val Value)) {}
    • func InjectContext(fpname string, ctx context.Context, fpblock func(val Value)) {}
    • func Break(label ...string) {}
    • func Goto(label string) {}
    • func Continue(label ...string) {}
    • func Fallthrough() {}
    • func Return(results ...interface{}) {}
    • func Label(label string) {}

How to inject a failpoint to your program

  • You can call failpoint.Inject to inject a failpoint to the call site, where failpoint-name is used to trigger the failpoint and failpoint-closure will be expanded as the body of the IF statement.

    failpoint.Inject("failpoint-name", func(val failpoint.Value) {
        failpoint.Return("unit-test", val)
    })

    The converted code looks like:

    if val, ok := failpoint.Eval(_curpkg_("failpoint-name")); ok {
        return "unit-test", val
    }
  • failpoint.Value is the value that passes by failpoint.Enable("failpoint-name", "return(5)") which can be ignored.

    failpoint.Inject("failpoint-name", func(_ failpoint.Value) {
        fmt.Println("unit-test")
    })

    OR

    failpoint.Inject("failpoint-name", func() {
        fmt.Println("unit-test")
    })

    And the converted code looks like:

    if _, ok := failpoint.Eval(_curpkg_("failpoint-name")); ok {
        fmt.Println("unit-test")
    }
  • Also, the failpoint closure can be a function which takes context.Context. You can do some customized things with context.Context like controlling whether a failpoint is active in parallel tests or other cases. For example,

    failpoint.InjectContext(ctx, "failpoint-name", func(val failpoint.Value) {
        fmt.Println("unit-test", val)
    })

    The converted code looks like:

    if val, ok := failpoint.EvalContext(ctx, _curpkg_("failpoint-name")); ok {
        fmt.Println("unit-test", val)
    }
  • You can ignore context.Context, and this will generate the same code as above non-context version. For example,

    failpoint.InjectContext(nil, "failpoint-name", func(val failpoint.Value) {
        fmt.Println("unit-test", val)
    })

    Becomes

    if val, ok := failpoint.EvalContext(nil, _curpkg_("failpoint-name")); ok {
        fmt.Println("unit-test", val)
    }
  • You can control a failpoint by failpoint.WithHook

    func (s *dmlSuite) TestCRUDParallel() {
        sctx := failpoint.WithHook(context.Backgroud(), func(ctx context.Context, fpname string) bool {
            return ctx.Value(fpname) != nil // Determine by ctx key
        })
        insertFailpoints = map[string]struct{} {
            "insert-record-fp": {},
            "insert-index-fp": {},
            "on-duplicate-fp": {},
        }
        ictx := failpoint.WithHook(context.Backgroud(), func(ctx context.Context, fpname string) bool {
            _, found := insertFailpoints[fpname] // Only enables some failpoints.
            return found
        })
        deleteFailpoints = map[string]struct{} {
            "tikv-is-busy-fp": {},
            "fetch-tso-timeout": {},
        }
        dctx := failpoint.WithHook(context.Backgroud(), func(ctx context.Context, fpname string) bool {
            _, found := deleteFailpoints[fpname] // Only disables failpoints. 
            return !found
        })
        // other DML parallel test cases.
        s.RunParallel(buildSelectTests(sctx))
        s.RunParallel(buildInsertTests(ictx))
        s.RunParallel(buildDeleteTests(dctx))
    }
  • If you use a failpoint in the loop context, maybe you will use other marker functions.

    failpoint.Label("outer")
    for i := 0; i < 100; i++ {
        inner:
            for j := 0; j < 1000; j++ {
                switch rand.Intn(j) + i {
                case j / 5:
                    failpoint.Break()
                case j / 7:
                    failpoint.Continue("outer")
                case j / 9:
                    failpoint.Fallthrough()
                case j / 10:
                    failpoint.Goto("outer")
                default:
                    failpoint.Inject("failpoint-name", func(val failpoint.Value) {
                        fmt.Println("unit-test", val.(int))
                        if val == j/11 {
                            failpoint.Break("inner")
                        } else {
                            failpoint.Goto("outer")
                        }
                    })
            }
        }
    }

    The above code block will generate the following code:

    outer:
        for i := 0; i < 100; i++ {
        inner:
            for j := 0; j < 1000; j++ {
                switch rand.Intn(j) + i {
                case j / 5:
                    break
                case j / 7:
                    continue outer
                case j / 9:
                    fallthrough
                case j / 10:
                    goto outer
                default:
                    if val, ok := failpoint.Eval(_curpkg_("failpoint-name")); ok {
                        fmt.Println("unit-test", val.(int))
                        if val == j/11 {
                            break inner
                        } else {
                            goto outer
                        }
                    }
                }
            }
        }
  • You may doubt why we do not use label, break, continue, and fallthrough directly instead of using failpoint marker functions.

    • Any unused symbol like an ident or a label is not permitted in Golang. It will be invalid if some label is only used in the failpoint closure. For example,

      label1: // compiler error: unused label1
          failpoint.Inject("failpoint-name", func(val failpoint.Value) {
              if val.(int) == 1000 {
                  goto label1 // illegal to use goto here
              }
              fmt.Println("unit-test", val)
          })
    • break and continue can only be used in the loop context, which is not legal in the Golang code if we use them in closure directly.

Some complicated failpoints demo

  • Inject a failpoint to the IF INITIAL statement or CONDITIONAL expression

    if a, b := func() {
        failpoint.Inject("failpoint-name", func(val failpoint.Value) {
            fmt.Println("unit-test", val)
        })
    }, func() int { return rand.Intn(200) }(); b > func() int {
        failpoint.Inject("failpoint-name", func(val failpoint.Value) int {
            return val.(int)
        })
        return rand.Intn(3000)
    }() && b < func() int {
        failpoint.Inject("failpoint-name-2", func(val failpoint.Value) {
            return rand.Intn(val.(int))
        })
        return rand.Intn(6000)
    }() {
        a()
        failpoint.Inject("failpoint-name-3", func(val failpoint.Value) {
            fmt.Println("unit-test", val)
        })
    }

    The above code block will generate something like this:

    if a, b := func() {
        if val, ok := failpoint.Eval(_curpkg_("failpoint-name")); ok {
            fmt.Println("unit-test", val)
        }
    }, func() int { return rand.Intn(200) }(); b > func() int {
        if val, ok := failpoint.Eval(_curpkg_("failpoint-name")); ok {
            return val.(int)
        }
        return rand.Intn(3000)
    }() && b < func() int {
        if val, ok := failpoint.Eval(_curpkg_("failpoint-name-2")); ok {
            return rand.Intn(val.(int))
        }
        return rand.Intn(6000)
    }() {
        a()
        if val, ok := failpoint.Eval(_curpkg_("failpoint-name-3")); ok {
            fmt.Println("unit-test", val)
        }
    }
  • Inject a failpoint to the SELECT statement to make it block one CASE if the failpoint is active

    func (s *StoreService) ExecuteStoreTask() {
        select {
        case <-func() chan *StoreTask {
            failpoint.Inject("priority-fp", func(_ failpoint.Value) {
                return make(chan *StoreTask)
            })
            return s.priorityHighCh
        }():
            fmt.Println("execute high priority task")
    
        case <- s.priorityNormalCh:
            fmt.Println("execute normal priority task")
    
        case <- s.priorityLowCh:
            fmt.Println("execute normal low task")
        }
    }

    The above code block will generate something like this:

    func (s *StoreService) ExecuteStoreTask() {
        select {
        case <-func() chan *StoreTask {
            if _, ok := failpoint.Eval(_curpkg_("priority-fp")); ok {
                return make(chan *StoreTask)
            })
            return s.priorityHighCh
        }():
            fmt.Println("execute high priority task")
    
        case <- s.priorityNormalCh:
            fmt.Println("execute normal priority task")
    
        case <- s.priorityLowCh:
            fmt.Println("execute normal low task")
        }
    }
  • Inject a failpoint to dynamically extend SWITCH CASE arms

    switch opType := operator.Type(); {
    case opType == "balance-leader":
        fmt.Println("create balance leader steps")
    
    case opType == "balance-region":
        fmt.Println("create balance region steps")
    
    case opType == "scatter-region":
        fmt.Println("create scatter region steps")
    
    case func() bool {
        failpoint.Inject("dynamic-op-type", func(val failpoint.Value) bool {
            return strings.Contains(val.(string), opType)
        })
        return false
    }():
        fmt.Println("do something")
    
    default:
        panic("unsupported operator type")
    }

    The above code block will generate something like this:

    switch opType := operator.Type(); {
    case opType == "balance-leader":
        fmt.Println("create balance leader steps")
    
    case opType == "balance-region":
        fmt.Println("create balance region steps")
    
    case opType == "scatter-region":
        fmt.Println("create scatter region steps")
    
    case func() bool {
        if val, ok := failpoint.Eval(_curpkg_("dynamic-op-type")); ok {
            return strings.Contains(val.(string), opType)
        }
        return false
    }():
        fmt.Println("do something")
    
    default:
        panic("unsupported operator type")
    }
  • More complicated failpoints

    • There are more complicated failpoint sites that can be injected to
      • for the loop INITIAL statement, CONDITIONAL expression and POST statement
      • for the RANGE statement
      • SWITCH INITIAL statement
    • Anywhere you can call a function

Failpoint name best practice

As you see above, _curpkg_ will automatically wrap the original failpoint name in failpoint.Eval call. You can think of _curpkg_ as a macro that automatically prepends the current package path to the failpoint name. For example,

package ddl // which parent package is `github.com/pingcap/tidb`

func demo() {
	// _curpkg_("the-original-failpoint-name") will be expanded as `github.com/pingcap/tidb/ddl/the-original-failpoint-name`
	if val, ok := failpoint.Eval(_curpkg_("the-original-failpoint-name")); ok {...}
}

You do not need to care about _curpkg_ in your application. It is automatically generated after running failpoint-ctl enable and is deleted with failpoint-ctl disable.

Because all failpoints in a package share the same namespace, we need to be careful to avoid name conflict. There are some recommended naming rules to improve this situation.

  • Keep name unique in current subpackage

  • Use a self-explanatory name for the failpoint

    You can enable failpoints by environment variables

    GO_FAILPOINTS="github.com/pingcap/tidb/ddl/renameTableErr=return(100);github.com/pingcap/tidb/planner/core/illegalPushDown=return(true);github.com/pingcap/pd/server/schedulers/balanceLeaderFailed=return(true)"

Implementation details

  1. Define a group of marker functions
  2. Parse imports and prune a source file which does not import a failpoint
  3. Traverse AST to find marker function calls
  4. Marker function calls will be rewritten with an IF statement, which calls failpoint.Eval to determine whether a failpoint is active and executes failpoint code if the failpoint is enabled

rewrite-demo

Acknowledgments

  • Thanks gofail to provide initial implementation.
You can’t perform that action at this time.