Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add error group wrapper for panic handling #4426

Merged
merged 11 commits into from
Mar 8, 2024
Merged

Conversation

dirkkul
Copy link
Contributor

@dirkkul dirkkul commented Mar 8, 2024

First part of #4329

What's being changed:

Adds

  • a custom wrapper that automatically:
    • adds a recover method that
      • logs the panic including local variables that have been given
      • prints the stacktrace
    • return an error, so users will not simply get an empty result
    • has the same syntax as error groups, only a logger needs to be provided when creating the wrapper. Adds the ability to add variables to the logging
  • replaces all direct usages of errorgroups with the wrapper. Only a single place had any panic handling
  • adds a custom linter that errors if error groups are used directly (example) and adds it to pre-commit+ci pipeline

Example with a panic("test panic") somewhere in the stack:

code:

	eg := enterrors.NewErrorGroupWrapper(logger, "filters:", filters) // additional info (filters) is optional
        eg.Go(func() error {....code.with.panic...}, shardName) //additional info (shardname) is optional

server log:

{"level":"error","msg":"Recovered from panic: test panic, local variables [5TosHWPHUFQG], additional localVars [filters: \u003cnil\u003e]\n","panic":"test panic","time":"2024-03-08T06:35:25+01:00"}
goroutine 473 [running]:
runtime/debug.Stack()
        /opt/homebrew/opt/go/libexec/src/runtime/debug/stack.go:24 +0x64
runtime/debug.PrintStack()
        /opt/homebrew/opt/go/libexec/src/runtime/debug/stack.go:16 +0x1c
github.com/weaviate/weaviate/entities/errors.(*ErrorGroupWrapper).Go.func1.1()
        /Users/dirkkulawiak/Documents/weaviate/weaviate/entities/errors/error_group_wrapper.go:46 +0x1b4
panic({0x10288ec80?, 0x102be50e0?})
        /opt/homebrew/opt/go/libexec/src/runtime/panic.go:920 +0x254
github.com/weaviate/weaviate/adapters/repos/db.(*Shard).ObjectSearch(0x14002aa2b00, {0x102bfd1b0, 0x140031839e0}, 0x64, 0x0, 0x0, {0x0, 0x0, 0x0}, 0x0, ...)
        /Users/dirkkulawiak/Documents/weaviate/weaviate/adapters/repos/db/shard_read.go:168 +0x68
github.com/weaviate/weaviate/adapters/repos/db.(*LazyLoadShard).ObjectSearch(0x14003ac7760, {0x102bfd1b0, 0x140031839e0}, 0x64, 0x0, 0x0, {0x0, 0x0, 0x0}, 0x0, ...)
        /Users/dirkkulawiak/Documents/weaviate/weaviate/adapters/repos/db/shard_lazyloader.go:209 +0x12c
github.com/weaviate/weaviate/adapters/repos/db.(*Index).objectSearchByShard.func1()
        /Users/dirkkulawiak/Documents/weaviate/weaviate/adapters/repos/db/index.go:1257 +0x1a4
github.com/weaviate/weaviate/entities/errors.(*ErrorGroupWrapper).Go.func1()
        /Users/dirkkulawiak/Documents/weaviate/weaviate/entities/errors/error_group_wrapper.go:50 +0xbc
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /Users/dirkkulawiak/Documents/weaviate/weaviate/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x80
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 472
        /Users/dirkkulawiak/Documents/weaviate/weaviate/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0xfc


in python client:

weaviate.exceptions.WeaviateQueryError: Query call with protocol GRPC search failed with message explorer: list class: search: object search at index test2: panic occurred: test panic.
sys:1: ResourceWarning: unclosed <socket.socket fd=6, family=2, type=1, proto=6, laddr=('127.0.0.1', 52472), raddr=('127.0.0.1', 8080)>

Review checklist

  • Documentation has been updated, if necessary. Link to changed documentation:
  • Chaos pipeline run or not necessary. Link to pipeline:
  • All new code is covered by tests where it is reasonable.
  • Performance tests have been run or not necessary.

@dirkkul dirkkul requested a review from a team as a code owner March 8, 2024 09:06
Copy link

sonarcloud bot commented Mar 8, 2024

Quality Gate Failed Quality Gate failed

Failed conditions
3.2% Duplication on New Code (required ≤ 3%)

See analysis details on SonarCloud

@dirkkul dirkkul merged commit 367c997 into stable/v1.24 Mar 8, 2024
34 of 36 checks passed
@dirkkul dirkkul deleted the panic_handling branch March 8, 2024 15:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants