Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

check for failure on mysql query #2105

Merged
merged 1 commit into from Nov 17, 2016
Merged

Conversation

kgoess
Copy link
Contributor

@kgoess kgoess commented Nov 16, 2016

That query used to have a check for err, but it was removed in 46ba8d1.
The caller, expireID in vault/expiration.go:565, retries after 1 second, and
there's other failures in this method that return nil,err so this approach
looks valid.

We were seeing these panics/crashes since our vault is lightly used and the connection goes stale:

[mysql] 2016/11/03 21:39:01 packets.go:33: unexpected EOF
[mysql] 2016/11/03 21:39:01 packets.go:124: write tcp x.x.x.x:xxxxx->x.x.x.x:xxxxx: write: broken pipe
[mysql] 2016/11/03 21:39:01 statement.go:27: invalid connection
[mysql] 2016/11/03 21:39:01 packets.go:33: unexpected EOF
[mysql] 2016/11/03 21:39:01 statement.go:27: invalid connection
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x20 pc=0x9a42a2]

goroutine 422144 [running]:
panic(0x148b560, 0xc8200100b0)
/goroot/src/runtime/panic.go:481
database/sql.(*Rows).Next(0x0, 0xc8207cf818)
/goroot/src/database/sql/sql.go:1751
github.com/hashicorp/vault/physical.(*MySQLBackend).List(0xc820162420, 0xc820a269c0, 0x3a, 0x0, 0x0, 0x0, 0x0, 0x0)
physical/mysql.go:177
physical/cache.go:83
vault/barrier_aes_gcm.go:678
vault/barrier_view.go:44
vault/token_store.go:786
vault/token_store.go:775
vault/expiration.go:581
vault/expiration.go:197
vault/expiration.go:177
vault/expiration.go:565
vault/expiration.go:538
created by time.goFunc
/goroot/src/time/sleep.go:129 +0x3a

This addresses #2063

That query used to have a check for err, but it was removed in 46ba8d1.
The caller, expireID in vault/expiration.go:565, retries after 1 second, and
there's other failures in this method that return nil,err so this approach
looks valid.

We were seeing these panics/crashes since our vault is lightly used:

    [mysql] 2016/11/03 21:39:01 packets.go:33: unexpected EOF
    [mysql] 2016/11/03 21:39:01 packets.go:124: write tcp x.x.x.x:xxxxx->x.x.x.x:xxxxx: write: broken pipe
    [mysql] 2016/11/03 21:39:01 statement.go:27: invalid connection
    [mysql] 2016/11/03 21:39:01 packets.go:33: unexpected EOF
    [mysql] 2016/11/03 21:39:01 statement.go:27: invalid connection
    panic: runtime error: invalid memory address or nil pointer dereference
    [signal 0xb code=0x1 addr=0x20 pc=0x9a42a2]

    goroutine 422144 [running]:
    panic(0x148b560, 0xc8200100b0)
    /goroot/src/runtime/panic.go:481
    database/sql.(*Rows).Next(0x0, 0xc8207cf818)
    /goroot/src/database/sql/sql.go:1751
    github.com/hashicorp/vault/physical.(*MySQLBackend).List(0xc820162420, 0xc820a269c0, 0x3a, 0x0, 0x0, 0x0, 0x0, 0x0)
    physical/mysql.go:177
    physical/cache.go:83
    vault/barrier_aes_gcm.go:678
    vault/barrier_view.go:44
    vault/token_store.go:786
    vault/token_store.go:775
    vault/expiration.go:581
    vault/expiration.go:197
    vault/expiration.go:177
    vault/expiration.go:565
    vault/expiration.go:538
    created by time.goFunc
    /goroot/src/time/sleep.go:129 +0x3a

This addresses hashicorp#2063
@jefferai jefferai added this to the 0.6.3 milestone Nov 17, 2016
@jefferai
Copy link
Member

Thanks!

@jefferai jefferai merged commit 2b374af into hashicorp:master Nov 17, 2016
@jefferai jefferai changed the title check for failure on that query check for failure on mysql query Nov 17, 2016
chrishoffman added a commit to chrishoffman/vault that referenced this pull request Nov 24, 2016
* upstream/master:
  Add version sha to server startup output
  Change current version number
  changelog++
  Don't say mlock is supported on OSX when it isn't. (hashicorp#2120)
  Change command examples for First Secrets hashicorp#2116 (hashicorp#2117)
  changelog++
  cli: fix bug with 'vault read -field=...' when the field value contains a printf formatting verb (hashicorp#2109)
  Update docs to fix hashicorp#2102
  Updating changelog
  http: increase request limit from 8MB to 32MB
  http: limit maximum request size
  Update libraries doc for Haskell community library (hashicorp#2101)
  Don't exclude 0 from the set of valid polynomials in Shamir. This leads to a potential (although extremely trivial) amount of information leakage.
  Bump proto files after update
  changelog++
  check for failure on that mysql query (hashicorp#2105)
  Bump deps
  Document bug causing certain LDAP settings to be forgotten on upgrade to 0.6.1+.
  Set number of pester retries to zero by default and make seal command… (hashicorp#2093)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants