Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

database static roles and credential rotation bug #7222

Closed
ivanchuang opened this issue Jul 31, 2019 · 4 comments · Fixed by #8098
Closed

database static roles and credential rotation bug #7222

ivanchuang opened this issue Jul 31, 2019 · 4 comments · Fixed by #8098
Labels
bug Used to indicate a potential bug secret/database

Comments

@ivanchuang
Copy link

ivanchuang commented Jul 31, 2019 •
edited
Loading

Describe the bug
when rotation_period is expired
run "vault write -f" & "vault delete" have error.

To Reproduce
Steps to reproduce the behavior:

  1. create database static-role

root@vault:/# vault write database/static-roles/test_wi \
db_name=mariadb10_2 \
rotation_statements="SET PASSWORD FOR '{{name}}'@'%' = PASSWORD('{{password}}');" \
username="test_wi" \
rotation_period=1m

Success! Data written to: database/static-roles/test_wi
  1. read static role: test_wi

root@vault:/# vault read database/static-roles/test_wi
==============================================
Key                    Value
==============================================
last_vault_rotation    2019-07-31T18:12:25.385338874+08:00
password               A1a-hWJ76oZ5NrLoomFV
rotation_period        1m
ttl                    58s
username               test_wi
  1. read static credential: test_wi

root@vault:/# vault read database/static-creds/test_wi
==============================================
Key                    Value
==============================================
last_vault_rotation    2019-07-31T18:12:35.667951775+08:00
password               A1a-RS02vPzwtp6ol2uW
rotation_period        1m
ttl                    40s
username               test_wi
  1. read static credential test_wi again (ttl is 0s)

root@vault:/# vault read database/static-creds/test_wi
==============================================
Key                    Value
==============================================
last_vault_rotation    2019-07-31T18:13:36.447644545+08:00
password               A1a-7rGrU8cC8Wd5qLFN
rotation_period        1m
ttl                    0s
username               test_wi
  1. Manually rotate the password

root@vault:/# vault write -f database/rotate-role/test_wi

Error writing data to database/rotate-role/test_wi: Put http://127.0.0.1:8200/v1/database/rotate-role/test_wi: EOF
  1. try to delete static role

root@vault:/# vault delete database/static-roles/test_wi

Error deleting database/static-roles/test_wi: Delete http://127.0.0.1:8200/v1/database/static-roles/test_wi: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

Server syslog


Aug 15 11:32:44 vault-2 vault[5497]: 2019-08-15T11:32:44.723+0800 [INFO]  http: panic serving 127.0.0.1:33534: runtime error: invalid memory address or nil pointer dereference
Aug 15 11:32:44 vault-2 vault[5497]: goroutine 2768822 [running]:
Aug 15 11:32:44 vault-2 vault[5497]: net/http.(*conn).serve.func1(0xc0001f3040)
Aug 15 11:32:44 vault-2 vault[5497]: #011/goroot/src/net/http/server.go:1769 +0x139
Aug 15 11:32:44 vault-2 vault[5497]: panic(0x2c125c0, 0x5b694f0)
Aug 15 11:32:44 vault-2 vault[5497]: #011/goroot/src/runtime/panic.go:522 +0x1b5
Aug 15 11:32:44 vault-2 vault[5497]: github.com/hashicorp/vault/builtin/logical/database.(*databaseBackend).pathRotateRoleCredentialsUpdate.func1(0x37b7480, 0xc000e691a0, 0xc000102dc0, 0xc000
7bc2f0, 0xc000959c28, 0x923e8750, 0x100000000000000)
Aug 15 11:32:44 vault-2 vault[5497]: #011/gopath/src/github.com/hashicorp/vault/builtin/logical/database/path_rotate_credentials.go:139 +0x5da
Aug 15 11:32:44 vault-2 vault[5497]: github.com/hashicorp/vault/vendor/github.com/hashicorp/vault/sdk/framework.(*Backend).HandleRequest(0xc000959c70, 0x37b7480, 0xc000e691a0, 0xc000102dc0, 0
x0, 0x0, 0x0)
Aug 15 11:32:44 vault-2 vault[5497]: #011/gopath/src/github.com/hashicorp/vault/vendor/github.com/hashicorp/vault/sdk/framework/backend.go:253 +0x492
Aug 15 11:32:44 vault-2 vault[5497]: github.com/hashicorp/vault/vault.(*Router).routeCommon(0xc0003be910, 0x37b7480, 0xc000e691a0, 0xc000102dc0, 0x0, 0x0, 0x2000000, 0x0, 0x0)
Aug 15 11:32:44 vault-2 vault[5497]: #011/gopath/src/github.com/hashicorp/vault/vault/router.go:676 +0x919
Aug 15 11:32:44 vault-2 vault[5497]: github.com/hashicorp/vault/vault.(*Router).Route(...)
Aug 15 11:32:44 vault-2 vault[5497]: #011/gopath/src/github.com/hashicorp/vault/vault/router.go:476
Aug 15 11:32:44 vault-2 vault[5497]: github.com/hashicorp/vault/vault.(*Core).doRouting(0xc000756000, 0x37b7480, 0xc000e691a0, 0xc000102dc0, 0xc000266c60, 0x0, 0x0)
Aug 15 11:32:44 vault-2 vault[5497]: #011/gopath/src/github.com/hashicorp/vault/vault/request_handling.go:558 +0x5a
Aug 15 11:32:44 vault-2 vault[5497]: github.com/hashicorp/vault/vault.(*Core).handleRequest(0xc000756000, 0x37b7480, 0xc000e691a0, 0xc000102dc0, 0x0, 0x0, 0x0, 0x0)
Aug 15 11:32:44 vault-2 vault[5497]: #011/gopath/src/github.com/hashicorp/vault/vault/request_handling.go:699 +0xc1d
Aug 15 11:32:44 vault-2 vault[5497]: github.com/hashicorp/vault/vault.(*Core).handleCancelableRequest(0xc000756000, 0x37b7480, 0xc000e691a0, 0x5b6ece0, 0xc000102dc0, 0x5b6ece0, 0x37b7480, 0xc
000e691a0)
Aug 15 11:32:44 vault-2 vault[5497]: #011/gopath/src/github.com/hashicorp/vault/vault/request_handling.go:450 +0xd9f
Aug 15 11:32:44 vault-2 vault[5497]: github.com/hashicorp/vault/vault.(*Core).switchedLockHandleRequest(0xc000756000, 0x37b7480, 0xc000e68ff0, 0xc000102dc0, 0x1, 0x0, 0x0, 0x0)
Aug 15 11:32:44 vault-2 vault[5497]: #011/gopath/src/github.com/hashicorp/vault/vault/request_handling.go:417 +0x288
Aug 15 11:32:44 vault-2 vault[5497]: github.com/hashicorp/vault/vault.(*Core).HandleRequest(...)
Aug 15 11:32:44 vault-2 vault[5497]: #011/gopath/src/github.com/hashicorp/vault/vault/request_handling.go:382
Aug 15 11:32:44 vault-2 vault[5497]: github.com/hashicorp/vault/http.request(0xc000756000, 0x37981c0, 0xc00012a000, 0xc00066b500, 0xc000102dc0, 0x0, 0x0)
Aug 15 11:32:44 vault-2 vault[5497]: #011/gopath/src/github.com/hashicorp/vault/http/handler.go:620 +0x80
Aug 15 11:32:44 vault-2 vault[5497]: github.com/hashicorp/vault/http.handleLogicalInternal.func1(0x37981c0, 0xc00012a000, 0xc00066b500)
Aug 15 11:32:44 vault-2 vault[5497]: #011/gopath/src/github.com/hashicorp/vault/http/logical.go:278 +0x32d
Aug 15 11:32:44 vault-2 vault[5497]: net/http.HandlerFunc.ServeHTTP(0xc0006538a0, 0x37981c0, 0xc00012a000, 0xc00066b500)
Aug 15 11:32:44 vault-2 vault[5497]: #011/goroot/src/net/http/server.go:1995 +0x44
Aug 15 11:32:44 vault-2 vault[5497]: github.com/hashicorp/vault/http.handleRequestForwarding.func1(0x37981c0, 0xc00012a000, 0xc00066b500)
Aug 15 11:32:44 vault-2 vault[5497]: #011/gopath/src/github.com/hashicorp/vault/http/handler.go:554 +0x424
Aug 15 11:32:44 vault-2 vault[5497]: net/http.HandlerFunc.ServeHTTP(0xc0006538c0, 0x37981c0, 0xc00012a000, 0xc00066b500)
Aug 15 11:32:44 vault-2 vault[5497]: #011/goroot/src/net/http/server.go:1995 +0x44
Aug 15 11:32:44 vault-2 vault[5497]: net/http.(*ServeMux).ServeHTTP(0xc0006a3cc0, 0x37981c0, 0xc00012a000, 0xc00066b500)
Aug 15 11:32:44 vault-2 vault[5497]: #011/goroot/src/net/http/server.go:2375 +0x1d6
Aug 15 11:32:44 vault-2 vault[5497]: github.com/hashicorp/vault/http.wrapHelpHandler.func1(0x37981c0, 0xc00012a000, 0xc00066b500)
Aug 15 11:32:44 vault-2 vault[5497]: #011/gopath/src/github.com/hashicorp/vault/http/help.go:24 +0x156
Aug 15 11:32:44 vault-2 vault[5497]: net/http.HandlerFunc.ServeHTTP(0xc000653960, 0x37981c0, 0xc00012a000, 0xc00066b500)
Aug 15 11:32:44 vault-2 vault[5497]: #011/goroot/src/net/http/server.go:1995 +0x44
Aug 15 11:32:44 vault-2 vault[5497]: github.com/hashicorp/vault/http.wrapCORSHandler.func1(0x37981c0, 0xc00012a000, 0xc00066b500)
Aug 15 11:32:44 vault-2 vault[5497]: #011/gopath/src/github.com/hashicorp/vault/http/cors.go:29 +0x9e1
Aug 15 11:32:44 vault-2 vault[5497]: net/http.HandlerFunc.ServeHTTP(0xc000653980, 0x37981c0, 0xc00012a000, 0xc00066b500)
Aug 15 11:32:44 vault-2 vault[5497]: #011/goroot/src/net/http/server.go:1995 +0x44
Aug 15 11:32:44 vault-2 vault[5497]: github.com/hashicorp/vault/http.wrapGenericHandler.func1(0x37981c0, 0xc00012a000, 0xc00066b300)
Aug 15 11:32:44 vault-2 vault[5497]: #011/gopath/src/github.com/hashicorp/vault/http/handler.go:204 +0x339
Aug 15 11:32:44 vault-2 vault[5497]: net/http.HandlerFunc.ServeHTTP(0xc0006775f0, 0x37981c0, 0xc00012a000, 0xc00066b300)
Aug 15 11:32:44 vault-2 vault[5497]: #011/goroot/src/net/http/server.go:1995 +0x44
Aug 15 11:32:44 vault-2 vault[5497]: github.com/hashicorp/vault/vendor/github.com/hashicorp/go-cleanhttp.PrintablePathCheckHandler.func1(0x37981c0, 0xc00012a000, 0xc00066b300)
Aug 15 11:32:44 vault-2 vault[5497]: #011/gopath/src/github.com/hashicorp/vault/vendor/github.com/hashicorp/go-cleanhttp/handlers.go:42 +0xbb
Aug 15 11:32:44 vault-2 vault[5497]: net/http.HandlerFunc.ServeHTTP(0xc0006539a0, 0x37981c0, 0xc00012a000, 0xc00066b300)
Aug 15 11:32:44 vault-2 vault[5497]: #011/goroot/src/net/http/server.go:1995 +0x44
Aug 15 11:32:44 vault-2 vault[5497]: github.com/hashicorp/vault/http.WrapForwardedForHandler.func1(0x37981c0, 0xc00012a000, 0xc00066b300)
Aug 15 11:32:44 vault-2 vault[5497]: #011/gopath/src/github.com/hashicorp/vault/http/handler.go:215 +0x1e5
Aug 15 11:32:44 vault-2 vault[5497]: net/http.HandlerFunc.ServeHTTP(0xc00004e460, 0x37981c0, 0xc00012a000, 0xc00066b300)
Aug 15 11:32:44 vault-2 vault[5497]: #011/goroot/src/net/http/server.go:1995 +0x44
Aug 15 11:32:44 vault-2 vault[5497]: net/http.serverHandler.ServeHTTP(0xc000694c30, 0x37981c0, 0xc00012a000, 0xc00066b300)
Aug 15 11:32:44 vault-2 vault[5497]: #011/goroot/src/net/http/server.go:2774 +0xa8
Aug 15 11:32:44 vault-2 vault[5497]: net/http.(*conn).serve(0xc0001f3040, 0x37b73c0, 0xc0006b5ec0)
Aug 15 11:32:44 vault-2 vault[5497]: #011/goroot/src/net/http/server.go:1878 +0x851
Aug 15 11:32:44 vault-2 vault[5497]: created by net/http.(*Server).Serve
Aug 15 11:32:44 vault-2 vault[5497]: #011/goroot/src/net/http/server.go:2884 +0x2f4

Environment:
Vault Server Version (retrieve with vault status): 1.2.0
Vault CLI Version (retrieve with vault version): 1.2.0
Server Operating System/Architecture: ubuntu 16.04.6 x86_64
@chrishoffman
Copy link
Contributor

Can you please provide your server logs? Usually you will get the EOF when there is a panic or some other error in the server logs.

@chrishoffman chrishoffman added the bug Used to indicate a potential bug label Aug 9, 2019
@snahelou
Copy link

I have exactly the same problem.

$ vault secrets enable -path=postgresql database
$ vault write postgresql/config/connection plugin_name=postgresql-database-plugin connection_url="postgresql://{{username}}:{{password}}@pgsql:5432/demo?sslmode=disable" username="postgres" password="password"
$ vault write postgresql/config/connection allowed_roles="springstatic"
$ vault write postgresql/static-roles/springstatic \
  db_name=connection \
  rotation_statements="ALTER USER \"{{name}}\" WITH PASSWORD '{{password}}';" \
  username="springstatic" \
  rotation_period=90s

The first refresh works fine. But the second stuck with TTL 0s.

Key                    Value
---                    -----
last_vault_rotation    2019-08-13T19:25:41.6650333Z
password               A1a-siq5p6QYqwDgQ9LU
rotation_period        1m30s
ttl                    0s
username               springstatic

I'm trying to refresh manually:

vault write -f postgresql/rotate-role/springstatic
Error writing data to postgresql/rotate-role/springstatic: Put http://127.0.0.1:8200/v1/postgresql/rotate-role/springstatic: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

Server logs:

2019-08-13T19:15:36.240Z [INFO]  secrets.database.database_27974dd7: initializing database rotation queue
2019-08-13T19:15:36.241Z [INFO]  secrets.database.database_27974dd7: populating role rotation queue
2019-08-13T19:15:36.242Z [INFO]  secrets.database.database_27974dd7: starting periodic ticker
2019-08-13T19:21:51.455Z [INFO]  http: panic serving 127.0.0.1:45578: runtime error: invalid memory address or nil pointer dereference
goroutine 3356 [running]:
net/http.(*conn).serve.func1(0xc0001e1a40)
	/goroot/src/net/http/server.go:1769 +0x139
panic(0x2c13560, 0x5b6b250)
	/goroot/src/runtime/panic.go:522 +0x1b5
github.com/hashicorp/vault/builtin/logical/database.(*databaseBackend).pathRotateRoleCredentialsUpdate.func1(0x37b8580, 0xc000aac780, 0xc0000f5a40, 0xc000019af0, 0xc00081ad88, 0x32383462398a759e, 0x100352d38383063)
	/gopath/src/github.com/hashicorp/vault/builtin/logical/database/path_rotate_credentials.go:139 +0x5da
github.com/hashicorp/vault/vendor/github.com/hashicorp/vault/sdk/framework.(*Backend).HandleRequest(0xc00081add0, 0x37b8580, 0xc000aac780, 0xc0000f5a40, 0x0, 0x0, 0x0)
	/gopath/src/github.com/hashicorp/vault/vendor/github.com/hashicorp/vault/sdk/framework/backend.go:253 +0x492
github.com/hashicorp/vault/vault.(*Router).routeCommon(0xc00022d2c0, 0x37b8580, 0xc000aac780, 0xc0000f5a40, 0x0, 0x0, 0x0, 0x0, 0x0)
	/gopath/src/github.com/hashicorp/vault/vault/router.go:676 +0x919
github.com/hashicorp/vault/vault.(*Router).Route(...)
	/gopath/src/github.com/hashicorp/vault/vault/router.go:476
github.com/hashicorp/vault/vault.(*Core).doRouting(0xc0003c2000, 0x37b8580, 0xc000aac780, 0xc0000f5a40, 0xc000848120, 0x0, 0x0)
	/gopath/src/github.com/hashicorp/vault/vault/request_handling.go:558 +0x5a
github.com/hashicorp/vault/vault.(*Core).handleRequest(0xc0003c2000, 0x37b8580, 0xc000aac780, 0xc0000f5a40, 0x0, 0x0, 0x0, 0x0)
	/gopath/src/github.com/hashicorp/vault/vault/request_handling.go:699 +0xc1d
github.com/hashicorp/vault/vault.(*Core).handleCancelableRequest(0xc0003c2000, 0x37b8580, 0xc000aac780, 0x5b70a40, 0xc0000f5a40, 0x5b70a40, 0x37b8580, 0xc000aac780)
	/gopath/src/github.com/hashicorp/vault/vault/request_handling.go:450 +0xd9f
github.com/hashicorp/vault/vault.(*Core).switchedLockHandleRequest(0xc0003c2000, 0x37b8580, 0xc000aac5d0, 0xc0000f5a40, 0x1, 0x0, 0x0, 0x0)
	/gopath/src/github.com/hashicorp/vault/vault/request_handling.go:417 +0x288
github.com/hashicorp/vault/vault.(*Core).HandleRequest(...)
	/gopath/src/github.com/hashicorp/vault/vault/request_handling.go:382
github.com/hashicorp/vault/http.request(0xc0003c2000, 0x37992c0, 0xc0008a1c00, 0xc000acc900, 0xc0000f5a40, 0x0, 0x0)
	/gopath/src/github.com/hashicorp/vault/http/handler.go:620 +0x80
github.com/hashicorp/vault/http.handleLogicalInternal.func1(0x37992c0, 0xc0008a1c00, 0xc000acc900)
	/gopath/src/github.com/hashicorp/vault/http/logical.go:278 +0x32d
net/http.HandlerFunc.ServeHTTP(0xc000966180, 0x37992c0, 0xc0008a1c00, 0xc000acc900)
	/goroot/src/net/http/server.go:1995 +0x44
github.com/hashicorp/vault/http.handleRequestForwarding.func1(0x37992c0, 0xc0008a1c00, 0xc000acc900)
	/gopath/src/github.com/hashicorp/vault/http/handler.go:545 +0x2b7
net/http.HandlerFunc.ServeHTTP(0xc0009661a0, 0x37992c0, 0xc0008a1c00, 0xc000acc900)
	/goroot/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xc000a272c0, 0x37992c0, 0xc0008a1c00, 0xc000acc900)
	/goroot/src/net/http/server.go:2375 +0x1d6
github.com/hashicorp/vault/http.wrapHelpHandler.func1(0x37992c0, 0xc0008a1c00, 0xc000acc900)
	/gopath/src/github.com/hashicorp/vault/http/help.go:24 +0x156
net/http.HandlerFunc.ServeHTTP(0xc000966240, 0x37992c0, 0xc0008a1c00, 0xc000acc900)
	/goroot/src/net/http/server.go:1995 +0x44
github.com/hashicorp/vault/http.wrapCORSHandler.func1(0x37992c0, 0xc0008a1c00, 0xc000acc900)
	/gopath/src/github.com/hashicorp/vault/http/cors.go:29 +0x9e1
net/http.HandlerFunc.ServeHTTP(0xc000966260, 0x37992c0, 0xc0008a1c00, 0xc000acc900)
	/goroot/src/net/http/server.go:1995 +0x44
github.com/hashicorp/vault/http.wrapGenericHandler.func1(0x37992c0, 0xc0008a1c00, 0xc000acc700)
	/gopath/src/github.com/hashicorp/vault/http/handler.go:204 +0x339
net/http.HandlerFunc.ServeHTTP(0xc00095e4b0, 0x37992c0, 0xc0008a1c00, 0xc000acc700)
	/goroot/src/net/http/server.go:1995 +0x44
github.com/hashicorp/vault/vendor/github.com/hashicorp/go-cleanhttp.PrintablePathCheckHandler.func1(0x37992c0, 0xc0008a1c00, 0xc000acc700)
	/gopath/src/github.com/hashicorp/vault/vendor/github.com/hashicorp/go-cleanhttp/handlers.go:42 +0xbb
net/http.HandlerFunc.ServeHTTP(0xc000966280, 0x37992c0, 0xc0008a1c00, 0xc000acc700)
	/goroot/src/net/http/server.go:1995 +0x44
net/http.serverHandler.ServeHTTP(0xc0005e55f0, 0x37992c0, 0xc0008a1c00, 0xc000acc700)
	/goroot/src/net/http/server.go:2774 +0xa8
net/http.(*conn).serve(0xc0001e1a40, 0x37b84c0, 0xc000aa87c0)
	/goroot/src/net/http/server.go:1878 +0x851
created by net/http.(*Server).Serve
	/goroot/src/net/http/server.go:2884 +0x2f4

I use docker image : Vault 1.2.1

I hope it can help you...

@ncabatoff
Copy link
Collaborator

I spent a little bit of time looking at this. It looks like pathRotateRoleCredentialsUpdate calls popFromRotationQueueByKey expecting to get either a non-nil item or err, but it's actually possible to get back nil item and nil err. This happens when credRotationQueue.PopByKey doesn't find a queued item with the given name. I think the fix would be to make popFromRotationQueueByKey return queue.ErrEmpty if nothing is found, but I'm not familiar enough with the code to be confident of that.

@ivanchuang
Copy link
Author

Can you please provide your server logs? Usually you will get the EOF when there is a panic or some other error in the server logs.

Server log is provided~ :)

@michelvocks michelvocks mentioned this issue Jan 6, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Used to indicate a potential bug secret/database
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix potential panic in database credential role rotation hashicorp/vault
6 participants
@catsby @chrishoffman @snahelou @michelvocks @ncabatoff @ivanchuang