Performance issue after writing 5k records?

[kokizzu]

Describe the bug
Everytime writing 5k records, always get connection reset by peer

Get "http://127.0.0.1:8200/v1/secret/data/keys1/yfsjuiqF.com": read tcp 127.0.0.1:46068->127.0.0.1:8200: read: connection reset by peer

To Reproduce
Steps to reproduce the behavior:

1. Clone this repo https://github.com/kokizzu/vaultdist1
2. Run docker compose up --build
3. Run ./fetch-tokens.sh
4. Run go run main.go benchmark

Expected behavior
No error

Environment:

• Vault Server Version (retrieve with vault status): latest docker
• Vault CLI Version (retrieve with vault version): latest docker
• Server Operating System/Architecture: Linux/amd64 (Ubuntu/Pop!_OS 22.04)

Vault server configuration file(s):

#reader1-policy.hcl
path "secret/data/keys1/*" {
  capabilities = ["read"]
}

path "secret/keys1/*" { # v1
  capabilities = ["read"]
}

#writer1-policy.hcl

path "auth/approle/role/reader1/secret-id" {
  capabilities = ["update"]
}

path "auth/approle/role/writer1/secret-id" {
  capabilities = ["update"]
}

path "secret/data/keys1/*" {
  capabilities = ["create","update","read","patch","delete"]
}

path "secret/keys1/*" { # v1
  capabilities = ["create","update","read","patch","delete"]
}

path "secret/metadata/keys1/*" {
  capabilities = ["list"]
}

[raskchanky]

Running ./fetch-tokens.sh produces 2 errors, both 403s. Running go run main.go benchmark produces similar errors.

[kokizzu]

weird, is the vault running on 127.0.0.1:8200 from docker compose up --build?

[raskchanky]

@kokizzu Apologies for the noise. That was my bad - I had multiple Vault servers running in the background and your tools were connecting to those instead. I've re-run your tools again, making sure your Vault server from the docker-compose file was the only one running. This time I received a different error, like so:

I started to wonder if the problem was in the benchmarking tool, so I ran a 5 minute benchmark using https://github.com/hashicorp/vault-benchmark. It processed 429K requests and didn't experience any errors. Here's the output:

Based on this, I'm wondering if the problem might be in your benchmarking tool, rather than in Vault itself.

[kokizzu]

i think not, because when i'm using my own secret engine plugin it works fine without issue, but with default kv plugin it has those issue

the one with my plugin is go run main.go benchmarkplugin
the code is 90% similar, just difference in secret engine