Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

inputs:sqlserver memguard panic when dealing with many connection strings #12982

Closed
spaghettidba opened this issue Mar 29, 2023 · 2 comments
Closed

inputs:sqlserver memguard panic when dealing with many connection strings #12982

spaghettidba opened this issue Mar 29, 2023 · 2 comments
Assignees
@srebhan
Labels
bug unexpected problem or unintended behavior plugin/secretstores

Comments

@spaghettidba
Copy link
Contributor

spaghettidba commented Mar 29, 2023 •
edited

Relevant telegraf.conf

# use a large number of servers here to reproduce
[[inputs.sqlserver]]
  servers = [
      'log=1;connection timeout=8;server=SERVER01',
      'log=1;connection timeout=8;server=SERVER02',
      'log=1;connection timeout=8;server=SERVER03',
      'log=1;connection timeout=8;server=SERVER04',
      'log=1;connection timeout=8;server=SERVER05',
      'log=1;connection timeout=8;server=SERVER06',
      'log=1;connection timeout=8;server=SERVER07',
      'log=1;connection timeout=8;server=SERVER08',
      'log=1;connection timeout=8;server=SERVER09',
      'log=1;connection timeout=8;server=SERVER10',
      'log=1;connection timeout=8;server=SERVER11',
      'log=1;connection timeout=8;server=SERVER12',
      'log=1;connection timeout=8;server=SERVER13',
      'log=1;connection timeout=8;server=SERVER14',
      'log=1;connection timeout=8;server=SERVER15',
      'log=1;connection timeout=8;server=SERVER16',
      'log=1;connection timeout=8;server=SERVER17',
      'log=1;connection timeout=8;server=SERVER18',
      'log=1;connection timeout=8;server=SERVER19',
      'log=1;connection timeout=8;server=SERVER20',
      'log=1;connection timeout=8;server=SERVER21',
      'log=1;connection timeout=8;server=SERVER22',
      'log=1;connection timeout=8;server=SERVER23',
      'log=1;connection timeout=8;server=SERVER24',
      'log=1;connection timeout=8;server=SERVER25',
      'log=1;connection timeout=8;server=SERVER26',
      'log=1;connection timeout=8;server=SERVER27',
      'log=1;connection timeout=8;server=SERVER28',
      'log=1;connection timeout=8;server=SERVER29',
      'log=1;connection timeout=8;server=SERVER30',
      'log=1;connection timeout=8;server=SERVER31',
      'log=1;connection timeout=8;server=SERVER32',
      'log=1;connection timeout=8;server=SERVER33',
      'log=1;connection timeout=8;server=SERVER34',
      'log=1;connection timeout=8;server=SERVER35',
      'log=1;connection timeout=8;server=SERVER36',
      'log=1;connection timeout=8;server=SERVER37',
      'log=1;connection timeout=8;server=SERVER38',
      'log=1;connection timeout=8;server=SERVER39',
      'log=1;connection timeout=8;server=SERVER40',
      'log=1;connection timeout=8;server=SERVER41',
      'log=1;connection timeout=8;server=SERVER42',
      'log=1;connection timeout=8;server=SERVER43',
      'log=1;connection timeout=8;server=SERVER44',
      'log=1;connection timeout=8;server=SERVER45',
      'log=1;connection timeout=8;server=SERVER46',
      'log=1;connection timeout=8;server=SERVER47',
      'log=1;connection timeout=8;server=SERVER48',
      'log=1;connection timeout=8;server=SERVER49',
      'log=1;connection timeout=8;server=SERVER50',
      'log=1;connection timeout=8;server=SERVER51',
      'log=1;connection timeout=8;server=SERVER52',
      'log=1;connection timeout=8;server=SERVER53',
      'log=1;connection timeout=8;server=SERVER54',
      'log=1;connection timeout=8;server=SERVER55',
      'log=1;connection timeout=8;server=SERVER56',
      'log=1;connection timeout=8;server=SERVER57',
      'log=1;connection timeout=8;server=SERVER58',
      'log=1;connection timeout=8;server=SERVER59',
      'log=1;connection timeout=8;server=SERVER60',
      'log=1;connection timeout=8;server=SERVER61',
      'log=1;connection timeout=8;server=SERVER62',
      'log=1;connection timeout=8;server=SERVER63',
      'log=1;connection timeout=8;server=SERVER64',
      'log=1;connection timeout=8;server=SERVER65',
      'log=1;connection timeout=8;server=SERVER66',
      'log=1;connection timeout=8;server=SERVER67',
      'log=1;connection timeout=8;server=SERVER68',
      'log=1;connection timeout=8;server=SERVER69',
      'log=1;connection timeout=8;server=SERVER70',
      'log=1;connection timeout=8;server=SERVER71',
      'log=1;connection timeout=8;server=SERVER72',
      'log=1;connection timeout=8;server=SERVER73',
      'log=1;connection timeout=8;server=SERVER74',
      'log=1;connection timeout=8;server=SERVER75',
      'log=1;connection timeout=8;server=SERVER76',
      'log=1;connection timeout=8;server=SERVER77',
      'log=1;connection timeout=8;server=SERVER78',
      'log=1;connection timeout=8;server=SERVER79',
      'log=1;connection timeout=8;server=SERVER80',
      'log=1;connection timeout=8;server=SERVER81',
      'log=1;connection timeout=8;server=SERVER82',
      'log=1;connection timeout=8;server=SERVER83',
      'log=1;connection timeout=8;server=SERVER84',
      'log=1;connection timeout=8;server=SERVER85',
      'log=1;connection timeout=8;server=SERVER86',
      'log=1;connection timeout=8;server=SERVER87',
      'log=1;connection timeout=8;server=SERVER88',
      'log=1;connection timeout=8;server=SERVER89',
      'log=1;connection timeout=8;server=SERVER90',
      'log=1;connection timeout=8;server=SERVER91',
      'log=1;connection timeout=8;server=SERVER92',
      'log=1;connection timeout=8;server=SERVER93',
      'log=1;connection timeout=8;server=SERVER94',
      'log=1;connection timeout=8;server=SERVER95',
      'log=1;connection timeout=8;server=SERVER96',
      'log=1;connection timeout=8;server=SERVER97',
      'log=1;connection timeout=8;server=SERVER98',
      'log=1;connection timeout=8;server=SERVER99',
  ]

  database_type = "SQLServer"

  include_query = [
    'SQLServerRequests'
  ]

[[outputs.file]]
  files = ["stdout"]
  data_format = "influx"

Logs from Telegraf

2023-03-29T14:54:24Z I! Loading config file: telegraf.conf
2023-03-29T14:54:24Z I! Starting Telegraf 1.3.0-1e9319f4 (customized)
2023-03-29T14:54:24Z I! Available plugins: 12 inputs, 0 aggregators, 3 processors, 6 parsers, 5 outputs, 0 secret-stores
2023-03-29T14:54:24Z I! Loaded inputs: sqlserver
2023-03-29T14:54:24Z I! Loaded aggregators:
2023-03-29T14:54:24Z I! Loaded processors:
2023-03-29T14:54:24Z I! Loaded secretstores:
2023-03-29T14:54:24Z I! Loaded outputs: file
2023-03-29T14:54:24Z I! Tags enabled: host=QDWS01
2023-03-29T14:54:24Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"QDWS01", Flush Interval:10s
2023-03-29T14:54:24Z I! [inputs.sqlserver] Config: database_type: SQLServer , query_version:0 , azuredb: false
2023-03-29T14:54:24Z I! [inputs.sqlserver] Config: Effective Queries: []string{"SQLServerRequests"}

2023-03-29T14:54:32Z E! [inputs.sqlserver] Error in plugin: query SQLServerRequests failed for server: SERVER10 and database: <empty-database-name> with Error: lookup SERVER10: no such host

... many more identical errors (it's all fictional servers) ...

2023-03-29T15:00:10Z W! [inputs.sqlserver] Collection took longer than expected; not complete after interval of 10s
2023-03-29T15:00:20Z W! [inputs.sqlserver] Collection took longer than expected; not complete after interval of 10s
2023-03-29T15:00:30Z W! [inputs.sqlserver] Collection took longer than expected; not complete after interval of 10s
2023-03-29T15:00:40Z W! [inputs.sqlserver] Collection took longer than expected; not complete after interval of 10s
2023-03-29T15:00:50Z W! [inputs.sqlserver] Collection took longer than expected; not complete after interval of 10s
2023-03-29T15:00:00Z W! [inputs.sqlserver] Collection took longer than expected; not complete after interval of 10s
2023-03-29T15:00:10Z W! [inputs.sqlserver] Collection took longer than expected; not complete after interval of 10s
2023-03-29T15:00:20Z W! [inputs.sqlserver] Collection took longer than expected; not complete after interval of 10s
2023-03-29T15:00:30Z W! [inputs.sqlserver] Collection took longer than expected; not complete after interval of 10s
2023-03-29T15:00:40Z W! [inputs.sqlserver] Collection took longer than expected; not complete after interval of 10s
2023-03-29T15:00:50Z W! [inputs.sqlserver] Collection took longer than expected; not complete after interval of 10s
2023-03-29T15:01:00Z W! [inputs.sqlserver] Collection took longer than expected; not complete after interval of 10s

...

2023-03-29T15:01:36Z  W! [inputs.sqlserver] Collection took longer than expected; not complete after interval of 10s

panic: <memcall> could not acquire lock on 0x21c2c9b1000, limit reached? [Err: Insufficient quota to complete the requested service.]

goroutine 713 [running]:
github.com/awnumar/memguard/core.Panic({0x9f6800, 0xc00021f3b0})
        C:/Users/gsartori/go/pkg/mod/github.com/awnumar/memguard@v0.22.3/core/exit.go:86 +0x34
github.com/awnumar/memguard/core.NewBuffer(0x3c)
        C:/Users/gsartori/go/pkg/mod/github.com/awnumar/memguard@v0.22.3/core/buffer.go:73 +0x2d5
github.com/awnumar/memguard/core.Open(0xc000008390)
        C:/Users/gsartori/go/pkg/mod/github.com/awnumar/memguard@v0.22.3/core/enclave.go:105 +0x2d
github.com/awnumar/memguard.(*Enclave).Open(0x1d9582cdf12cc8b?)
        C:/Users/gsartori/go/pkg/mod/github.com/awnumar/memguard@v0.22.3/enclave.go:43 +0x25
github.com/influxdata/telegraf/config.(*Secret).Get(0xc00025bba0)
        C:/github/telegraf_custom/config/secret.go:128 +0x8e
github.com/influxdata/telegraf/plugins/inputs/sqlserver.(*SQLServer).Gather.func1(0xc0009a2140?, {{0xb0fee4, 0x11}, {0xb70dc3, 0x10f1}, 0x0, {0x0, 0x0, 0x0}}, 0x2e)
        C:/github/telegraf_custom/plugins/inputs/sqlserver/sqlserver.go:275 +0x11d
created by github.com/influxdata/telegraf/plugins/inputs/sqlserver.(*SQLServer).Gather
        C:/github/telegraf_custom/plugins/inputs/sqlserver/sqlserver.go:273 +0x15d

System info

Telegraf 1.26.0 on Windows 11

Docker

No response

Steps to reproduce

Run telegraf with the included conf

Expected behavior

Should continue gathering data

Actual behavior

Panics and quits

Additional info

Looks a lot like #12980, except there is no way to increase the limit on windows
@spaghettidba spaghettidba added the bug unexpected problem or unintended behavior label Mar 29, 2023
@srebhan
Copy link
Contributor

srebhan commented Mar 29, 2023

Probably the same root cause as #12980...

@srebhan srebhan self-assigned this Mar 29, 2023
This was referenced Mar 29, 2023
Closed
Closed
Closed
Open
@spaghettidba
Copy link
Contributor Author

spaghettidba commented Apr 26, 2023 •
edited

Seems ok with #12988
Closing for now. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@srebhan srebhan

Labels
bug unexpected problem or unintended behavior plugin/secretstores
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@spaghettidba @srebhan