Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Response status code does not indicate success: 556 #280

Closed
bajayb4u opened this issue Mar 28, 2022 · 28 comments
Closed

Response status code does not indicate success: 556 #280

bajayb4u opened this issue Mar 28, 2022 · 28 comments
Labels
bug Issues that have been confirmed to be bugs in PrtgAPI and will be fixed in a future version

Comments

@bajayb4u
Copy link

bajayb4u commented Mar 28, 2022

Describe the bug

Hello,

Using PrtgAPI module v0.9.17, I am trying to add devices and sensors. No issues when adding device but only with add-sensors throwing this error. This error is not consistent even on the same device when added multiple times. Though most of times script is throwing this error sometimes it is working just fine.

Error:
Failed CMD Line: $getDevice | Add-Sensor $sensorparams
Error message: Response status code does not indicate success: 556 (Unknown Response Code).

Steps to reproduce

$PRTGGroupID = <PRTGGroupID> 
$deviceName = <deviceName> 
$ServerIP = <ServerIP> 
Connect-PrtgServer $prtgURL (New-Credential $prtgUserName $prtgPass) 
$addDevice = Get-Group -id $PRTGGroupID | Add-Device -Name $deviceName -Host $ServerIP 
$addDeviceId = $addDevice.Id 
$getDevice = Get-Device -Id $addDeviceId 
  
$SatSensor = "snmpcpu" 
  
$sensorparams = $getDevice | New-SensorParameters -RawType $SatSensor 
$cpusensor = $getDevice | Add-Sensor $sensorparams 
$cpuchannel = Get-Channel -Sensor $cpusensor 
$cputotal = $cpuchannel | Where {$_.Name -eq "Total"} 
Set-ChannelProperty -Channel $cputotal -Property UpperErrorLimit -Value 90 
Set-ChannelProperty -Channel $cputotal -Property UpperWarningLimit -Value 75

What is the output of 'Get-PrtgClient -Diagnostic'?

PSVersion      : 5.1.14393.4583
PSEdition      : Desktop
OS             : Microsoft Windows Server 2016 Standard
PrtgAPIVersion : 0.9.17
Culture        : en-US
CLRVersion     : .NET Framework 4.7.2 (461814)
PrtgVersion    : 22.1.74.1869
PrtgLanguage   : english.lng

Additional context

Above sample code has only snmpcpu sensor in it but we are also encountering same with ping and other SNMP sensors. Please let me know if I can provide any further information. Thanks for the help!

@bajayb4u bajayb4u added the alleged-bug Bugs raised by people helpfully using the Bug Report template! label Mar 28, 2022
@lordmilko
Copy link
Owner

Did this issue only start happening recently? Or did you only start using PrtgAPI and/or this feature recently and so this has always occurred?

@bajayb4u
Copy link
Author

We have recently upgraded Prtg to v22.1.74.1869. Before the upgrade we had v20.4.64.1402 and used PrtgAPI PS module v0.9.16 to add devices and sensors without any issues. After the upgrade we start seeing error "Could not authenticate to PRTG; the specified username and password were invalid", so we now using module v0.9.17. Now the authentication issue was resolved but getting this new error.

@carlbeechvuw
Copy link

Hi
Don't know whether this helps, but here's the full exception that I'm seeing when getting the same problem (i.e. randomly when adding sensors - and the sensors are present, even though there's an exception reported:

System.Net.Http.HttpRequestException: Response status code does not indicate success: 556 (Unknown Response Code).
at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
at PrtgAPI.Request.RequestEngine.ValidateHttpResponse(HttpResponseMessage responseMessage, PrtgResponse response)
at PrtgAPI.Request.RequestEngine.ExecuteRequest(PrtgRequestMessage request, CancellationToken token, Func2 responseParser) at PrtgAPI.Request.RequestEngine.ExecuteRequest(ICommandParameters parameters, Func2 responseParser, CancellationToken token)
at PrtgAPI.Request.VersionClient18_1.AddSensorInternal(ICommandParameters internalParams, Int32 index, CancellationToken token)
at PrtgAPI.PrtgClient.AddObjectInternal(Either2 parent, NewObjectParameters parameters, CancellationToken token) at PrtgAPI.PrtgClient.<>c__DisplayClass151_01.b__0(CancellationToken t)
at PrtgAPI.PrtgClient.ResolveWithDiff[T](Action1 createObject, Func2 getObjects, Func3 exceptFunc, CancellationToken token, Action2 errorCallback, Func1 shouldStop, Boolean allowMultiple, String enhancedResolutionError) at PrtgAPI.PrtgClient.AddObject[T](Either2 parent, NewObjectParameters parameters, Func3 getObjects, Boolean resolve, CancellationToken token, Action2 errorCallback, Func1 shouldStop, Boolean allowMultiple) at PrtgAPI.PowerShell.Base.NewObjectCmdlet.<>c__DisplayClass6_01.b__0()
at PrtgAPI.PowerShell.Base.NewObjectCmdlet.AddAndResolveRunner[T](Func1 addObject) at PrtgAPI.PowerShell.Base.AddObject2.ExecuteOperationAction(Int32 destinationId)
at PrtgAPI.PowerShell.Base.PrtgOperationCmdlet.ExecuteOperation(Action action, String progressMessage, Boolean complete, Boolean incrementRecord)
at PrtgAPI.PowerShell.Base.AddObject`2.AddObjectInternal(Int32 destinationId)
at PrtgAPI.PowerShell.Cmdlets.AddSensor.ProcessRecordEx()
at PrtgAPI.PowerShell.Base.PrtgCmdlet.ExecuteWithCoreState(Action action)
at System.Management.Automation.CommandProcessor.ProcessRecord()

PrtgAPI 0.9.17
PRTG: 22.1.74.1869

@lordmilko
Copy link
Owner

I haven't been able to reproduce this issue myself, however are you able to advise

  • Whether upgrading to PRTG 22.1.75 resolves the issue?
  • Whether the sensor is actually created when you get this error?

@bajayb4u
Copy link
Author

  • We just upgraded our PRTG to v22.1.74.1869 and it might a take while for one more upgrade
  • Yes, sensor is getting created however it is failing to get channel for the sensor and add any sensor properties. Also, as we add more sensors through a loop, script won't get pass the failed sensor.

We are not seeing full exception that was reported by @carlbeechvuw above, but based on that can you can any suggestion that we can try?

@lordmilko
Copy link
Owner

@bajayb4u,

I have no idea what the issue is; when trying to repro this you should be running one liner commands to narrow down what the source of the issue is. I tried reproducing the issue using your repro steps above; there is a mistake in your code: $DeviceId is never declared, however even if I manually fixup the code I'm unable to replicate the issue

@lordmilko
Copy link
Owner

@carlbeechvuw,

What sort of sensors are you trying to add? Can you provide the output of specifying -Verbose to the Add-Sensor cmdlet when it crashes (please omit the server, username and passhash)

@carlbeechvuw
Copy link

carlbeechvuw commented Mar 31, 2022 via email

@bajayb4u
Copy link
Author

bajayb4u commented Apr 1, 2022

Hi @lordmilko, here is the verbose output when adding the snmpcpu sensor; Reproduced using the same code I provided above (fixed the $DeviceId, it was a copy paste mistake). Script is throwing error at Add-sensor command.

Is it possible to ass some wait time to add-sensor?

VERBOSE: Performing the operation "Add-Sensor" on target "SNMP CPU Load 70 (Destination:2454566-pragtwb4 (XXX.XXX) (ID: 22222))".
VERBOSE: Add-Sensor: Synchronously executing request https://prtg.hostname.org/api/table.xml?content=sensors&columns=objid,name,probe,group,favorite,lastvalue,device,downtime,downtimetime,downtimesince,uptime,uptimetime,uptimesince,knowntime,cumsince,lastcheck,lastup,lastdown,minigraph,schedule,basetype,baselink,notifiesx,intervalx,access,dependency,position,status,comments,priority,message,parentid,tags,type,active&count=*&filter_parentid=55555&filter_type=snmpcpu&username=**********&passhash=**********
VERBOSE: Add-Sensor: Synchronously executing request https://prtg.hostname.org/controls/addsensor2.htm?id=98452&sensortype=snmpcpu&username=**********&passhash=**********
VERBOSE: Add-Sensor: Synchronously executing request https://prtg.hostname.org/addsensor5.htm?name_=SNMPCPULoad70&priority_=3&inherittriggers=1&intervalgroup=1&interval_=60%7C60seconds&errorintervalsdown_=1&tags_=snmpcpucpuloadsensor&anti-csrf-token=8567bfq385b696	98fe5bn0MmEyNGQ2OGE4ZTY4MWVjYjc1OA%3D%3D&sensortype=snmpcpu&id=98452&tmpid=129

Add-Sensor : Response status code does not indicate success: 556 (Unknown Response Code).
At C:\work\Powershell\PRTG_AddDevice.ps1:53 char:49
 ...           $cpusensor = $getDevice | Add-Sensor $sensorparams -Verbose
     CategoryInfo          : NotSpecified: (:) [Add-Sensor], HttpRequestException
     FullyQualifiedErrorId : System.Net.Http.HttpRequestException,PrtgAPI.PowerShell.Cmdlets.AddSensor
   
   Get-Channel : PRTG was unable to complete the request. The server responded with the following error: There is no object with the specified ID.
At C:\work\Powershell\PRTG_AddDevice.ps1:57 char:37
                       $cpuchannel = Get-Channel -Sensor $cpusensor
     CategoryInfo          : InvalidOperation: (:) [Get-Channel], PrtgRequestException
     FullyQualifiedErrorId : PrtgRequestException,PrtgAPI.PowerShell.Cmdlets.GetChannel
 
	SNMP CPU Sensor channel:
Set-ChannelProperty : Cannot bind argument to parameter 'Channel' because it is null.
At C:\work\Powershell\PRTG_AddDevice.ps1:61 char:52
                       Set-ChannelProperty -Channel $cputotal -Propert ...
     CategoryInfo          : InvalidData: (:) [Set-ChannelProperty], ParameterBinding ValidationException
     FullyQualifiedErrorId : ParameterArgumentValidationErrorNullNotAllowed,PrtgAPI.P owerShell.Cmdlets.SetChannelProperty
 
Set-ChannelProperty : Cannot bind argument to parameter 'Channel' because it is null.
At C:\work\Powershell\PRTG_AddDevice.ps1:62 char:52
                       Set-ChannelProperty -Channel $cputotal -Propert ...
     CategoryInfo          : InvalidData: (:) [Set-ChannelProperty], ParameterBinding ValidationException
     FullyQualifiedErrorId : ParameterArgumentValidationErrorNullNotAllowed,PrtgAPI.PowerShell.Cmdlets.SetChannelProperty

@lordmilko
Copy link
Owner

Can you see if you can replicate this issue with HTTP, WMI Service or Advanced EXE/Script sensors? I would like to see whether you can reproduce the issue when using dynamic sensor parameters for these 3 (New-SensorParameters with a -RawType of http, exexml or wmiservice)

If you can reproduce the issue with one of these sensor types, can you then see if you can reproduce the issue using a "normal" set of sensor parameters for them instead (i.e. New-SensorParameters Http rather than New-SensorParameters -RawType http)

This will help tell us whether it's got something to do with something in the dynamic sensor parameters. I note that the anti-csrf-token in the URL above appears to have a tab in it. Something else worth doing might be validating that is in fact the case and clearing out this property and seeing if that has an effect

$sensorparams = $getDevice | New-SensorParameters -RawType $SatSensor 

# What is the output of this?
$sensorparams ."anti-csrf-token".contains("`t")

# What happens if you do this then try and add the sensor?
$sensorparams ."anti-csrf-token" = $null

@bajayb4u
Copy link
Author

bajayb4u commented Apr 1, 2022

There is no tab in anti-csrf-token, kindly ignore that; I have edited it. But I will try to reproduce with dynamic and normal sensors as you suggested.

@bajayb4u
Copy link
Author

bajayb4u commented Apr 1, 2022

@lordmilko, this error is also reproducible with adding ping sensor; Here also I am using raw type,

$SatSensor = "ping"
$sensorparams = $getDevice | New-SensorParameters -RawType $SatSensor
$getDevice | Add-Sensor $sensorparams

@lordmilko
Copy link
Owner

Please see my post above and advise what occurs if you test using raw HTTP/WMI Service/EXE Script Advanced sensors using raw + non-raw types

@bajayb4u
Copy link
Author

bajayb4u commented Apr 4, 2022

@lordmilko,

I have tried http sensor few times with raw and non-raw types and in both the case the issue reproducible. Below is what I am getting,

Add-Sensor : Response status code does not indicate success: 556 (Unknown Response Code).
At line:17 char:14
$getDevice | Add-Sensor $sensorparams
~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Add-Sensor], HttpRequestException
+ FullyQualifiedErrorId : System.Net.Http.HttpRequestException,PrtgAPI.PowerShell.Cmdlets.AddSensor

@bajayb4u
Copy link
Author

bajayb4u commented Apr 5, 2022

I was able to reproduce the same error for WMI Service sensor with both raw type and non-raw type.

@bajayb4u
Copy link
Author

bajayb4u commented Apr 6, 2022

Update, we had a call with Paessler, and they suspect the issue may relate to system resources. we have thousands of sensors in our PRTG, as system is busy taking care of other tasks it might creating some read and write locks which causing this issue. They made some recommendation to offload sensors on core server local probe to remote probe to free up the load on core server. I will check the script again once we made the recommended changes.

Thank you @lordmilko, for your help

@lordmilko
Copy link
Owner

PrtgAPI should be able to handle any scenario regardless of load; the common scenario is that API requests time out (as opposed to returning random error codes). PrtgAPI has special logic for retrying API requests that fail due to a suspected timeout, so if that is indeed the case here an appropriate error handler needs to be implemented

@carlbeechvuw how many sensors do you have in your PRTG server?

@bajayb4u
Copy link
Author

Our PRTG server has 22300 sensors.

@Silex
Copy link

Silex commented May 2, 2022

We have the same issue (add-sensor failing randomly), our PRTG is huge and has way too many devices/sensors, maybe that's the issue.

    Write-Output ('Processing "{0}"' -f $sensor.Device)
    $device = Get-Device -Probe $probe -Name $sensor.Device -Group $sensor.Group

    $params = New-SensorParameters ExeXml 'My Sensor'
    $params.ExeFile = 'script.bat'
    $params.ExeParameters = '%host'
    $params.UseWindowsAuthentication = $true
    $params.Timeout = 120
    $ErrorActionPreference = 'Continue'
    Add-Sensor -ErrorAction Continue -Destination $device -Parameters $params | Out-Null
    $ErrorActionPreference = 'Stop'

Output:

Processing "DEVICE1"
Processing "DEVICE2"
Processing "DEVICE3"
Add-Sensor : Response status code does not indicate success: 556 (Unknown Response Code).
At C:\CCTV\Scripts\PRTG\PrtgReconfigureStorages.ps1:26 char:5
+     Add-Sensor -ErrorAction Continue -Destination $device -Parameters ...
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Add-Sensor], HttpRequestException
    + FullyQualifiedErrorId : System.Net.Http.HttpRequestException,PrtgAPI.PowerShell.Cmdlets.AddSensor

What makes NO SENSE to me is that despite all my attemps to tell it "PLEASE CONTINUE" it stops at the first error. As if $ErrorActionPreference and -ErrorAction was ignored.

EDIT: ok, it makes sense because it's actually an exception. Also it can be ignored because in PRTG the sensor is added (even tho it's reported as an error HTTP 556).

EDIT2: question, if you only have a sensor, is $device = Get-Device -Probe $probe -Name $sensor.Device -Group $sensor.Group the best way to get the device?

@Silex
Copy link

Silex commented May 2, 2022

Hello,

I wrapped the Add-Sensor in a try/catch which warns "Supposedly cannot add sensor" and here's the output:

Processing "DEVICE1"
Processing "DEVICE2"
WARNING: Supposedly cannot add sensor
Processing "DEVICE3"
WARNING: 'Add-Sensor' failed to resolve sensor: object is still being created. Retries remaining: 4
WARNING: 'Add-Sensor' failed to resolve sensor: object is still being created. Retries remaining: 3
Processing "DEVICE4"
Processing "DEVICE5"

It's very likely that this bug is triggered by an overloaded PRTG server, and HTTP 556 means something like "Ok but wait".

@lordmilko
Copy link
Owner

Based on the reports made in this thread it sounds to me as if HTTP status code 556 can indeed be safely ignored as objects are actually created properly, however I've raised a case with Paessler to get their input on this as well. Users need to be confident their objects were indeed created successfully; PrtgAPI suppressing an error because it thinks its "probably" fine is not really an ideal outcome, and implementing the necessary logic to "double check" seems like it would be very painful

@lordmilko
Copy link
Owner

@Silex

In regards to your question, if you have a sensor then the best way to get the device is to do Get-Device -Id $sensor.ParentId

@bajayb4u
Copy link
Author

bajayb4u commented May 3, 2022

Hello @lordmilko,

How to ignore only status code 556 and continue with the rest of the script? It appears add-sensor is ignoring "-ErrorAction Continue". In my case I have to set error limits to the sensor once it added. but script errors out at add-sensor with status code 556. Even if we are able to ignore the 556, will get-channel works for that sensor?

@lordmilko
Copy link
Owner

Hi @bajayb4u,

-ErrorAction only applies to non-terminating exceptions (i.e. emitted ErrorRecord objects). To ignore an exception you need to wrap the relevant code in a try/catch block

try
{
    # do stuff
}
catch
{
    Write-Host $_.Exception.Message
}

lordmilko added a commit that referenced this issue May 8, 2022
…vice.htm fails witth HTTP status 556 as a result of the PRTG server being under high load. As this redirect is simply a byproduct of adding a sensor (meaning the sensor was in fact successfully created) the failure on this redirect can be safely ignored (For #280)
@lordmilko
Copy link
Owner

Hi all,

Please be advised I have been able to reproduce this issue and have released a new pre-release build of PrtgAPI which should hopefully include a fix for this issue

Can you please follow the manual installation instructions and advise whether the issue is now resolved? If you've successfully followed the manual installation instructions, Get-PrtgClient -Diagnostic should show 0.9.18-preview.4

@lordmilko lordmilko added bug Issues that have been confirmed to be bugs in PrtgAPI and will be fixed in a future version and removed alleged-bug Bugs raised by people helpfully using the Bug Report template! labels May 8, 2022
@bajayb4u
Copy link
Author

Hi @lordmilko,

I have tested the 0.9.18-preview.4. It appears status code 556 is no longer reproducible but getting timeout error (which may not related to the status code 556) when adding the sensors. At the end prtgapi is adding the snmp sensors but adding it TWICE and failing to get the channel to further apply sensor limits. Here is the script and error I am getting. Is there any workaorund to fix these issues?

$SatSensor = "snmpcpu"
$getDevice = Get-Device -Id $addDeviceId
$sensorparams = $getDevice | New-SensorParameters -RawType $SatSensor
$cpusensor = $getDevice | Add-Sensor $sensorparams
$cpuchannel = Get-Channel -Sensor $cpusensor
$cputotal = $cpuchannel | Where {$_.Name -eq "Total"}
Set-ChannelProperty -Channel $cputotal -Property UpperErrorLimit -Value 90
Set-ChannelProperty -Channel $cputotal -Property UpperWarningLimit -Value 75

Adding Sensor: snmpcpu
WARNING: 'Add-Sensor' timed out: The server timed out while executing request. Retries remaining: 6

following error returned by "Get-Channel -Sensor $cpusensor" cmd

Error message: Cannot convert 'System.Object[]' to the type 'PrtgAPI.Sensor' required by parameter 'Sensor'. Specified method is not supported. 
PositionMessage: At line: char:
+ ...                  $cpuchannel = Get-Channel -Sensor $cpusensor

@lordmilko
Copy link
Owner

Hi @bajayb4u,

You should not be doing

$cpuchannel = Get-Channel -Sensor $cpusensor

If $cpusensor contains multiple objects you'll get the error you got above. This is an error the PowerShell Engine throws due to how you've written your script, as opposed to being anything to do with PrtgAPI specifically.

The recommended way of using the Get-Channel cmdlet - and indeed any cmdlet in PowerShell, is as so

$cpusensor | Get-Channel

By piping the $cpusensor variable, the PowerShell engine will enumerate through each object in the variable, assigning each one to the -Sensor parameter one at a time

I can't see any reason why the sensor would be created twice; are you potentially able to test with another sensor type such as HTTP and see if it is added twice as well?

@bajayb4u
Copy link
Author

Hi @lordmilko, This issue has been resolved after we have upgraded our PRTG version. But it is only working with 0.9.18-preview.4 version. Thanks for your help,

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Issues that have been confirmed to be bugs in PrtgAPI and will be fixed in a future version
Projects
None yet
Development

No branches or pull requests

4 participants