[BUG] API len(cb.select(Sensor)) causes error when >1000 sensors

[atchutameka]

I am getting errors using select(Sensor) API, requiring paging for >1000

• OS: Not applicable
• Carbon Black Product: [Carbon Black EDR]
• Python Version: 3.8

Describe the bug
I use cb.select(Sensor) to acquire a count of sensors using the API, I get an error if there are >1000 sensors. I would like to use released code rather than patching the SDK. Can you please provide a release that has SensorPaginatedQuery enabled? Thanks

why is SensorPaginatedQuery disabled here

Steps to Reproduce
Steps to reproduce the behavior (Provide a log message if relevant):

1. use the API len(cb.select(Sensor))
2. See error "Unpaged request has #### results, exceeds limit of 1000. Please consider using paging."

Expected behavior
To get the count, it should use the paging ({"start":0,"end":0}) to get the count.

Additional context
Applying the following change to the models.py fixes the issue. Is there a release that has paging enabled for Sensor query?

% diff -u response/models.py.orig response/models.py
--- response/models.py.orig	2021-12-13 09:18:32.000000000 -0600
+++ response/models.py	2021-12-13 09:18:51.000000000 -0600
@@ -658,7 +658,7 @@
         #     return SensorPaginatedQuery(cls, cb)
         # else:
         #     return SensorQuery(cls, cb)
-        return SensorQuery(cls, cb)
+        return SensorPaginatedQuery(cls, cb)
 
     @property
     def group(self):

[cforbes-tw]

See also #67 the reason for the comment.

[mlodic]

got the same error.

[sevdog]

Note that this error is only raised by CB EDR Server 7.6.0, previous versions does not have this issue.

This change was not documented in the changelog.

[kebringer-cb]

v1.7.6 has been released with this fix.
Master branch and pypi: https://pypi.org/project/cbapi/1.7.6/
Any problems, please let us know.
Thanks for your contribution to get this fixed.

[OevreFlataeker]

While v1.7.6 fixes the paging problem, another problem emerged which is quite severe for us. Retrieving the list of all sensors returns a different set on every invocation. Running the following script multiple times in a row produces a different set of unique hostnames. This problem only appeared after upgrading to v1.7.6 but downgrading to an earlier version is no longer an option due to the paging error that v1.7.6 fixed. The problem only appears when using cbapi. Querying the API endpoint directly delivers the correct amount of hosts every time

(All following hostnames removed on purpose)

(python_virtualenv) scriptbot@xxx:~/CB/CarbonBlackExporter> cat debug2.py
from cbapi.response.models import Sensor
from cbapi.response import CbEnterpriseResponseAPI

if __name__ == '__main__':
    carbonblack = CbEnterpriseResponseAPI(profile='default')
    cb_sensors = carbonblack.select(Sensor)

    for cb_sensor in cb_sensors:
       print(cb_sensor.hostname)


// This returns a different set of unique hostnames every time, albeit the TOTAL amount of hostnames stays constant!

(python_virtualenv) scriptbot@xxx:~/CB/CarbonBlackExporter> vi debug2.py
(python_virtualenv) scriptbot@xxx:~/CB/CarbonBlackExporter> python3 debug2.py | sort | uniq| wc -l
3161
(python_virtualenv) scriptbot@xxx:~/CB/CarbonBlackExporter> python3 debug2.py | sort | uniq| wc -l
3156
(python_virtualenv) scriptbot@xxx:~/CB/CarbonBlackExporter> python3 debug2.py | sort | uniq| wc -l
2992
(python_virtualenv) scriptbot@xxx:~/CB/CarbonBlackExporter> python3 debug2.py | sort | uniq| wc -l
3119

// This attempt produces the correct amount of 3259 hostnames

(python_virtualenv) scriptbot@xxx:~/CB/CarbonBlackExporter> curl https://xxx:8443/api/v1/sensor?start=1&rows=5000 -H "X-Auth-Token: mysecrettoken" | grep computer_name | sort | uniq | wc -l
3259
(python_virtualenv) scriptbot@xxx:~/CB/CarbonBlackExporter> curl https://xxx:8443/api/v1/sensor?start=1&rows=5000 -H "X-Auth-Token: mysecrettoken " | grep computer_name | sort | uniq | wc -l
3259
(python_virtualenv) scriptbot@xxx:~/CB/CarbonBlackExporter> curl https://xxx:8443/api/v1/sensor?start=1&rows=5000 -H "X-Auth-Token: mysecrettoken " | grep computer_name | sort | uniq | wc -l
3259
(python_virtualenv) scriptbot@xxx:~/CB/CarbonBlackExporter> curl https://xxx:8443/api/v1/sensor?start=1&rows=5000 -H "X-Auth-Token: mysecrettoken " | grep computer_name | sort | uniq | wc -l
3259

[atchutameka]

@OevreFlataeker I believe the issue you are running into is due to #67 pagination is using /api/v2/sensor I believe that still needs to be addressed in #67

[OevreFlataeker]

Thanks @atchutameka - I second the assumption it has to do with the new paging handling. So we're out of luck for the time being until #67 is fixed as well?

[zacharyestep]

Thanks @atchutameka - I second the assumption it has to do with the new paging handling. So we're out of luck for the time being until #67 is fixed as well?

Should be fixed in the next release

[kebringer-cb]

v1.7.7 has been released with Zach's update to set the sort order.