ActiveAlertQuery("Shared Domain").fetch_as_element() never stops

[shibumi]

Hi,
I have encountered the following issue with:

• fp-ngfw-smc-python version: 0.7.0-b23
• python: 3.8.3

When calling smc_monitoring.monitors.alerts.ActiveAlertQuery("Shared Domain").fetch_as_element() I will get a generator
over the current active alerts as expected, but the generator will never stop.

Corresponding debug output looks like this:

{'fetch': {},
 'format': {'field_format': 'id',
            'resolving': {'senders': True},
            'type': 'texts'},
 'query': {'definition': 'ACTIVE_ALERTS', 'target': 'Shared Domain'}}
Query returned 31 records.

Note the last line that says:"Query returned 31 records" this is equal to the number of alerts I retrieve, but the generator will stuck in an endless loop after running over all alerts. I tried to debug this, but the debugger refuses to jump into the corresponding thread.

Is this an issue with python 3.8.3? I saw, that you only support 3.6..

[shibumi]

Ok,
I have tested this issue with docker container python:3.6-buster and I receive the same error. Looks like an issue with our SMC.
Do you have any idea how I can further debug this?

[shibumi]

Another addition.. when I give the query enough time. I see this:

{'fetch': {},
 'format': {'field_format': 'id',
            'resolving': {'senders': True},
            'type': 'texts'},
 'query': {'definition': 'ACTIVE_ALERTS', 'target': 'Shared Domain'}}
Query returned 31 records.
Query returned 0 records.
Caught exception in receive: <class 'TimeoutError'> -> [Errno 110] Connection timed out
Resetting dropped connection: 10.88.5.5
Incremented Retry for (url='/6.5/logout'): Retry(total=4, connect=4, read=5, redirect=None, status=None)
Retrying (Retry(total=4, connect=4, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc44b69f940>: Failed to establish a new connection: [Errno 101] Network is unreachable')': /6.5/logout
....

especially the part with "Query returned 0 records" is interesting. Did the generator get invoked again?

My code looks like this:

  active_alerts = alerts.ActiveAlertQuery(config['smc']['domain']).fetch_as_element()
  for alert in active_alerts:
      if alert.source == None:
          continue
  .... do more...

with config['smc']['domain'] = "Shared Domain"

[shibumi]

Small addition. Same happens when I use API version 6.6 and 6.7. The only difference is that the connection is more stable and instead of a timeout I just see:

"Query returned 0 records" over and over again..

[shibumi]

ping @gabstopper :)

[shibumi]

Interesting observation:

query.fetch_batch() exits after a few minute and the code after it continues.
query.fetch_as_element() does not exit and I am further stuck in a loop

0.7.0-b23
Starting new HTTP connection (1): 10.88.5.5:8082
http://10.88.5.5:8082 "GET /api HTTP/1.1" 200 None
Starting new HTTP connection (1): 10.88.5.5:8082
http://10.88.5.5:8082 "POST /6.6/login HTTP/1.1" 200 0
Using SMC API version: 6.6
http://10.88.5.5:8082 "GET /6.6/api HTTP/1.1" 200 None
Loaded entry points with obtained session.
http://10.88.5.5:8082 "GET /6.6/system/current_user HTTP/1.1" 500 0
Login succeeded for admin: 7647560106141 in domain: Shared Domain, session: JSESSIONID=C7A5BB70621D9A45BD9FCF02E473EFC2
<generator object Query.fetch_batch at 0x6f4960e92820>
{'fetch': {'quantity': 0},
 'format': {'field_ids': [1, 602, 14, 4, 7, 8, 27, 11, 9, 10, 1000, 20000],
            'type': 'detailed'},
 'query': {}}
Event loop terminating.
Closed web socket connection normally.
{'fetch': {},
 'format': {'field_format': 'pretty',
            'resolving': {'senders': True},
            'type': 'texts'},
 'query': {'definition': 'ACTIVE_ALERTS',
           'filter': {'left': {'id': 7, 'type': 'field'},
                      'right': [],
                      'type': 'in'},
           'target': 'Shared Domain'}}
Query returned 0 records.
Event loop terminating.
Closed web socket connection normally.
(avency-alertctl) chris motoko ~/avency/avency-alertctl 12:35:28 7d44d2e1 master  
❯  cd /home/chris/avency/avency-alertctl ; env /home/chris/.virtualenvs/avency-alertctl/bin/python /home/chris/.vscode-oss/extensions/ms-python.python-2020.7.96456/pythonFiles/lib/python/debugpy/launcher 35295 -- /home/chris/avency/avency-alertctl/avency-alertctl.py -d -f prod.yml 
0.7.0-b23
Starting new HTTP connection (1): 10.88.5.5:8082
http://10.88.5.5:8082 "GET /api HTTP/1.1" 200 None
Starting new HTTP connection (1): 10.88.5.5:8082
http://10.88.5.5:8082 "POST /6.6/login HTTP/1.1" 200 0
Using SMC API version: 6.6
http://10.88.5.5:8082 "GET /6.6/api HTTP/1.1" 200 None
Loaded entry points with obtained session.
http://10.88.5.5:8082 "GET /6.6/system/current_user HTTP/1.1" 500 0
Login succeeded for admin: 7260571189649 in domain: Shared Domain, session: JSESSIONID=989E2C279E64547880809F0BA9188E53
{'fetch': {},
 'format': {'field_format': 'id',
            'resolving': {'senders': True},
            'type': 'texts'},
 'query': {'definition': 'ACTIVE_ALERTS',
           'filter': {'left': {'id': 7, 'type': 'field'},
                      'right': [],
                      'type': 'in'},
           'target': 'Shared Domain'}}
Query returned 0 records.
Query returned 0 records.
Query returned 0 records.
Query returned 0 records.
Query returned 0 records.
Query returned 0 records.
Query returned 0 records.
Query returned 0 records.
Query returned 0 records.
Query returned 0 records.
Query returned 0 records.
Query returned 0 records.
Query returned 0 records.
Query returned 0 records.
Query returned 0 records.
Query returned 0 records.
Query returned 0 records.

ping @lmartinson

[ad1rie1]

I have the same behaviour with VPNSA when I have 0 results. A ticket is open on Forcepoint support portal

[EtienneMILON]

Hello,

I have solved this issue with the parameter "max_recv=1".

query = VPNSAQuery(firewall_name)
for sa in query.fetch_as_element(max_recv=1):
     print(sa)

Does it work for you?

[shibumi]

@EtienneMILON I need to try this out, thanks a lot!

[ad1rie1]

it's working but only return one element

[EtienneMILON]

When you say one element it is one SA?
Do you have more SA in the VPN SA of the SMC GUI client?

[ad1rie1]

hello, see : #12 :)

[ggrimaux]

Hello,

@ad1rie1 you should not wait for 1.
Let's have following example:
In your case when checking VPNSAs monitoring you usually have 5 entries. So the max_recv should be at least 5 or a little more just to be sure.

BR,
/Greg.

[ad1rie1]

if set max_recv is the number max of SA (or log entry) returned.
if max_recv is set to 5 and you have 6 SA, if you see the returned result, you only see 5 Result

i dont remember if you set max_recv = 7 with 6 SA block or nor sorry

If you remplace you wsocket.py by the MR one, you have just to do :
query = VPNSAQuery("PACS-SDWAN", query_timeout=10)
tab = query.fetch_raw(query_timeout=10)

after 10 sec of work it stop and return the result

[ggrimaux]

I did try it with max_recv=10 with 6 VPN SAs.

Yes, what you propose is an enhancement to the current method that could be done.

[EtienneMILON]

Hello,

I don't agree with your explanation of "max_recv".
If I execute this:

query = VPNSAQuery(firewall_name)
for sa in query.fetch_as_element(max_recv=1):
     print(sa)

I don't get only 1 SA but all the SA corresponding to "firewall_name".
As I understand, it doesn't limit the number of results but the number of "loops" done.

However, the timeout is a good improvement!

Etienne

[ggrimaux]

Yes sorry

From SMCSocketProtocol

        :param int max_recv: for queries that are not 'live', set
            this to supply a max number of receive iterations.

It is not number but iteration.

[ad1rie1]

Its strange your doc is different from fetchraw one ... test and see :)
From fetchraw:
param int max_recv: max number of socket receive calls before
returning from this query. If you want to wait longer for
results before returning, increase max_iterations (default: 0

[ad1rie1]

I precise, i do not have the same behaviour on 6.5 that return all , and 6.8 that not return all ;)
i have the problem due to the smc upgrade , i have coded the MR because of that