Inconsistent Event Reporting - Unknown error

[wrong1man]

Hello,

I've been using the Facebook Business SDK for Python to report events, but I'm experiencing inconsistent results. Some events are reported successfully, while others fail without a clear explanation from the traceback.

here's the traceback:

Traceback (most recent call last):
  File "/home/azureuser/projects/live/gunicorn/project/app1/models.py", line 648, in initialize
    resp = self.send_meta_event(client_ip, client_user_agent, fbc, fbp)
  File "/home/azureuser/projects/live/gunicorn/project/app1/models.py", line 673, in send_meta_event
    resp= meta_api_send_event(
  File "/home/azureuser/projects/live/gunicorn/project/app1/meta_api.py", line 87, in meta_api_send_event
    event_response = event_request.execute()
  File "/home/azureuser/projects/live/gunicorn/project/venv/lib/python3.8/site-packages/facebook_business/adobjects/serverside/event_request.py", line 281, in execute
    response = AdsPixel(self.__pixel_id).create_event(
  File "/home/azureuser/projects/live/gunicorn/project/venv/lib/python3.8/site-packages/facebook_business/adobjects/adspixel.py", line 408, in create_event
    return request.execute()
  File "/home/azureuser/projects/live/gunicorn/project/venv/lib/python3.8/site-packages/facebook_business/api.py", line 668, in execute
    response = self._api.call(
  File "/home/azureuser/projects/live/gunicorn/project/venv/lib/python3.8/site-packages/facebook_business/api.py", line 311, in call
    response = self._session.requests.request(
  File "/home/azureuser/projects/live/gunicorn/project/venv/lib/python3.8/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/azureuser/projects/live/gunicorn/project/venv/lib/python3.8/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/home/azureuser/projects/live/gunicorn/project/venv/lib/python3.8/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/home/azureuser/projects/live/gunicorn/project/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 791, in urlopen
    response = self._make_request(
  File "/home/azureuser/projects/live/gunicorn/project/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 537, in _make_request
    response = conn.getresponse()
  File "/home/azureuser/projects/live/gunicorn/project/venv/lib/python3.8/site-packages/urllib3/connection.py", line 461, in getresponse
    httplib_response = super().getresponse()
  File "/usr/lib/python3.8/http/client.py", line 1348, in getresponse
    response.begin()
  File "/usr/lib/python3.8/http/client.py", line 316, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.8/http/client.py", line 277, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.8/socket.py", line 669, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.8/ssl.py", line 1270, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.8/ssl.py", line 1128, in read
    return self._sslobj.read(len, buffer)
  File "/home/azureuser/projects/live/gunicorn/project/venv/lib/python3.8/site-packages/gunicorn/workers/base.py", line 203, in handle_abort
    sys.exit(1)
SystemExit: 1

and here's the implementation:

def meta_api_send_event(emails, client_ip,user_id,ph,date_of_birth,first_name, client_user_agent, fbc, fbp,timestamp,
               event_name, event_source_url, action_source,quantity=None,product_id=None,currency=None, value=None, description=None):
    '''

    :param emails: list[str]
    :param client_ip: str
    :param client_user_agent: str
    :param product_id: str
    :param quantity: int
    :param timestamp: timezone.now().timestamp()
    :param currency: str
    :param value: float
    :param event_name: str
    :param event_source_url: str
    :param action_source: str
    :return:

    see example usage bellow
    '''
    user_data = UserData(
        emails=emails,
        client_ip_address=client_ip,
        client_user_agent=client_user_agent,
        external_id=user_id,
        # phones=ph if ph else [],
        date_of_birth=date_of_birth,
        first_name=first_name,
        fbc=fbc,
        fbp=fbp,
    )
    if ph != None and ph[0] != "None" and ph[0] != None:
        user_data.phones=ph
    if product_id and quantity and value:
        content = [Content(
            product_id=product_id,
            quantity=quantity,
            item_price=value,
            description=description,
        )]
    else:
        content=None

    custom_data = CustomData(
        contents=content,
        currency=currency,
        value=value,
    )

    event = Event(
        event_name=event_name,
        event_time=int(timestamp),
        user_data=user_data,
        custom_data=custom_data,
        event_source_url=event_source_url,
        action_source=ActionSource[action_source],
    )

    events = [event]

    event_request = EventRequest(
        events=events,
        pixel_id=pixel_id,
    )
    event_response = event_request.execute()
    return event_response

The issue seems to occur inconsistently. Some events pass through without any issues, while others fail. I haven't been able to identify a pattern or specific cause for these failures.

The lack of detailed information in the traceback makes it difficult to diagnose and address the issue. I'm hoping for some guidance or insight into what could be causing these inconsistent results.

Thank you for your assistance!

[stcheng]

Hi @wrong1man are there any errors returned from the API call? what are the errors?
according to the stack trace, it appears that the request call failed on getting the response. was it taking too long time? have you tried to reproduce by sending raw requests without using sdk?

[wrong1man]

hey @stcheng

Thank you for your response. Unfortunately, there is no specific error message returned from the API call. The issue that I am facing is that roughly 31% of around 220 events reported by my backend in the last three days have failed. The error traceback I shared earlier is what I get for these failed

I can manually re-trigger them and they usually go through.

To handle this, I've implemented a try-except block to catch all exceptions and log either the API response or the error traceback to a database. Here's the code snippet for your reference:

        resp=None
        try:
            resp = self.send_meta_event(client_ip, client_user_agent, fbc, fbp)
            received=resp["events_received"] == True # this allways fails - Logging all events
        except:
            trace = traceback.format_exc() if not resp else resp
            Generic_error_log.objects.create(
                function_code="Purchase - send_meta_event",
                traceback = str(trace),
              related_element_id=f"Dinner_booking - {self.id}"
          )
          pass

I've recently updated the code to include more information in case of exceptions (but i don't have one of these yet):

except Exception as e:
#...
      trace=traceback.format_exc() + "\n"+str(e) if not resp else resp
#...

I am currently logging all events to understand the rate of success and failure. Any suggestions on how to further investigate this issue would be highly appreciated.

Would it be beneficial to log the time it takes to run the meta_api_send_event function? I would expect a timeout error to present itself in the traceback, wouldn't it?

I appreciate your suggestion of attempting to reproduce the issue manually by sending raw requests without using the SDK. However, given the sporadic occurrence of the error, it's been challenging to reproduce it consistently even with the SDK. Furthermore, testing outside of the SDK might not necessarily aid in identifying or resolving the root cause of the issue.

My primary goal is to understand why this error is happening and why the process abruptly aborts. If it's a matter of a timeout, for instance, I could potentially adjust the timeout duration or defer the event reporting to a later time.

In this context, any insights or suggestions that could help me further diagnose this problem would be immensely valuable. Thanks again for your help and cooperation.

[wrong1man]

Futher investigation revealed it was indeed a timeout.
I placed the event reporting on a task queue (celery) sometimes requests take over 3 minutes to complete.

I can re-launch the event report, but its a bit worrying to me that it takes so long to report an event, any ideas on what s happening?