Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

No Performance metrics are being collected #60779

Closed
codingmoh opened this issue Nov 29, 2023 · 8 comments
Closed

No Performance metrics are being collected #60779

codingmoh opened this issue Nov 29, 2023 · 8 comments
Labels
Product Area: Performance

Comments

@codingmoh
Copy link

How do you use Sentry?

Sentry Saas (sentry.io)

Version

sentry-sdk[fastapi]==1.37.1

Steps to Reproduce

We're running a fastAPI and want to use performance tracking but unfortunately there are not transactions being collected at all - errors/issues however are being collected - I've tried figuring out what's been missing but couldn't pinpoint the problem - I saw in other discussions that initializing the fastAPI app first may be help resolve the issue - but it didn't, I've also tried using the fastApiIntegration (from sentry_sdk.integrations.fastapi import FastApiIntegration) - but it didn't resolve the problem either.

This what our code looks like right now:

What are we missing?

app = FastAPI()
app.include_router(atelier_router)
app.include_router(qr_code_router)

app.add_middleware(
  .....
  ....
)

sentry_sdk.init(
    dsn="https://xxxxxxx@xxxxxxxxxxxx.ingest.sentry.io/xxxxxxxxx",
    # Enable performance monitoring
    enable_tracing=True,
    traces_sample_rate=1.0,
    integrations=[FastApiIntegration()],
    profiles_sample_rate=1.0,
)

Thanks!

Expected Result

Performance and transaction metrics are being collected

Actual Result

image
@sentrivana
Copy link
Contributor

sentrivana commented Nov 29, 2023 •
edited
Loading

Hey @codingmoh, thanks for raising this.

The sentry_sdk.init should always happen as early as possible, in this case before your app = FastAPI(). Using FastApiIntegration directly shouldn't make a difference since this should be auto-enabled (and if it's collecting errors, then the integration is active).

Could you move the init to the top, add debug=True to it, call an endpoint, and see what it says? It should output a debug log for each transaction that has been either sent or dropped.

@codingmoh
Copy link
Author

codingmoh commented Nov 29, 2023 •
edited
Loading

Thanks so much for your very quick response and support!

So, I moved sentry init to the top - first thing after the imports

and this is how the log output with debug=True looks like - I'm not able to pinpoint the reason, are you?

I have also provoked an error to see if it would make any difference - the error is being collected but no performance metric unfortunately

[sentry] DEBUG: Setting up integrations (with default = True)
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.boto3.Boto3Integration: botocore is not i
nstalled
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.bottle.BottleIntegration: Bottle not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.celery.CeleryIntegration: Celery not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.django.DjangoIntegration: Django not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.falcon.FalconIntegration: Falcon not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.flask.FlaskIntegration: Flask is not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.pyramid.PyramidIntegration: Pyramid not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.sanic.SanicIntegration: Sanic not installed
 [sentry] DEBUG: Did not import default integration sentry_sdk.integrations.tornado.TornadoIntegration: Tornado not installed
 [sentry] DEBUG: Setting up previously not enabled integration fastapi
 [sentry] DEBUG: Setting up previously not enabled integration argv
 [sentry] DEBUG: Setting up previously not enabled integration atexit
 [sentry] DEBUG: Setting up previously not enabled integration dedupe
 [sentry] DEBUG: Setting up previously not enabled integration excepthook
 [sentry] DEBUG: Setting up previously not enabled integration logging
 [sentry] DEBUG: Setting up previously not enabled integration modules
 [sentry] DEBUG: Setting up previously not enabled integration stdlib
 [sentry] DEBUG: Setting up previously not enabled integration threading
 [sentry] DEBUG: Setting up previously not enabled integration aiohttp
 [sentry] DEBUG: Setting up previously not enabled integration httpx
 [sentry] DEBUG: Setting up previously not enabled integration redis
 [sentry] DEBUG: Setting up previously not enabled integration rq
 [sentry] DEBUG: Setting up previously not enabled integration sqlalchemy
 [sentry] DEBUG: Setting up previously not enabled integration starlette
 [sentry] DEBUG: Enabling integration fastapi
 [sentry] DEBUG: Enabling integration argv
 [sentry] DEBUG: Enabling integration atexit
 [sentry] DEBUG: Enabling integration dedupe
 [sentry] DEBUG: Enabling integration excepthook
 [sentry] DEBUG: Enabling integration logging
 [sentry] DEBUG: Enabling integration modules
 [sentry] DEBUG: Enabling integration stdlib
 [sentry] DEBUG: Enabling integration threading
 [sentry] DEBUG: Enabling integration aiohttp
 [sentry] DEBUG: Enabling integration httpx
 [sentry] DEBUG: Enabling integration redis
 [sentry] DEBUG: Enabling integration rq
 [sentry] DEBUG: Enabling integration sqlalchemy
 [sentry] DEBUG: Enabling integration starlette
 [sentry] DEBUG: Setting SDK name to 'sentry.python.fastapi'
 [sentry] DEBUG: [Profiling] Setting up profiler in thread mode
INFO:     Started server process [6094]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
 [sentry] DEBUG: [ASGI] Created transaction (continuing trace): <Transaction(name='http://127.0.0.1:8000/.../messages', op='http.server', trace_id='11cfaecee5b24c53bf4757d204eb4e95', span_id='a5c34de21bca5d1e', parent_span_id=None, sampled=None, source='url')>
DEBUG:sentry_sdk.errors:[ASGI] Created transaction (continuing trace): <Transaction(name='http://127.0.0.1:8000/.../messages', op='http.server', trace_id='11cfaecee5b24c53bf4757d204eb4e95', span_id='a5c34de21bca5d1e', parent_span_id=None, sampled=None, source='url')>
 [sentry] DEBUG: [ASGI] Set transaction name and source on transaction: 'http://127.0.0.1:8000/.../messages' / 'url'
DEBUG:sentry_sdk.errors:[ASGI] Set transaction name and source on transaction: 'http://127.0.0.1:8000/.../messages' / 'url'
 [sentry] DEBUG: [Tracing] Starting <http.server> transaction <http://127.0.0.1:8000/.../messages>
DEBUG:sentry_sdk.errors:[Tracing] Starting <http.server> transaction <http://127.0.0.1:8000/.../messages>
 [sentry] DEBUG: [Profiling] Initializing profile
DEBUG:sentry_sdk.errors:[Profiling] Initializing profile
 [sentry] DEBUG: [Profiling] Starting profile
DEBUG:sentry_sdk.errors:[Profiling] Starting profile
 [sentry] DEBUG: [ASGI] Started transaction: <Transaction(name='http://127.0.0.1:8000/.../messages', op='http.server', trace_id='11cfaecee5b24c53bf4757d204eb4e95', span_id='a5c34de21bca5d1e', parent_span_id=None, sampled=True, source='url')>
DEBUG:sentry_sdk.errors:[ASGI] Started transaction: <Transaction(name='http://127.0.0.1:8000/.../messages', op='http.server', trace_id='11cfaecee5b24c53bf4757d204eb4e95', span_id='a5c34de21bca5d1e', parent_span_id=None, sampled=True, source='url')>
 [sentry] DEBUG: [FastAPI] Set transaction name and source on scope: /cube/{cube_id}/../{session_id}/messages / route
DEBUG:sentry_sdk.errors:[FastAPI] Set transaction name and source on scope: /cube/{cube_id}/../{session_id}/messages / route
content="....." id='bc65c901-4c08-43e2-957d-02ac83729e7c' role='assistant' created_at=datetime.datetime(2023, 11, 29, 12, 19, 55, 389870)
INFO:     127.0.0.1:50098 - "GET /.../messages HTTP/1.1" 200 OK
 [sentry] DEBUG: [Profiling] Stopping profile
DEBUG:sentry_sdk.errors:[Profiling] Stopping profile
 [sentry] DEBUG: Sending envelope [envelope with 2 items (profile, transaction)] project:4506262642163712 host:o4506262642032640.ingest.sentry.io
DEBUG:sentry_sdk.errors:Sending envelope [envelope with 2 items (profile, transaction)] project:4506262642163712 host:o4506262642032640.ingest.sentry.io
 [sentry] DEBUG: Sending envelope [envelope with 1 items (default)] project:4506262642163712 host:o4506262642032640.ingest.sentry.io
DEBUG:sentry_sdk.errors:Sending envelope [envelope with 1 items (default)] project:4506262642163712 host:o4506262642032640.ingest.sentry.io
 [sentry] DEBUG: [ASGI] Created transaction (continuing trace): <Transaction(name='http://127.0.0.1:8000/.../messages', op='http.server', trace_id='f5ffee66d46f41ec833756db0195c831', span_id='89db8c0b2456fcc1', parent_span_id=None, sampled=None, source='url')>
DEBUG:sentry_sdk.errors:[ASGI] Created transaction (continuing trace): <Transaction(name='http://127.0.0.1:8000/.../messages', op='http.server', trace_id='f5ffee66d46f41ec833756db0195c831', span_id='89db8c0b2456fcc1', parent_span_id=None, sampled=None, source='url')>
 [sentry] DEBUG: [ASGI] Set transaction name and source on transaction: 'http://127.0.0.1:8000/.../messages' / 'url'
DEBUG:sentry_sdk.errors:[ASGI] Set transaction name and source on transaction: 'http://127.0.0.1:8000/.../messages' / 'url'
 [sentry] DEBUG: [Tracing] Starting <http.server> transaction <http://127.0.0.1:8000/.../messages>
DEBUG:sentry_sdk.errors:[Tracing] Starting <http.server> transaction <http://127.0.0.1:8000/.../messages>
 [sentry] DEBUG: [Profiling] Initializing profile
DEBUG:sentry_sdk.errors:[Profiling] Initializing profile
 [sentry] DEBUG: [Profiling] Starting profile
DEBUG:sentry_sdk.errors:[Profiling] Starting profile
 [sentry] DEBUG: [ASGI] Started transaction: <Transaction(name='http://127.0.0.1:8000/.../messages', op='http.server', trace_id='f5ffee66d46f41ec833756db0195c831', span_id='89db8c0b2456fcc1', parent_span_id=None, sampled=True, source='url')>
DEBUG:sentry_sdk.errors:[ASGI] Started transaction: <Transaction(name='http://127.0.0.1:8000/.../messages', op='http.server', trace_id='f5ffee66d46f41ec833756db0195c831', span_id='89db8c0b2456fcc1', parent_span_id=None, sampled=True, source='url')>
 [sentry] DEBUG: [FastAPI] Set transaction name and source on scope: /cube/{cube_id}/../{session_id}/messages / route
DEBUG:sentry_sdk.errors:[FastAPI] Set transaction name and source on scope: /cube/{cube_id}/../{session_id}/messages / route
 [sentry] DEBUG: [Tracing] Adding `sentry-trace` header f5ffee66d46f41ec833756db0195c831-861a25d3f7b9b9bb-1 to outgoing request to .../api/...
DEBUG:sentry_sdk.errors:[Tracing] Adding `sentry-trace` header f5ffee66d46f41ec833756db0195c831-861a25d3f7b9b9bb-1 to outgoing request to .../api/...
 [sentry] DEBUG: [Tracing] Adding `baggage` header sentry-trace_id=f5ffee66d46f41ec833756db0195c831,sentry-environment=production,sentry-release=8e8aed71dae9c37870306bdaa76fc0a45c3ab88b,sentry-public_key=0479f036cadfd944447ff454b7feb6e2,sentry-transaction=/cube/%7Bcube_id%7D/../%7Bsession_id%7D/messages,sentry-sample_rate=1.0,sentry-sampled=true to outgoing request to .../api/...
DEBUG:sentry_sdk.errors:[Tracing] Adding `baggage` header sentry-trace_id=f5ffee66d46f41ec833756db0195c831,sentry-environment=production,sentry-release=8e8aed71dae9c37870306bdaa76fc0a45c3ab88b,sentry-public_key=0479f036cadfd944447ff454b7feb6e2,sentry-transaction=/cube/%7Bcube_id%7D/../%7Bsession_id%7D/messages,sentry-sample_rate=1.0,sentry-sampled=true to outgoing request to .../api/...
INFO:httpx:HTTP Request: POST .../api/.. "HTTP/1.1 200 OK"
content='you goof' id='673ba1d06-8a6c-3a7a-99c0-37e86b6d0d7f' role='assistant' created_at=datetime.datetime(2023, 11, 27, 22, 0, 4, 230480)
INFO:     127.0.0.1:50748 - "POST /.../messages HTTP/1.1" 500 Internal Server Error
 [sentry] DEBUG: [Profiling] Stopping profile
DEBUG:sentry_sdk.errors:[Profiling] Stopping profile
 [sentry] DEBUG: Sending envelope [envelope with 1 items (error)] project:4506262642163712 host:o4506262642032640.ingest.sentry.io
DEBUG:sentry_sdk.errors:Sending envelope [envelope with 1 items (error)] project:4506262642163712 host:o4506262642032640.ingest.sentry.io
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File ".../../backend/env/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 408, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../../backend/env/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../../backend/env/lib/python3.11/site-packages/fastapi/applications.py", line 289, in __call__
    await super().__call__(scope, receive, send)
  File ".../../backend/env/lib/python3.11/site-packages/sentry_sdk/integrations/starlette.py", line 364, in _sentry_patched_asgi_app
    return await middleware(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../../backend/env/lib/python3.11/site-packages/sentry_sdk/integrations/asgi.py", line 146, in _run_asgi3
    return await self._run_app(scope, receive, send, asgi_version=3)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../../backend/env/lib/python3.11/site-packages/sentry_sdk/integrations/asgi.py", line 241, in _run_app
    raise exc from None
  File ".../../backend/env/lib/python3.11/site-packages/sentry_sdk/integrations/asgi.py", line 234, in _run_app
    return await self.app(
           ^^^^^^^^^^^^^^^
  File ".../../backend/env/lib/python3.11/site-packages/starlette/applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File ".../../backend/env/lib/python3.11/site-packages/sentry_sdk/integrations/starlette.py", line 157, in _create_span_call
    return await old_call(app, scope, new_receive, new_send, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../../backend/env/lib/python3.11/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File ".../../backend/env/lib/python3.11/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File ".../../backend/env/lib/python3.11/site-packages/sentry_sdk/integrations/starlette.py", line 157, in _create_span_call
    return await old_call(app, scope, new_receive, new_send, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../../backend/env/lib/python3.11/site-packages/starlette/middleware/cors.py", line 83, in __call__
    await self.app(scope, receive, send)
  File ".../../backend/env/lib/python3.11/site-packages/sentry_sdk/integrations/starlette.py", line 256, in _sentry_exceptionmiddleware_call
    await old_call(self, scope, receive, send)
  File ".../../backend/env/lib/python3.11/site-packages/sentry_sdk/integrations/starlette.py", line 157, in _create_span_call
    return await old_call(app, scope, new_receive, new_send, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../../backend/env/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File ".../../backend/env/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File ".../../backend/env/lib/python3.11/site-packages/sentry_sdk/integrations/starlette.py", line 157, in _create_span_call
    return await old_call(app, scope, new_receive, new_send, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../../backend/env/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in __call__
    raise e
  File ".../../backend/env/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in __call__
    await self.app(scope, receive, send)
  File ".../../backend/env/lib/python3.11/site-packages/starlette/routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File ".../../backend/env/lib/python3.11/site-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File ".../../backend/env/lib/python3.11/site-packages/starlette/routing.py", line 66, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File ".../../backend/env/lib/python3.11/site-packages/sentry_sdk/integrations/fastapi.py", line 136, in _sentry_app
    return await old_app(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../../backend/env/lib/python3.11/site-packages/fastapi/routing.py", line 273, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../../backend/env/lib/python3.11/site-packages/fastapi/routing.py", line 190, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../../backend/app.py", line 100, in post_message
    db.save_message(session, message, "user", session_id, cube_id)
  File ".../../backend/db.py", line 73, in save_message
    db.execute(text(
  File ".../../backend/env/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2308, in execute
    return self._execute_internal(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File ".../../backend/env/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2199, in _execute_internal
    result = conn.execute(
             ^^^^^^^^^^^^^
  File ".../../backend/env/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1416, in execute
    return meth(
           ^^^^^
  File ".../../backend/env/lib/python3.11/site-packages/sqlalchemy/sql/elements.py", line 516, in _execute_on_connection
    return connection._execute_clauseelement(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../../backend/env/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1639, in _execute_clauseelement
    ret = self._execute_context(
          ^^^^^^^^^^^^^^^^^^^^^^
  File ".../../backend/env/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1848, in _execute_context
    return self._exec_single_context(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../../backend/env/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1988, in _exec_single_context
    self._handle_dbapi_exception(
  File ".../../backend/env/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 2343, in _handle_dbapi_exception
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File ".../../backend/env/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1969, in _exec_single_context
    self.dialect.do_execute(
  File ".../../backend/env/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 922, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "messages_pkey"
DETAIL:  Key (id)=(673ba1d06-8a6c-3a7a-99c0-37e86b6d0d7f) already exists.

[SQL: INSERT INTO messages (id, session_id, cube_id, role, content, created_at) 
            VALUES (%(id)s, %(session_id)s, %(cube_id)s, %(role)s, %(content)s, %(created_at)s)]
[parameters: {'id': '673ba1d06-8a6c-3a7a-99c0-37e86b6d0d7f', 'session_id': 'xxxx', 'cube_id': 'xxxxx', 'role': 'user', 'content': ''....', 'created_at': datetime.datetime(2023, 11, 27, 22, 0, 4, 230480)}]
(Background on this error at: https://sqlalche.me/e/20/gkpj)
 [sentry] INFO: event processor (<function DedupeIntegration.setup_once.<locals>.processor at 0x10542f600>) dropped event
INFO:sentry_sdk.errors:event processor (<function DedupeIntegration.setup_once.<locals>.processor at 0x10542f600>) dropped event
 [sentry] DEBUG: Sending envelope [envelope with 2 items (profile, transaction)] project:4506262642163712 host:o4506262642032640.ingest.sentry.io
DEBUG:sentry_sdk.errors:Sending envelope [envelope with 2 items (profile, transaction)] project:4506262642163712 host:o4506262642032640.ingest.sentry.io
 [sentry] WARNING: Rate-limited via x-sentry-rate-limits
WARNING:sentry_sdk.errors:Rate-limited via x-sentry-rate-limits
 [sentry] DEBUG: [Monitor] health check negative, downsampling with a factor of 1
DEBUG:sentry_sdk.errors:[Monitor] health check negative, downsampling with a factor of 1
 [sentry] DEBUG: [ASGI] Created transaction (continuing trace): <Transaction(name='http://127.0.0.1:8000/.../messages', op='http.server', trace_id='df141200b4e3422d8670e0184f9350bd', span_id='a7ad4cc6757264d8', parent_span_id=None, sampled=None, source='url')>
DEBUG:sentry_sdk.errors:[ASGI] Created transaction (continuing trace): <Transaction(name='http://127.0.0.1:8000/.../messages', op='http.server', trace_id='df141200b4e3422d8670e0184f9350bd', span_id='a7ad4cc6757264d8', parent_span_id=None, sampled=None, source='url')>
 [sentry] DEBUG: [ASGI] Set transaction name and source on transaction: 'http://127.0.0.1:8000/.../messages' / 'url'
DEBUG:sentry_sdk.errors:[ASGI] Set transaction name and source on transaction: 'http://127.0.0.1:8000/.../messages' / 'url'
 [sentry] DEBUG: [Tracing] Starting <http.server> transaction <http://127.0.0.1:8000/.../messages>
DEBUG:sentry_sdk.errors:[Tracing] Starting <http.server> transaction <http://127.0.0.1:8000/.../messages>
 [sentry] DEBUG: [Profiling] Initializing profile
DEBUG:sentry_sdk.errors:[Profiling] Initializing profile
 [sentry] DEBUG: [Profiling] Starting profile
DEBUG:sentry_sdk.errors:[Profiling] Starting profile
 [sentry] DEBUG: [ASGI] Started transaction: <Transaction(name='http://127.0.0.1:8000/.../messages', op='http.server', trace_id='df141200b4e3422d8670e0184f9350bd', span_id='a7ad4cc6757264d8', parent_span_id=None, sampled=True, source='url')>
DEBUG:sentry_sdk.errors:[ASGI] Started transaction: <Transaction(name='http://127.0.0.1:8000/.../messages', op='http.server', trace_id='df141200b4e3422d8670e0184f9350bd', span_id='a7ad4cc6757264d8', parent_span_id=None, sampled=True, source='url')>
 [sentry] DEBUG: [FastAPI] Set transaction name and source on scope: /cube/{cube_id}/../{session_id}/messages / route
DEBUG:sentry_sdk.errors:[FastAPI] Set transaction name and source on scope: /cube/{cube_id}/../{session_id}/messages / route
 [sentry] DEBUG: [Tracing] Adding `sentry-trace` header df141200b4e3422d8670e0184f9350bd-90589cb80c93c075-1 to outgoing request to .../api/...
DEBUG:sentry_sdk.errors:[Tracing] Adding `sentry-trace` header df141200b4e3422d8670e0184f9350bd-90589cb80c93c075-1 to outgoing request to .../api/...
 [sentry] DEBUG: [Tracing] Adding `baggage` header sentry-trace_id=df141200b4e3422d8670e0184f9350bd,sentry-environment=production,sentry-release=8e8aed71dae9c37870306bdaa76fc0a45c3ab88b,sentry-public_key=0479f036cadfd944447ff454b7feb6e2,sentry-transaction=/cube/%7Bcube_id%7D/../%7Bsession_id%7D/messages,sentry-sample_rate=0.5,sentry-sampled=true to outgoing request to .../api/...
DEBUG:sentry_sdk.errors:[Tracing] Adding `baggage` header sentry-trace_id=df141200b4e3422d8670e0184f9350bd,sentry-environment=production,sentry-release=8e8aed71dae9c37870306bdaa76fc0a45c3ab88b,sentry-public_key=0479f036cadfd944447ff454b7feb6e2,sentry-transaction=/cube/%7Bcube_id%7D/../%7Bsession_id%7D/messages,sentry-sample_rate=0.5,sentry-sampled=true to outgoing request to .../api/...
INFO:httpx:HTTP Request: POST .../api/.. "HTTP/1.1 200 OK"
content='....' id='673ba1d06-8a6c-3a7a-99cc0-37e86b6d0d7f' role='assistant' created_at=datetime.datetime(2023, 11, 27, 22, 0, 4, 230480)
 [sentry] DEBUG: [Tracing] Adding `sentry-trace` header df141200b4e3422d8670e0184f9350bd-a8a4909cae5279ff-1 to outgoing request to .../api/...
DEBUG:sentry_sdk.errors:[Tracing] Adding `sentry-trace` header df141200b4e3422d8670e0184f9350bd-a8a4909cae5279ff-1 to outgoing request to .../api/...
 [sentry] DEBUG: [Tracing] Adding `baggage` header sentry-trace_id=df141200b4e3422d8670e0184f9350bd,sentry-environment=production,sentry-release=8e8aed71dae9c37870306bdaa76fc0a45c3ab88b,sentry-public_key=0479f036cadfd944447ff454b7feb6e2,sentry-transaction=/cube/%7Bcube_id%7D/../%7Bsession_id%7D/messages,sentry-sample_rate=0.5,sentry-sampled=true to outgoing request to .../api/...
DEBUG:sentry_sdk.errors:[Tracing] Adding `baggage` header sentry-trace_id=df141200b4e3422d8670e0184f9350bd,sentry-environment=production,sentry-release=8e8aed71dae9c37870306bdaa76fc0a45c3ab88b,sentry-public_key=0479f036cadfd944447ff454b7feb6e2,sentry-transaction=/cube/%7Bcube_id%7D/../%7Bsession_id%7D/messages,sentry-sample_rate=0.5,sentry-sampled=true to outgoing request to .../api/...
INFO:httpx:HTTP Request: POST .../api/.. "HTTP/1.1 200 OK"
content='....' id='1e2760ee-14e8-4c64-ace9-75793027dc84' role='assistant' created_at=datetime.datetime(2023, 11, 29, 12, 28, 40, 77728)
INFO:     127.0.0.1:50759 - "POST /.../messages HTTP/1.1" 200 OK
 [sentry] DEBUG: [Profiling] Stopping profile
DEBUG:sentry_sdk.errors:[Profiling] Stopping profile
 [sentry] DEBUG: [Monitor] health check negative, downsampling with a factor of 2
DEBUG:sentry_sdk.errors:[Monitor] health check negative, downsampling with a factor of 2
 [sentry] DEBUG: [Monitor] health check negative, downsampling with a factor of 3
DEBUG:sentry_sdk.errors:[Monitor] health check negative, downsampling with a factor of 3
 [sentry] DEBUG: Sending envelope [envelope with 1 items (default)] project:4506262642163712 host:o4506262642032640.ingest.sentry.io
DEBUG:sentry_sdk.errors:Sending envelope [envelope with 1 items (default)] project:4506262642163712 host:o4506262642032640.ingest.sentry.io
 [sentry] DEBUG: [Monitor] health check negative, downsampling with a factor of 4
DEBUG:sentry_sdk.errors:[Monitor] health check negative, downsampling with a factor of 4
 [sentry] DEBUG: [Monitor] health check negative, downsampling with a factor of 5
DEBUG:sentry_sdk.errors:[Monitor] health check negative, downsampling with a factor of 5
 [sentry] DEBUG: [Monitor] health check negative, downsampling with a factor of 6
DEBUG:sentry_sdk.errors:[Monitor] health check negative, downsampling with a factor of 6
 [sentry] DEBUG: [Monitor] health check positive, reverting to normal sampling
DEBUG:sentry_sdk.errors:[Monitor] health check positive, reverting to normal sampling

@sentrivana
Copy link
Contributor

sentrivana commented Nov 29, 2023 •
edited
Loading

Thanks! Looking at the SDK output it all seems fine, the transactions are being sent since I'm seeing these:

[sentry] DEBUG: Sending envelope [envelope with 2 items (profile, transaction)]

So the transactions leave the SDK as expected.

That'd indicate that they are either being dropped sometime later or possibly filtered away in the UI. Is it possible you have some sort of filtering active in Sentry? We've had something similar happen with messages recently.

The transactions could also be getting dropped because of size. I assume you're just using the out-of-the-box transactions from the FastApiIntegration, right? You're not adding anything on top or creating your own transactions/spans? (I don't think you should be hitting any size limits in either case for what it's worth.)

@codingmoh
Copy link
Author

Thanks @sentrivana

That'd indicate that they are either being dropped sometime later or possibly filtered away in the UI. Is it possible you have some sort of filtering active in Sentry? We've had https://github.com/getsentry/sentry-python/issues/2451#issuecomment-1766517275 happen with messages recently.

I've checked - there's no active filter apart from the health check filter enabled, which I have disabled for now to see if it would change anything.

also we're not creating or adding additionals spans or transactions

@sentrivana
Copy link
Contributor

I see, thanks for checking! I'd be surprised if it was the health check filter.

I'll route this to our support folks, they might have more ideas. (TL;DR: the transactions leave the SDK ok, unsure why they're not popping up in Sentry.)

@getsantry
Copy link
Contributor

getsantry bot commented Nov 29, 2023

Assigning to @getsentry/support for routing ⏲️

@sentrivana sentrivana transferred this issue from getsentry/sentry-python Nov 29, 2023
@getsantry
Copy link
Contributor

getsantry bot commented Nov 29, 2023

Routing to @getsentry/product-owners-performance for triage ⏲️

@kerenkhatiwada
Copy link
Member

Hey @codingmoh, this would require taking a look at your organization. If this issue is still occurring, please write in to support@sentry.io with your issue.

@sentrivana sentrivana mentioned this issue Dec 12, 2023
Closed
@github-actions github-actions bot locked and limited conversation to collaborators Dec 21, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Product Area: Performance
Projects
GitHub Issues with 👀
Archived in project
Milestone
No milestone
Development

No branches or pull requests
4 participants
@codingmoh @JLuse @kerenkhatiwada @sentrivana