Environment
self-hosted (https://develop.sentry.dev/self-hosted/)
What are you trying to accomplish?
Our Setup
-
Sentry Self Hosted running on Docker.
-
An additional Symbolicator instance running in Docker that acts as our Symbol Server.
-
An NGINX proxy that sits in front of Symbolicator, and also statically serves the symbol files. A host volume is mounted to a local path that contains the symbols.
All of these are running on the same machine.
As a part of our CI/CD, we use Symsorter to generate the symbols, and these are then copied over to the path that is mounted in the Docker container running NGINX.
Our project is Unreal Engine based, and we're using 0.15.1 Sentry Unreal Plugin.
This works most of the time
The Problem
Occasionally, Sentry will start requesting files that don't exist. When this happens, it causes Symbolicator to block our local Symbol Proxy / Cache, and then legitimate crashes fail to Symbolicate with errors like this:
2024-03-06T21:28:44.296783833Z INFO stackwalk_minidump:fetch_object_file:fetch_file: symbolicator_service::download: Blocking host due to too many download failures host=REDACTED block_time=1day file_id=HTTP source 'e026b4be-65c9-4e31-b473-898c714801ae' location '4d/43cab87cb573b14b1f37ffeaf274731/debuginf_' file_id=HTTP source 'e026b4be-65c9-4e31-b473-898c714801ae' location '4d/43cab87cb573b14b1f37ffeaf274731/debuginf_'
It looks like at least some of these files are redundant requests that Sentry makes, where it'll request something like executabl_ and then immediately request executable.
I don't think that's the main issue, however. My assumption is that when engineers are testing builds they've made locally, if those builds happen to cause issues that get reported to sentry, that causes problems. In this case, symbols never would have been uploaded (we have automatic symbol uploading disabled).
How are you getting stuck?
I think one of the following would help us, if my assumptions above are correct:
-
A way to block Sentry from sending reports for local builds. I think this is something I can work out on my end, because our CI/CD system is already injecting some additional version information into the builds, and I can check for the existence of that.
-
A way to tell Sentry or Symbolicator never to block a particular server. I'd rather have it always query and fail.
As far as confirming my assumptions, it would be nice if there was an easy way to trace back what Symbolication request ultimately triggered the requests that caused the server to get blocked. Maybe this is already possible in the logs, and I just haven't pieced it together yet.
Where in the product are you?
Issues
Link
No response
DSN
No response
Version
24.2.0
Environment
self-hosted (https://develop.sentry.dev/self-hosted/)
What are you trying to accomplish?
Our Setup
Sentry Self Hosted running on Docker.
An additional Symbolicator instance running in Docker that acts as our Symbol Server.
An NGINX proxy that sits in front of Symbolicator, and also statically serves the symbol files. A host volume is mounted to a local path that contains the symbols.
All of these are running on the same machine.
As a part of our CI/CD, we use Symsorter to generate the symbols, and these are then copied over to the path that is mounted in the Docker container running NGINX.
Our project is Unreal Engine based, and we're using 0.15.1 Sentry Unreal Plugin.
This works most of the time
The Problem
Occasionally, Sentry will start requesting files that don't exist. When this happens, it causes Symbolicator to block our local Symbol Proxy / Cache, and then legitimate crashes fail to Symbolicate with errors like this:
2024-03-06T21:28:44.296783833Z INFO stackwalk_minidump:fetch_object_file:fetch_file: symbolicator_service::download: Blocking host due to too many download failures host=REDACTED block_time=1day file_id=HTTP source 'e026b4be-65c9-4e31-b473-898c714801ae' location '4d/43cab87cb573b14b1f37ffeaf274731/debuginf_' file_id=HTTP source 'e026b4be-65c9-4e31-b473-898c714801ae' location '4d/43cab87cb573b14b1f37ffeaf274731/debuginf_'It looks like at least some of these files are redundant requests that Sentry makes, where it'll request something like
executabl_and then immediately requestexecutable.I don't think that's the main issue, however. My assumption is that when engineers are testing builds they've made locally, if those builds happen to cause issues that get reported to sentry, that causes problems. In this case, symbols never would have been uploaded (we have automatic symbol uploading disabled).
How are you getting stuck?
I think one of the following would help us, if my assumptions above are correct:
A way to block Sentry from sending reports for local builds. I think this is something I can work out on my end, because our CI/CD system is already injecting some additional version information into the builds, and I can check for the existence of that.
A way to tell Sentry or Symbolicator never to block a particular server. I'd rather have it always query and fail.
As far as confirming my assumptions, it would be nice if there was an easy way to trace back what Symbolication request ultimately triggered the requests that caused the server to get blocked. Maybe this is already possible in the logs, and I just haven't pieced it together yet.
Where in the product are you?
Issues
Link
No response
DSN
No response
Version
24.2.0