seaweed empty .dat after restart #3990
Description
Self-Hosted Version
25.9.0
CPU Architecture
x86_64
Docker Version
28.2.2
Docker Compose Version
2.36.2
Machine Specification
- My system meets the minimum system requirements of Sentry
Steps to Reproduce
Hi everyone,
Just a restart after the last update (migration completed).
Expected Result
Keep data post restart.
Actual Result
After restarting my Sentry stack, all event details for issues disappeared.
In my browser's network, I had a 500 on “/api/0/organizations/company/issues/AAAAA/events/recommended/?collapse=fullRelease.”
botocore.exceptions.ClientError: An error occurred (InternalError) when calling the GetObject operation (reached max retries: 3): We encountered an internal error, please try again.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/.venv/lib/python3.13/site-packages/sentry_sdk/serializer.py", line 218, in _serialize_node
return _serialize_node_impl(
obj,
...<4 lines>...
remaining_breadth=remaining_breadth,
)
File "/.venv/lib/python3.13/site-packages/sentry_sdk/serializer.py", line 315, in _serialize_node_impl
obj = dict(obj.items())
File "<frozen _collections_abc>", line 898, in __iter__
File "/usr/src/sentry/src/sentry/db/models/fields/node.py", line 63, in __iter__
return iter(self.data)
^^^^^^^^^
File "/.venv/lib/python3.13/site-packages/django/utils/functional.py", line 47, in __get__
res = instance.__dict__[self.name] = self.func(instance)
~~~~~~~~~^^^^^^^^^^
File "/usr/src/sentry/src/sentry/db/models/fields/node.py", line 92, in data
self.bind_data(nodestore.backend.get(self.id) or {})
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
File "/usr/src/sentry/src/sentry/utils/metrics.py", line 227, in inner
return f(*args, **kwargs)
File "/usr/src/sentry/src/sentry/utils/metrics.py", line 227, in inner
return f(*args, **kwargs)
File "/usr/src/sentry/src/sentry/services/nodestore/base.py", line 150, in get
bytes_data = self._get_bytes(id)
File "/.venv/lib/python3.13/site-packages/sentry_nodestore_s3/backend.py", line 67, in _get_bytes
return self.__read_from_bucket(id)
~~~~~~~~~~~~~~~~~~~~~~~^^^^
File "/.venv/lib/python3.13/site-packages/sentry_nodestore_s3/backend.py", line 96, in __read_from_bucket
obj = self.client.get_object(
Key=self.__get_key_for_id(id),
Bucket=self.bucket_name,
)
File "/.venv/lib/python3.13/site-packages/botocore/client.py", line 565, in _api_call
return self._make_api_call(operation_name, kwargs)
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
File "/.venv/lib/python3.13/site-packages/botocore/client.py", line 1021, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InternalError) when calling the GetObject operation: We encountered an internal error, please try again.
15:09:28 [ERROR] sentry_sdk.errors: Internal error in sentry_sdk
So I turned to seaweedfs (newly added to the stack).
I discovered that at the same time my requests end in 500, I get this message appearing in the console (logs of seaweedfs -v=2):
needle_read.go:46 /tmp/nodestore_34.dat read 0 dataSize 8784 offset 140044520 fileSize 8: EOF
I1007 15:14:32.291285 filechunk_manifest.go:162 request_id:8e1f0a06-d80e-4c7b-aba2-86f4b819d006 read http://127.0.0.1:8080/34,24d316d5d4efe0 failed, err: http://127.0.0.1:8080/34,24d316d5d4efe0?readDeleted=true: 500 Internal Server Error
E1007 15:14:32.295678 filer_server_handlers_read.go:269 request_id:8e1f0a06-d80e-4c7b-aba2-86f4b819d006 failed to stream content /buckets/nodestore/nodestore/a2ac397027acd644310cc8e31ef36a79: read chunk: http://127.0.0.1:8080/34,24d316d5d4efe0?readDeleted=true: 500 Internal Server Error
E1007 15:14:32.312670 common.go:316 ProcessRangeRequest: read chunk: http://127.0.0.1:8080/34,24d316d5d4efe0?readDeleted=true: 500 Internal Server Error
And indeed, when I look at “/tmp/nodestore_34.dat”, the file appears to be empty.
$ ls /tmp
...
-rw-r--r-- 1 root root 8 Oct 7 15:02 nodestore_34.dat
...
The fact that these files were present in /tmp explains why everything disappears after restarting.
The data never comes back, but the new events are present in Sentry, which leads me to believe that there is either a problem with the post-restart restoration or that my data is stored incorrectly (I did not edit the mount points or configurations related to the path).
I don't know if it's related, but when I restart the seaweedfs container, I get these kinds of errors (logs grepped on "volume 34"):
: failed to write to local disk: volume 34 is read only
I1007 15:22:35.081016 store_replicate.go:61 failed to write to local disk: volume 34 is read only
I1007 15:22:35.081038 common.go:124 error JSON response status 500: failed to write to local disk: volume 34 is read only
I1007 15:22:35.081055 common.go:81 response method:POST URL:/34,24a4ca9f9f8d6b with httpStatus:500 and JSON:{"error":"failed to write to local disk: volume 34 is read only"}
W1007 15:22:35.081183 upload_content.go:192 request_id:cf18e83d-7e4f-41c4-baf8-01f597db4ab1 uploading 2 to http://127.0.0.1:8080/34,24a4ca9f9f8d6b: unmarshalled error http://127.0.0.1:8080/34,24a4ca9f9f8d6b: failed to write to local disk: volume 34 is read only
E1007 15:22:35.081219 filer_server_handlers_write_upload.go:230 request_id:cf18e83d-7e4f-41c4-baf8-01f597db4ab1 upload error: unmarshalled error http://127.0.0.1:8080/34,24a4ca9f9f8d6b: failed to write to local disk: volume 34 is read only
I1007 15:22:35.081240 filer_server_handlers_write_upload.go:152 request_id:cf18e83d-7e4f-41c4-baf8-01f597db4ab1 upload file 97a8c8e12855a46b79c5508c2894dffb error: unmarshalled error http://127.0.0.1:8080/34,24a4ca9f9f8d6b: failed to write to local disk: volume 34 is read only
I1007 15:22:35.081253 common.go:124 error JSON response status 500: unmarshalled error http://127.0.0.1:8080/34,24a4ca9f9f8d6b: failed to write to local disk: volume 34 is read only
I1007 15:22:35.081273 common.go:81 response method:PUT URL:/buckets/nodestore/nodestore/97a8c8e12855a46b79c5508c2894dffb with httpStatus:500 and JSON:{"error":"unmarshalled error http://127.0.0.1:8080/34,24a4ca9f9f8d6b: failed to write to local disk: volume 34 is read only"}
If you have any ideas about what to do, I'm all ears.
Event ID
No response