-
Notifications
You must be signed in to change notification settings - Fork 338
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Immudb instance doesn't load data from S3 storage after reboot (randomly, discarding snapshots and other data) #1856
Comments
Update: according to the logs we found next:
And a lot of other mentions about "discarded" data
So why it may happen and how to fix this? |
We'll review it asap. Some data seems to be loaded and thus the mismatch described in the logs. Non-fully committed transactions may be discarded as the client shouldn't have received any confirmation. |
Hi! Thanks a lot! |
Hi! We investigated the issue. Steps to reproduce:
The rootcause that from the one side ImmuDB anyway host some recent data files (like tx or ay else) on the Docker conatiner (in our case - Fargate site) File system and from another side - instead docker or docker-compose - AWS Fargate create completly new instance of the tasks . Old data (volume) is just vanished without any way to restore them. So, looks like it impossible to use ImmuDB running it on AWS Fargate and to be sure that data will not be lost after Fargate task crushes. |
Currently, immudb asynchronously upload data to S3, so as you describe data may be lost in such cases. Replication could be used to mitigate this scenarios but master election is not implemented so it may require manual intervention or an external tooling to determine the best instance. @SimoneLazzaris you may be able to extend in this aspect. Another possibility is to implement a synchronous operation mode when using S3 but a noticeable performance degradation would be expected. Nevertheless it seems a nice capability for use cases were degraded performance is still acceptable. |
What happened
We host Immudb as AWS Fargate task (literally - docker container). As back-end storage we use S3 bucket as storage.
When container/task restarted Immudb doesn't load existed data from storage, it become just empty. But in the same time looks like it's continue update storage.
What you expected to happen
After reload Immudb instance should load data from S3 storage.
How to reproduce it (as minimally and precisely as possible)
Environment
Additional info (any other context about the problem)
The text was updated successfully, but these errors were encountered: