Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

404 when using limitedStore and hitting the storage limit #249

Closed
chendo opened this Issue Mar 13, 2019 · 4 comments

Comments

Projects
None yet
2 participants
@chendo
Copy link

chendo commented Mar 13, 2019

When using -store-size, we're seeing 404s when the disk usage hits the limit.

tusd log snippet:

[tusd] event="ResponseOutgoing" status="201" method="POST" path="" 
[tusd] event="HookInvocationStart" type="post-create" id="2c04ab7bdb5db06e50821caa67668565" 
[tusd] event="HookInvocationFinish" type="post-create" id="2c04ab7bdb5db06e50821caa67668565" 
[tusd] event="RequestIncoming" method="PATCH" path="2c04ab7bdb5db06e50821caa67668565" 
[tusd] event="ChunkWriteStart" id="2c04ab7bdb5db06e50821caa67668565" maxSize="465096" offset="0" 
[tusd] event="ChunkWriteComplete" id="2c04ab7bdb5db06e50821caa67668565" bytesWritten="465096" 
[tusd] event="ResponseOutgoing" status="204" method="PATCH" path="2c04ab7bdb5db06e50821caa67668565" 
[tusd] event="UploadFinished" id="2c04ab7bdb5db06e50821caa67668565" size="465096" 
[tusd] event="HookInvocationStart" type="post-finish" id="2c04ab7bdb5db06e50821caa67668565" 
[tusd] event="HookInvocationFinish" type="post-finish" id="2c04ab7bdb5db06e50821caa67668565" 
[tusd] event="HookInvocationStart" type="post-receive" id="2c04ab7bdb5db06e50821caa67668565" 
[tusd] event="HookInvocationFinish" type="post-receive" id="2c04ab7bdb5db06e50821caa67668565" 
[tusd] event="RequestIncoming" method="HEAD" path="7049c60f64fee683ac96acf52c598f66" 
[tusd] event="ResponseOutgoing" status="200" method="HEAD" path="7049c60f64fee683ac96acf52c598f66" 
[tusd] event="RequestIncoming" method="POST" path="" 
[tusd] event="HookInvocationStart" type="pre-create" id="" 
[tusd] event="HookInvocationFinish" type="pre-create" id="" 
[tusd] event="ResponseOutgoing" status="404" method="POST" path="" error="upload not found" 
[tusd] event="RequestIncoming" method="POST" path="" 
[tusd] event="HookInvocationStart" type="pre-create" id="" 
[tusd] event="HookInvocationFinish" type="pre-create" id="" 
[tusd] event="ResponseOutgoing" status="404" method="POST" path="" error="upload not found" 
[tusd] event="RequestIncoming" method="POST" path="" 
[tusd] event="HookInvocationStart" type="pre-create" id="" 
[tusd] event="HookInvocationFinish" type="pre-create" id="" 
[tusd] event="ResponseOutgoing" status="404" method="POST" path="" error="upload not found" 

Version:

Version: 0.11.0
Commit: 27c9c4aab534a9802a6334e99c4119f876ace8ff
Date: Mon Nov 19 16:44:52 UTC 2018

Environment:

Running in docker, with arguments: tusd --hooks-dir /srv/tusd-hooks -port 8080 -max-size 31457280 -store-size 1073741824 -behind-proxy

Disk usage:

/srv/tusd-data/data $ du -hc
1001.6M	.
1001.6M	total

@Acconut Acconut closed this in 0116e4e Mar 15, 2019

@Acconut

This comment has been minimized.

Copy link
Member

Acconut commented Mar 15, 2019

Thanks for reporting this! I was able to reproduce this by creating uploads until the data store hit the specified limit. After that I deleted some files from the data directory. Now, tusd is not able to delete some old uploads when it tries to reclaim disk space for new uploads.

It seems as if you have another service/user which deletes files from the data directory and collided with tusd. This should be fixed in the latest commit in the master branch, please test it out and tell me if it fixes your problem.

BTW, we are considering dropping support for the limitedstore package in the next major version (see #212). Due to limitedstore's design, it is not able to persist the internal state between restarts and therefore is not able to cleanup files which were created by previous tusd processes. Furthermore, limitedstore only works if you only have a single tusd process running, which is rarely the case. We have had problems with these limitations in the past and therefore have never used it in production. We also recommend you to reconsider your setup, even if the next major version will take some more time.

@chendo

This comment has been minimized.

Copy link
Author

chendo commented Mar 16, 2019

Ahhhh, I had manually deleted some images after doing initial testing with it. My bad, thanks!

I understand that this isn't the ideal setup. This was a quick test run to see if tusd would solve the issues we had when using tus-ruby-server with an S3-compatible endpoint but appeared to have consistency issues which was causing Upload-Offset errors (janko/tus-ruby-server#12). I may still use limitedstore until I have time to reconfigure tusd for an S3 endpoint but thank you for the heads up!

@Acconut

This comment has been minimized.

Copy link
Member

Acconut commented Mar 16, 2019

You're welcome. I just want to add that we have been using tusd with AWS S3 in production for years without any problems related to the integration with AWS S3, so I am pretty happy about that.

@chendo

This comment has been minimized.

Copy link
Author

chendo commented Mar 21, 2019

Oh I don't doubt that, the problem for us is I'm not convinced S3-compatible endpoints properly emulate S3 so potential for issues there for us.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.