404 when using limitedStore and hitting the storage limit

[chendo]

When using -store-size, we're seeing 404s when the disk usage hits the limit.

tusd log snippet:

[tusd] event="ResponseOutgoing" status="201" method="POST" path="" 
[tusd] event="HookInvocationStart" type="post-create" id="2c04ab7bdb5db06e50821caa67668565" 
[tusd] event="HookInvocationFinish" type="post-create" id="2c04ab7bdb5db06e50821caa67668565" 
[tusd] event="RequestIncoming" method="PATCH" path="2c04ab7bdb5db06e50821caa67668565" 
[tusd] event="ChunkWriteStart" id="2c04ab7bdb5db06e50821caa67668565" maxSize="465096" offset="0" 
[tusd] event="ChunkWriteComplete" id="2c04ab7bdb5db06e50821caa67668565" bytesWritten="465096" 
[tusd] event="ResponseOutgoing" status="204" method="PATCH" path="2c04ab7bdb5db06e50821caa67668565" 
[tusd] event="UploadFinished" id="2c04ab7bdb5db06e50821caa67668565" size="465096" 
[tusd] event="HookInvocationStart" type="post-finish" id="2c04ab7bdb5db06e50821caa67668565" 
[tusd] event="HookInvocationFinish" type="post-finish" id="2c04ab7bdb5db06e50821caa67668565" 
[tusd] event="HookInvocationStart" type="post-receive" id="2c04ab7bdb5db06e50821caa67668565" 
[tusd] event="HookInvocationFinish" type="post-receive" id="2c04ab7bdb5db06e50821caa67668565" 
[tusd] event="RequestIncoming" method="HEAD" path="7049c60f64fee683ac96acf52c598f66" 
[tusd] event="ResponseOutgoing" status="200" method="HEAD" path="7049c60f64fee683ac96acf52c598f66" 
[tusd] event="RequestIncoming" method="POST" path="" 
[tusd] event="HookInvocationStart" type="pre-create" id="" 
[tusd] event="HookInvocationFinish" type="pre-create" id="" 
[tusd] event="ResponseOutgoing" status="404" method="POST" path="" error="upload not found" 
[tusd] event="RequestIncoming" method="POST" path="" 
[tusd] event="HookInvocationStart" type="pre-create" id="" 
[tusd] event="HookInvocationFinish" type="pre-create" id="" 
[tusd] event="ResponseOutgoing" status="404" method="POST" path="" error="upload not found" 
[tusd] event="RequestIncoming" method="POST" path="" 
[tusd] event="HookInvocationStart" type="pre-create" id="" 
[tusd] event="HookInvocationFinish" type="pre-create" id="" 
[tusd] event="ResponseOutgoing" status="404" method="POST" path="" error="upload not found" 

Version:

Version: 0.11.0
Commit: 27c9c4aab534a9802a6334e99c4119f876ace8ff
Date: Mon Nov 19 16:44:52 UTC 2018

Environment:

Running in docker, with arguments: tusd --hooks-dir /srv/tusd-hooks -port 8080 -max-size 31457280 -store-size 1073741824 -behind-proxy

Disk usage:

/srv/tusd-data/data $ du -hc
1001.6M	.
1001.6M	total

[Acconut]

Thanks for reporting this! I was able to reproduce this by creating uploads until the data store hit the specified limit. After that I deleted some files from the data directory. Now, tusd is not able to delete some old uploads when it tries to reclaim disk space for new uploads.

It seems as if you have another service/user which deletes files from the data directory and collided with tusd. This should be fixed in the latest commit in the master branch, please test it out and tell me if it fixes your problem.

BTW, we are considering dropping support for the limitedstore package in the next major version (see #212). Due to limitedstore's design, it is not able to persist the internal state between restarts and therefore is not able to cleanup files which were created by previous tusd processes. Furthermore, limitedstore only works if you only have a single tusd process running, which is rarely the case. We have had problems with these limitations in the past and therefore have never used it in production. We also recommend you to reconsider your setup, even if the next major version will take some more time.

[chendo]

Ahhhh, I had manually deleted some images after doing initial testing with it. My bad, thanks!

I understand that this isn't the ideal setup. This was a quick test run to see if tusd would solve the issues we had when using tus-ruby-server with an S3-compatible endpoint but appeared to have consistency issues which was causing Upload-Offset errors (janko/tus-ruby-server#12). I may still use limitedstore until I have time to reconfigure tusd for an S3 endpoint but thank you for the heads up!

[Acconut]

You're welcome. I just want to add that we have been using tusd with AWS S3 in production for years without any problems related to the integration with AWS S3, so I am pretty happy about that.

[chendo]

Oh I don't doubt that, the problem for us is I'm not convinced S3-compatible endpoints properly emulate S3 so potential for issues there for us.