worker: Remove tons of work arounds and quirks in upload #826
Conversation
Just use blocking post - possibly all this was necessary with mojo < 7, but surely not in current: http://mojolicious.org/perldoc/Mojolicious/Guides/Cookbook#Large-file-upload
|
LGTM |
|
the coverage increase is fake - I just replaced uncovered code :) |
|
Finally not deleting covered code ;-P |
|
I implemented a chksum check - verify link: https://tortuga.suse.de/tests/427#downloads See the log: Hopefully we can find out the root problem with this - uploading works 100% on my staging instance ;( |
|
I applied the patch on 4 workers on openqa.opensuse.org and we didn't see a single failure since then ;( So I claim f2547e9 is the actual fix - but to prove that we would need to have the two other commits run independent of it. |
|
could very well be. LGTM |
|
we have this deployed on osd as well as my change to not always auto-duplicate and we see problems saving and uploading images, e.g. |
|
I think this is exactly what @nadvornik tried to solve by that removed hacks. :) |
|
I just checked the new Mojo. The default boundary calculation is still the same: Mojo generates a random string and then checks if the file contains it - that means the file is completely read from disk before sending anything. It can take up to several minutes and the server closes the connection meanwhile. So I think the boundary hack is still needed. |
|
Maybe the most correct solution is to split the file to ~16MB chunks and upload each in separate request. |
|
some of the stuff coolo applied certainly make sense. What he failed to do is reproduce the error first locally to confirm that his "fix" is really a fix. OTOH http://mojolicious.org/perldoc/Mojolicious/Guides/Cookbook#Large-file-upload should still apply. Maybe the images we are handling are still a bit too big and 16MB might make sense. Maybe these references can help: |
|
This concrete problem is mentioned here: The official mojo documentation speaks about large files in size of megabytes, we need gigabytes. I don't see any way how to override the boundary calculation in ua->post() api. |
|
I am pretty sure in perl everything can be monkey patched :-) |
|
See #843 |
Just use blocking post - possibly all this was necessary with mojo < 7, but
surely not in current:
http://mojolicious.org/perldoc/Mojolicious/Guides/Cookbook#Large-file-upload