Unexpected termination of the app results in corrupt files

[yatharthranjan]

If the gzip compression is enabled, and the app terminates unexpectedly, it causes certain files to become corrupt.
Running a gzip -t test for all the files showed that some files are corrupt and corruption can be different types -
invalid compressed data--format violated and unexpected end of file

we should investigate why this is happening.

[blootsvoets]

Unfortunately, this is expected due to the file cache that is being used. Files will not be flushed until they are evicted from the cache. This is important for GZIP compression, because if it has too little data to compress, compression will be very bad.

The alternative, which I think would be good to pursue, is to write data into a staging area and replace the original files when the app finishes correctly. But there would have to be some logic to clean up the staging area. Also, we can choose whether to move the files when the cache gets closed or when the app is done completely, at which time we also update the offsets file.

[blootsvoets]

Measure to try to recover some of the data:

gunzip < some.csv.gz > some.csv

will try and unzip as much as possible before the corruption is encountered. See https://stackoverflow.com/questions/201392/how-can-i-recover-files-from-a-corrupted-tar-gz-archive for more solutions.
Alternatively try and install gzrecover: http://www.urbanophile.com/arenn/coding/gzrt/gzrt.html. On Ubuntu, it is available with sudo apt install gzrt.