Cant interact with file during upload process

[MeijeSibbel]

Users should be able to interact with their files without getting these sort of errors. Maybe copy the file to temp and upload it from there.

[MeijeSibbel]

This issue still persists for large files, i suggest we leave it for now and get back to it later, eventually users should be able to do whatever they want with the files, deleting them, renaming them, moving them..., the app has to do what the user wants, not the other way around.

[jkawamoto]

I'm curious we can delete or rename files while we are copying those files. If we cannot, even if we make a copy of a large file to prevent this issue, OS will block until copying the file to some temporal directory.

[MeijeSibbel]

Can we cancel the copy process on the fly?, So file is copying -> cancel it -> execute user action.

[jkawamoto]

The file watch service we're using notifies a file was created/modified/deleted. To cancel the copy process, we need to get events in advance e.g. a file is going to be created/modified/deleted. So, we need to investigate other libraries.

[jasonwee]

Is file in use is o/s specific? I cannot reproduce in linux.

[MeijeSibbel]

@jasonwee,

1. You should use relativel large files. E.g. 2
   1 to 4 gb.
2. This was tested on windows.

[jasonwee]

During uploading a repeating text file of about 3.2GB, I removed the file using
command line, it works okay. Of cause, this cancel upload is in the PR123.

Which class that lock the file during uploading? afaik, g-s-s does not do that?
It's a pity I don't have windows nor mac to test this though. But later I will
try using a video file which is around 1-4GB and open using vlc in linux.

[Thread-1] INFO  io.goobox.sync.storj.UploadFileTask -     1%        42516480/3221225472 bytes
[Thread-1] INFO  io.goobox.sync.storj.UploadFileTask -     1%        42598400/3221225472 bytes
[Thread-1] INFO  io.goobox.sync.storj.UploadFileTask -     1%        42680320/3221225472 bytes
[Thread-1] INFO  io.goobox.sync.storj.UploadFileTask -     1%        42762240/3221225472 bytes
[ForkJoinPool.commonPool-worker-1] DEBUG io.methvin.watcher.DirectoryWatcher - ENTRY_DELETE [/home/jason/Goobox/file3.txt]
[Thread-1] INFO  io.goobox.sync.storj.UploadFileTask -     1%        42844160/3221225472 bytes
[Thread-1] DEBUG io.goobox.sync.storj.UploadFileTask - file is not exists (rename, delete or moved), cancelling upload
[Thread-1] INFO  io.goobox.sync.storj.UploadFileTask - canceling upload
[Thread-1] DEBUG o.dizitart.no2.internals.DataService - Filter EqualsObjectFilter(field=name, value=file3.txt) found total 1 document(s) to update with options UpdateOptions(upsert=false, justOnce=false) in io.goobox.sync.storj.db.SyncFile
[Thread-1] DEBUG o.dizitart.no2.internals.DataService - Document to update {name=file3.txt, storjId=null, storjCreatedTime=0, storjSize=0, localModifiedTime=1526032127000, localSize=3391619072, state=FOR_UPLOAD, _id=175108323055922, _revision=2, _modified=1526032622248} in io.goobox.sync.storj.db.SyncFile
[Thread-1] DEBUG o.dizitart.no2.internals.DataService - Document {name=file3.txt, storjId=null, storjCreatedTime=0, storjSize=0, localModifiedTime=1526032127000, localSize=3391619072, state=UPLOAD_FAILED, _id=175108323055922, _revision=3, _modified=1526033013231} updated in io.goobox.sync.storj.db.SyncFile
[Thread-1] DEBUG o.dizitart.no2.internals.DataService - Notifying UPDATE event for items [ChangedItem(document={name=file3.txt, storjId=null, storjCreatedTime=0, storjSize=0, localModifiedTime=1526032127000, localSize=3391619072, state=UPLOAD_FAILED, _id=175108323055922, _revision=3, _modified=1526033013231}, changeType=UPDATE, changeTimestamp=1526033013231)] from io.goobox.sync.storj.db.SyncFile
[Thread-1] DEBUG o.dizitart.no2.internals.DataService - Returning write result WriteResultImpl(nitriteIdList=[[175108323055922]NO₂]) for collection io.goobox.sync.storj.db.SyncFile
[Thread-1] DEBUG org.dizitart.no2.Nitrite - Unsaved changes committed successfully.
[Thread-1] ERROR io.goobox.sync.storj.UploadFileTask - Upload failed: File transfer canceled (1)
[Thread-2] INFO  io.goobox.sync.storj.CheckStateTask - Skip checking for changes - local file operations in progress
[Timer-0] INFO  io.goobox.sync.storj.FileWatcher - 3 seconds after the last file event
[Thread-2] INFO  io.goobox.sync.storj.CheckStateTask - Checking for changes
[Thread-1] DEBUG o.dizitart.no2.internals.DataService - Filter EqualsObjectFilter(field=name, value=file3.txt) found total 1 document(s) to remove with options RemoveOptions(justOne=false) from io.goobox.sync.storj.db.SyncFile
[Thread-1] DEBUG o.dizitart.no2.internals.DataService - Document removed {name=file3.txt, storjId=null, storjCreatedTime=0, storjSize=0, localModifiedTime=1526032127000, localSize=3391619072, state=UPLOAD_FAILED, _id=175108323055922, _revision=4, _modified=1526033017146} from io.goobox.sync.storj.db.SyncFile
[Thread-1] DEBUG o.dizitart.no2.internals.DataService - Notifying REMOVE event for items [ChangedItem(document={name=file3.txt, storjId=null, storjCreatedTime=0, storjSize=0, localModifiedTime=1526032127000, localSize=3391619072, state=UPLOAD_FAILED, _id=175108323055922, _revision=4, _modified=1526033017146}, changeType=REMOVE, changeTimestamp=1526033017146)] from io.goobox.sync.storj.db.SyncFile
[Thread-1] DEBUG o.dizitart.no2.internals.DataService - Returning write result WriteResultImpl(nitriteIdList=[[175108323055922]NO₂]) for collection io.goobox.sync.storj.db.SyncFile
[Thread-1] DEBUG org.dizitart.no2.Nitrite - Unsaved changes committed successfully.
[Thread-1] DEBUG io.goobox.sync.storj.ipc.IpcExecutor - Command sent: {"method":"syncState","args":{"newState":"idle"}}
{"method":"syncState","args":{"newState":"idle"}}
[Thread-2] INFO  io.goobox.sync.storj.SleepTask - Sleeping for 1 minute

[MeijeSibbel]

@kaloyan-raev , @jkawamoto ^

[jkawamoto]

@jasonwee Yes, this only happens on Windows. It blocks renaming/deleting files while they are opened. So, in sync-sia, files are blocked while it calculates hash values
https://github.com/GooBox/goobox-sync-sia/blob/master/src/main/java/io/goobox/sync/sia/FileWatcher.java#L139

[jasonwee]

@jkawamoto , do you mean because of sync-sia calculate hash values of a file then the file is blocked? if you removed this file hashing calculation, will the file in use error still complaint by windows?

[jkawamoto]

I think during the hash value calculation the file is blocked but I haven't tested what happens if the hash calculation is removed.