FUSE+ACD/B2: Support for writing

[jkaberg]

Now that we have a properly working FUSE mount for reading, how about beeing able to write to it? :-)

Writing works for all except B2 and ACD?

[ncw]

Thanks for making a ticket about this. I agree it needs to be done, but it might not be easy! The two approaches are

• play with the uploading code for acd / b2 to see if it can work without knowing the length in advance - probably won't work
• cache the upload on disk first (as in Enhancement: Optional Caching of upload/downloads to improve stream stability #711) - will be more reliable but has several disadvantages

[jkaberg]

The first part would be pretty fast and easy to test I assume? I'm no go expert but from my python experience it seems straightforward

The 2nd part has some limitations as you mentioned in a commit, I'd be fine with that as long as the cache is always emptied when upload has finished

[xelra]

I like the solution with the cache. A proper writeback LRU cache, with a user-adjustable size. Anything from 1 MB to 1 TB. That would also reduce network usage significantly for many use-cases. Most of the time the files would be served locally.

I'm guessing you used hanwen/go-fuse, so maybe using a union fs could be useful for creating a cached mount. There's an example of a union fs in its repo.

I've also seen that there is an LRU example in the respository of groupcache from the standard library.

I hope this is helpful.

[jkaberg]

@xelra this is mainly a write cache so that rclone can stat the file(s) properly as ACD and B2 doesn't support uploading unknown file sizes.

[calisro]

The original intent of #711 was both a read and write cache. Probably should best be in two issues though depending on the complexity and the purpose. I can see it being useful for writes but the original intent was to an approach to allow caching locally with some way to clean it up (potentially an optional LRU flag or allow the user to do it manually).

[breunigs]

How many tickets are there for this? Regardless, see #890 for writing support for ACD.

[docloulou]

Hi,
(with the last #890 mod)
Write support seems working perfectly now, what about seeking write ?
I tried two method to download torrent directly on my mounted acd drive

1. NOT WORKING: preallocated file space --> before download, torrent client (qbittorrent in my case) write a blank file with 1 byte per 1 byte (this may take a very long time to build a 1go blank file) but after a while, the blank file upload failed with this error ...pacer: Rate limited, increasing sleep....
2. WORKS with issues.: not preallocated file space ---> the torrent download and upload in same time on acd mounted drive pretty well but if i close my torrent client or pause and restart on incompleted download, i have an I/O error.
   NB: if i don't stop the download, the download is fine.

I will provide a complete log this afternoon.

[docloulou]

1. Preallocated disk space method:

.......
.......
2016/11/19 11:40:17 mediafile.mkv: WriteFileHandle.Write len=1
2016/11/19 11:40:17 mediafile.mkv: WriteFileHandle.Write OK (1 bytes written)
2016/11/19 11:40:17 mediafile.mkv: WriteFileHandle.Write len=1
2016/11/19 11:40:17 mediafile.mkv: WriteFileHandle.Write OK (1 bytes written)
2016/11/19 11:40:17 mediafile.mkv: WriteFileHandle.Write len=1
2016/11/19 11:40:17 mediafile.mkv: WriteFileHandle.Write OK (1 bytes written)
2016/11/19 11:40:17 mediafile.mkv: WriteFileHandle.Write len=1
2016/11/19 11:40:17 mediafile.mkv: WriteFileHandle.Write OK (1 bytes written)
2016/11/19 11:40:17 mediafile.mkv: WriteFileHandle.Write len=1
2016/11/19 11:40:17 mediafile.mkv: WriteFileHandle.Write OK (1 bytes written)
2016/11/19 11:40:17 mediafile.mkv: WriteFileHandle.Write len=1
2016/11/19 11:40:17 mediafile.mkv: WriteFileHandle.Write OK (1 bytes written)
2016/11/19 11:40:17 mediafile.mkv: Upload error detected but didn't finish upload: HTTP code 400: "400 Bad Request": response body: "{\"logref\":\"dxxxxxxxe-ae4c-1xxx6-9c74-3d4xxxxx78\",\"message\":\"A node cannot be added to trashed parent\",\"code\":\"\"}" ("400 Bad Request")
2016/11/19 11:40:17 pacer: Rate limited, sleeping for 672.546632ms (1 consecutive low level retries)
2016/11/19 11:40:17 pacer: low level retry 1/1 (error HTTP code 400: "400 Bad Request": response body: "{\"logref\":\"d8xxxc4e-axc-1x6-9cx4-3d4xxxxxx78\",\"message\":\"A node cannot be added to trashed parent\",\"code\":\"\"}")
2016/11/19 11:40:17 mediafile.mkv: WriteFileHandle.Write len=1

2. Direct write in acd mounted volume method

.........
.........
2016/11/19 11:32:39 mediafile.mkv: WriteFileHandle.Write OK (16384 bytes written)
2016/11/19 11:32:39 mediafile.mkv: WriteFileHandle.Write len=32768
2016/11/19 11:32:39 mediafile.mkv: WriteFileHandle.Write OK (32768 bytes written)
2016/11/19 11:32:39 mediafile.mkv: WriteFileHandle.Write len=32768
2016/11/19 11:32:39 mediafile.mkv: WriteFileHandle.Write OK (32768 bytes written)
2016/11/19 11:32:39 mediafile.mkv: WriteFileHandle.Write len=131072
2016/11/19 11:32:39 mediafile.mkv: WriteFileHandle.Write OK (131072 bytes written)
2016/11/19 11:32:39 mediafile.mkv: WriteFileHandle.Write len=131072
2016/11/19 11:32:39 mediafile.mkv: WriteFileHandle.Write OK (131072 bytes written)
2016/11/19 11:32:39 mediafile.mkv: WriteFileHandle.Write len=131072
2016/11/19 11:32:39 mediafile.mkv: WriteFileHandle.Write OK (131072 bytes written)
2016/11/19 11:32:39 mediafile.mkv: WriteFileHandle.Write len=114688
2016/11/19 11:32:39 mediafile.mkv: WriteFileHandle.Write OK (114688 bytes written)
2016/11/19 11:32:39 mediafile.mkv: WriteFileHandle.Flush
2016/11/19 11:32:50 mediafile.mkv: WriteFileHandle.Flush OK
2016/11/19 11:32:50 mediafile.mkv: WriteFileHandle.Release nothing to do  
<<<---- I pause torrent download

......
......

I resume the torrent download --->>>
2016/11/19 11:34:14 mediafile.mkv: Dir.Lookup OK
2016/11/19 11:34:14 mediafile.mkv: File.Attr
2016/11/19 11:34:14 mediafile.mkv: File.Open
2016/11/19 11:34:14 mediafile.mkv: File.Open
2016/11/19 11:34:15 mediafile.mkv: ReadFileHandle.Read size 4096 offset 406290432
2016/11/19 11:34:15 mediafile.mkv: ReadFileHandle.Read attempt to read beyond end of file: 406290432 > 281149440
2016/11/19 11:34:15 mediafile.mkv: ReadFileHandle.Flush
2016/11/19 11:34:15 mediafile.mkv: ReadFileHandle.Flush OK
2016/11/19 11:34:15 mediafile.mkv: ReadFileHandle.Release closing
2016/11/19 11:34:15 mediafile.mkv: ReadFileHandle.Release OK

[breunigs]

There is no seeking write support in AmazonDrive. So even if pre-allocating would finish, you cannot change individual blocks later on and the file would be reported as corrupt. For the latter case I assume your client writes data consecutively and either fills gaps with 0s or temporarily caches the file locally.

Instead I suggest you download the file locally and then later upload it.

[docloulou]

thanks for answer: @breunigs

Instead I suggest you download the file locally and then later upload it.

(This may be difficult for big file >20gb)

yeah i will, but i currently use a DigitalOcean droplet with a low disk space (20gb), then i will write a automation bash script to create volume block from DO when needed for temp download file to save money :)

[thechile]

@breunigs just wondering when you said

no seeking write support in AmazonDrive

... can't Content-Range be used to provide both random file read/write access ? So in same way that rclone upload already allows resume... can't that be retro fitted with magic to allow overwrite feature ?

[breunigs]

I haven't tried, but you're welcome to do so. Also, which rclone upload do you speak of; is it in some separate branch? Current master says Error: unknown command "upload" for "rclone"

[ncw]

This is now implemented with the various --vfs-cache-mode options. You'll need the latest beta to use it. I used the strategy outlined in #711