Add large file support

[Xuanwo]

We read all content into memory for now, it's not acceptable for real users.

The logic looks the same as https://github.com/beyondstorage/beyond-fs/blob/master/vfs/cache.go, maybe we can extract it as a new library.

[Xuanwo]

I'm working the lib: https://github.com/beyondstorage/go-converge

GitHub

beyondstorage/go-converge

A lib that converge input from upperlying storage to underlying storage. - beyondstorage/go-converge

[Xuanwo]

Hi @xiongjiwei, the 0.1.0 version for go-stream has been released: https://github.com/beyondstorage/go-stream/releases/tag/v0.1.0

Could you give it a try?

GitHub

Release v0.1.0 · beyondstorage/go-stream

v0.1.0 - 2021-08-23 Added

feat: Implement basic stream logic (#1)
feat: Implement stream logic via Multipart (#4)
feat: Add check for multipart (#7)
test: Implement integration tests (#8)
feat: Ad...

[xiongjiwei]

that's great

[xiongjiwei]

Hi @Xuanwo, seems it only supports storager which implements Multiparter, maybe I should wait for the PersistMethodWrite PersistMethod to be supported?

[Xuanwo]

Hi @Xuanwo, seems it only supports storager which implements Multiparter, maybe I should wait for the PersistMethodWrite PersistMethod to be supported?

I think it's OK to add support for Multipart first.

[xiongjiwei]

Got a question about the stream lib, does it support append data to the under storager？ How can I control the behavior to append or overwrite? I find the PersistMethodAppend which seems will be implemented in the future. if I should use this field to control the stream behavior, seems I should create 2 streams, one is used for overwriting and another one is used for append.

[Xuanwo]

The go-storage design limited that:

• We can't append an object unless the object is created by CreateAppend, aka, has Append object mode.
• We can't do multipart on an object unless the object is created by CreateMultipart, aka, has Multipart object mode.

The limitation is introduced by most storage services (except local fs) can't support append to object.

So, we always need to choose a persist mode before we start to create the file. And it's could be PersistMethodMutlipart or PersistMethodAppend, of course, multipart only for now.

It's OK for BeyondFTP to return errors if the underlying storage doesn't have append supports.