Virtual file system or object storage to fetch variables and state between live deployments

[Maly-Lemire]

Would it be possible to provide a virtual file system with about 1mb available for free?
You could charge for more space.

I could do a pull request that provides a wrapper around File class or a simple Dictionary for saving variables with size and available space check. I would need a server side directory with proper access for saving the data.

Thanks

[jaredbroad]

Thanks @Maly-Lemire -- to understand the use case:

• Are you storing data between backtests? (e.g. training data, train once, use it multiple times)
• Or looking to literally model a file system?

"Temporary store" is a popular request and would be a well used feature. To make it compatible with our web platform need a design like this: A IStorageProvider which with Initialize, Get and Set methods for a key-value pair. Initialize could take the api class.

It should be MEF'd in during the start up and set to the algorithm in in the SetupHandler via IAlgorithm. For LEAN you could implement a "FileSystemStorageProvider" which saves to a local directory.

[4real]

Key value storage, preferably possible to share via backtest and live instances, and even edit externally, would be great. I'm using Amazon S3 for it now but I can't say I recommend it for the average user given the amount of time necessary to set it up.

Use cases:

• Saving/restoring algo state over restarts
• Warming up algorithm with backtest to present day for computation too heavy for Initialize w/ history requests or warmup
• Another way to give parameters to algorithm if some keys can be made accessible externally via GUI or REST

In fact, building this on S3 would be very viable, although using a DB service would perhaps be more performant.

[jaredbroad]

• Would also be useful if this object was available as a stream. (For Accord library neural net loading)

[Maly-Lemire]

Sorry about the lack of reply but I've managed to use warm up for what I needed to do. Since I won't need this feature, I may never do it. Sorry about that.

[DMTSource]

Would be great to be able to work with files either taken from local(is that a major security problem?), or more importantly created in Research, or during a Backtest/Live.
For example: You are working with DEAP, TensorFlow, or Scikit-Learn to perform machine learning of some kind. In these cases a binary checkpoint file(TensorFlow) or pickled objects file(sklearn, Deap) is created to persist the model in order to continue training, or to be used for inference.
These files may be non trivial in size however, maybe a storage add-on sub in QC?

[jaredbroad]

👍 PR might look like this:

IObjectStorage {
object Get(string key)
void Put(string key, object value)
}

QCAlgorithm {
IObjectStorage Storage;
}

//Usage like:
Algorithm.Storage.Get()
Algorithm.Storage.Put()

Then using LeanEngineAlgorithmHandlers from configuration to load the type;

Apply the algorithmHandler.Storage in Engine.cs