Recovery.db.mv.db size crashes Mist

[utkarshsaraf19]

I have setup a VM having following configuration : Redhat 7.4, 4 GB RAM

I have visualized that the size of Recovery.db.mv.db increases which is obvious as i run more jobs.

It is crashing when the size reaches 37 MB with java heap space error.

I wanted to know the reason of it.Is it due to browser loading this whole file or mist itself and what configuration changes/factors i have to keep in mind while deploying it?

[dos65]

Does it always crash after restart?

[utkarshsaraf19]

Yes..once this starts,it never gets normal.I have to delete Recovery.db.mv.db manually to make it work again.

[dos65]

As a workaround you could increase Xmx for mist master process.
But we definitely should reconsider our way to store jobs history.

[utkarshsaraf19]

I have some suggestion for the same:

a. Bucketing recoverydb on basis of number of jobs and output size of it.
b. Storing recoverydb on basis of folder named with endpoint
c. Showing it in front end with pagination feature as one page will utilize only one recoverydb at a time.
d. Seperate jvm for history logs.

[dos65]

Yes bucketing or limiting could help.
I think at least we should provide a way to configure history storage (use external databases).

There is also a complicated question - should we continue storing jobs results inside database? They can very large and using databases may be an inefficient approach.

@spushkarev @mkf-simpson - This may be interesting for you - if we found another way how to store job results, it can be possible to build pipelines over datasets with that feature.

[mkf-simpson]

I cannot understand how job history relates to pipelines?

[dos65]

To invoke pipeline stages on different spark-contexts we need to store jobs results somewhere.

[mkf-simpson]

Ok, MistWarehouse? :) But this discussion is for another ticket I guess.

[utkarshsaraf19]

@mkf-simpson @dos65 @spushkarev Whichever way you choose, just one thing should be considered that job should not be deployed every time we hit it with endpoint due to this seperation. As of now,it takes 25 seconds for first run and less then 2 seconds for after run for same endpoint which is better in case of production use.