Store partial reconciliation results

[ettorerizza]

Reconciliation (eg with Wikidata) is sometimes a long process. At the slightest connection problem, or if Open Refine crashes, you have to start all over again. Would it not be possible to preserve the reconciliations already made, or to store them in a kind of cache? Unless it is already the case and I ignore it ? (I've never noticed that a failed reconciliation was faster the second time).

[wetneb]

I agree this is an important problem. But it's not entirely clear to me what the solution should look like.

• I can put a server-side cache on top of the reconciliation queries (but that would only solve the problem for Wikidata)
• We can put a client-side cache in OpenRefine but it would probably not survive crashes. It would also need to be persistent across operations (which is not the case, for instance, for the cache I have added to the URL fetching operation).

As always with caches, we have to be careful about the size, invalidation strategy, time to live, and so on…

[wetneb]

Related issue: #580

[thadguidry]

@wetneb If I recall, I think I asked David H. to consider Fetch URLs and Reconcile as long running operations and hence no automatic project save occurs. Honestly, I cannot recall...so...We need to

• check the code on that

and could change the logic so that project saves are configurable and could occur more rapidly than default (ex. every 4 mins).

[wetneb]

I don't think that running an autosave while the operation is running would do anything - the reconciliation results are stored in memory and they are only added to the project at the very end of the operation. Even if the column was populated gradually, you would still need to write the change to the history, and recover from an interrupted run (that could be done by faceting, but it would be quite manual).

[thadguidry]

@wetneb your probably right, like I said...I cannot recall, and I am not very intimate anymore with our code in a lot of areas. But I can give lots of pointers to folks that have more time so they can get intimate with our code :)

[jackyq2015]

Maybe some tweak around line: https://github.com/OpenRefine/OpenRefine/blob/master/main/src/com/google/refine/process/ProcessManager.java#L152
could save partial reconciliation results. Or even better to use Interceptor design pattern make it more flexible and clean. Just a wild idea. :)

[magdmartin]

this will help to address #1235

[wetneb]

This is implemented in the 4.0 branch. See the videos and discussion there: https://forum.openrefine.org/t/partial-results-of-long-running-operations/482