You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently we serialize scigraph results as a list of JSON documents, and then post the entire JSON file to solr. This has worked well in the past, but seems to cause issues as these JSON documents have gotten larger.
As an alternative, we can use the SolrJ API to construct SolrInputDocument objects and post these the server in batches of 100k or so. It's unclear if this will result in any performance boost, as SolrJ appears to be sending them to the server using http regardless. At a minimum this should fix #27 and possibly #30.
As a test I've reworked the golr worker to convert the JSON documents to SolrInputDocuments - which is a pretty minimal change but is slightly less performant. After chatting with @kltm and @cmungall I'm planning on moving ahead with the larger refactor of removing the JSON intermediate step.
@benwbooth I'm wondering if it will conflict with your work on #17 ?
The text was updated successfully, but these errors were encountered:
I don't think it would. I'm only adding some optional keys to the yaml, then adding extra closure relationships to the query. In any case, I'm happy to do any required refactoring as needed.
Currently we serialize scigraph results as a list of JSON documents, and then post the entire JSON file to solr. This has worked well in the past, but seems to cause issues as these JSON documents have gotten larger.
As an alternative, we can use the SolrJ API to construct SolrInputDocument objects and post these the server in batches of 100k or so. It's unclear if this will result in any performance boost, as SolrJ appears to be sending them to the server using http regardless. At a minimum this should fix #27 and possibly #30.
As a test I've reworked the golr worker to convert the JSON documents to SolrInputDocuments - which is a pretty minimal change but is slightly less performant. After chatting with @kltm and @cmungall I'm planning on moving ahead with the larger refactor of removing the JSON intermediate step.
@benwbooth I'm wondering if it will conflict with your work on #17 ?
The text was updated successfully, but these errors were encountered: