Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix AutoCommit & soft AutoCommit #157

Closed
wants to merge 1 commit into from
Closed

Fix AutoCommit & soft AutoCommit #157

wants to merge 1 commit into from

Conversation

Scarfmonster
Copy link
Contributor

Current settings disabled both AutoCommit and soft AutoCommit with the currently used version of Solr.
This should ensure that YaCy always gets fresh index without needing to explicitly commit, as it is recommended not to do that by the official Sorl docs.

Current settings disabled both AutoCommit and soft AutoCommit with the currently used version of Solr.
This should ensure that YaCy always gets fresh index without needing to explicitly commit, as it is recommended not to do that by the official Sorl docs.
Scarfmonster referenced this pull request Jan 28, 2018
This ensures up-to-date simulation query results, and recrawl
processing.
@luccioman
Copy link
Member

HI @Scarfmonster , I would have a few questions about your proposal :

  • can you give a link to the official Solr doc where it is talking about avoiding explicit commits?
  • you say AutoCommit is currently disabled. What led you to think so? On my own tests the solrconfig.xml setting <maxTime>${solr.autoCommit.maxTime:15000}</maxTime> is properly parsed and used : when debugging the SolrConfig.UpdateHandlerInfo() constructor, autoCommmitMaxTime has indeed 15000 as value.
  • did you try to measure performance impacts, notably on crawl operations for example, after enabling soft autocommit?

@Scarfmonster
Copy link
Contributor Author

@luccioman
Here is the part in the docs I am talking about

I tried delving into it deeper outside of the docs and based on some stackoverflow answers and comments it seems that the cause of the index updates not being visible my have been the disabled soft commit. Although AutoCommit does write to the hard drive, it may not make the changes visible with openSearcher set to false.

As for the performance impacts, this article has some interesting points and answers. Although even here at the end they boldly state "Be very careful committing from the client! In fact, don’t do it."

@luccioman
Copy link
Member

Ok thanks for the details @Scarfmonster .
By the way, current YaCy code base relies on occasional explicit hard and soft commits spread among various classes. It is not the recommended way, but it is the current situation. So to my mind activating automated soft commit every second as a default should be considered with great care, especially while explicit manual commits remain in code.

@luccioman
Copy link
Member

@Scarfmonster soft autocommit is now enabled in the default Solr configuration with an interval of 5 seconds which I found to be sufficient and reasonable regarding system load overhead (I shared some measurements in issue #251).
Thank you anyway and sorry for the long merge delay.

@luccioman luccioman closed this Nov 20, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants