Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

introduce stand-in indexer #10

Merged
merged 13 commits into from
Mar 13, 2019
Merged

introduce stand-in indexer #10

merged 13 commits into from
Mar 13, 2019

Conversation

mtbc
Copy link
Member

@mtbc mtbc commented Feb 27, 2019

With Hibernate 3.6 the OMERO 5.4 indexer fails with a Sealed WorkQueue exception. This PR introduces a stand-in replacement to index adequately until the problem can be better addressed. See https://trello.com/c/XdwEzrvs/73-indexer-log-hibernate-error.

CLI foreground reindexing (with --foreground) is not fixed but background should suffice.

Fails to circumvent Sealed WorkQueue exception but
highlights that the simple act of trying to index
suffices to trigger it.
configurable analyzer used in @ClassBridge(analyzer = ...)
Also reuses fixed set of jobs in the scheduler.
For the cron expression the indexer just notes if it is blank.
Can purge not-included types in case they used to be included.
* now respects omero.search.include_actions
* still ignores omero.search.max_partition_size
  but now avoids unnecessary indexing
* minor refactor of getWhen into FTI2.Event enum
* handle unknown entityType values
LimitTokenCountAnalyzer would be an alternative should we
ever wish to set this kind of property to a lower value.
In Hibernate Search 3.4.2's Lucene 3.1.0 the code now reads,
DEFAULT_MAX_FIELD_LENGTH = MaxFieldLength.UNLIMITED.getLimit();
@mtbc
Copy link
Member Author

mtbc commented Feb 28, 2019

To see more of its activity can add to etc/logback-indexing.xml,

<logger name="ome.services.fulltext.FullTextIndexer2" level="DEBUG"/>

@joshmoore
Copy link
Member

Currently testing via ome/omero-build#6

@rgozim rgozim mentioned this pull request Mar 12, 2019
9 tasks
@mtbc
Copy link
Member Author

mtbc commented Mar 12, 2019

Now running into just a couple of integration test failures, mostly intermittent. Probably nothing major, will investigate. Consistent failure is of testFilename.

Update: Naturally, now it passes after cleaning out the DB. Probably some bad assumptions. Still some intermittent issues with test3164* though.


/**
* An indexer bean replacing the 5.4 full-text thread with adequate functionality.
* Exists as a stand-in while Hibernate / Spring upgrade issues remain unresolved.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mtbc : could you extend this a bit with:

  • how this differs from the previous indexer (trade-offs, etc)
  • and how the implementation functions ("first A, then ...")

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, though I may miss some differences and maybe not in time for m4.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Extended in #23.

@mtbc
Copy link
Member Author

mtbc commented Mar 12, 2019

The issue with test3164* seems somehow related to the use of IUpdate.indexObject.

@mtbc
Copy link
Member Author

mtbc commented Mar 12, 2019

Ha, IUpdate.indexObject fires up the legacy indexer. For now I'll tweak it to instead create a REINDEX entry in the event log. Alternatively could plug actual indexer more clearly into update service if preferred.

@mtbc
Copy link
Member Author

mtbc commented Mar 12, 2019

Heh, so: interesting issue: my simple approach to fixing IUpdate.indexObject fails because some tests like test3721Ordering expect that method to index things like tags which are not in omero.search.include_types so the indexer ignores those requests. (They would get indexed if the request were to index a tagged image but that test doesn't have such.) I assume that IUpdate.indexObject should override include_types? In which case, expect some IUpdate.indexObject-using tests to fail until I push a fix tomorrow.

@joshmoore
Copy link
Member

Green with jars from artifactory after conflict resolution. Merging proactively for M4 and the more significant testing can happen there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants