Custom Lucene Extensions Blocked

[mattweber]

It appears that the introduction of centralized lucene extensions (#71416) we are no longer able to use non-default lucene codecs that introduce additional file extensions. For example, uniform split that introduces the ustd and ustb extensions or uniform split shared terms which introduces stustd and stustb extensions.

Can we support custom extensions via plugins? Maybe just remove the assert on unknown extension which is tripping in my plugin tests.

cc @tlrx @jpountz

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[tlrx]

Thanks for reporting @mattweber and sorry to hear it breaks your own tests. We added this assertion to catch file extensions that were not explicitly declared so that we are forced to declare them and to configure some options about them: should the file be memory mapped, should it be cached differently than other files etc

I think this assertion is valuable because it prevents us from missing some Lucene files. If the file extension is provided by a codec maintained in Lucene I think it is perfectly fine to update the list of file extensions and just add them. If users who maintain their own Lucene codecs are blocked by this assertion we can add a system prop to disable the assertion for their tests. Providing a support for custom extensions via plugins is possible but looks too heavy to me for now.

I'm deferring the decision to the es-search team but I'd be glad to help implementing the solution.

[ywelsch]

+1 on all points here. Ideally all file extensions are known to the system so that it can make the right decision when it comes to caching / memory mapping the files and provide optimal performance. I'm inclined to further lock this down in the future (not only assert, but also check at runtime when assertions are disabled). At the same time it's important to allow plugins to use codecs that introduce additional file extensions. While adding an explicit plugin extension point for new file extensions is certainly an option (and a requirement if we further lock this down), there's some plumbing required to make that happen. To get @mattweber unblocked as quickly as possible, a system property for opting out of this check sounds like a good short-term solution to me. @mattweber WDYT?

[mattweber]

@tlrx @ywelsch thanks for the quick review and discussion. I think a system property would be perfectly fine. In the future, I would love better support for custom codecs (and stores) which could have hooks into this extension map. Thank you!

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)