Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move FaissService and NmslibService in jni directory and make proper sub-gradle project #1540

Open
jmazanec15 opened this issue Mar 14, 2024 · 0 comments
Labels
Infrastructure Changes to infrastructure, testing, CI/CD, pipelines, etc.

Comments

@jmazanec15
Copy link
Member

Description

Following up on discussion here: #1529 (comment)

I think we can get rid of the whole JNIService and move the calls to the respective Engines. If we were to do this, we would then just have FaissService and NmslibService in the jni module.

If we were to do this, we could move FaissService and NmslibService into the jni directory in the root project and create dedicated gradle sub-projects. Then, for the plugin itself, we could take a proper dependency on the JNI related code itself.

I think this would properly decouple the JNI library/Java bindings from the plugin. This would allow us to more easily:

  1. Separate plugin build from library build (this has been an issue with build infra in many instances cc: @peterzhuamazon).
  2. This would allow us to decouple library versioning from plugin versioning. If we dont change the library, we dont need to release it with every release
  3. Take a step towards making other custom engines pluggable (similar to [FEATURE] Is possible to add new KNNs using microservices? That broadens the KNNs that can be used as backends #1328)
  4. Publish lib based artifacts separate from the plugin. We could then properly avoid issues around libs missing for plugin installs ([BUG] Missing lib folder after plugin installation using opensearch-plugin CLI tool #1408) without doing hacks like adding to our zip in the build script
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Infrastructure Changes to infrastructure, testing, CI/CD, pipelines, etc.
Projects
Status: Backlog
Development

No branches or pull requests

1 participant