Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove getClassLoader permissions in plugin-security.policy #656

Merged
merged 12 commits into from
Mar 23, 2024

Conversation

alexklibisz
Copy link
Owner

@alexklibisz alexklibisz commented Mar 23, 2024

Related Issue

#393

Changes

Avoiding the use of scala's Array and ArrayBuffer because they require the use of ClassTag, which requires extra permissions to be used in Elasticsearch. These permissions have been enabled for a while now, and I went down the rabbithole of potentially adding more for Scala 3, so I'd rather get rid of them now that I know how.

Specifically:

  • changing vector parsing from using ArrayBuffer to using custom specialized Java implementations. Also added a trick to set the array buffer's initial array capacity to be the final capacity of the previous array. This exploits the fact that we're usually indexing and querying vectors of the same length. This is about ~2.8x faster than using a scala ArrayBuffer initialized to the final size, and ~3.3x faster than using a scala ArrayBuffer initialized to the default size. But this only shows up in a microbenchmark. It doesn't show up in the ann-benchmarks as JSON deserialization is not really a bottleneck.
  • changing some plugin internals that were mapping and flatMapping Arrays to instead use foreach with ListBuffers. This avoids the invocation of methods that require a ClassTag.

Testing and Validation

Standard CI and benchmarking.

@alexklibisz alexklibisz changed the title Plugin: optimize decoder array buffers Plugin: stop using scala arrays so we can get rid of extra permissions in plugin-security.policy Mar 23, 2024
@alexklibisz alexklibisz marked this pull request as ready for review March 23, 2024 21:25
@alexklibisz alexklibisz changed the title Plugin: stop using scala arrays so we can get rid of extra permissions in plugin-security.policy Stop using scala arrays so we can get rid of extra permissions in plugin-security.policy Mar 23, 2024
@alexklibisz alexklibisz changed the title Stop using scala arrays so we can get rid of extra permissions in plugin-security.policy Remove getClassLoader permissions in plugin-security.policy Mar 23, 2024
@alexklibisz alexklibisz merged commit 6434945 into main Mar 23, 2024
5 checks passed
@alexklibisz alexklibisz deleted the decoder-array-buffers branch March 23, 2024 21:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant