[BUG] Unable to register extension with OpenSearch version 2.15.0 #166
Comments
mzainab24
added
bug
|
Hey @mzainab24 thanks for this report. It's been a while since I mucked with this code, but it looks like the transport handshakes use the 3.0.0 version: I took a quick look at the Version Class in OpenSearch and it's a bit confusing to me how all that math works out. But I'd suggest as a first try, set that Version constant in this extension to 2150000 and see if that resolves the issue. |
|
OK, spent a little bit more time looking into this. I think this is indeed a bug, specifically: this extension uses the same version in the response as it received in the request: The version being sent, however, is a "minCompatVersion" rather than the current one. } else if (version.onOrAfter(Version.V_2_0_0)) {
minCompatVersion = Version.fromId(7099999);
}
handshakeRequestSender.sendRequest(node, channel, requestId, minCompatVersion);This runs afoul of 2.x version checking which requires 7.10.0 minimum. That version checking happened to work on 3.x but doesn't work on 2.x. TLDR, we should not use the inbound "min compatibility" in our response. I think if we pass |
|
@dbwiddis Isn't there an easier workaround for @mzainab24 in the extension itself? |
Nothing really configurable, this is our hardcoded TCP handshake. We just sent back the same version which worked on 3.x so I never really dug into why it worked and assumed it would. It's clearly not the same thing. I believe the bugfix is probably to change this line to something else, probably I haven't had any time to set it up to test yet. |
|
@dbwiddis and @dblock thank you for the insights. I am currently using the opensearch-sdk-py as a Python library and building my extension on top of it. Could I inquire if there are any upcoming commits that might address this issue? Another thing to mention is that the extension registration works perfectly fine when I clone the whole code of OpenSearch, the latest version 2.15.0 from https://github.com/opensearch-project/OpenSearch.git, and in that way I do not have to change the below version in the json file. |
|
@mzainab24 I can certainly make a PR to change that version number, but as mentioned I haven't had time to test it so it's an informed guess that it may fix the problem. I'll make that commit now. |
dbwiddis
removed
the
untriaged
|
@dblock @mzainab24 FYI, I just spent some time digging through all the related Java code (again) and it's a bit more complex. The proposed PR fix might work as a quick unblocker but is not the right long term fix, which I'm still trying to wrap my head around. Key point is that there are two different version numbers being exchanged in these handshakes. Here's the inbound bytes from the handshake header from a 3.0.0 cluster: It's easier to read with Netty debug. Here's just the header part. The Here's the full handshake: The As an extension we are not supposed to care about compatibility, and since the transport layer hasn't changed, we don't have to. So we can send back the same version (which we tried to do) as long as we distinguish between the two types. The bug here is that we were sending back the minimum compatible version in both fields. This happened to work on 3.x because the minimum compatible version being sent was the lastest 2.x version (2.10.0) in this case so the extension was responding "Hey, I'm running version 2.10.0 and that's my minimum compatible!" Except when we connect from a 2.x cluster, the min compat is (ES) 7.9 and we're reporting that as our minimum compatible (OK) and our current version (not OK, as OpenSearch 2.x is only compatible with the latest (7.10) previous minor version. TLDR: the PR fix as it is, is wrong.
So the only change needed is TransportHandshakerHandshakeResponse(Version(Version.CURRENT)),but we can also hardcode the min compat version to 7.9 if we want, which I think I'll do... I'll build in some more tests and docs this weekend. |
|
@dbwiddis 👨🎤 thanks for fixing this |
What is the bug?
I have a running distribution of OpenSearch version 2.15.0 which was installed using the method on the following link.
https://opensearch.org/docs/latest/install-and-configure/install-opensearch/tar/
I enabled the experimental extension feature in the opensearch.yml file and then run the distribution.
Then while trying to register the extension, it gave me version incompatibility error so I changed the versions in hello.json extension sample file.
After that, when I again tried to register the extension, it gave the following errors on the running OpenSearch and the registration was unsuccessful, as I could not get a response from the sample extension.
How can one reproduce the bug?
https://opensearch.org/docs/latest/install-and-configure/install-opensearch/tar/
opensearch.yml. Search foropensearch.experimental.feature.extensions.enabled, uncomment it, and set it totrue.https://github.com/opensearch-project/opensearch-sdk-py/tree/main/samples/hello
What is the expected behavior?
Error logs on OpenSearch.
What is your host/environment?
Distributor ID: Ubuntu
Description: Ubuntu 22.04.4 LTS
Release: 22.04
Architecture: x86-64
Do you have any screenshots?
If applicable, add screenshots to help explain your problem.
Do you have any additional context?
Add any other context about the problem.
The text was updated successfully, but these errors were encountered: