-
Notifications
You must be signed in to change notification settings - Fork 16
Node position offset is not bytes #122
Comments
@m09 Thanks for reporting it! It is indeed a bug in the Java driver. Moving to that repository. |
@dennwc Would you be so kind to elaborate, why do you think it's not a bug in JS driver instead? |
@bzz Babelfish defines positions as byte offsets, and it's consistent with how JS does this, according to the bug report:
|
I should add that we maintain custom logic to do the conversion (since the bytes offset is rarely what we want) in several projects right now. I think an option to specify whether we want utf8 or bytes position would be appreciated by the ML team (and utf8 default would be the handiest for us). |
I'm playing the devil's advocate here, so bare with me for a while pease - my question was more like: how is the user supposed to figure this one out, without asking? :) I know the answer, https://doc.bblf.sh/uast/semantic-uast.html#position but I also think it may be a signal for us, that it's something not clear/under-communicated to our users (even given the fact of bblfsh/libuast#102) I also presume that this issue could be an opportunity to verify and post results of this behavior across all the drivers - who knows, may be as with string and number literals before, it could be treated differently in more different drivers. WDYT? |
@m09 I'm working on the conversion function that returns UTF-8 offsets for the Python client, but it may take some time. @bzz You are right, the docs should give a bit more insight into how positions work. Also, it's definitely worth adding a test for UTF-8 offsets. I remember doing a pass over some drivers, but maybe I missed Java that time. |
…h#122 Signed-off-by: Denys Smirnov <denys@sourced.tech>
…h#122 Signed-off-by: Denys Smirnov <denys@sourced.tech>
Signed-off-by: Denys Smirnov <denys@sourced.tech>
In Java, the position is returned based on the offset in the utf8 content, as shown here (no variation between an ascii string and a more complex utf8 string):
381
:381
:In JS, the position is returned based on the offset in the bytes content, as shown here (variation between an ascii string and a more complex utf8 string):
93
:97
:The text was updated successfully, but these errors were encountered: