New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Script: ulong via fields API #76519

Merged

stu-elastic merged 16 commits into elastic:master from stu-elastic:script/69742-06-ulong-as

Aug 17, 2021

Contributor

stu-elastic commented Aug 13, 2021

Exposes unsigned long via the fields API.

Unsigned longs default to java signed longs. That means the upper range
appears negative. Consumers should use Long.compareUnsigned(long, long)
Long.divideUnsigned(long, long) and Long.remainderUnsigned(long, long)
to correctly work with values known to be unsigned long.

Alternatively, users may treat the unsigned long type as BigInteger using
the field API, field('ul').as(Field.BigInteger).getValue(BigInteger.ZERO).

field('ul').as(Field.BigInteger).getValue(BigInteger.valueOf(1000))
field('ul').getValue(1000L)

This change also implements the beginning of the converters for the fields
API. The following conversions have been added:

ulong <-> BigInteger
long <-> BigInteger
double -> BigInteger
String (parsed as long or double) -> BigInteger
double -> long
String (parsed as long or double) -> long
Date (epoch milliseconds) -> long
Nano Date (epoch nanoseconds) -> long
boolean (1L for true, 0L for false) -> long

Fixes: #64361

stu-elastic added 2 commits

August 13, 2021 18:07


          Script: ulong via fields API

e8663a9

Exposes unsigned long via the fields API.

Unsigned longs default to java signed longs.  That means the upper range
appears negative.  Consumers should use `Long.compareUnsigned(long, long)`
`Long.divideUnsigned(long, long)` and `Long.remainderUnsigned(long, long)`
to correctly work with values known to be unsigned long.

Alternatively, users may treat the unsigned long type as `BigInteger` using
the field API, `field('ul').as(Field.BigInteger).getValue(BigInteger.ZERO)`.
```
field('ul').as(Field.BigInteger).getValue(BigInteger.valueOf(1000))
field('ul').getValue(1000L)
```

This change also implements the beginning of the converters for the fields
API.  The following conversions have been added:
```
ulong <-> BigInteger
long <-> BigInteger
double -> BigInteger
String (parsed as long or double) -> BigInteger
double -> long
String (parsed as long or double) -> long
Date (epoch milliseconds) -> long
Nano Date (epoch nanoseconds) -> long
boolean (1L for true, 0L for false) -> long
```

Fixes: elastic#64361


          Merge branch 'master' of github.com:elastic/elasticsearch into script…

f3f5700

…/69742-06-ulong-as

stu-elastic added >enhancement :Core/Infra/Scripting v8.0.0 v7.15.0 labels

stu-elastic requested review from rjernst and jdconrad

August 13, 2021 23:08

elasticmachine added the Team:Core/Infra label

Collaborator

elasticmachine commented Aug 13, 2021

Pinging @elastic/es-core-infra (Team:Core/Infra)


          Script: dense vector not implemented yet

7adc6d4

This was referenced Aug 13, 2021

Script: ulong support via Fields API (boxing) #76260

Closed

Script: ulong support via Fields API #76205

Closed

stu-elastic added the auto-backport label

stu-elastic added 3 commits

August 16, 2021 10:14


          javadoc

60ebf61


          Merge branch 'master' of github.com:elastic/elasticsearch into script…

5155e51

…/69742-06-ulong-as


          don't cast EmptyField, create one

29dad53

jdconrad approved these changes

View reviewed changes

Contributor

jdconrad left a comment

Walked through this fully with @stu-elastic and this looks great! Thanks for this change.

stu-elastic added 6 commits

August 16, 2021 14:01


          doc['ul'].value syntax

4904abc


          remove date tests

aa52c00


          date milli and nano tests

2b1dc8a


          remove unused TimeUnit

69ec48d


          lenient big integer parsing

b5a2965


          revert number parsing fanciness

aef03a1

rjernst approved these changes

View reviewed changes

Member

rjernst left a comment

Looks good! Most of my comments are small nits.

server/src/main/java/org/elasticsearch/script/Converter.java Outdated

+               * Converts between one scripting {@link Field} type and another,  {@code CF}, with a different underlying
+               * value type, {@code CT}.
+               */
+              public interface Converter<CT, CF extends Field<CT>> {

Member

rjernst Aug 16, 2021

nit: maybe these should be TC for target class and FC for Field class?

Contributor Author

stu-elastic Aug 17, 2021

Changed.

server/src/main/java/org/elasticsearch/script/Converters.java Outdated

+                  // No instances, please
+                  private Converters() {}
+                  public static BigIntegerField LongToBigInteger(LongField sourceField) {

Member

rjernst Aug 16, 2021

These can be private (or package private if we want tests for these), since they are only called in this file right?

Contributor Author

stu-elastic Aug 17, 2021

Changed to package private.

server/src/main/java/org/elasticsearch/script/Converters.java

+               * {@link Converters} for scripting fields.  These constants are exposed as static fields on {@link Field} to
+               * allow a user to convert via {@link Field#as(Converter)}.
+               */
+              public class Converters {

Member

rjernst Aug 16, 2021

I think this can be package private since it is only accessed by Field?

Contributor Author

stu-elastic Aug 17, 2021

It's used by by UnsignedLongField in x-pack, so we have to leave it public.

server/src/main/java/org/elasticsearch/script/Converters.java Outdated

+                          }
+                          @Override
+                          public java.math.BigInteger getNonPrimitiveValue() {

Member

rjernst Aug 16, 2021

nit: why are these fully qualified?

Contributor Author

stu-elastic Aug 17, 2021

Removed package prefixes.

server/src/main/java/org/elasticsearch/script/Converters.java Outdated

+                      return new BigIntegerField(sourceField.getName(), new DelegatingFieldValues<BigInteger, String>(fv) {
+                          protected long parseNumber(String str) {
+                              try {
+                                  return Long.parseLong(str);

Member

rjernst Aug 16, 2021

This will fail on values larger than Long.MAX_VALUE, and then lose precision (or fail again) when trying to parse as double.

I think for string, since the value can be arbitrarily large, we should completely use BigInteger and BigDecimal. So something like this:

try {
    return new BigInteger(str);
} catch (NumberFormatException e) {
    return new BigDecimal(str).toBigInteger();
}

Contributor Author

stu-elastic Aug 17, 2021

Ahh nice, changed.

server/src/main/java/org/elasticsearch/script/Converters.java Outdated

+                          public double getDoubleValue() {
+                              String str = values.getNonPrimitiveValue();
+                              try {
+                                  return Double.parseDouble(str);

Member

rjernst Aug 16, 2021

What's the reason for first trying double? The underlying data could likewise be a BigInteger or BigDecimal that isn't parsable as long. The user asked to convert to long, so if the data can't fit it, it seems like getting a NumberFormatException is appropriate?

Contributor Author

stu-elastic Aug 17, 2021

Makes sense, changed to using getLongValue(), which will attempt to parse the long.

server/src/main/java/org/elasticsearch/script/Field.java Outdated

+                  /**
+                   * Convert this {@code Field} into another {@code Field} type using a {@link Converter} of the target type.
+                   *
+                   * As this is called from user scripts, {@code as} may be called to convert a field into it's same type, if

Member

rjernst Aug 17, 2021

nit: it's -> its

Contributor Author

stu-elastic Aug 17, 2021

Changed.

server/src/main/java/org/elasticsearch/script/Field.java Outdated

+                   * Convert this {@code Field} into another {@code Field} type using a {@link Converter} of the target type.
+                   *
+                   * As this is called from user scripts, {@code as} may be called to convert a field into it's same type, if
+                   * so {@code this} is cast via that converters {@link Converter#getFieldClass()}.

Member

rjernst Aug 17, 2021

nit: I think "that" can be removed

Contributor Author

stu-elastic Aug 17, 2021

Removed.

...pper-unsigned-long/src/main/java/org/elasticsearch/xpack/unsignedlong/UnsignedLongField.java Outdated

+                  // UnsignedLongFields must define their own conversions as they are in x-pack
+                  @Override
+                  public <CT, CF extends Field<CT>> Field<CT> as(Converter<CT, CF> converter) {
+                      if (converter.getFieldClass().isInstance(this)) {

Member

rjernst Aug 17, 2021

It's unfortunate we have to repeat this. I wonder if we could instead have as be final on Field, and have an implementation method (eg convert) that is protected which can be overridden here, so that the identity check can always be first.

Contributor Author

stu-elastic Aug 17, 2021

Added convert which is what subclasses should override.

...pper-unsigned-long/src/main/java/org/elasticsearch/xpack/unsignedlong/UnsignedLongField.java

+                          return converter.getFieldClass().cast(bigIntegerField);
+                      }
+                      return super.as(converter);

Member

rjernst Aug 17, 2021

Should there be a special case for converting to Long, since the getLongValue() doesn't return the right long in this case? ie if the value is under Long.MAX_VALUE then long is returned, otherwise an error should be raised, not a negative number returned? String would be similar, otherwise large values would show up as negative numbers?

Contributor Author

stu-elastic Aug 17, 2021 •

edited

Loading

UnsignedLongScriptDocValues/UnsignedLongField returns longs with exactly this property, values may be negative.

I expect diligent users will perform "confirming" type conversions and never take the original type from field('fieldname'), in this case the a user who wants to handle possible negative values themselves would only be able to do a field('ul').as(Field.Long) confirming type conversion.

If we want to allow field('ul').as(Field.UnsignedLong), we'll need to implement Field augmentation for painless.

Member

rjernst Aug 17, 2021

I understand the complication with Long type since that is the underlying native type returned here, but I don’t think conversion to string will work correctly? It should not show negative values, yet I think it will since it would just stringify the raw long value. A minimal test for this conversion would be good.

Contributor Author

stu-elastic Aug 17, 2021

Added minimal test.

Member

rjernst Aug 17, 2021 •

edited

Loading

Can you point at the test? I can't find it. I was specifically talking about conversion to string, to ensure that unsigned long values do not show up as negative values with this conversion.

Contributor Author

stu-elastic Aug 17, 2021

This PR does not implement conversion to String fields, only from String fields. To BigInteger and to Long are implemented here.

Member

rjernst commented Aug 17, 2021

One general comment I forgot to make is I think we should have tests for every method/conversion we have exposed. I realize that is a lot, it doesn’t have to all come in this PR, but at least some minimal verification that this all works as we think it does.

stu-elastic added 4 commits

August 17, 2021 07:58


          pr comments

e60fe36


          minimal UnsignedLongField.as(Field.Long) test

5d82d1a


          unused import


          format

369e58b

stu-elastic merged commit aea8bff into elastic:master

Collaborator

elasticsearchmachine commented Aug 17, 2021

💔 Backport failed

Status	Branch	Result
❌	7.x	Commit could not be cherrypicked due to conflicts

To backport manually run backport --upstream elastic/elasticsearch --pr 76519

stu-elastic added a commit to stu-elastic/elasticsearch that referenced this pull request


          Script: ulong via fields API (elastic#76519)

f64d370

Exposes unsigned long via the fields API.

Unsigned longs default to java signed longs.  That means the upper range
appears negative.  Consumers should use `Long.compareUnsigned(long, long)`
`Long.divideUnsigned(long, long)` and `Long.remainderUnsigned(long, long)`
to correctly work with values known to be unsigned long.

Alternatively, users may treat the unsigned long type as `BigInteger` using
the field API, `field('ul').as(Field.BigInteger).getValue(BigInteger.ZERO)`.
```
field('ul').as(Field.BigInteger).getValue(BigInteger.valueOf(1000))
field('ul').getValue(1000L)
```

This change also implements the beginning of the converters for the fields
API.  The following conversions have been added:
```
ulong <-> BigInteger
long <-> BigInteger
double -> BigInteger
String (parsed as long or double) -> BigInteger
double -> long
String (parsed as long or double) -> long
Date (epoch milliseconds) -> long
Nano Date (epoch nanoseconds) -> long
boolean (1L for true, 0L for false) -> long
```

Backport: aea8bff

stu-elastic added a commit that referenced this pull request


          Script: ulong via fields API (#76519) (#76613)

5491d94

Exposes unsigned long via the fields API.

Unsigned longs default to java signed longs.  That means the upper range
appears negative.  Consumers should use `Long.compareUnsigned(long, long)`
`Long.divideUnsigned(long, long)` and `Long.remainderUnsigned(long, long)`
to correctly work with values known to be unsigned long.

Alternatively, users may treat the unsigned long type as `BigInteger` using
the field API, `field('ul').as(Field.BigInteger).getValue(BigInteger.ZERO)`.
```
field('ul').as(Field.BigInteger).getValue(BigInteger.valueOf(1000))
field('ul').getValue(1000L)
```

This change also implements the beginning of the converters for the fields
API.  The following conversions have been added:
```
ulong <-> BigInteger
long <-> BigInteger
double -> BigInteger
String (parsed as long or double) -> BigInteger
double -> long
String (parsed as long or double) -> long
Date (epoch milliseconds) -> long
Nano Date (epoch nanoseconds) -> long
boolean (1L for true, 0L for false) -> long
```

Backport: aea8bff

Contributor Author

stu-elastic commented Aug 17, 2021

master (8.x): aea8bff
7.x (7.15.0): 5491d94

stu-elastic mentioned this pull request

QL: Add support for unsigned_long field type #63312

Closed

jakelandis added v8.0.0-alpha2 and removed v8.0.0 labels

jdconrad mentioned this pull request

Update Painless Fields API design to use mapped types #78950

Merged

mayya-sharipova mentioned this pull request

Add scripted_metric agg context to unsigned_long #64422

Merged

mayya-sharipova mentioned this pull request

Document script_values for unsigned_long #89627

Open

2lambda123 pushed a commit to 2lambda123/elastic-elasticsearch that referenced this pull request


          Document script_values for unsigned_long

6b0d03e

We introduced script values support for unsigned_long from 7.15
in elastic/elasticsearch#76519.
But forgot add this document for 7.x.

This just backport documentation from
elastic/elasticsearch#64422.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment