Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BigQuery Storage Write API: JsonStreamWriter cannot transform BigDecimal with 9+ decimals to BigNumeric #2013

Closed
roy-tc opened this issue Feb 23, 2023 · 0 comments · Fixed by #2031 or #2033
Labels
api: bigquerystorage Issues related to the googleapis/java-bigquerystorage API.

Comments

@roy-tc
Copy link
Contributor

roy-tc commented Feb 23, 2023

Environment details

  1. OS type and version: OSX 11.6.3
  2. Java version: OpenSDK 11
  3. version(s): v2.28.3 and later

Steps to reproduce

  1. Define a TableSchema with a field of type BIGNUMERIC
  2. Serialize a JSON message to Proto containing more than 9 decimals, eg. the number 24.6789012345

Code example

This unit test could be added to JsonToProtoMessageTest to reproduce it:

@Test
  public void testBigDecimalToBigNumericConversion() {
    TableSchema ts =
        TableSchema.newBuilder()
            .addFields(
                0,
                TableFieldSchema.newBuilder()
                    .setName("bignumeric")
                    .setType(TableFieldSchema.Type.BIGNUMERIC)
                    .setMode(TableFieldSchema.Mode.REPEATED)
                    .build())
            .build();
    TestBignumeric expectedProto =
        TestBignumeric.newBuilder()
            .addBignumeric(
                BigDecimalByteStringEncoder.encodeToBigNumericByteString(
                    new BigDecimal("24.6789012345")))
            .build();
    JSONObject json = new JSONObject();
    json.put("bignumeric", Collections.singletonList(new BigDecimal("24.6789012345")));
    DynamicMessage protoMsg =
        JsonToProtoMessage.convertJsonToProtoMessage(TestBignumeric.getDescriptor(), ts, json);
    assertEquals(expectedProto, protoMsg);
  }

Stack trace

[ERROR] com.google.cloud.bigquery.storage.v1.JsonToProtoMessageTest.testBigDecimalToBigNumericConversion  Time elapsed: 0.007 s  <<< ERROR!
com.google.cloud.bigquery.storage.v1.Exceptions$FieldParseError: ByteString scale cannot exceed 9: 24.6789012345
	at com.google.cloud.bigquery.storage.v1.JsonToProtoMessage.convertJsonToProtoMessageImpl(JsonToProtoMessage.java:247)
	at com.google.cloud.bigquery.storage.v1.JsonToProtoMessage.convertJsonToProtoMessage(JsonToProtoMessage.java:139)
	at com.google.cloud.bigquery.storage.v1.JsonToProtoMessageTest.testBigDecimalToBigNumericConversion(JsonToProtoMessageTest.java:1437)
...

Any additional information below

On the following lines, encodeToBigNumericByteString should be called instead of encodeToNumericByteString:

If you agree on this proposed approach, we can create PR with the following fix:
main...tradecloud:java-bigquerystorage:TC-8710-serialize-bigdecimal

@product-auto-label product-auto-label bot added the api: bigquerystorage Issues related to the googleapis/java-bigquerystorage API. label Feb 23, 2023
gcf-merge-on-green bot pushed a commit that referenced this issue Mar 8, 2023
…2031)

- [x] Make sure to open an issue as a [bug/issue](https://togithub.com/googleapis/java-bigquerystorage/issues/new/choose) before writing your code!  That way we can discuss the change, evaluate designs, and agree on the general idea
- [x] Ensure the tests and linter pass
- [x] Code coverage does not decrease (if any source code was changed)
- [x] Appropriate docs were updated (if necessary)

Fixes #2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment