Java Client byte[] encoding throws Illegal character

[Burtan]

Hi,
when trying to send byte[] parameters as body information to appengine I'm getting the following error:

com.google.api.client.googleapis.json.GoogleJsonResponseException: 400 Bad Request
{
  "code" : 400,
  "errors" : [ {
    "domain" : "global",
    "message" : "com.fasterxml.jackson.databind.JsonMappingException: Illegal character '_' (code 0x5f) in base64 content\n at [Source: N/A; line: -1, column: -1] (through reference chain: backend.apis.PutSongRequest[\"sheetsData\"])",
    "reason" : "badRequest"
  } ],
  "message" : "com.fasterxml.jackson.databind.JsonMappingException: Illegal character '_' (code 0x5f) in base64 content\n at [Source: N/A; line: -1, column: -1] (through reference chain: backend.apis.PutSongRequest[\"sheetsData\"])"
}

In another project, receiving byte[] works fine.

[Burtan]

I only get this error when I use the "encode" function by the java client library. When I encode it myself using java.util.Base64.getEncoder().encodeToString(), it is working fine.

[tangiel]

Can you be more specific about the Java client encode? Specifically which class it's in?

[Burtan]

When you create a class to use in the body of an endpoint api, the java client generates a class for it. If you specify a parameter byte[] in the endpoint api body, the java client generated class will have a function encodeparameter with byte[] as value and a function set parameter with string as function.

[Burtan]

There seem to be changes in the javascript client behaviour as well. My images are not displayed correctly anymore because the resulting base64string from the byte[] remains url encoded.

[tangiel]

I believe 2.0.5 should fix this, but let me know if it doesn't.

[Burtan]

The changes in the javascript client bahaviour were introduced with 2.0.5. 2.0.4 is fine.

[tangiel]

@Burtan can you give an example of what you mean? Seems like base64 shouldn't be url encoded.

[Burtan]

I have small thumbnails that I send from the backend as byte[], so they get encoded as base64 for transmission. The datastore content of an example is the following:
/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAUDB...
With 2.0.5 the resulting base64 string is url encoded like this:
_9j_4AAQSkZJRgABAQAAAQABAAD_2wBDAAUDB...
With 2.0.4 it remains the same as in the datastore

[Burtan]

The log for 2.0.5 says

use web-safe base64 encoding for byte arrays

I think that is the relevant change. But if web-safe base64 encoding is wanted for transmission, the client should convert it back to the standard base64 style.

[tangiel]

@Burtan in the initial comment, are you sending the base64 as a body field or a query parameter? Perhaps the client changes the base64 encoding based on the location of the field.

[Burtan]

The code in the initial comment was similiar to this:

PutByteRequest request = new PutByteRequest()
request.setByteArray(bytes)

I guess this sends the data as a body field.

[Burtan]

Does this help?

[tangiel]

@Burtan apologies for the slowness in replying. Can you show the @ApiMethod and method signature that has issues? Let's recap the current state. It sounds like you're saying the base64 using - and _ is not recognized by the JS client or Java client. Is that correct?

[Burtan]

@ApiMethod(httpMethod = ApiMethod.HttpMethod.POST)
public RbRefResponse getRbRef(RbRefRequest req, User user) {
}

public static class RbRefRequest {
    public String rbId;
    public Date entryTime;
    public boolean updatePurchase;
    public boolean skipProfiles;
    public int imageWidth;
    public int imageHeight;
}

public static class RbRefResponse {
    public RbRef rbRef;
}

RbRef contains various fields, one of them is an byte[]. The byte[] is transferred as a base64 String. The encoding was always non-url (using + and /). Since 2.0.5 the encoding is url (using - and _ instead). This problem occurs for the JS library and the iOS library (I just found out). It is fine in Android because it has a wrapper method decode which delivers a byte[] instead of a base64 string. My current workaround is just replacing all - by + and all / by _.