Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve memory efficiency of base64-encoded byte array decoding operations #967

clementdenis opened this issue Sep 3, 2015 · 2 comments


None yet
5 participants
Copy link

commented Sep 3, 2015

Currently, decoding base64-encoded byte array with the Java API client is very inefficient in terms of memory usage.

Reading big attachments from the GMail API in memory-constrained environments like App Engine is quite a challenge because of that.

On a 64 bit Java 7 VM, it required at least 115 MB of heap (-Xmx115M) to read a 12Mb attachment from the Gmail API (which is a 17MB base64 string in the response from the API).

The code to test it is dead simple (MESSAGE_ID / ATTACHMENT_ID references a big attachment):

gmail.users().messages().attachments().get("me", MESSAGE_ID, ATTACHMENT_ID).execute()

Here is the stacktrace when trying to load the attachment with only 110MB or heap:
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange(
at java.lang.String.(
at java.lang.StringBuilder.toString(
at com.fasterxml.jackson.core.util.TextBuffer.contentsAsString(
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText(
at LoadEmail.main(

Obtaining a stream from the data field in the response would help keep the memory usage lower.


This comment has been minimized.

Copy link

commented Jun 2, 2017

Any update or workaround about that important concern?
Ideally, providing similar API than for downloading file as stream from drive would be a good solution.


This comment has been minimized.

Copy link

commented Jun 5, 2019

Use a manual parsing code:

Gmail.Users.Messages.Attachments.Get get = api.getGmail().users().messages()
                .attachments().get(ME, gid, attachId);

// Very important!!

InputStream is = get.executeAsInputStream();

is = InputStreamJsonField.getBase64DataStream(is, "data");

And InputStreamJsonField is just a custom InputStream wrapper that ignores the first JSON chars and delegates to org.apache.commons.codec.binary.Base64InputStream.Base64InputStream(InputStream)

 * Very simple utility to stream data from a single field JSON,
 * now only used for Gmail Api where we can get a JSON like this:
 * In a single line without spaces, then we can easily extract the data
public class InputStreamJsonField extends InputStream {
    private String field;
    private String expectedPrefix;
    private InputStream is;

    public static InputStream getBase64DataStream(InputStream is, String field) {
        return new Base64InputStream(new InputStreamJsonField(is, field));

    public InputStreamJsonField(InputStream is, String field) {
        this.field = field; = is;

    public int read() throws IOException {

        if (expectedPrefix == null) {
            // i.e: {"FIELD":"DATA_RETURNED"}
            expectedPrefix = "{\"" + field + "\":\"";

            byte[] buff = new byte[expectedPrefix.length()];

            String prefix;
            int pos = 0;
            do {
                int c =;

                if (c == -1) return -1;

                if (c == ' ' || c == '\t' || c == '\n' || c == '\r') {
                    // swallow all blanks
                } else {
                    buff[pos++] = (byte) c;

                prefix = new String(buff, 0, pos);
                if (!expectedPrefix.startsWith(prefix)) {
                    // error

            } while (!expectedPrefix.equals(prefix));

            if (!prefix.equals(expectedPrefix)) {
                throw new IllegalStateException(prefix + " != " + expectedPrefix);


        int c =;
        if ('"' == c) {
            // read the }
            c =;
            // read -1 EOF
            c =;
        return c;

    public void close() throws IOException {
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.