GenericJson.clone() method blows up when trying to clone Google reads

[droazen]

When we copy a Google read (ie., a com.google.api.services.genomics.model.Read), we currently call into the GenericJson.clone() method, which uses reflection to deep copy the fields. However, this method seems to incorrectly try to instantiate a java.util.Arrays$ArrayList instead of a java.util.ArrayList, causing it to blow up during BQSR runs (as reported by @jean-philippe-martin):

    [Jul-13 1:36 PM] JP Martin: 
    2015-07-13T17:36:00.151Z: Error:  (5361981f0726257c): java.lang.IllegalArgumentException: unable to create new instance of class java.util.Arrays$ArrayList possibly because it is not public
    at com.google.api.client.util.Types.handleExceptionForNewInstance(Types.java:165)
    at com.google.api.client.util.Types.newInstance(Types.java:120)
    at com.google.api.client.util.Data.clone(Data.java:220)
    at com.google.api.client.util.Data.deepCopy(Data.java:307)
    at com.google.api.client.util.Data.clone(Data.java:222)
    at com.google.api.client.util.Data.deepCopy(Data.java:293)
    at com.google.api.client.util.GenericData.clone(GenericData.java:172)
    at com.google.api.client.json.GenericJson.clone(GenericJson.java:90)
    at com.google.api.services.genomics.model.Read.clone(Read.java:548)
    at      
    org.broadinstitute.hellbender.utils.read.GoogleGenomicsReadToGATKReadAdapter.copy(GoogleGenomicsReadToGATKReadAdapter.java:619)

[droazen]

@davidadamsphd is contacting some Google folks who can hopefully shed some light on this error. If all else fails, we can file a new issue against the repository that owns GenericJson: https://github.com/google/google-http-java-client

[davidadamsphd]

This is now also tracked at:
googleapis/google-http-java-client#293

[droazen]

Is this resolved?

[jean-philippe-martin]

Yes, in the latest version of the Dataflow SDK. Once we update to it, we're good.

[droazen]

Was this included in #775?

[droazen]

@jean-philippe-martin confirms that the fix was included in #775 -- closing this!

[jean-philippe-martin]

I thought it was, but it turns out I was wrong! Thanks to David Tester for pointing it out. We have to wait for them to release their new version, or use their jar for a bit.

[jean-philippe-martin]

Mystery now completely solved! It was com.google.appengine.tools:appengine-gcs-client:0.4.4 that was asking for the latest available version of com.google.http-client:google-http-client-jackson2. This is what caused the problem to appear solved to me when in fact the fix wasn't in maven yet.

[droazen]

Reopening -- this is still a problem according to @jean-philippe-martin

[droazen]

According to @jean-philippe-martin, the following workaround is still necessary in GoogleGenomicsReadToGATKReadAdapter:

    @Override
    public GATKRead copy() {
        // workaround until https://github.com/google/google-http-java-client/issues/293 is released
        genomicsRead.setAlignedQuality(new ArrayList<>(genomicsRead.getAlignedQuality()));
        Map<String, List<String>> infoCopy = new HashMap<>();
        for (Map.Entry<String, List<String>> entry : genomicsRead.getInfo().entrySet()) {
            infoCopy.put(entry.getKey(), new ArrayList<>(entry.getValue()));
        }
        genomicsRead.setInfo(infoCopy);
        genomicsRead.getAlignment().setCigar(new ArrayList<>(genomicsRead.getAlignment().getCigar()));
        // end workaround

        // clone() actually makes a deep copy of all fields here (via GenericData.clone())
        return new GoogleGenomicsReadToGATKReadAdapter(genomicsRead.clone());
    }

[jean-philippe-martin]

The class that's giving us trouble this time is com.google.common.collect.Lists$TransformingRandomAccessList. Here's the stack trace:

Error:   (e0086ea6f7eea4e6): java.lang.IllegalArgumentException: unable to create new instance of class com.google.common.collect.Lists$TransformingRandomAccessList possibly because it is not public
    at com.google.api.client.util.Types.handleExceptionForNewInstance(Types.java:165)
    at com.google.api.client.util.Types.newInstance(Types.java:120)
    at com.google.api.client.util.Data.clone(Data.java:229)
    at com.google.api.client.util.Data.deepCopy(Data.java:302)
    at com.google.api.client.util.GenericData.clone(GenericData.java:172)
    at com.google.api.client.json.GenericJson.clone(GenericJson.java:90)
    at com.google.api.services.genomics.model.LinearAlignment.clone(LinearAlignment.java:128)
    at com.google.api.services.genomics.model.LinearAlignment.clone(LinearAlignment.java:34)
    at com.google.api.client.util.Data.clone(Data.java:212)
    at com.google.api.client.util.Data.deepCopy(Data.java:302)
    at com.google.api.client.util.GenericData.clone(GenericData.java:172)
    at com.google.api.client.json.GenericJson.clone(GenericJson.java:90)
    at com.google.api.services.genomics.model.Read.clone(Read.java:548)
    at org.broadinstitute.hellbender.utils.read.GoogleGenomicsReadToGATKReadAdapter.copy(GoogleGenomicsReadToGATKReadAdapter.java:637)
    at org.broadinstitute.hellbender.tools.dataflow.transforms.bqsr.ApplyBQSRTransform$ApplyBQSR.processElement(ApplyBQSRTransform.java:65)

(see #835 for context)

[droazen]

I've added a comment in the upstream ticket (googleapis/google-http-java-client#297) suggesting a general fix for this issue using reflection.

[akiezun]

This issue was moved to broadinstitute/gatk-dataflow#38