TypeError: decoded.map is not a function

[splawndive]

Environment details

• OS: Google Container-Optimized OS
• Node.js version: 14.15 (also experienced on 12.18)
• npm version: 6.14
• @google-cloud/spanner version: 5.7.0

Steps to reproduce

1. ?
2. ?

We're seeing an issue more-or-less identical to this one opened by a coworker of mine a ways back: #198. It happens pretty inconsistently, so I'm not sure of how exactly to reproduce it. The error message looks like:

TypeError: decoded.map is not a function at Object.decode (/opt/app/node_modules/@google-cloud/spanner/build/src/codec.js:299:31)

Possibly related, we're seeing a separate error message, though much less frequently:

TypeError: tail.shift is not a function at Function.mergeLists (/opt/app/node_modules/@google-cloud/spanner/build/src/partial-result-stream.js:239:77)

The code in our application looks something like this:

// query returns ~90k rows -- each `arrayOfStrings` value consists of 0-4 short strings
const stream = spannerDb.runStream({
  sql: '
    SELECT string1, string2, bool1, arrayOfStrings
    FROM <table>
  ',
  json: true,
});

Please let me know what other information may be helpful

[skuruppu]

@olavloite would you be able to take a look and see if you can reproduce this issue? I think you did some investigations in the stream code before so I thought you might have some luck.

[olavloite]

It seems that this is caused by something in the decoding of chunked string arrays. A string array can be split over multiple PartialResultSets, and it seems that this can cause this error when the array is split in some specific way. That also explains why it is not easy to reproduce it consistently, as there is no guarantee exactly where/how Cloud Spanner will chunk the returned values.

I'll try to find the exact chunking that is needed to reproduce this problem.

EDIT: The problem is only partly related to decoding chunked string arrays. The problem occurs in the following scenario:

1. The client receives a stream from Cloud Spanner where the last value that it receives is chunked.
2. The client has not yet received a resume token.
3. The stream breaks with a retryable error (UNAVAILABLE).
4. The query is retried, but the status that the last value was chunked is not cleared, which means that during the retry it will try to merge the first value with the last value that it received during the first attempt. That is however not possible, as the entire stream is restarted because of the lack of a resume token.

---

TypeError: decoded.map is not a function at Object.decode (/opt/app/node_modules/@google-cloud/spanner/build/src/codec.js:299:31)

will be thrown if the chunked value was one of the string values (not one of the strings in the array).

---

TypeError: tail.shift is not a function at Function.mergeLists (/opt/app/node_modules/@google-cloud/spanner/build/src/partial-result-stream.js:239:77)

will be thrown if the chunked value was the array of strings.

[skuruppu]

Nice work @olavloite. Is there anything we can do from the client side to prevent the error? I.e. clear the state such that the client doesn't attempt to merge values from the previous attempt? Or does that happen in the gax layer?

[olavloite]

Nice work @olavloite. Is there anything we can do from the client side to prevent the error? I.e. clear the state such that the client doesn't attempt to merge values from the previous attempt? Or does that happen in the gax layer?

It happens in the client library, so we can (must) fix it there. I'm currently working on a fix, but it is a little bit more complicated than I had first anticipated. I think I have a solution now, but it needs some more testing. I'll try to put in a fix today.

[splawndive]

Thanks for getting this one fixed so quickly @skuruppu @olavloite -- we'll try out this version next week and report back

[olavloite]

Thanks for getting this one fixed so quickly @skuruppu @olavloite -- we'll try out this version next week and report back

@splawndive Did you have a chance to try this version? And if so, were you able to detect any improvement?