spanner: Session not found on Read with KeySet

[ericwenn]

Client

Spanner (v1.7.0)

Environment

Cloud Run (fully managed)

Code

keySet := spanner.KeySets(
	spanner.Key{ /* ... */ },
	spanner.Key{ /* ... */ },
	spanner.Key{ /* ... */ }
)
err = spannerClient.Single().Read(
	ctx,
	/* ... */,
	keySet,
	/* ... */,
).Do(/* ... */)

Expected behavior

Request should succeed, and the client handles maintaining the session pool.

Actual behavior

Intermittent failure, client returns:

code = "NotFound", desc = "Session not found: projects/xxx/instances/xxx/databases/xxx/sessions/AN4G3x93_EfsUJcOlBpnvbOmZELeqImNJjvJv2UiEX5r0rvMzqTU1Juhug7qOA"

Additional context
Re-running the same request succeeds but sometimes, very seldom, the error is returned.
There are many Read() calls being made by our service concurrently in the call-chain, and when the error occurs it is only for one of the calls.

[skuruppu]

@olavloite, would you please be able to take a look at this one?

CC @hengfengli

[ericwenn]

@olavloite thanks for quick fix.

Have some more context in case you need it:

• Issue is not related to using KeySet as the title of this issue indicates. We've seen the same issue when using KeyRanges as well.
• We are using the default Spanner config with not tuning of the parameters.

[olavloite]

@ericwenn
Thanks for the additional information. I've been able to confirm and reproduce the issue, and as you noted, it can occur regardless of whether you use a KeySet or KeyRange. We added automatic retries for Session not found earlier in the client library for most methods, but the Read method on a single-use read-only transaction was not included in that fix. That also explains why you are only experiencing this error on this specific method occasionally.

[ericwenn]

Thanks for quick triage and turnaround! Looking forward to bumping our service to next release 🚀