BQ Storage API: backend does not support small query results

[yan-hic]

Similar to https://stackoverflow.com/questions/57367139/bigquery-storage-api-the-table-has-a-storage-format-that-is-not-supported, we experience the same issue using the Python client, namely, presumably, the backend cannot serialize small query results saved in the temporary anonymous table.

See in our case:

[2020-01-17 15:04:42,088] {{base_task_runner.py:115}} INFO - Job 4337: Subtask calculate_tax [2020-01-17 15:04:42,088] {{taxcalcs.py:104}} INFO - Loading 19 rows from _9a9427bb1813a244d22255961ee8725af72f9feb.anond337c45b_401a_4358_9195_97f9a7f27b38
[2020-01-17 15:04:42,418] {{taskinstance.py:1058}} ERROR - 400 request failed: the table has a storage format that is not supported
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 824, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 726, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    status = StatusCode.FAILED_PRECONDITION
    details = "request failed: the table has a storage format that is not supported"
    debug_error_string = "{"created":"@1579273482.418031949","description":"Error received from peer ipv4:xxxxx:443","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"request failed: the table has a storage format that is not supported","grpc_status":9}"

I understand the workaround is to create "temporarily" a permanent table on the client-side - which is suboptimal - but since issue still exists months after, maybe the client library should handle e.g. fallback to regular readrows().

Thoughts ?

[shollyman]

Following up on this one. As of this week, the BigQuery storage API no longer has the limitation of being sometimes unable to create sessions for small query results.