BigQuery Storage: "400 Request is missing 'x-goog-request-params' header" when using dictionary as table reference

[tswast]

I'm unable to create a BigQuery Storage API read session when I use a dictionary instead of a TableReference class as the first argument.

Environment details

1. Specify the API at the beginning of the title (for example, "BigQuery: ...")
   General, Core, and Other are also allowed as types
2. OS type and version.

macOS 10.14.3

3. Python version and virtual environment information: python --version

$ python --version
Python 3.6.4

4. google-cloud- version: pip show google-<service> or pip freeze

$ pip freeze | grep google
You are using pip version 18.0, however version 19.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
google-api-core==1.7.0
google-auth==1.4.1
google-auth-oauthlib==0.2.0
google-cloud-bigquery==1.8.1
google-cloud-bigquery-storage==0.2.0
google-cloud-core==0.29.1
google-resumable-media==0.3.1
googleapis-common-protos==1.5.3

# also
grpcio==1.18.0
protobuf==3.6.1

Steps to reproduce

1. ?

Code example

from google.cloud import bigquery_storage_v1beta1
client = bigquery_storage_v1beta1.BigQueryStorageClient()

# This call fails with InvalidArgument: 400 Request is missing 'x-goog-request-params' header.
session = client.create_read_session(
    {"project_id": "swast-scratch", "dataset_id": "schema_examples", "table_id": "array_struct"},
    "projects/swast-scratch")

table_ref = bigquery_storage_v1beta1.types.TableReference()
table_ref.project_id = "swast-scratch"
table_ref.dataset_id = "schema_examples"
table_ref.table_id = "array_struct"

# This call succeeds, but the previous call should have been equivalent to this one.
session = client.create_read_session(table_ref, "projects/swast-scratch")

Stack trace

In [3]: session = client.create_read_session({"project_id": "swast-scratch", "dataset_id": "schema_examples", "table_id": "array_
   ...: struct"}, "projects/swast-scratch")
---------------------------------------------------------------------------
_Rendezvous                               Traceback (most recent call last)
~/src/google-cloud-python-private/api_core/google/api_core/grpc_helpers.py in error_remapped_callable(*args, **kwargs)
     56         try:
---> 57             return callable_(*args, **kwargs)
     58         except grpc.RpcError as exc:

~/.pyenv/versions/3.6.4/envs/scratch/lib/python3.6/site-packages/grpc/_channel.py in __call__(self, request, timeout, metadata, credentials, wait_for_ready)
    549                                       wait_for_ready)
--> 550         return _end_unary_response_blocking(state, call, False, None)
    551

~/.pyenv/versions/3.6.4/envs/scratch/lib/python3.6/site-packages/grpc/_channel.py in _end_unary_response_blocking(state, call, with_call, deadline)
    466     else:
--> 467         raise _Rendezvous(state, None, None, deadline)
    468

_Rendezvous: <_Rendezvous of RPC that terminated with:
        status = StatusCode.INVALID_ARGUMENT
        details = "Request is missing 'x-goog-request-params' header."
        debug_error_string = "{"created":"@1548798018.047716000","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1036,"grpc_message":"Request is missing 'x-goog-request-params' header.","grpc_status":3}"
>

The above exception was the direct cause of the following exception:

InvalidArgument                           Traceback (most recent call last)
<ipython-input-3-2dbd1fe7a082> in <module>()
----> 1 session = client.create_read_session({"project_id": "swast-scratch", "dataset_id": "schema_examples", "table_id": "array_struct"}, "projects/swast-scratch")

~/src/google-cloud-python-private/bigquery_storage/google/cloud/bigquery_storage_v1beta1/gapic/big_query_storage_client.py in create_read_session(self, table_reference, parent, table_modifiers, requested_streams, read_options, format_, retry, timeout, metadata)
    295
    296         return self._inner_api_calls["create_read_session"](
--> 297             request, retry=retry, timeout=timeout, metadata=metadata
    298         )
    299

~/src/google-cloud-python-private/api_core/google/api_core/gapic_v1/method.py in __call__(self, *args, **kwargs)
    141             kwargs["metadata"] = metadata
    142
--> 143         return wrapped_func(*args, **kwargs)
    144
    145

~/src/google-cloud-python-private/api_core/google/api_core/retry.py in retry_wrapped_func(*args, **kwargs)
    268                 sleep_generator,
    269                 self._deadline,
--> 270                 on_error=on_error,
    271             )
    272

~/src/google-cloud-python-private/api_core/google/api_core/retry.py in retry_target(target, predicate, sleep_generator, deadline, on_error)
    177     for sleep in sleep_generator:
    178         try:
--> 179             return target()
    180
    181         # pylint: disable=broad-except

~/src/google-cloud-python-private/api_core/google/api_core/timeout.py in func_with_timeout(*args, **kwargs)
    212             """Wrapped function that adds timeout."""
    213             kwargs["timeout"] = next(timeouts)
--> 214             return func(*args, **kwargs)
    215
    216         return func_with_timeout

~/src/google-cloud-python-private/api_core/google/api_core/grpc_helpers.py in error_remapped_callable(*args, **kwargs)
     57             return callable_(*args, **kwargs)
     58         except grpc.RpcError as exc:
---> 59             six.raise_from(exceptions.from_grpc_error(exc), exc)
     60
     61     return error_remapped_callable

~/.pyenv/versions/3.6.4/envs/scratch/lib/python3.6/site-packages/six.py in raise_from(value, from_value)

InvalidArgument: 400 Request is missing 'x-goog-request-params' header.

Suspected culprit

google-cloud-python/bigquery_storage/google/cloud/bigquery_storage_v1beta1/gapic/big_query_storage_client.py

Lines 284 to 287 in 4e8bc85

	routing_header = [
	("table_reference.project_id", table_reference.project_id),
	("table_reference.dataset_id", table_reference.dataset_id),
	]

I believe these are the suspect lines (in the generated code, unfortunately). When table_reference is a dictionary, table_reference.project_id is not accurate. Instead, it would have to be table_reference["project_id"]. Likewise for dataset_id.

The true fix is probably to have the generator read off the request object, since the dict -> protobuf conversion happens in that constructor. That is, request.table_reference.project_id and request.table_reference.dataset_id instead of reading off the table_reference object.

[tswast]

Added the triage me tag because I'm sure this bug probably belongs somewhere else since it's in the generated code.

[tswast]

I think this is a problem with https://github.com/googleapis/gapic-generator/blob/master/src/main/resources/com/google/api/codegen/py/main.snip but it looks like we might actually have to get deep into some Java code to use the request object. Hopefully someone more familiar with the gapic-generator codebase can help out.

[tseaver]

@tswast Should we close this issue, now that you've opened the one against gapic-generator?

[tswast]

I think the problem is in the generator. We'll have to regenerate google-cloud-bigquery-storage once that's fixed, but maybe that'll happen automatically?

[tseaver]

We'll have to regenerate google-cloud-bigquery-storage once that's fixed, but maybe that'll happen automatically?

@yoshi-automation will helpfully do that the next morning after the gapic-generator fix lands.