No way to set max receive message size in MetadataStore

[daikeshi]

We use ml-metadata python lib to access MLMD. When there're many executions or some large-size properties stored in MLMD, we are getting Received message larger than max error. It would be great if max receive message size can be configurable in MetadataStore.

Relevant stack trace:

Traceback (most recent call last):
 ...
  File "/usr/local/lib/python3.6/dist-packages/ml_metadata/metadata_store/metadata_store.py", line 697, in get_executions_by_type
    self._call('GetExecutionsByType', request, response)
  File "/usr/local/lib/python3.6/dist-packages/ml_metadata/metadata_store/metadata_store.py", line 131, in _call
    return self._call_method(method_name, request, response)
  File "/usr/local/lib/python3.6/dist-packages/ml_metadata/metadata_store/metadata_store.py", line 162, in _call_method
    raise _make_exception(e.details(), e.code().value[0])
tensorflow.python.framework.errors_impl.ResourceExhaustedError: Received message larger than max (4333469 vs. 4194304)

[hughmiao]

hi @daikeshi , it looks your case hit the default grpc message size, 4MB.
to reproduce it, what is the client, server setting in this case?
qq, what's the use case for put_executions here? is it bulk inserting many executions or a single execution with large size properties? does the traffic generated from tfx-oss or custom client application?

[daikeshi]

Hey @hughmiao, we are using MetadataStore python client to access MLMD. The query is from tfx and it calls get_executions_by_type method in MetadataStore. Due to some executions having large size custom execution properties, the received message size exceeds the default max limit.

I wonder if we could allow users to config these kinds of properties when the channel is created https://github.com/google/ml-metadata/blob/master/ml_metadata/metadata_store/metadata_store.py#L105

[hughmiao]

hi @daikeshi , if it is tfx, after 0.15, we made a series optimization to address the tfx caching lookup performance, and as a result, we dropped all unbounded call. The only place it used was in tfx 0.15 [1]. Even if we change here at head and make a new release, it may not work with tfx 0.15.

[1] https://github.com/tensorflow/tfx/blob/r0.15/tfx/orchestration/metadata.py#L497

[daikeshi]

Yes, @hughmiao after tfx 0.15, we will probably switch to use get_executions_by_context which can limit the # of executions retrieved based on the context. I guess it will be less likely to hit the max size (it also depends on how we define context), but the edge case issue still remains. Users can save large-size execution properties in MLMD (e.g. a large BQ query), the return value of get_executions_by_context can still blow up the default maximum message size.

Just want to point it out, this is nice to have but not urgent for us.

[hughmiao]

agreed, @daikeshi . let's keep it open and resolve it in the coming releases.

[nlarusstone]

I'm encountering this same error, but I'm not using tfx. We're using the MetadataStore as part of Kubeflow, but we're using the ml-metadata package to access the MetadataStore.

Specifically we're calling self.store.get_artifacts_by_type(artifact_type_name) to list all of our models, and it's throwing this error: tensorflow.python.framework.errors_impl.ResourceExhaustedError: Received message larger than max (4194643 vs. 4194304)

What's the recommendation to solve this?

[nlarusstone]

Note that normally I would want to delete artifacts to clean up the database, but that doesn't seem to be allowed: #38

[daikeshi]

Hey @hughmiao, I wonder if we can bump this issue's priority. It actually happens quite often now, even in the KFP UI when I try to fetch 50-100 runs.

[hughmiao]

hi @nlarusstone @daikeshi on the kfp side, we are adding pagination support recently, the unbounded return will be addressed in the coming releases. cc some KFP folks.

/cc @dushyanthsc