tests.system.aiplatform.test_featurestore.TestFeaturestore: test_ingest_feature_values_from_df_using_feature_time_datetime_and_online_read_single_entity failed

[flaky-bot]

Note: #2153 was also for this test, but it was closed more than 10 days ago. So, I didn't mark it flaky.

---

commit: b4eba68
buildURL: Build Status, Sponge
status: failed

Test output

args = (entity_type: "projects/580378083368/locations/us-central1/featurestores/temp_vertex_sdk_e2e_featurestore_test_movie_p...on_c57e4/entityTypes/movies"
entity_id: "movie_04"
feature_selector {
  id_matcher {
    ids: "average_rating"
  }
}
,)
kwargs = {'metadata': [('x-goog-request-params', 'entity_type=projects/580378083368/locations/us-central1/featurestores/temp_ve...ityTypes/movies'), ('x-goog-api-client', 'model-builder/1.26.0 gl-python/3.8.13 grpc/1.47.0 gax/1.32.0 gapic/1.26.0')]}
@six.wraps(callable_)
def error_remapped_callable(*args, **kwargs):
    try:


      return callable_(*args, **kwargs)


.nox/system-3-8/lib/python3.8/site-packages/google/api_core/grpc_helpers.py:67:

self = <grpc._channel._UnaryUnaryMultiCallable object at 0x7f0f77f8ffa0>

request = entity_type: "projects/580378083368/locations/us-central1/featurestores/temp_vertex_sdk_e2e_featurestore_test_movie_pr...tion_c57e4/entityTypes/movies"

entity_id: "movie_04"

feature_selector {

id_matcher {

ids: "average_rating"

}

}
timeout = None

metadata = [('x-goog-request-params', 'entity_type=projects/580378083368/locations/us-central1/featurestores/temp_vertex_sdk_e2e_...tityTypes/movies'), ('x-goog-api-client', 'model-builder/1.26.0 gl-python/3.8.13 grpc/1.47.0 gax/1.32.0 gapic/1.26.0')]

credentials = None, wait_for_ready = None, compression = None
def __call__(self,
             request,
             timeout=None,
             metadata=None,
             credentials=None,
             wait_for_ready=None,
             compression=None):
    state, call, = self._blocking(request, timeout, metadata, credentials,
                                  wait_for_ready, compression)


  return _end_unary_response_blocking(state, call, False, None)


.nox/system-3-8/lib/python3.8/site-packages/grpc/_channel.py:946:

state = <grpc._channel._RPCState object at 0x7f0f180c91f0>

call = <grpc._cython.cygrpc.SegregatedCall object at 0x7f0f180a5e00>

with_call = False, deadline = None
def _end_unary_response_blocking(state, call, with_call, deadline):
    if state.code is grpc.StatusCode.OK:
        if with_call:
            rendezvous = _MultiThreadedRendezvous(state, call, None, deadline)
            return state.response, rendezvous
        else:
            return state.response
    else:


      raise _InactiveRpcError(state)


E           grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:

E           	status = StatusCode.RESOURCE_EXHAUSTED

E           	details = "Online store doesn't have enough serving resource. Check the CPU usage and provision more online serving nodes may help."

E           	debug_error_string = "{"created":"@1686195239.951057148","description":"Error received from peer ipv4:173.194.202.95:443","file":"src/core/lib/surface/call.cc","file_line":966,"grpc_message":"Online store doesn't have enough serving resource. Check the CPU usage and provision more online serving nodes may help.","grpc_status":8}"

E           >
.nox/system-3-8/lib/python3.8/site-packages/grpc/_channel.py:849: _InactiveRpcError
The above exception was the direct cause of the following exception:
self = <tests.system.aiplatform.test_featurestore.TestFeaturestore object at 0x7f0f1a6a4940>

shared_state = {'bigquery_client': <google.cloud.bigquery.client.Client object at 0x7f0f1a7a76a0>, 'bigquery_dataset': Dataset(Datase...1819_4005_b779_7724ef24cb4e', 'bucket': <Bucket: temp_vertex_sdk_e2e_featurestore_test-f8cc9103-3538-4fe6-be62-2>, ...}

caplog = <_pytest.logging.LogCaptureFixture object at 0x7f0f77fdd6d0>
def test_ingest_feature_values_from_df_using_feature_time_datetime_and_online_read_single_entity(
    self, shared_state, caplog
):
    assert shared_state["movie_entity_type"]
    movie_entity_type = shared_state["movie_entity_type"]

    caplog.set_level(logging.INFO)

    aiplatform.init(
        project=e2e_base._PROJECT,
        location=e2e_base._LOCATION,
    )

    movies_df = pd.DataFrame(
        data=[
            {
                "movie_id": "movie_03",
                "average_rating": 4.5,
                "title": "Cinema Paradiso",
                "genres": ["Romance"],
            },
            {
                "movie_id": "movie_04",
                "average_rating": 4.6,
                "title": "The Dark Knight",
                "genres": ["Action"],
            },
        ],
        columns=["movie_id", "average_rating", "title", "genres"],
    )

    feature_time_datetime_str = datetime.datetime.now().isoformat(
        sep=" ", timespec="milliseconds"
    )
    feature_time_datetime = datetime.datetime.strptime(
        feature_time_datetime_str, "%Y-%m-%d %H:%M:%S.%f"
    )

    movie_entity_type.ingest_from_df(
        feature_ids=[
            _TEST_MOVIE_TITLE_FEATURE_ID,
            _TEST_MOVIE_GENRES_FEATURE_ID,
            _TEST_MOVIE_AVERAGE_RATING_FEATURE_ID,
        ],
        feature_time=feature_time_datetime,
        df_source=movies_df,
        entity_id_field="movie_id",
    )


  movie_entity_views_df_avg_rating = movie_entity_type.read(


        entity_ids="movie_04",
        feature_ids="average_rating",
    )

tests/system/aiplatform/test_featurestore.py:428:

google/cloud/aiplatform/featurestore/_entity_type.py:1463: in read

self._featurestore_online_client.read_feature_values(

google/cloud/aiplatform_v1/services/featurestore_online_serving_service/client.py:570: in read_feature_values

response = rpc(

.nox/system-3-8/lib/python3.8/site-packages/google/api_core/gapic_v1/method.py:145: in call

return wrapped_func(*args, **kwargs)

.nox/system-3-8/lib/python3.8/site-packages/google/api_core/grpc_helpers.py:69: in error_remapped_callable

six.raise_from(exceptions.from_grpc_error(exc), exc)

value = None

from_value = <_InactiveRpcError of RPC that terminated with:

status = StatusCode.RESOURCE_EXHAUSTED

details = "Online store doesn...ave enough serving resource. Check the CPU usage and provision more online serving nodes may help.","grpc_status":8}"



???

E   google.api_core.exceptions.ResourceExhausted: 429 Online store doesn't have enough serving resource. Check the CPU usage and provision more online serving nodes may help.

:3: ResourceExhausted

[flaky-bot]

Looks like this issue is flaky. 😟

I'm going to leave this open and stop commenting.

A human should fix and close this.

---

When run at the same commit (b4eba68), this test passed in one build (Build Status, Sponge) and failed in another build (Build Status, Sponge).

[ucdmkt]

Flakiness was due to resource exhaustion, which is since then addressed. Closing.