Skip to content

[autoscaler][core] Debug flag to log resource message in autoscaler #22136

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Feb 5, 2022
Merged

[autoscaler][core] Debug flag to log resource message in autoscaler #22136

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a745929
Log resource message
DmitriGekhtman Feb 5, 2022
c31c7d8
Merge branch 'master' into dmitri/option-to-vomit-resource-message
DmitriGekhtman Feb 5, 2022
ae4cc2c
Retry on image build hiccup
DmitriGekhtman Feb 5, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions python/ray/autoscaler/_private/monitor.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@
from ray.autoscaler._private.util import format_readonly_node_type

from ray.core.generated import gcs_service_pb2, gcs_service_pb2_grpc
from ray.core.generated import gcs_pb2
import ray.ray_constants as ray_constants
from ray._private.ray_logging import setup_component_logger
from ray._private.gcs_pubsub import gcs_pubsub_enabled, GcsPublisher
Expand Down Expand Up @@ -268,6 +269,7 @@ def update_load_metrics(self):
request = gcs_service_pb2.GetAllResourceUsageRequest()
response = self.gcs_node_resources_stub.GetAllResourceUsage(request, timeout=60)
resources_batch_data = response.resource_usage_data
log_resource_batch_data_if_desired(resources_batch_data)

# Tell the readonly node provider what nodes to report.
if self.readonly_config:
Expand Down Expand Up @@ -512,6 +514,15 @@ def run(self):
raise


def log_resource_batch_data_if_desired(
resources_batch_data: gcs_pb2.ResourceUsageBatchData,
) -> None:
if os.getenv("AUTOSCALER_LOG_RESOURCE_BATCH_DATA") == "1":
logger.info("Logging raw resource message pulled from GCS.")
logger.info(resources_batch_data)
logger.info("Done logging raw resource message.")


if __name__ == "__main__":
parser = argparse.ArgumentParser(
description=("Parse Redis server for the " "monitor to connect to.")
Expand Down