Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ModelExplainabilityMonitor cannot execute one-time monitoring job due to lack of data_analysis_start_time/data_analysis_end_time in _create_monitoring_schedule_from_job_definition #4431

Open
hightensan opened this issue Feb 16, 2024 · 0 comments
Labels

Comments

@hightensan
Copy link

hightensan commented Feb 16, 2024

Describe the bug
SageMaker Model Monitor provides one-time monitoring job schedule[Link].
However, ValueError are raised when specifying CronExpressionGenerator.now().

To reproduce

Using SageMaker-Monitoring-Feature-Attribution-Drift-for-Endpoint.ipynb:

  • Change schedule_expression to CronExpressionGenerator.now()
  • Specify data_analysis_start_time and data_analysis_end_time in ModelExplainabilityMonitor's create_monitoring_schedule
type(model_explainability_monitor) # sagemaker.model_monitor.clarify_model_monitoring.ModelExplainabilityMonitor

schedule_expression = sagemaker.model_monitor.CronExpressionGenerator.now()

model_explainability_monitor.create_monitoring_schedule(
    analysis_config=model_explainability_analysis_config,
    endpoint_input=sagemaker.model_monitor.EndpointInput(
        endpoint_name=endpoint_name,
        destination="/opt/ml/processing/input/endpoint",
        features_attribute=features_jmespath,
        inference_attribute=confidence_score_jmespath,
    ),
    output_s3_uri=monitor_output_s3_uri,
    schedule_cron_expression=schedule_expression,
    data_analysis_start_time="-PT2H",
    data_analysis_end_time="-PT1H"
)

Error Message:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[63], line 3
      1 schedule_expression = sagemaker.model_monitor.CronExpressionGenerator.now()
----> 3 model_explainability_monitor.create_monitoring_schedule(
      4     analysis_config=model_explainability_analysis_config,
      5     endpoint_input=sagemaker.model_monitor.EndpointInput(
      6         endpoint_name=endpoint_name,
      7         destination="/opt/ml/processing/input/endpoint",
      8         features_attribute=features_jmespath,
      9         inference_attribute=confidence_score_jmespath,
     10     ),
     11     output_s3_uri=monitor_output_s3_uri,
     12     schedule_cron_expression=schedule_expression,
     13     data_analysis_start_time="-PT2H",
     14     data_analysis_end_time="-PT1H"
     15 )

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/sagemaker/model_monitor/clarify_model_monitoring.py:1104, in ModelExplainabilityMonitor.create_monitoring_schedule(self, endpoint_input, analysis_config, output_s3_uri, constraints, monitor_schedule_name, schedule_cron_expression, enable_cloudwatch_metrics, batch_transform_input, data_analysis_start_time, data_analysis_end_time)
   1102 # create schedule
   1103 try:
-> 1104     self._create_monitoring_schedule_from_job_definition(
   1105         monitor_schedule_name=monitor_schedule_name,
   1106         job_definition_name=new_job_definition_name,
   1107         schedule_cron_expression=schedule_cron_expression,
   1108     )
   1109     self.job_definition_name = new_job_definition_name
   1110     self.monitoring_schedule_name = monitor_schedule_name

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/sagemaker/model_monitor/model_monitoring.py:1562, in ModelMonitor._create_monitoring_schedule_from_job_definition(self, monitor_schedule_name, job_definition_name, schedule_cron_expression, data_analysis_start_time, data_analysis_end_time)
   1559 message = "Creating Monitoring Schedule with name: {}".format(monitor_schedule_name)
   1560 logger.info(message)
-> 1562 self._check_monitoring_schedule_cron_validity(
   1563     schedule_cron_expression=schedule_cron_expression,
   1564     data_analysis_start_time=data_analysis_start_time,
   1565     data_analysis_end_time=data_analysis_end_time,
   1566 )
   1568 monitoring_schedule_config = {
   1569     "MonitoringJobDefinitionName": job_definition_name,
   1570     "MonitoringType": self.monitoring_type(),
   1571 }
   1572 if schedule_cron_expression is not None:

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/sagemaker/model_monitor/model_monitoring.py:1536, in ModelMonitor._check_monitoring_schedule_cron_validity(self, schedule_cron_expression, data_analysis_start_time, data_analysis_end_time)
   1531 message = (
   1532     "Both data_analysis_start_time and data_analysis_end_time are required "
   1533     "for one time monitoring schedule "
   1534 )
   1535 _LOGGER.error(message)
-> 1536 raise ValueError(message)

ValueError: Both data_analysis_start_time and data_analysis_end_time are required for one time monitoring schedule 

Expected behavior
Create one-time monitoring job schedule.

ModelBiasMonitor (Model Bias Monitoring Job) can create one-time monitoring job schedule, because data_analysis_start_time and data_analysis_end_time are correctly passed in ModelBiasMonitor's _create_monitoring_schedule_from_job_definition function.
ModelBiasMonitor's _create_monitoring_schedule_from_job_definition

However, ModelExplainabilityMonitor's _create_monitoring_schedule_from_job_definition function does not pass data_analysis_start_time and data_analysis_end_time params, consequently ValueError was raised in _check_monitoring_schedule_cron_validity called in _create_monitoring_schedule_from_job_definition.
data_analysis_start_time and data_analysis_end_time params are defined in ModelExplainabilityMonitor's create_monitoring_schedule args.
ModelExplainabilityMonitor's _create_monitoring_schedule_from_job_definition
ModelMonitor's _check_monitoring_schedule_cron_validity

In this commit, codes to support for one time monitoring schedule were merged, however ModelExplainabilityMonitor's one time monitoring schedule looks missed (data_analysis_start_time and data_analysis_end_time as ModelExplainabilityMonitor's create_monitoring_schedule args are added).

System information

  • SageMaker Python SDK version: v2.0.8 (latest)
  • Framework name (eg. PyTorch) or algorithm (eg. KMeans): N/A
  • Framework version: N/A
  • Python version: 3.10.13 | packaged by conda-forge
  • CPU or GPU: CPU
  • Custom Docker image (Y/N): N

Additional context
N/A

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant