-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Open
Labels
Description
Describe the bug
The documentation for SKLearn constructor mentions that we can use custom metric_definitions
with a regex. I checked that this kwarg is passed to Framework
which then passes it to EstimatorBase.
This regex only captures numbers and not strings. This should either be mentioned in the documentation or support fort arbitrary strings needs to be present.
To reproduce
cli.py
if __name__ == "__main__":
print("canary=1.01;")
print(f"cat=dog;")
sagemaker.ipynb
from sagemaker.sklearn.estimator import SKLearn
import sagemaker
from sagemaker import get_execution_role
from copy import deepcopy
import itertools
sagemaker_session = sagemaker.Session()
role = get_execution_role()
FRAMEWORK_VERSION = "1.0-1"
script_path = "cli.py"
LOCAL=False
kwargs=dict(
instance_type="ml.m5.large",
instance_count=1,
sagemaker_session=sagemaker_session,
base_job_name="test-job-",
)
base_model_gen = lambda: EstimatorBase(
entry_point=script_path,
framework_version=FRAMEWORK_VERSION,
role=role,
source_dir="./",
metric_definitions=[dict(Name=m, Regex=f"{m}=(.*?);") for m in ["canary",'cat']],
**kwargs
)
base_model_gen().fit()
Expected behavior
Algorithm metrics should've displayed both canary
and cat
Only canary
is displayed
System information
A description of your system. Please provide:
- SageMaker Python SDK version: '2.97.0'
- Framework name (eg. PyTorch) or algorithm (eg. KMeans): SKLearn
- Framework version: 1.0-1
- Python version: 3.8.12
- CPU or GPU: CPU
- Custom Docker image (Y/N): N
Additional context
Add any other context about the problem here.