Can not set endpoint environment variables when deploying a model package

[l3ku]

Describe the bug
I am unable to set environment variables for an endpoint deployment based on a model package registered in SageMaker Pipelines. I can not set the environment variables to the model package directly, because they are generated only when the endpoint is being deployed. I get the following validation error from the CreateModel API when I am trying to create a model based on the model package:

ClientError: An error occurred (ValidationException) when calling the CreateModel operation: Environment variable map cannot be specified both at Model Package 
{}
 and Model {ENDPOINT_CONFIG_NAME=REDACTED, ENDPOINT_NAME=REDACTED, MODEL_NAME=REDACTED}

To reproduce
I have registered a Model Package as part of a SageMaker Pipelines execution using the following code:

...
model = Model(
    image_uri=params.inference_image_uri,
    model_data=step_train.properties.ModelArtifacts.S3ModelArtifacts,
    role=params.sagemaker_role_arn,
    sagemaker_session=pipeline_session,
)

register_args = model.register(
    content_types=["application/json"],
    response_types=["application/json"],
    model_package_group_name="REDACTED",
    approval_status="PendingManualApproval",
    description=f"Model version for REDACTED model",
    domain=params.model_domain,
    task=params.model_task,
    metadata_properties=metadata_properties.MetadataProperties(
        commit_id=params.git_commit_sha,
        generated_by=pipeline.ExecutionVariables.PIPELINE_EXECUTION_ARN,
        repository=params.git_repository_name,
    ),
    customer_metadata_properties={"Environment": params.env_name},
)

As a result, I am able to see the model package in the model registry (output from aws sagemaker describe-model-package --model-package-name REDACTED):

{
    "ModelPackageGroupName": "REDACTED",
    "ModelPackageVersion": 3,
    "ModelPackageArn": "REDACTED",
    "ModelPackageDescription": "Model version for REDACTED model",
    "CreationTime": "2022-09-05T16:35:46.036000+03:00",
    "InferenceSpecification": {
        "Containers": [
            {
                 ...
                "Environment": {}
            }
        ],
        "SupportedContentTypes": [
            "application/json"
        ],
        "SupportedResponseMIMETypes": [
            "application/json"
        ]
    },
   ...
}

I am not specifying any environment variables to the model, and they seem to be empty when looking at the first container in the InferenceSpecification. When the model version is approved in the model registry, a Lambda is triggered which tries to deploy the model version to an endpoint, and hence requires using the CreateModel API:

...

client.create_model(
    ModelName="REDACTED",
    ExecutionRoleArn=role_arn,
    PrimaryContainer={
        "ModelPackageName": model_package_arn,
        "Environment": {
            "MODEL_NAME": sm_model_name,
            "ENDPOINT_CONFIG_NAME": endpoint_config_name,
            "ENDPOINT_NAME": endpoint_name,
        },
    },
)

When attempting to do this, I am getting an error in the Lambda logs for specifying environment variables for both the model package and the model even though I didn't specify any:

ClientError: An error occurred (ValidationException) when calling the CreateModel operation: Environment variable map cannot be specified both at Model Package 
{}
 and Model {ENDPOINT_CONFIG_NAME=REDACTED, ENDPOINT_NAME=REDACTED, MODEL_NAME=REDACTED}

When I look at the generated SageMaker Pipelines definition JSON, it indeed has the "Environment" as an empty mapping for the inference container:

{
    "Name": "REDACTED-RegisterModel",
    "Type": "RegisterModel",
    "Arguments": {
        "ModelPackageGroupName": "REDACTED",
        "ModelPackageDescription": "Model version for REDACTED model",
        ...
        "InferenceSpecification": {
            "Containers": [
                {
                    "Image": "REDACTED",
                    "Environment": {},
                    "ModelDataUrl": {
                        "Get": "Steps.training.ModelArtifacts.S3ModelArtifacts"
                    }
                }
            ],
            "SupportedContentTypes": [
                "application/json"
            ],
            "SupportedResponseMIMETypes": [
                "application/json"
            ]
        },
        "ModelApprovalStatus": "PendingManualApproval"
    },
}

Expected behavior
I am able to set environment variables to the endpoint without any error message.

Screenshots or logs
Error message from logs:

ClientError: An error occurred (ValidationException) when calling the CreateModel operation: Environment variable map cannot be specified both at Model Package 
{}
 and Model {ENDPOINT_CONFIG_NAME=REDACTED, ENDPOINT_NAME=REDACTED, MODEL_NAME=REDACTED}

System information
A description of your system. Please provide:

• SageMaker Python SDK version: 2.111.0
• Framework name (eg. PyTorch) or algorithm (eg. KMeans): -
• Framework version: -
• Python version: Python 3.8.13
• CPU or GPU:
• Custom Docker image (Y/N): Y

Additional context
Add any other context about the problem here.

[l3ku]

Any updates on this? It is currently preventing us from deploying models that were registered in the model registry during a SageMaker Pipelines execution as endpoints. There is a workaround to simply manually register a new version of the model in the model registry using the artifacts from the pipeline execution, but this is suboptimal in the long run.

[l3ku]

It seems that this has been fixed in the SageMaker APIs, by allowing to provide environment variables for both the SageMaker model package and the model object (which seem to be merged?). I couldn't find any docs about this, though.