SageMakerModel.transform() Not Working When Model Constructed With SageMakerModel.fromEndpoint()

[robperc]

I've trained and deployed an XGBoost model for classifying the MNIST dataset. I can call transform() successfully on my test dataset when I use the model object returned by the XGBoostSageMakerEstimator() constructor, however when I try to useSageMakerModel.fromEndpoint() to create my model object and transform the same dataframe I get the following error:

'DataFrame' object has no attribute '_get_object_id'
Traceback (most recent call last):
File "/usr/lib/spark/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1124, in call
args_command, temp_args = self._build_args(*args)
File "/usr/lib/spark/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1094, in _build_args
[get_command_part(arg, self.pool) for arg in new_args])
File "/usr/lib/spark/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py", line 289, in get_command_part
command_part = REFERENCE_TYPE + parameter._get_object_id()
File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/dataframe.py", line 1020, in getattr
"'%s' object has no attribute '%s'" % (self.class.name, name))
AttributeError: 'DataFrame' object has no attribute '_get_object_id'

Attached is a .zip containg screenshots of this error and the .ipynb that can be used to reproduce it.
issue.zip

Here are the screenshots for a quick look:

[ChoiByungWook]

Hello,

Thank you so much for reporting this issue. We are aware of this, and working towards getting a fix out.

As a workaround, a model can still be constructed from an endpoint by instantiating a SageMakerModel.

from sagemaker_pyspark import SageMakerModel, EndpointCreationPolicy
from sagemaker_pyspark.transformation.deserializers import XGBoostCSVRowDeserializer
from sagemaker_pyspark.transformation.serializers import LibSVMRequestRowSerializer

my_endpoint = "endpoint-name-for-created-model"

xgboost_model = SageMakerModel(
    endpointInstanceType=None,
    endpointInitialInstanceCount=None,
    requestRowSerializer=LibSVMRequestRowSerializer(),
    responseRowDeserializer=XGBoostCSVRowDeserializer(),
    existingEndpointName=my_endpoint,
    endpointCreationPolicy=EndpointCreationPolicy.DO_NOT_CREATE
)

Let me know if there is anything I can clarify.

[robperc]

Thanks for the workaround! This worked for me (see below image). Should we leave this open until the bug fix is merged so that it is apparent to others if they run into this same issue?

[ChoiByungWook]

Thanks for the update!

Yes, lets leave it open. I'll tag it, to indicate that a fix is being worked on.

[yangaws]

Hi @robperc

Pull request #39 to fix this bug is submitted. Thanks for using sagemaker and reporting this issue!

[ChoiByungWook]

The fix has been released to pypi.