Pyspark Mnist XGBoost tutorial

Please fill out the form below.

System Information

• Framework (e.g. TensorFlow) / Algorithm (e.g. KMeans): I am running the sample notebook provided when running new sagemaker notebooks: sample-notebooks/sagemaker-spark/pyspark_mnist/pyspark_mnist_xgboost.ipynb
• Framework Version:
• Python Version: Python 3.6.4
• CPU or GPU: CPU
• Python SDK Version: sagemaker==1.2.2, sagemaker-pyspark==1.0.4
• Are you using a custom image: No

Describe the problem

When I am running Training and Hosting a Model as it is, I get a ClientError. If i provide the IAM role myself (hence not using role = sagemaker.get_execution_role()) the algorithm is trained without any problem.
More precisely, the two roles are:

• from execution_role(): arn:aws:iam::<ACCOUNT>:role/service-role/AmazonSageMaker-ExecutionRole
• provided by me: arn:aws:iam::<ACCOUNT>:role/AmazonSageMaker-ExecutionRole

Minimal repro / logs

Py4JJavaError: An error occurred while calling o139.fit.
: java.lang.RuntimeException: Training job couldn't be completed.
	at com.amazonaws.services.sagemaker.sparksdk.SageMakerEstimator.runTrainingJob(SageMakerEstimator.scala:407)
	at com.amazonaws.services.sagemaker.sparksdk.SageMakerEstimator.fit(SageMakerEstimator.scala:311)
	at com.amazonaws.services.sagemaker.sparksdk.SageMakerEstimator.fit(SageMakerEstimator.scala:175)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:214)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Training job 'trainingJob-665ebe56c361-2018-05-09T10-53-21-799' failed for reason: 'ClientError: SageMaker was unable to assume the role 'arn:aws:iam::<ACCOUNT>:role/service-role/AmazonSageMaker-ExecutionRole''
	at com.amazonaws.services.sagemaker.sparksdk.SageMakerEstimator.awaitTrainingCompletion(SageMakerEstimator.scala:435)
	at com.amazonaws.services.sagemaker.sparksdk.SageMakerEstimator.runTrainingJob(SageMakerEstimator.scala:409)
	... 13 more

• Exact command to reproduce:
  The cell Training and Hosting a Model on the tutorial sample-notebooks/sagemaker-spark/pyspark_mnist/pyspark_mnist_xgboost.ipynb on a new sagemaker notebook instance

[andremoeller]

Hi @Pompelmo ,

Hi, thanks for the report! Unfortunately, I couldn't reproduce this, so I have a few questions to help us figure this out:

1. Does the role returned by sagemaker.get_execution_role() exist? If it does, does it name "sagemaker.amazon.com" as a trusted entity under Trust Relationships?
2. Does the role returned by sagemaker.get_execution_role() have a timestamp appended to it, or does it look just like you pasted (but with the account ID)?
3. Did you manually create the role you provided (arn:aws:iam::<ACCOUNT>:role/AmazonSageMaker-ExecutionRole), or was that role automatically created for you?

The reason I ask is: if "AmazonSageMaker-ExecutionRole" happens to be in your role ARN, the Python SDK prepends adds the "service-role" part to the role ARN, so it might be returning a non-existent role:

sagemaker-python-sdk/src/sagemaker/session.py

Lines 541 to 553 in 607059d

	def get_caller_identity_arn(self):
	"""Returns the ARN user or role whose credentials are used to call the API.
	Returns:
	(str): The ARN uer or role
	"""
	assumed_role = self.boto_session.client('sts').get_caller_identity()['Arn']
	if 'AmazonSageMaker-ExecutionRole' in assumed_role:
	role = re.sub(r'^(.+)sts::(\d+):assumed-role/(.+?)/.*$', r'\1iam::\2:role/service-role/\3', assumed_role)
	return role
	role = re.sub(r'^(.+)sts::(\d+):assumed-role/(.+?)/.*$', r'\1iam::\2:role/\3', assumed_role)
	return role

Thanks!

Hi @andremoeller

I am not really an expert on IAM roles but:
1- the role arn:aws:iam::<ACCOUNT>:role/AmazonSageMaker-ExecutionRole does exist, but arn:aws:iam::<ACCOUNT>:role/service-role/AmazonSageMaker-ExecutionRole, that is the one returned by ``sagemaker.get_execution_role(), doesn't. 2- The role has no timestamp appended to it, it's just as I pasted it (I just copied the notebook cell output of a cell print(role)```)
3 - The role was manually created

Yes I guess that was what it happened! I thought that service-role was some sort of "sub-role" of the role we created. Hence, should there be two different role?

Thank you so much for your time!

[andremoeller]

Hi @Pompelmo ,

Got it, thanks!

The SageMaker console lets you auto-create an IAM role with that string ("service-role/AmazonSageMaker-ExecutionRole") when creating a Notebook Instance.
I think the "service-role" part is just meant to indicate that it's been created by the AWS service. The code in the get_execution_role() function in the Python SDK tries to find that auto-created role, but there was a naming collision with your own role, so the Python SDK mistakenly prepends "service-role".

We should improve the if 'AmazonSageMaker-ExecutionRole' in assumed_role to be less permissive to avoid cases like this.

I believe that if you create a role through the console, this will work fine. Let us know if you have any more questions

Thanks!