Disable Horovod stderr in sagemaker container

[vbvg2008]

It is a known issue that when using horovod in docker container, there will be a bunch of stderr message printing out like this(you can reproduce it by executing sagemaker's built-in example with multi-gpu instance: tensorflow_script_mode_horovod.ipynb):

• [1,1]:[algo-1:00062] Read -1, expected 36864, errno = 1
• [1,3]:[algo-1:00064] Read -1, expected 36864, errno = 1
• [1,2]:[algo-1:00063] Read -1, expected 36864, errno = 1
• [1,1]:[algo-1:00062] Read -1, expected 36864, errno = 1
• [1,3]:[algo-1:00064] Read -1, expected 36864, errno = 1
• [1,2]:[algo-1:00063] Read -1, expected 36864, errno = 1

Most of the time, we can simply ignore the message or filter them out in cloud watch. But sometimes the printing happens so frequent that it affects the training speed significantly (I observed ~50% training speed decrease when using p3.16xlarge).

One way to disable the message is by providing --privileged flag when calling nvidia-docker run, as suggested in official horovod github .

Is there a way to add the --privileged flag in our container or suppress the message in some other ways such that the training speed won't be affected by the printing?

[ChoiByungWook]

Hello @vbvg2008,

Apologies for the late response.

Let me start a conversation in regards to allowing the container to be run with --privileged. I'm not too sure if this functionality can be added, as it looks like it gives the container more access then we would potentially be comfortable with.

Apologies that this is affecting your training speed, I'll see what we can do.

Thanks!

[dickren123]

@ChoiByungWook Hi facing the same issue, just want to make sure --privileged can be added when docker run in sagemaker now

[ChaiBapchya]

@dickren123 can you paste the command which allows you to add privileged to docker run?

[ShwetaSingh801]

Hi @vbvg2008,

Thanks for using SageMaker and taking the time to suggest ways to improve SageMaker Python SDK. We have added your feature request to our backlog of feature requests and may consider putting it into future SDK versions. I will go ahead and close the issue now, please let me know if you have any more feedback. Let me know if you have any other questions.

Best,
Shweta