Skip to content

Commit

Permalink
Merge pull request #473 from sony/fix/20190613-fix-stalled-multigpu-t…
Browse files Browse the repository at this point in the history
…raining

Set timeout to kill itself after command is aborted
  • Loading branch information
TE-YoshiyukiKobayashi committed Jun 25, 2019
2 parents 887e1a7 + e455bcf commit 7cbd0d0
Showing 1 changed file with 6 additions and 1 deletion.
7 changes: 6 additions & 1 deletion python/src/nnabla/utils/cli/cli.py
Expand Up @@ -13,9 +13,11 @@
# limitations under the License.

import argparse
import os
import sys
import warnings
warnings.filterwarnings("ignore", message="numpy.dtype size changed")
from nnabla.logger import logger


def _nnabla_version():
Expand Down Expand Up @@ -128,7 +130,10 @@ def cli_main():
except:
import traceback
print(traceback.format_exc())
comm.abort()

logger.log(99, "ABORTED")
os.kill(os.getpid(), 9)
# comm.abort()
else:
try:
return_value = args.func(args)
Expand Down

0 comments on commit 7cbd0d0

Please sign in to comment.