Skip to content

Latest commit

 

History

History
8 lines (6 loc) · 623 Bytes

TROUBLESHOOTING.md

File metadata and controls

8 lines (6 loc) · 623 Bytes
Raw

Troubleshooting

RuntimeError: merge_sort: failed to synchronize: an illegal memory access was encountered

This is caused in multi-box loss. The sort method failed due to NaN numbers. This may be a bug in log_softmax: pytorch/pytorch#14335 .Three ways to solve :

  1. Use a smaller warmup factor, like 0.1. (append SOLVER.WARMUP_FACTOR 0.1 to your train cmd's end).
  2. Use a longer warmup iters, like 1000. (append SOLVER.WARMUP_ITERS 1000 to your train cmd's end).
  3. Described in the forums by Jinserk Baik