I'm a bit confused as to what could be causing this. Can you reduce the scan to a smaller set that still produces the error? For example, do a port scan for 5900, then do your --script vnc-brute against each of the discovered systems separately. Does one of them cause the error, or do they all finish?
If you can get it down to just one target that causes the error, then further try to reduce the problem:
Does it happen with --script-args unpwdb.passlimit=5
Does it happen with --script-args unpwdb.timelimit=5s