[RPC] Report RPC Session Timeout to Client Instead of "kShutdown"

[Johnson9009]

By using RPC server in NPU board, at some time a compiled model will hang the NPU, because of the buggy operator libraries of NPU toolchain, so we must to use the session_timeout to ensure the board resource can be released by the hang jobs.

Currently the handling of session timeout error in RPC server is not good, it just kill the server loop sub process, then in the destructor of class RPCEndpoint will send the code of kShutdown to the RPC client, but the RPC client expect receive the code of kReturn or kException, so users will see the error message that like the one reported in #15151, this error report will make users very confused and don't know what's happened.

When using tuning to search a good schedule for operators, we only want to ignore the RPC session timeout error that indicate the schedule generated is an illegal one, but other error reported by the RPC server may help us find the potential bug of our tool chain built on top of TVM, so the RPC session timeout error should be split to a standalone TVM error class.

This PR implemented these requirements by sending the RPC session timeout error message as a PRC server exception to the RPC client before kill the server loop sub process.

[tvm-bot]

Thanks for contributing to TVM! Please refer to the contributing guidelines https://tvm.apache.org/docs/contribute/ for useful information and tips. Please request code reviews from Reviewers by @-ing them in a comment.

• No users to tag found in teams: rpc _{See #10317 for details}

_{Generated by tvm-bot}

[junrushao]

This is a nice addition! I'm curious what its implication is to the existing auto tuning system though - for example, will it affect AutoTVM's time out mechanism? CC @zxybazh

[Johnson9009]

This is a nice addition! I'm curious what its implication is to the existing auto tuning system though - for example, will it affect AutoTVM's time out mechanism? CC @zxybazh

@junrushao In my opinion, there isn't any effect for the existing auto tuning system, because currently they all catch the very general error type, e.g., Exception, TVMError, and the RPCSessionTimeoutError is a subclass of TVMError, so it will be caught too.

tvm/python/tvm/autotvm/measure/measure_methods.py

Lines 369 to 375 in e178375

	for future in futures:
	try:
	res = future.result()
	results.append(res)
	except Exception as ex: # pylint: disable=broad-except
	tb = traceback.format_exc()
	results.append(

tvm/python/tvm/autotvm/measure/measure_methods.py

Lines 683 to 693 in e178375

	costs.sort()
	costs = tuple(costs[1:-1])
	except TVMError as exc:
	msg = str(exc)
	if "Stack trace returned" in msg:
	msg = msg[: msg.index("Stack trace returned")]
	if "CUDA Source" in msg:
	msg = msg[: msg.index("CUDA Source")]
	costs = (traceback.format_exc(), RuntimeError(msg[:1024]))
	errno = MeasureErrorNo.RUNTIME_DEVICE
	tstamp = time.time()

tvm/python/tvm/auto_scheduler/measure.py

Lines 1150 to 1159 in e178375

	remote.remove("")
	dev.free_raw_stream(stream)
	# pylint: disable=broad-except
	except Exception:
	dev.free_raw_stream(stream)
	costs = (MAX_FLOAT,)
	error_no = MeasureErrorNo.RUNTIME_DEVICE
	error_msg = make_traceback_info()
	shutil.rmtree(os.path.dirname(build_res.filename))

[junrushao]

Yep, and that's why I am curious. Thanks for the explanation!