With the new ReplExecutor the code is not evaluated asynchronously anymore, this functionality needs to be added back.