[BUG] stop throwing when configuring default host mr

[abellina]

In this PR #15665 we added a function config_default_host_memory_resource that takes in parameters that allow us to disable the default pinned behavior in cuDF.

I tested this manually and neglected to run the unit tests in spark. The unit tests will, in the same process, re-initialize our plugin. Since all of the storage is static in this cuDF code around the pinned resource, we wind up calling config_default_host_memory_resource multiple times, causing the exception.

We'd like to propose a change and I can put up a bug fix for this today. Instead of a config_default_host_memory_resource, we'd like to pass the host_mr_options that were being passed here, directly to set_host_memory_resource. We'd also like set_host_memory_resource to not throw if called multiple times.

Here is a high level change that I can try:

rmm::host_async_resource_ref set_host_memory_resource(rmm::host_async_resource_ref mr, host_mr_options const& default_mr_opts)
{
  std::scoped_lock lock{host_mr_mutex()};
  auto last_mr = host_mr(opts);
  host_mr()    = mr;
  return last_mr;
}

CUDF_EXPORT rmm::host_async_resource_ref& host_mr(std::optional<host_mr_options> const& opts)
{
  static rmm::host_async_resource_ref mr_ref = make_host_mr(opts);
  return mr_ref;
}

I think we can also remove the exception from make_host_mr and simplify that a bit.

I think this will unblock us and the api is cleaner: call set_host_memory_resource and pass the default options (which in our case are to always disable the default pool).