Fix --remote-auth-plugin and --remote-oauth-bearer-token-path to execute every run with Pantsd

[Eric-Arellano]

As found in #12018, --remote-auth-plugin and --remote-oauth-bearer-token-path are both impure functions with side effects like reading from the file system. Consequently, it's necessary to rerun the two mechanisms every Pants run, even with pantsd. For example, currently we will not pick up changes made to the oauth file unless you kill Pantsd.

This PR fixes it so that we always do the correct things of re-evaluating the DynamicRemoteExecutionOptions. If those have changed from the prior Pantsd run, we re-initialize the scheduler.

Naively, however, this has a negative implication for the Toolchain plugin. Because Toolchain's auth logic is stateless, it generates a new short-lived token every call, which updates --remote-store-header. That means that this PR will cause Pantsd to be invalidated every single run, making Pantsd useless. This will be fixed in a followup by passing the prior AuthPluginResult to the auth plugin function + allowing the plugin to set a token expiration time, which will allow the plugin to decide if it's safe to resuse the prior token.

[Eric-Arellano]

How I tested this:

diff --git a/pants.toml b/pants.toml
index f147114fe..40451c99e 100644
--- a/pants.toml
+++ b/pants.toml
@@ -53,7 +53,7 @@ build_ignore.add = [
 # NB: Users must still set `--remote-cache-{read,write}` to enable the remote cache.
 remote_store_address = "grpcs://cache.toolchain.com:443"
 remote_instance_name = "main"
-remote_auth_plugin = "toolchain.pants.auth.plugin:toolchain_auth_plugin"
+remote_auth_plugin = "pants.backend.python.goals.pytest_runner:auth"
 
 [anonymous-telemetry]
 enabled = true
diff --git a/src/python/pants/backend/python/goals/pytest_runner.py b/src/python/pants/backend/python/goals/pytest_runner.py
index 7b2245fd7..25abd028b 100644
--- a/src/python/pants/backend/python/goals/pytest_runner.py
+++ b/src/python/pants/backend/python/goals/pytest_runner.py
@@ -59,7 +59,7 @@ from pants.engine.process import (
 from pants.engine.rules import Get, MultiGet, collect_rules, rule
 from pants.engine.target import TransitiveTargets, TransitiveTargetsRequest
 from pants.engine.unions import UnionRule
-from pants.option.global_options import GlobalOptions
+from pants.option.global_options import AuthPluginResult, AuthPluginState, GlobalOptions
 from pants.python.python_setup import PythonSetup
 from pants.util.logging import LogLevel
 
@@ -308,3 +308,10 @@ async def debug_python_test(field_set: PythonTestFieldSet) -> TestDebugRequest:
 
 def rules():
     return [*collect_rules(), UnionRule(TestFieldSet, PythonTestFieldSet)]
+
+
+def auth(initial_execution_headers, initial_store_headers, options, env):
+    logger.error("HELLO!")
+    import datetime
+
+    return AuthPluginResult(AuthPluginState.OK, {"foo": str(datetime.datetime.now())}, {})

Then running ./pants multiple times.

[tdyas]

lgtm