Fix FlowRunView task run queries when all task runs are cached

[zanieb]

Closes bug reported in https://prefecthq.slack.com/archives/C028BPL85C3/p1661452093250219 where if all of the task runs are cached an error is thrown when retrieving cache runs

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/prefect/engine/task_runner.py", line 880, in get_task_run_state
    value = prefect.utilities.executors.run_task_with_timeout(
  File "/usr/local/lib/python3.9/site-packages/prefect/utilities/executors.py", line 468, in run_task_with_timeout
    return task.run(*args, **kwargs)  # type: ignore
  File "/redacted.py", line 122, in redacted
    tasks = view.get_all_task_runs()
  File "/usr/local/lib/python3.9/site-packages/prefect/backend/flow_run.py", line 739, in get_all_task_runs
    task_run_data = TaskRunView._query_for_task_runs(
  File "/usr/local/lib/python3.9/site-packages/prefect/backend/task_run.py", line 367, in _query_for_task_runs
    raise ValueError(
ValueError: No task runs found while querying for task runs where ....

This is easily fixed by toggling the error_on_empty bool.
Includes a change that excludes of cached task runs from the load_static_tasks query.
Includes a fix that prevents duplicate task runs from being returned when the object is shared by multiple threads.

Example

from prefect.backend import FlowRunView

view = FlowRunView.from_flow_run_id("030ee8ba-3e2c-439f-96c4-44c0ef2ad618")

# Calling repeatedly does not fail
view.get_all_task_runs()
view.get_all_task_runs()  # Previously would fail here
view.get_all_task_runs()

# Getting the latest is fine
view.get_latest()

# Getting the latest and making a query for static tasks is fine
view.get_latest(load_static_tasks=True)

# Instantiating a new instance with static loading works still
view = FlowRunView.from_flow_run_id("030ee8ba-3e2c-439f-96c4-44c0ef2ad618", load_static_tasks=True)

Checklist

• [ ] This pull request references any related issue by including "closes <link to issue>"
  • If no issue exists and your change is not a small fix, please create an issue first.
• This pull request includes tests or only affects documentation.
• This pull request includes a bug, feature, enhancement, docs, or maintenance label categorizing the change.

[bunchesofdonald]

Assuming my assumptions are right this looks good to me.

[bunchesofdonald]

Forgive my v1 ignorance, but I assume that _nin means not in? So this is effectively saying "Get all of the task runs associated with this flow run, except for the ones that are cached."

[zanieb]

Yep! We only cache finished task runs so excluding them from the query results will reduce the amount of traffic with no change in behavior.

[zanieb]

@bunchesofdonald A new change here that prevents a race condition if self._cached_task_runs is updated by another thread while the query is running. We wouldn't include those task runs in the nin call so they'd be returned by the API then on L759 we'd append the cached runs resulting in duplicates.