Add filters to job/task Rest API

[jcdiazvelez]

I would like to be able to retrieve a subset of jobs or tasks. The primary need at the moment is to restrict the number of tasks when analyzing benchmark statistics (/datasets/{dset.did}/bulk/task_stats). For large datasets this becomes prohibitively expensive.
I can imagine other more general uses using filters:

For the example in mind you might filter by "jobnumber < 100"

or something like "taskname == ppc".

[dsschult]

Due to the database structure and contents, it's hard to do this directly with a single API query, but I could certainly write an example script and modify the API to make something like this possible:

ret = await client.request("GET", f"/datasets/{dataset_id}/tasks", {"status": "complete"})
task_ids = []
for task_id, task in ret.items():
  if task['name'] == "propagate" and task["job_index"] < 1000:
    task_ids.append(task_id)
task_stats = {}
while task_ids:
  query_task_ids = task_ids[:100]
  task_ids = task_ids[100:]
  for line in client.request_stream("GET", f"/datasets/{dataset_id}/bulk/task_stats", {"last": True, "task_ids": query_task_ids}):
    stat = json.loads(line)
    task_stats[stat["task_id"]] = stat