More tests on dataset state

[severo]

• changes in the Graph specification:
  • provides_dataset_config_names field indicates if the step is providing the list of config names for a dataset
  • provides_config_split_names field indicates if the step is providing the list of split names for a config
  • requires field has been renamed to triggered_by: indeed, the relation between steps is that, when step B is triggered_by step A, if A has been updated, a new job will be created for step B.
• ProcessingGraph:
  • the concept of ProcessingStep is simplified: it's a data class, and provides the name, job runner version, input type (as well as the job type and cache kind, which are equal to the processing step name)
  • most of the methods are provided by ProcessingGraph, in particular those related to the edges, or to filtering nodes.
  • the data structure is a networkx.digraph: the nodes are the step names, and we use the attributes to store some properties
• remove the /admin/jobs_duration endpoint, because the finished jobs are now deleted quickly, so: this statistic does not mean much now.
• add tests for state.py. In particular: use small processing graphs with the special cases (children, grand-children, multiple parents, fan-in, fan-out, parallel steps, etc). The "real" processing graph now only has one test. This should reduce code to change when a step is added, modified or deleted.

[HuggingFaceDocBuilder]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[albertvillanova]

Thanks for the huge contribution. In principle, all proposed changes seem sensible.

[AndreaFrancis]

Thanks!
There are some commented lines, will those be removed?
I dont quite understand the advantage of having processing graph in all job runners to make one call to get the children but maybe in the future, it will be done by the orchestrator.

[severo]

Thanks for the huge contribution. In principle, all proposed changes seem sensible.

Yes, the PR is too big, I should have broken it into some smaller parts... Thanks for reviewing it all (same @AndreaFrancis)!

[severo]

There are some commented lines, will those be removed?

Yes! I removed them. And part of them was due to me having forgotten to push the last commit (finishing the tests in tests/state). I'll let you review the last changes, before merging.

I dont quite understand the advantage of having processing graph in all job runners to make one call to get the children but maybe in the future, it will be done by the orchestrator.

Yes, in the future (the PR is nearly ready #1157), we will not get the children, but instead do a full DatasetState analysis + backfill at the end of each job. It's possibly too much, but it will be better than the current incoherencies we have in the cache.