Metadata validation

[theo-m]

• pydantic metadata schema with dedicated validators against our taxonomy
• ci script to validate new changes against this schema and start a vertuous loop
• soft validation on tasks ids since we expect the taxonomy to undergo some changes in the near future

for reference with the current validation we have 365 378 datasets with invalid metadata! full error report here.

[lhoestq]

This is really cool thanks :)

I left a few comments.

Also I was wondering this is really needed to have utils.metadata as a submodule of datasets ? This is only used by the CI so I'm not sure we should have this in the actual datasets package.

[theo-m]

Also I was wondering this is really needed to have utils.metadata as a submodule of datasets ? This is only used by the CI so I'm not sure we should have this in the actual datasets package.

I'm unclear on the suggestion, would you rather have a root-level ./metadata.py file? I think it's well where it is, if anything we could move it out of utils and into datasets as it could be used by e.g. DatasetDict so that users can pull the metadata easily rather than have to reparse the readme.

[lhoestq]

Ok that makes sense if we want to have functions that parse the metadata for users

[gchhablani]

Hi @theo-m @lhoestq

This seems very interesting. Should I add the descriptions to the PR on datasets-tagging? Alternatively, I can also create a google-sheet/markdown table :)

Sorry for the delay in responding.

Thanks,
Gunjan

[theo-m]

Hi @theo-m @lhoestq

This seems very interesting. Should I add the descriptions to the PR on datasets-tagging? Alternatively, I can also create a google-sheet/markdown table :)

Sorry for the delay in responding.

Thanks,
Gunjan

Hi @gchhablani, yes I think at the moment the best solution is for you to write in datasets-tagging, as the PR will allow us to discuss and review, even though the work will be ported to this repo in the end.
Or we wait for this to be merged and you reopen the PR here, your call :)

[lhoestq]

Looks good :)
Can you just add docstrings before we merge ?
Can you add tests as well ?

[lhoestq]

Suggested change

	def dict_from_readme(f: Path) -> Optional[Dict[str, List[str]]]:
	def dict_from_readme(path: Path) -> Optional[Dict[str, List[str]]]:

Use explicit argument names.
Can you also add a docstring ?

[SBrandeis]

cc @abhi1thakur

[lhoestq]

Thank you @theo-m @SBrandeis !