Expand control plane via admin resources

[fubhy]

The following feature requests have been identified:

• List, get by id, stop, and delete jobs (/jobs endpoints)

  • feat(admin-api): add idempotent job stop endpoint #507
  • feat(admin-api): add job deletion endpoints and database operations #643

• List, get by id, and delete locations (/locations endpoints)

  • feat(admin-api): add locations endpoints and homogenize handlers #592

• List by location, get by id, and delete files

  • feat(metadata-db): extract file metadata operations into dedicated module #689
  • feat(admin-api): add files CRUD endpoints #706

• List, get by id, create, and remove providers (/providers endpoints)

  • feat(dataset-store): extract providers management to separate module #669
  • feat(admin-api): add providers admin API resource #688

• List, get by id, register, and delete datasets (/datasets endpoints)

  • Dataset manifest definitions and store cleanup
    • refactor(common): make schemars dep optional and co-locate schemas #717
    • refactor(common): unify manifest schema types #721
    • feat(metadata-db): add semver version wrapper for PostgreSQL storage #736
    • refactor(dataset-store): rename dataset kind Manifest to Derived #821
    • refactor(datasets-common): simplify Debug output for Manifest struct #822
    • feat(datasets-derived): introduce datasets-derived crate #826
    • refactor(extractors): align manifest types with other datasets crates #828
    • feat(dataset-store): support loading datasets by name and version #796
  • Merge registry-service's /register endpoint into admin-api
    • feat(admin-api): merge registry service register endpoint into admin API #720
  • Merge registry-service's /output_schema endpoint into admin-api
    • feat(admin-api): add SQL schema analysis endpoint #729
  • Trigger a dataset dump
    • feat(admin-api): merge registry service register endpoint into admin API #720
  • List all dataset versions
    • feat(metadata-db): add semver version wrapper for PostgreSQL storage #736
    • feat(datasets): add pagination support for dataset listing and retrieval  #825
    • feat(admin-api): implement paginated datasets endpoints #750
  • Delete a dataset over the Admin API #1057
  • chore(registry-service): remove service and related configurations #735

• Job pause/resume #1058

• Explore OpenAPI spec file generation #626

  • feat(admin-api): implement OpenAPI spec generation with utoipa #645

Original issue description

We need a couple of added functionaity there like:

• Ability to flush all data & metadata (effectively reset a running server)
• Delete a specific dataset and all its data & metadata
• Dump all datasets that are currently deployed (without providing a dataset name)
• Get information and state of a specific dataset
• Get state and information of all datasets
• ...

These things would unblock a couple of features for the development environment. Can we get them addressed ASAP? ./cc @leoyvens

[fubhy]

@LNSD another one: I'd like to also deploy datasets via HTTP instead of having to run a child process for the CLI in the dev server.

[fubhy]

Sorry, I meant dump datasets

[fubhy]

The goal is to do this via HTTP: https://github.com/edgeandnode/project-nozzle/blob/main/typescript/nozzle/src/Nozzle.ts#L148-L154

[LNSD]

Adding this here for the record:

The idea is to turn the admin-api crate into a presentation layer that just calls business logic from the core crates (similarly to the bin commands are just shells around the core libs code)