Unified DB-backed cloud catalog with background provider refresh

[zdeneklapes]

Proposal: Unified DB-backed cloud catalog with background provider refresh

Summary

I would like to propose consolidating SkyPilot's cloud catalog handling around a single DB-backed catalog maintained by a dedicated background refresher service.

Today, SkyPilot already has API-backed catalog/data fetchers for several providers, but refresh, caching, persistence, and consumption are handled differently across providers.

The proposed architecture is:

AWS API       GCP API       Azure API       RunPod API       ...
   │             │              │               │
   └─────────────┴──────────────┴───────────────┘
                         │
                         ▼
              ProviderCatalogFetcher
                  common interface
                         │
                         ▼
                      normalize
                         │
                         ▼
                      validate
                         │
                         ▼
                atomic DB publish
                         │
                         ▼
              cloud_catalog_entries
                         │
                         ▼
                SkyPilot API server
                         │
                         ▼
                     optimizer

A separate catalog-refresher service would periodically query configured providers and update instance types, regions/zones, hardware information, prices, spot prices, and other provider-exposed catalog data.

The API server and optimizer would then always consume the latest validated catalog from the database.

The main goal is to consolidate the lifecycle:

fetch -> normalize -> validate -> persist -> consume

while keeping provider-specific API logic inside small adapters.

---

Motivation

Catalog freshness

The current catalog flow works well for relatively stable cloud data, but it is less suitable for dynamic providers.

RunPod is a good example. Its offerings and pricing can change much faster than SkyPilot's periodically refreshed catalog.

This was already discussed during the RunPod catalog work:

#6824

With the current hosted catalog approach, freshness depends on multiple refresh cycles:

provider
   │
   ▼
hosted SkyPilot catalog
   │
   ▼
local/server catalog
   │
   ▼
optimizer

For dynamic providers, it would be better for the SkyPilot server itself to maintain a much fresher catalog.

Provider-specific catalog implementations

SkyPilot already contains provider fetchers for several clouds:

• AWS: https://github.com/skypilot-org/skypilot/blob/master/sky/catalog/data_fetchers/fetch_aws.py
• GCP: https://github.com/skypilot-org/skypilot/blob/master/sky/catalog/data_fetchers/fetch_gcp.py
• Azure: https://github.com/skypilot-org/skypilot/blob/master/sky/catalog/data_fetchers/fetch_azure.py
• RunPod: https://github.com/skypilot-org/skypilot/blob/master/sky/catalog/data_fetchers/fetch_runpod.py

The provider API calls will always be provider-specific.

However, the following parts do not need to be:

• refresh scheduling;
• retries/backoff;
• jitter;
• normalization;
• validation;
• persistence;
• revisions;
• cache invalidation;
• observability;
• serving catalog data to the optimizer.

I think these should become common SkyPilot infrastructure.

---

Goals

1. Keep full provider catalog queries outside the launch/optimizer request path.
2. Keep cloud catalog data substantially fresher.
3. Use one common catalog refresh architecture across providers where their API/SDK supports discovery.
4. Reuse the existing SkyPilot catalog schema and optimizer logic.
5. Store one consistent catalog revision per provider.
6. Preserve the previous valid revision if refresh fails.
7. Support account/workspace-specific catalogs where required.
8. Make catalog freshness observable.

---

Non-goals

This proposal does not make catalog data an authoritative guarantee of capacity.

instance is listed by provider
            !=
instance can definitely be allocated now

Provisioning-time capacity errors, retries, and failover remain necessary.

Providers also do not have to expose identical information. Fields that are unavailable should remain NULL / UNKNOWN.

---

Architecture

1. Dedicated catalog refresher service

Introduce a separate background process, for example:

skypilot-catalog-refresher

For a remote SkyPilot deployment:

┌────────────────────────────┐
│ SkyPilot API Server        │
│                            │
│ requests / optimizer       │
└─────────────┬──────────────┘
              │
              │ reads
              ▼
┌────────────────────────────┐
│ SkyPilot DB                │
│                            │
│ cloud_catalog_entries      │
└─────────────▲──────────────┘
              │
              │ atomic updates
              │
┌─────────────┴──────────────┐
│ Catalog Refresher          │
│                            │
│ provider polling scheduler │
└─────────────┬──────────────┘
              │
      ┌───────┼─────────┬──────────┐
      ▼       ▼         ▼          ▼
     AWS     GCP      Azure      RunPod

The refresher would have no user-facing API.

Its responsibility is only:

fetch
normalize
validate
write latest catalog revision

The API server only reads catalog state.

---

2. Common provider fetcher interface

Provider implementations could expose a common interface such as:

class ProviderCatalogFetcher(Protocol):
    provider: str
    default_refresh_interval: timedelta

    def fetch_catalog(
        self,
        context: CatalogFetchContext,
    ) -> pd.DataFrame:
        ...

Example:

class RunPodCatalogFetcher:
    provider = 'runpod'
    default_refresh_interval = timedelta(minutes=10)

    def fetch_catalog(self, context):
        ...

The provider adapters remain responsible for translating their API into SkyPilot's normalized catalog format.

The important consolidation is everything after the adapter.

---

3. Reuse the current SkyPilot catalog schema

The existing catalog already represents fields such as:

InstanceType
AcceleratorName
AcceleratorCount
vCPUs
MemoryGiB
GpuInfo
Price
SpotPrice
Region
AvailabilityZone

The DB model should represent the same logical information.

This allows existing catalog helper functions and optimizer logic to remain mostly unchanged.

The main difference becomes:

today:
CSV -> DataFrame -> optimizer

proposal:
DB -> DataFrame -> optimizer

---

Database model

4. Dedicated catalog table

A possible schema:

CREATE TABLE cloud_catalog_entries (
    id                  BIGINT PRIMARY KEY,

    provider            TEXT NOT NULL,
    scope_key           TEXT NOT NULL DEFAULT 'global',

    revision            TEXT NOT NULL,
    fetched_at          TIMESTAMP NOT NULL,

    instance_type       TEXT,

    accelerator_name    TEXT,
    accelerator_count   REAL,

    vcpus               REAL,
    memory_gib          REAL,

    gpu_info            TEXT,

    price               REAL,
    spot_price          REAL,

    region              TEXT,
    availability_zone   TEXT,

    availability_status TEXT,

    provider_metadata   TEXT
);

provider_metadata can store provider-specific information that does not fit the common schema, for example:

stock status
available GPU counts
provider cloud type
provider-specific IDs
capacity hints

Unsupported fields remain NULL / UNKNOWN.

---

5. Catalog revisions

Every successful provider refresh should publish one coherent revision.

RunPod revision 41
        │
        │ refresh
        ▼
RunPod revision 42

Every row from one refresh uses the same revision.

The API server can cache the current DataFrame in memory and only reload it when the DB revision changes.

optimizer request
      │
      ▼
read current revision
      │
 ┌────┴─────┐
 │          │
same      changed
 │          │
 ▼          ▼
reuse     reload DB
DataFrame    │
             ▼
          DataFrame

This avoids rebuilding the DataFrame on every request.

---

Refresh behavior

6. Atomic provider replacement

A refresh should update a whole provider catalog atomically:

fetch full catalog
       │
       ▼
normalize
       │
       ▼
validate
       │
       ▼
transaction
       │
       ▼
publish new revision

Readers must never observe a partially updated catalog.

The implementation could either replace all rows in one transaction or insert immutable revisions and atomically switch the active revision.

---

7. Keep the last valid revision on failure

If fetching, validation, or DB publishing fails, the previous revision stays active.

provider API timeout
        │
        ▼
refresh fails
        │
        ▼
previous revision remains active

Only a complete, validated catalog is published.

---

Refresh scheduling

8. One scheduler, provider-specific intervals

The refresh architecture should be common, while frequency can remain provider-specific.

For example:

catalog:
  refresh:
    defaultInterval: 10m

    providers:
      runpod:
        interval: 10m

      aws:
        interval: 60m

      gcp:
        interval: 60m

      azure:
        interval: 60m

The exact intervals are not important to this proposal.

The important property is:

different provider frequency
          !=
different refresh architecture

Dynamic providers can refresh frequently, while slower or more expensive provider queries can run less often.

---

9. Refresh only configured providers

The service should only refresh providers enabled/configured on the SkyPilot server.

This avoids unnecessary credentials, API traffic, DB writes, and provider rate-limit usage.

---

10. Jitter and retries

Refresh schedules should include jitter so many SkyPilot installations do not query provider APIs simultaneously.

Failed calls should use exponential backoff while the current DB revision remains active.

---

Catalog consumption

11. DB-backed DataFrame cache

The optimizer does not need to query individual catalog rows directly.

The catalog layer can continue exposing a DataFrame:

class DatabaseCatalog:

    def load(self, provider, scope):
        revision = get_current_revision(provider, scope)

        if revision != self._loaded_revision:
            self._df = load_dataframe_from_db(
                provider,
                scope,
                revision,
            )
            self._loaded_revision = revision

        return self._df

The API-server path remains:

request
   │
   ▼
DB/in-memory catalog
   │
   ▼
existing optimizer
   │
   ▼
provisioner

No full provider catalog API query occurs during a launch.

---

Account/workspace scope

12. Support scoped catalogs

Some catalog information may depend on:

• account;
• project;
• tenant;
• workspace;
• negotiated/private pricing.

For that reason, the catalog should include scope_key.

A global provider catalog can use:

scope_key = global

while account-specific catalogs can use a stable account/project/tenant identity.

The exact scope identity can be discussed separately.

---

Why a separate service?

I prefer a separate refresher service rather than running full provider fetchers inside API request workers.

Isolation

Slow or failing provider APIs do not block SkyPilot API requests.

Independent lifecycle

The refresher can be restarted or upgraded independently.

Clear ownership

refresher -> writes catalog
API server -> reads catalog

Better observability

Catalog refresh behavior can have dedicated metrics and logs.

A Kubernetes deployment could use a single replica. A CronJob would also work, but a long-running service makes per-provider intervals, retries, jitter, and metrics easier to manage.

---

Why not query providers during each launch?

Querying provider catalogs synchronously during optimization would add:

• API latency;
• API availability as a launch dependency;
• provider rate limits;
• duplicated calls across concurrent jobs;
• unpredictable optimizer latency.

Instead:

provider APIs
     │
     │ background
     ▼
     DB
 ┌───┼────┐
 ▼   ▼    ▼
job A B    C

All jobs reuse the same recently refreshed data.

---

Why use the DB as the source of truth?

For a long-running SkyPilot API server, the DB provides:

• one consistent catalog across API workers;
• atomic revisions;
• simple cross-process cache invalidation;
• persistence across pod restarts;
• easy freshness metadata;
• straightforward monitoring;
• account-scoped catalogs;
• no second catalog-distribution cycle.

The server architecture becomes:

provider API
     │
     ▼
catalog refresher
     │
     ▼
SkyPilot DB
     │
     ▼
SkyPilot API server
     │
     ▼
optimizer

---

Relation to existing SkyPilot work

I think this proposal mostly consolidates patterns that already exist.

Existing provider fetchers

SkyPilot already has API-backed fetchers for several providers:

• AWS: https://github.com/skypilot-org/skypilot/blob/master/sky/catalog/data_fetchers/fetch_aws.py
• GCP: https://github.com/skypilot-org/skypilot/blob/master/sky/catalog/data_fetchers/fetch_gcp.py
• Azure: https://github.com/skypilot-org/skypilot/blob/master/sky/catalog/data_fetchers/fetch_azure.py
• RunPod: https://github.com/skypilot-org/skypilot/blob/master/sky/catalog/data_fetchers/fetch_runpod.py

The proposal changes mainly:

when they run
+
where their normalized output is stored
+
how the API server consumes it

RunPod freshness discussion

The RunPod work already identified the mismatch between fast-changing provider state and slower static catalog refresh:

#6824

Realtime Kubernetes discovery

SkyPilot also has realtime resource discovery for Kubernetes, where repeated provider queries and TTL caching were discussed:

#3499

The DB-backed background refresher addresses the same general tradeoff for cloud catalogs:

fresh provider information
            vs.
request latency / repeated API calls

---

Minimal implementation

I would keep the first implementation intentionally small.

1. Add cloud_catalog_entries

Add one dedicated DB table containing the current catalog entries and revision metadata.

2. Add ProviderCatalogFetcher

Wrap existing provider fetchers behind one common interface.

3. Add catalog-refresher

One background service handles:

scheduling
fetching
retries
normalization
validation
DB publishing
metrics

4. Add DB-backed catalog loading

The API server checks provider revision and reloads its DataFrame only when that revision changes.

5. Start with two providers

I suggest:

RunPod
    demonstrates the freshness benefit

AWS or GCP
    demonstrates that the same architecture works
    for a traditional hyperscaler

6. Keep optimizer/provisioner unchanged

The initial proposal should not require changes to resource selection semantics or provisioning logic.

The main change is the catalog source and lifecycle.

---

Failure model

Situation	Behavior
Provider API succeeds	Publish new revision
Provider API times out	Keep previous revision
Validation fails	Keep previous revision
DB write fails	Roll back
Refresher process is temporarily down	API server keeps serving the current DB revision
Catalog field unavailable	Store NULL / UNKNOWN
Provisioning reports no capacity	Existing retry/failover logic remains

The DB catalog therefore acts as the last-known-good catalog state.

---

Observability

Useful metrics could include:

skypilot_catalog_refresh_success_total{provider=...}
skypilot_catalog_refresh_failure_total{provider=...}
skypilot_catalog_refresh_duration_seconds{provider=...}
skypilot_catalog_age_seconds{provider=...}
skypilot_catalog_entries{provider=...}

Diagnostics could expose:

Provider: RunPod
Revision: 42
Last successful refresh: ...
Age: 4m
Entries: 1842

---

Tradeoffs

Provider API amplification

With server-side refresh, each SkyPilot deployment queries its configured providers independently.

Mitigations:

• refresh only configured providers;
• use provider-specific intervals;
• add jitter;
• use exponential backoff;
• respect provider rate limits.

Credentials

The refresher needs provider credentials for APIs that require authentication.

For account-specific catalogs, credentials need to map to a stable scope_key.

Additional DB data

Catalog rows move into the SkyPilot DB. This adds some storage and DB traffic, but refreshes are relatively infrequent and the catalog is read mostly through an in-memory DataFrame cache.

---

Open questions

I would like feedback on:

1. Does a DB-backed catalog fit the direction of the SkyPilot API server?
2. Would maintainers prefer a separate catalog-refresher service or an API-server internal daemon?
3. Should refresh intervals be configured centrally, defined by provider adapters, or both?
4. What should identify account-specific catalog scope: workspace, cloud account/project/tenant, credential fingerprint, or another abstraction?
5. Would RunPod + AWS/GCP be a reasonable MVP for validating the common interface?
6. Should revisions use transactional replacement or immutable revisions with an active-revision pointer?

---

Proposed MVP

1. Add cloud_catalog_entries DB table.

2. Add ProviderCatalogFetcher interface.

3. Add one catalog-refresher service.

4. Adapt the existing RunPod fetcher.

5. Atomically publish RunPod catalog revisions to the DB.

6. Make the API-server catalog reader revision-aware.

7. Convert DB rows to the existing DataFrame schema.

8. Add AWS or GCP as a second provider.

9. Keep optimizer and provisioner behavior unchanged.

The intended architecture is:

provider-specific APIs
         │
         ▼
small provider adapters
         │
         ▼
────────────────────────────────────
common SkyPilot catalog infrastructure
────────────────────────────────────
         │
         ├── scheduling
         ├── retries/backoff
         ├── normalization
         ├── validation
         ├── DB persistence
         ├── revisions
         ├── cache invalidation
         └── observability
         │
         ▼
existing optimizer

I think this would make catalog handling considerably more consistent and give long-running SkyPilot API servers much fresher provider data without putting provider catalog queries into the launch path.

Would this direction make sense for SkyPilot?