multiple postgres database & metadata storage separation

[rakeshkky]

TODO

• Avoid hdb_catalog.hdb_catalog.check_violation

• Cron/Scheduled triggers

• Console support - metadata storage data APIs

• Backwards compatibilty of DATABASE_URL

• v1/query API backwards compatibility

  • Make existing tests work

• pg_dump API to accept source

• Heterogeneous execution (i.e, merge master)

• Action queries/mutations execution in appropriate source (when there are relationships defined)

• Tests for multiple sources**

• Add/modify column issue:
  The server gets stuck with requests like the following one:

   {
        "type": "run_sql",
        "source": "default",
        "args": {
               "sql": "alter table \"public\".\"test\" add column \"test_col\" text null;",
               "cascade": false,
               "read_only": false
        }
    }
  

• pg_invoke_event_trigger results in the following error:

  {
   "internal": {
       "statement": "\n           SELECT hdb_catalog.insert_event_log($1, $2, $3, $4, $5)\n                ",
       "prepared": true,
       "error": {
           "exec_status": "FatalError",
           "hint": null,
           "message": "invalid input syntax for type json",
           "status_code": "22P02",
           "description": "The input string ended unexpectedly."
       },
       "arguments": [
           "(Oid 25,Just (\"public\",Binary))",
           "(Oid 25,Just (\"test\",Binary))",
           "(Oid 25,Just (\"lala\",Binary))",
           "(Oid 25,Just (\"MANUAL\",Binary))",
           "(Oid 114,Just (\"{\\\"row\\\":{\\\"id\\\":2}}\",Binary))"
       ]
   },
   "path": "$",
   "error": "database query error",
   "code": "unexpected"
  }
  

• can't set comments for computed fields. Request:

      {
           "type": "pg_add_computed_field",
           "args": {
               "table": {
                   "schema": "public",
                   "name": "users"
               },
               "name": "test",
               "definition": {
                   "function": {
                       "schema": "public",
                       "name": "author_full_name"
                   },
                   "table_argument": "author_row"
               },
               "comment": "some comment",
               "source": "my db 1"
           }
       }
  

** - can be done in parallel

Description

Metadata:-

• This PR overhauls the way in which the metadata is persisted in Postgres database. Now, the metadata is stored as a single json blob in hdb_catalog.hdb_metadata table in a single row. This reduces the Hasura footprint in the hdb_catalog schema.
• Metadata can be stored in a separate database, use --metadata-database-url or HASURA_GRAPHQL_METADATA_DATABASE_URL option. By default it is stored in the database specified via --database-url. One can start the server with only metadata database specified and configure the postgres database as sources (see below).

Multiple Databases

• Introduction of new notion in metadata - Source, a user Postgres database where all tables/views and functions can reside in. We can configure multiple sources via Metadata API or console UI.
• This change will pave the way for supporting sources of multiple backends other than Postgres. (MySQL, MSSQL etc.)

Changelog

• CHANGELOG.md is updated with user-facing content relevant to this PR. If no changelog is required, then add the no-changelog-required label.

Affected components

• Server
• Tests

Limitations, known bugs & workarounds

Actions:-

1. Action query subscriptions
2. Action with relationships to postgres tables

Metadata

The metadata version is bumped to 3. The only difference from version 2 is that now the tables and functions will be present inside new sources field for each time. The other top level fields remain as it is.
Example:-
version 2:

version: 2
tables:
- table:
    schema: public
    name: test
functions:
- function:
    schema: public
    name: search_test

version 3:

version: 3
sources:
- name: default
  tables:
  - table:
      schema: public
      name: test
  functions:
  - function:
      schema: public
      name: test
- name: chinook
  tables:
  - table:
      schema: public
      name: actors
  configuration:
    database_url: postgres://postgres@127.0.0.1:5432/chinook
    connection_pool_settings:
      retries: 1
      idle_timeout: 180
      max_connections: 50

Where default is the source configured via --database-url server option and chinook is newly added postgres source via metadata API.

New APIs

It is recommended to use newly added /v1/metadata to manage metadata and /v2/query to query postgres sources via JSON DSL, instead of /v1/query API.

/v1/metadata

It contains all query types in /v1/query except database DML operations like insert, select etc.
Query types for managing postgres related metadata are appended with pg_.
For example, track_table is changed to pg_track_table and similarly for all query types for managing tables, functions, relationships, remote relationships, event triggers and computed fields.
This also supports bulk query type for running multiple metadata queries in a single request.

APIs for console data requirements from metadata storage:

1. Get the console state:

request,

{
  "type": "get_catalog_state",
  "args": {}
}

response,

{
    "id": "fc0bc750-b343-44cc-a7f4-46637b0067f7",
    "cli_state": {},
    "console_state": {}
}

2. Set the console state:

request,

{
  "type": "set_catalog_state",
  "args": {
    "type": "console",
    "state": {"key": "value"}
  }
}

{ "message": "success" } is the response.

3. Get invocation logs for cron/scheduled (one-off) events.

request,

{
  "type": "get_event_invocations",
  "args": {
    "type": "one_off",
    "limit": 10,
    "offset": 10
  }
}

{
  "type": "get_event_invocations",
  "args": {
    "type": "cron",
    "trigger_name": "some_trigger_name",
    "limit": 10,
    "offset": 10
  }
}

By event id,

{
  "type": "get_event_invocations",
  "args": {
    "type": "cron",
    "event_id": "some_event_id"
  }
}

response,

{
    "invocations": <list of invocation objects>,
    "count": <count of total invocations"
}

By Event Id:

4. Get all scheduled events.

One off:-
pending

{
    "type": "get_scheduled_events",
    "args": {
        "type": "one_off",
        "limit": 10,
        "offset": 10,
        "status": ["scheduled"] 
    }
}

processed

{
    "type": "get_scheduled_events",
    "args": {
        "type": "one_off",
        "limit": 10,
        "offset": 10,
        "status": ["delivered", "error", "dead"] 
    }
}

Cron:-

{
    "type": "get_scheduled_events",
    "args": {
        "type": "cron",
        "trigger_name": "test"
    }
}

trigger_name is required if type is cron.

Response:-

{
    "events": <list of events, the object structure is different for cron and one_off>,
    "count": <total-count>
}

5. Delete cron/scheduled (one-off) event by event id

request:-

{
  "type": "delete_scheduled_event",
  "args": {
    "type": "one_off",
    "event_id": "00a8d34c-de0d-417a-ac9f-22df310af989"
  }
}

{"message": "success"} is the response.

/v2/query

All database DML operations like insert, select, delete, update, count and run_sql from /v1/query. An optional top level field source is accepted in payload and its absence treated as default source. These queries are run in the Postgres source specified.
Example:-

---
type: select
source: chinook
args:
  table: actor
  columns:
  - id
  - name

---

Console issues:

• get_invocation_logs: filter by cron trigger name, for one-off return all invocations
• get_invocation_logs: pagination
• get_invocation_logs: count all

// example req
    {
      "type": "get_event_invocations",
      "args": {
        "type": "one_off",
        "offset": 10,
        "limit": 10,
      }
    }

// response
{
  invocations: [...], count: 122,
}

• get_scheduled_events: filter by status - pending/processed
• get_scheduled_events: pagination
• get_scheduled_events: count