Hooks, typed callbacks, more builtin tables, no more tempstates

[droserasprout]

Improvements:

• New entity: hook. User-defined callback with arbitrary code. Same as handler but not linked to any index. Called either from ctx.fire_hook (from hadlers or by DupDup) or by scheduler (jobs config section). Optionally typed with args. Jobs are now schedules for hooks.

hooks:
  a:
    callback: a
    atomic: True
    args:
     foo: int
     bar: str

jobs:
  a_leet:
    hook: a
    interval: 1337
    args:
      foo: 1337
      bar: "1337"

• SQL hooks. hooks/callback.sql directory next to hooks/callback.py. Both Python and SQL hooks are always fired (but not nescessary non-empty). Executed by CallbackManager.

• Built-in hooks defined declaratively in config.

• Unified codegen based on a single template for all callbacks. Config section must implement CallbackMixin.

• Unified callbacks processing in CallbackManager. Optional dynamic type-checking of arguments (basic types and dotted notations, skipped for handlers).

• All above finally apply to big_map indexes, debt closed.

• Massive refactoring of dipdup.config module. Easier to understand stages of complex config initialization flow.

• Callback-specific context classes removed. Callbacks receive either HandlerContext or HookContext. Required context created in base context.

• Matching Websocket data with cached head block.

• Bunch of internal models:

  • Schema: just hash
  • Index: status, template, template values, full config
  • Head: last Websocket connection status
  • Contract: hack to keep dynamically spawned contracts after restart (ignored if presented in config)

• No more "temporary states"!

  1. Indexes from config: just verify hash
  2. Indexes from template: recreate from config template, check hash
  3. Indexes with template missing: reindex
  4. Indexes without template not in config: ignore (must be just commented out)

See edge cases in TODO.

• Datasource now maintains linked models.Head. More on int.models below.
• Completely rewritten DipDup boot sequence. Main entrypoint splitted to _set_up_smth methods using stack and task set.

TODO:

• Reindex if on_rollback hook is not implemented
• Fix ParentMixin generic
• Fix formatted loggers missing context due to init in base context
• Set ONESHOT status on complete
• Context reused between job runs
• Factory level updated, config not reloaded, indexer dies -> invalid state
• Project migration: move SQL scripts, remove default handlers
• Check migration on demos
• Spec and major version bump
• hooks: List[HookConfig]? Now name must be equal to callback.

[droserasprout]

Store block datetime in dipdup_state

[droserasprout]

Index level and sync level in DipDupState.

[droserasprout]

Save spawned indexes, drop stateless option

[droserasprout]

Typed index settings (template values)

[droserasprout]

quipuswap_fa2_guts: Config hash mismatch (config has been changed), reindexing

[m-kus]

Сложна

[droserasprout]

camelCase table columns break Hasura