Construction API Testing 2.0

[patrick-ogrady]

Replaces #86
Closes #119

Motivation

The recently added Construction API testing (#81) does not support multi-currency sends, on-chain address generation, testing non-transfer transactions, or testing account/utxo transfers on the same network. This makes it impossible to test the Construction API of many projects implementing the Rosetta API.

Solution

We need to redesign the constructor to allow for the generic execution of a core set of primitives (generate_key, save_address, derive, set_variable, find_account, find_coin, print_message, wait_for_funds, math, random_string) that can be stitched together into scripts (with some shared state). These "scripts" should be called scenarios and it should be possible to combine multiple scenarios together into a workflow where all execution state is shared. At the end of each scenario execution it should be possible to execute a transaction which will allow multiple transactions to be broadcasted as part of a workflow. The constructor will manage the execution of these workflows (instantiated as jobs) using a new storage backend called JobStorage.

Preview

Workflow
  []Scenario
    []Action

Job (instance of a plan)
  Identifier
  State
  Index
  []Scenario
  Status [ready, broadcasting, failed, complete]

  Create(Plan) Job
  Intent() [run after execute] -> sets status to broadcasting

Worker (executes job)
  Helper (fetcher, balanceStorage, coinStorage, broadcastStorage, keyStorage)
    Derive
    LockedAddresses
    AccountBalance
    CoinBalance
    StoreAddress

  processScenario(State, []Actions) -> NewState, Error
  Process(job) -> updatedJob, intent, error


Constructor (creates jobs, coordinates worker, manages Construction API Flow)
  Helper (broadcastStorage, keyStorage, flowStorage)
    ConstructionAPIFlow (-Submit)
    Broadcast()
    Sign()
    JobsProcessing() -> Count
    JobsReady() []jobs
  Handler
    AddressCreated
    TransactionCreated

  Loop:
    var job
    var intent
    for [
      ready jobs...,
      (new jobs excluding create_account and request_funds as long as concurrency lower than required)plans...,
      request funds (if no broadcasting jobs and broadcast completed hasn't been updated since start)
    ]:
      Worker.Process(Job) -> updatedJob, Intent, error
      if errInsufficientAccounts:
        run "create_account" then try jobs again (as long as not existing create account)
      if errInsufficientFunds, errUnsatisfiable:
        continue
      if err != nil:
        THROW ERROR

      job = updatedJob
      intent = intent
      break

    jobID = JobStorage.Update(job) -> only assign identifier and store once successful at least once/only update if successful processing
    if intent:
      networktransaction, transactionidentifier = CreateTransaction(intent) [existing]
      Broadcast(network transaction, intent, transactionIdentifier, jobID)

  Errors:
    ErrUnsatisfiable -> if some action can't be completed because of lack of resources, but not error
    ErrInsufficientFunds -> if all ready fail + no in progress + creating all new jobs fails (don't try create account and request funds), request funds
    ErrInsufficientAccounts -> means we should create a new account, returned when we don't max out (only some addresses are creatable)
  Methods:
    CreateTransaction(Intent) Broadcast, error

Todos

• Add new constructor package to rosetta-sdk-go
  • Add support for basic job (processing Add constructor package scaffold + basic primitives mesh-sdk-go#104)
  • Add primitives
    • generate_key (Add constructor package scaffold + basic primitives mesh-sdk-go#104)
    • save_address (Construction API Primitives [PART 2] mesh-sdk-go#109)
    • derive (Add constructor package scaffold + basic primitives mesh-sdk-go#104)
    • set_variable (Add constructor package scaffold + basic primitives mesh-sdk-go#104)
    • find_account
    • find_coin
    • find_balance (replaces all other find calls) (Construction API Primitives [Part 3] mesh-sdk-go#112)
    • print_message (Construction API Primitives [PART 2] mesh-sdk-go#109)
    • wait_for_funds (just use find_account with a "wait")
    • math (Construction API Primitives [PART 2] mesh-sdk-go#109)
    • random_string (Construction API Primitives [PART 2] mesh-sdk-go#109)
  • Implement coordinator (Construction API Coordinator mesh-sdk-go#114)
  • make all calls during execution loop transactional + make BroadcastComplete accept a dbtx as an arg (like BlockWorker) -> Make Coordinator Fully Transactional mesh-sdk-go#117
• Add JobStorage to rosetta-sdk-go/storage (Add JobStorage mesh-sdk-go#118)
  • allow for viewing the number of completed jobs of a certain workflow (considered an end condition) -> Add JobStorage mesh-sdk-go#118
• replace rosetta-cli/pkg/constructor with rosetta-sdk-go/constructor (Integrate Construction API 2.0 #120)
• add constructor end conditions (i.e. stop after x transfers) (Construction End Conditions #122)
  • add bitcoin configuration file example (Construction End Conditions #122)
• err txn too big ([BUG] Txn too big error when running check:data mesh-sdk-go#125)
• populate requested pubkeys ([Constructor] Inject Public Keys During Transaction Construction mesh-sdk-go#133)
• option to return funds to a single address when ending (check:construction improvements #119)
  • could be a reserved workflow ("shutdown") that is called after end condition is met (instead of just halting)
• (Post-Release) Allow for re-processing based on the result from SuggestedFee ([Constructor] Suggested Fee Dry Run mesh-sdk-go#144)
  • add a method to the worker that allows for "dry running" a preprocess + metadata to get the suggested fee...this can then be injected into the state for later construction
  • allow for saving certain currencies from suggested fee as variables, then rewriting operations (triggers a re-run of construction flow)
  • if suggested fee is too high, allow for aborting (account may not have enough amount)
  • allow for a fee tolerance to continue (i.e. if fee within X range of Y, continue)