Quest Framework

A python framework for long-running, asynchronous workflows.

Principles

Code written on quest essentially lives forever. It lives in the past, present, and future. Turning off your application does not end it, it merely suspends its consciousness. Thus, you never "restart" your application; rather, you evolve it from its past state to the current requirement.

However, to remember the entire past would be overly-cumbersome. The ability to forget the trivia of the past is an important part of having bandwidth for the present. Quest seeks to balance immortality with evolution. Quest forgets the details of a step when the step completes, thus limiting the size of the execution history.

When evolving your code, it is often necessary to version your code so the past can replay while the present follows new logic. Over time, the steps that run the old code will complete and prune the history. Eventually, the old code paths will no longer replay and the supporting code can be deleted.

Rules

• Don't change global state in a step
• Don't cancel a task step from another task
  • TODO - handle cancellations from other tasks as outside events so a canceled task finishes after the cancellation is sent
• step and task functions can only be called within the historian.run() call tree

Notes

• Quest uses a logging filter to print the current task name in log statements. the task_name_getter context variable is used to return the current task name. (see utils.py) The filter is applied to all logger instances to avoid errors when the task_name_getter context variable isn't set.

TODO

• How can we enable forgetting in long-running event loops?
  • Many applications listen to event sources and respond. Once the event is processed, it could be forgotten.
  • Perhaps we provide a mechanism for tying a step to an event so the event can be deleted when the step is.
    • Maybe a "callback" option on a queue.
• How will the framework evolve?
  • How can current framework code process histories generated by prior framework versions?
  • Simple field changes might be handled by a DB migration (or similar)
  • What about introduction of new record types?
    • If the client code is versioned, such that old code only uses old features, this might work?
    • We can't introduce new record types on existing features (unless a deterministic migration is possible)
  • What about fixing bugs requiring a breaking change?
    • Can we version the framework code itself, much like the client?
    • Is there any way to know when an old version of the framework is no longer in any client's history?
      • Not practically. So versioned framework code might persist forever...
      • But such changes might be rare enough we can tolerate that?
• Manager
  • The event loop pattern is very common
  • We need a way to fire off a workflow and resume it if it hasn't finished yet
    • But not need to track the state outside the workflow